When server logs misrepresent reality, it’s a DevOps problem, not a marketing failure. Learn how to fix NGINX and Apache logs to filter out health check noise, ensuring your traffic metrics accurately reflect real user engagement and conversions.

More Money, Less Traffic? Why Your Server Logs Are Lying and How to Fix It.

I’ll never forget the Monday morning meeting. We’d just rolled out a major infrastructure upgrade over the weekend—migrated our main e-commerce app to a new Kubernetes cluster behind a shiny new load balancer. I was expecting pats on the back. Instead, the head of marketing puts up a slide with a traffic graph that looks like it fell off a cliff. Panic. The CTO is looking at me. But then the head of finance chimes in, “That’s odd, because revenue is up 15% since Saturday and our conversion rate is the highest it’s been all quarter.”

That right there is the moment every DevOps engineer has faced in some form. The data says one thing, but reality says another. You’re not crazy; your tools are just telling a half-truth. Someone on Reddit was going through this exact thing, getting blamed for “low traffic” when the business was actually doing better than ever. Let’s break down why this happens and how to make sure it never happens to you again.

The “Why”: Your Servers Are Drowning in Health Check Chatter

The root of this problem isn’t malicious. It’s a side effect of modern, resilient infrastructure. To ensure your application is healthy, systems like load balancers (AWS ELB/ALB, Google Cloud Load Balancing) and container orchestrators (Kubernetes) constantly poke your servers. They send a tiny request every few seconds to an endpoint to ask, “Hey, you still alive?”

These are called health checks, liveness probes, or readiness probes. And your web server, by default, diligently logs every single one of them. The result? Your access logs get flooded with thousands of automated requests that have nothing to do with real customers.

When your analytics tool (be it a simple log parser or a fancy enterprise platform) ingests these raw logs, it sees a massive volume of “traffic.” After you implement a more robust health checking system, that automated traffic count might drop, making it look like you’ve lost users when, in fact, you’ve just cleaned up the noise. The real user traffic was there all along, and now it’s converting better because the platform is more stable.

The signal was always there; it was just buried under the noise.

Three Ways to Fix This Mess

You’ve got a few ways to tackle this, ranging from a quick-and-dirty fix to get marketing off your back to a proper architectural solution. Let’s walk through them.

1. The Quick Fix: Filter the Logs After the Fact

This is the “I need an answer by lunch” solution. You don’t change the server configuration; you simply filter the raw access logs to exclude the health check requests before you send them to the analytics team. Most health checkers have a unique, identifiable User-Agent string.

For example, an AWS Elastic Load Balancer uses something like ELB-HealthChecker/2.0. You can use a simple `grep` command to strip these out.

# This command finds all lines that do NOT contain the health checker user agent
# and saves them to a new, clean log file.
grep -v "ELB-HealthChecker/2.0" /var/log/nginx/access.log > /var/log/nginx/clean_access.log

Pros: Fast, requires no service restarts, and can be done on historical data.

Cons: It’s a manual band-aid. You have to do it every time, and it doesn’t fix the underlying problem of bloated log files on your server.

Warning: Be careful with this approach. Sometimes health checks can come from a specific private IP. If you filter by IP, make sure that IP range is only used for health checks and not for other internal traffic you might want to track.

2. The Permanent Fix: Conditional Logging in NGINX

This is the proper sysadmin solution. You tell your web server, “If a request is from a health checker, just don’t even bother logging it.” We can do this in NGINX using a `map` and a conditional access log. This is far more efficient and stops the noise at the source.

You would edit your `nginx.conf` file, usually in the `http` block:

http {
    # ... other http settings ...

    # 1. Create a map. If the User-Agent matches, the $loggable variable will be 0.
    #    Otherwise, it will be 1.
    map $http_user_agent $loggable {
        "~ELB-HealthChecker" 0;
        "~kube-probe"        0; # For Kubernetes probes
        default              1;
    }

    # ... other http settings ...
}

Then, in your `server` block for your site, you modify the `access_log` directive:

server {
    # ... server settings ...

    # 2. Point your access log to a variable, enabled by our map.
    #    The 'if=$loggable' part ensures NGINX only writes to the log
    #    if our variable is 1.
    access_log /var/log/nginx/access.log main if=$loggable;

    # ... other server settings ...
}

Pros: Solves the problem at the source. Your log files will be smaller, cleaner, and always accurate. It’s a set-it-and-forget-it solution.

Cons: Requires a configuration change and a reload of your NGINX service (`sudo systemctl reload nginx`). You need to be sure you identify all health check user agents.

3. The ‘Nuclear’ Option: A Dedicated Health Check Endpoint

This is the cloud architect’s answer. Why are you even letting the health checker hit your main application? That request still consumes a tiny bit of PHP-FPM or Node.js resources. A better pattern is to create a dedicated, lightweight endpoint that does nothing but confirm the server is running.

You could have your developers create a `/healthz` endpoint in the application that returns a simple `200 OK` without touching the database or complex logic. Or, even better, create a separate `server` block in NGINX that listens on a different port (e.g., 8081) just for health checks.

# New server block just for health checks, maybe on a private port
server {
    listen 8081;
    server_name localhost;

    # Disable logging entirely for this endpoint
    access_log off;

    location /healthz {
        # Simply return 200 OK. Nothing else.
        return 200 "OK";
    }
}

Then, you configure your load balancer and Kubernetes probes to hit `http://your-instance-ip:8081/healthz`. This traffic never even touches your main application’s logs.

Pros: The most robust and architecturally clean solution. It completely isolates health check traffic from user traffic and reduces load on your application.

Cons: Requires more setup. You need to coordinate with developers or have control over the infrastructure to expose a separate port and point the health checkers to it.

Which Path Should You Choose?

Here’s a quick breakdown to help you decide.

Solution	Effort	Effectiveness	Best For
1. Log Grepping	Low	Low (It’s a band-aid)	Emergency situations; proving your theory to management.
2. Conditional Logging	Medium	High	The standard, best-practice fix for most teams.
3. Dedicated Endpoint	High	Very High	High-traffic environments where performance and log purity are critical.

So, no, you’re not crazy. You’re just seeing the ghost in the machine. Your job isn’t just to keep the servers running; it’s to make sure the data they produce reflects reality. Now you have the tools to do just that.

---

🤖 Frequently Asked Questions