Stop drowning in useless metrics. Learn the three tiers of web app performance monitoring that actually tell you if things are broken, from user-facing impact down to the nitty-gritty infrastructure.

Beyond CPU: What We *Actually* Monitor in a Web App

I still remember the 3 AM page. Every single dashboard was a sea of green. CPU on the web fleet? 20%. Memory? Barely touching 40%. Disk I/O on our primary database, `prod-db-01`? Flat as a pancake. Yet, our support channels were on fire with customers complaining that the site was “unbearably slow” or “just timing out.” For an hour, we chased ghosts in the machine, convinced it was a network issue or some upstream provider. The problem? Our application had exhausted its database connection pool. The app servers were just sitting there, patiently waiting in a queue for a connection that would never come. The servers were perfectly healthy. The *application* was dead in the water. That night taught me a lesson I preach to every new engineer on my team: stop looking at the server first.

The Real Problem: Confusing a Healthy Server with a Healthy App

Look, the glut of monitoring tools today gives us an ocean of data. It’s easy to get mesmerized by beautiful Grafana dashboards showing CPU utilization across a hundred nodes. The fundamental mistake we make is assuming these infrastructure-level metrics tell us anything meaningful about the user’s experience. A server can be humming along at 5% CPU while your application is throwing 500 errors on every checkout attempt because of a misconfigured API key.

Your goal isn’t to monitor servers; it’s to monitor the *service* you’re providing. The server is just a component. We need to flip the script and monitor from the outside in, starting with what the user actually feels.

Tier 1: The User Experience (The “Is It Broken?” Monitor)

This is your first line of defense and the only thing that should be waking you up at night. If you monitor nothing else, monitor this. We’re talking about Application Performance Monitoring (APM) here. These are metrics that directly measure what a user experiences when they interact with your application.

• Request Latency (p95/p99): I don’t care about the average (mean) response time. Averages lie. They hide the outliers, and your angriest users are always the outliers. I want to know the 95th and 99th percentile latency. This tells me that 5% or 1% of my users are having a terrible experience. If p99 latency for the /api/v1/login endpoint jumps from 400ms to 3000ms, something is wrong. Period.
• Error Rate (HTTP 4xx/5xx): This is a no-brainer. What percentage of your requests are resulting in errors? A sudden spike in 500 errors is a clear “all hands on deck” signal. Don’t just track the total; segment it by endpoint. A high error rate on a critical endpoint like /api/v2/payment is an emergency.
• Transaction Traces: When things go wrong, you need to know why. A good APM tool will give you a detailed trace of a slow or failed request. You can see the exact database query that took 3 seconds, or the external API call that timed out. This moves you from “it’s slow” to “it’s slow because of this specific line of code.”

Pro Tip: Only page a human for a Tier 1 alert. A CPU spike is noise; a 5% error rate on your checkout API is a signal. Respect your team’s sleep and sanity by only escalating alerts that represent real, verified user impact.

Tier 2: The Application Internals (The “Why Is It Broken?” Monitor)

Okay, so your p99 latency is through the roof. The user is having a bad time. Now we move down the stack. These are the “white-box” metrics from within your application and its direct dependencies. They help you diagnose the *why* behind the Tier 1 alert.

This is where we look at the health of the components that make up our service:

These metrics rarely tell you if the user is happy, but they are absolutely critical for telling you *why* the user is unhappy.

Tier 3: The Infrastructure (The “Is the Foundation Crumbling?” Monitor)

Finally, we get to the classic metrics that everyone starts with. CPU, Memory, Disk, and Network. Do not get me wrong, these are important, but they are for *correlation*, not primary alerting.

I almost never look at these dashboards first. I only look at them after a Tier 1 alert has fired and the Tier 2 metrics aren’t pointing to an obvious application-level bug. The conversation goes like this:

“Okay, p99 latency for the web fleet is up, and I see in the transaction traces that database queries are slow. I check the query stats in Tier 2, and nothing looks out of the ordinary. Hmm. Let’s look at the infrastructure metrics for `prod-db-01`… whoa, CPU is pegged at 100% and Disk IOPS are maxed out. Now we have a theory: the database server itself is overloaded.”

See the difference? We used the infrastructure metric to confirm a theory that started with a user-facing problem. We didn’t get a meaningless alert that “CPU is high” and then have to spend an hour figuring out if it even mattered.

A “Hacky” but Effective Start: If you don’t have a full APM suite, you can build a very simple version of a Tier 1 check yourself. Create a /health endpoint in your app that does more than just return “OK”. Make it perform a quick, non-destructive check of its critical dependencies, like running SELECT 1 on the database or a `PING` to Redis. If any of those fail or take too long, the endpoint fails. Now you have a simple URL you can monitor with any basic uptime tool that gives you a much better signal than a raw server ping.

function handle_health_check(request):
    start_time = now()
    
    // Check Database
    db_ok = database.query("SELECT 1")
    if not db_ok:
        return HTTPResponse(503, "Database connection failed")

    // Check Cache
    cache_ok = redis.ping()
    if not cache_ok:
        return HTTPResponse(503, "Cache connection failed")

    // Check latency
    total_time = now() - start_time
    if total_time > 500ms:
        return HTTPResponse(503, "Health check latency too high")

    return HTTPResponse(200, "OK")

Stop monitoring your servers. Start monitoring your service. Start with the user experience and work your way down. You’ll catch real problems faster, spend less time chasing false alarms, and maybe even get a full night’s sleep.

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions