In a containerized world, traditional host-level metrics like CPU Percentage (CPC) are misleading. This guide explains why and reveals the container-specific metrics, like cgroup memory usage and working set bytes, that senior engineers check first to solve real-world problems.

Beyond The Host: What I Check Now That Node-Level CPU is Lying to Me

I remember it like it was yesterday. 3 AM. The PagerDuty alert screams about `prod-checkout-api-7f8c9…` being OOMKilled. Again. I hop on a call with a junior engineer, bless his heart, who is absolutely frantic. “Darian, I don’t get it! I’m SSH’d into the node, `k8s-worker-us-east-1c-04`, and `top` shows plenty of free memory and CPU is barely ticking over! It makes no sense!” I could hear the exhaustion and confusion in his voice. He was looking at the right screen, but at the wrong data. That night, I realized we’d failed to teach a fundamental lesson of the container era: the host is lying to you.

The “Why”: Your Tools Are Living in the Past

Let’s get this straight. Tools like top, htop, and free are not broken. They’re just old-school. They were built for a world where one server ran a handful of processes. They read from global system files like /proc/meminfo. In the world of Kubernetes and containers, this file tells you about the entire Virtual Machine or bare-metal server, but it’s completely oblivious to the carefully constructed resource prisons we call cgroups (control groups).

Your pod isn’t running on the “server”; it’s running inside a cgroup that has its own specific memory and CPU limits. When a pod gets OOMKilled, it’s not because the server ran out of memory, it’s because the pod itself hit its cgroup limit. Looking at the host’s free memory is like checking the water level in the city reservoir to see why your kitchen sink has no pressure. You’re looking at the wrong scope.

The Fixes: From Triage to Architecture

So, what do you look at instead? Here’s my playbook, from the “get me back to sleep” fix to the “let’s never get paged for this again” solution.

1. The Quick Fix: Go Straight to the Source (cgroups)

When you’re on a node and need the ground truth right now, bypass the old tools and read the cgroup data directly. This is my go-to for immediate triage during an incident.

First, find the pod’s unique ID on the node. You can usually do this by running crictl ps | grep <pod-name>. Then, you need to find its cgroup memory file. The exact path can vary, but you can find it.

For a Kubernetes pod, the path will look something like this. You can find the right directory using the pod’s UID which you can get from `kubectl get pod <pod-name> -o yaml`.

# Find the cgroup path for your pod
# The path often contains "kubepods" and the pod's UID
find /sys/fs/cgroup/memory/ -name "*<pod-uid>*"

# Once you have the path, cat the memory usage file
# This value is in bytes. Divide by 1024*1024 to get MiB.
cat /sys/fs/cgroup/memory/kubepods/burstable/pod<...uid...>/<...container-id...>/memory.usage_in_bytes

This command gives you the exact, undeniable memory usage for that specific container. No guesswork. If this number is bumping up against the limit you set in your manifest, you’ve found your culprit.

Pro Tip: On newer systems using cgroups v2, the file path and name are different. You’ll be looking for a file named memory.current instead of memory.usage_in_bytes. The principle is the same: find the container’s cgroup directory and read the file.

2. The Permanent Fix: Use Container-Aware Monitoring

Reading cgroup files is great for an emergency, but it’s not a monitoring strategy. For a permanent solution, your observability stack must be container-aware. In the Kubernetes world, this means Prometheus and cAdvisor.

The Kubelet on every node already embeds a service called cAdvisor that exposes exactly the metrics we need. Your Prometheus instance just needs to be configured to scrape them. The single most important metric I look at now is container_memory_working_set_bytes.

Why this one? It represents the amount of memory that is actively being used and cannot be easily reclaimed by the kernel. It’s a much more accurate predictor of OOMKills than generic `usage_in_bytes`, which can include reclaimable cache.

A typical PromQL query I’d use in Grafana or for an alert would be:

// Show memory usage as a percentage of the request for a specific deployment
sum(container_memory_working_set_bytes{namespace="prod", pod=~"prod-checkout-api.*"}) by (pod) / sum(kube_pod_container_resource_requests{resource="memory", namespace="prod", pod=~"prod-checkout-api.*"}) by (pod) * 100

Setting up alerts on this metric when it exceeds, say, 90% of the container’s memory limit is how you get ahead of the problem and stop the 3 AM pages.

3. The ‘Nuclear’ Option: The `kubectl` Quick-Check

Let’s say you can’t SSH to the node and your Grafana dashboard is down. There’s one more trick up your sleeve, provided you have the Kubernetes Metrics Server installed (and you absolutely should).

The command is kubectl top. It’s not as granular as cAdvisor, but it’s a fantastic “in a pinch” tool.

# Get the current CPU and Memory usage for a specific pod
kubectl top pod prod-checkout-api-7f8c9d... -n prod --containers

This command queries the Metrics Server API, which aggregates data from the Kubelets, and gives you a clean, simple readout of real CPU and Memory usage for the pod and its containers. It’s an abstraction, but it’s an abstraction over the *correct* underlying data. It’s fast, easy, and respects the cgroup boundaries.

Warning: The data from kubectl top is often slightly delayed and averaged over a short period. It’s perfect for a spot-check, but for alerting and deep analysis, stick to Prometheus.

Summary Table: Choosing Your Weapon

So, the next time a pod starts acting up, take a breath. Step away from top on the host node. You’re a container engineer now. Your new first metric isn’t on the host; it’s inside the container’s own little world. Go look there first.

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions