When a server becomes unresponsive and cloud costs unexpectedly spike, it’s often due to a single runaway process consuming all resources. Here’s a senior engineer’s guide to regaining control, from emergency hacks to permanent architectural fixes.

My Server is On Fire and I Can’t SSH In: A DevOps War Story

I still remember the feeling. It was 2:47 AM, and my phone was buzzing with a PagerDuty alert that could strip paint. The dashboard was a sea of red: our main Kubernetes cluster, `k8s-prod-us-east-1`, was showing a dozen nodes pegged at 100% CPU. The real kicker? I couldn’t SSH into a single one. `ssh -i my-key.pem ec2-user@<ip-address>` just timed out, leaving me blind and helpless. For a junior engineer, this is a nightmare scenario. For a senior, it’s a Tuesday, but a deeply annoying one. This isn’t just about a server being down; it’s about a cloud bill that’s about to go through the stratosphere as auto-scaling desperately tries to fix a problem by throwing more expensive hardware at it.

So, What’s Actually Happening Here?

Before you start blaming the cloud provider or a DDoS attack, let me tell you what’s almost always going on. One of your processes has gone rogue. It’s stuck in an infinite loop, a memory leak, or a deadlock, and it’s consuming every last CPU cycle. This is called “CPU starvation.”

When a process hits 100% CPU, it doesn’t leave any processing time for the other essential system daemons. That includes `sshd`, the very service that lets you log in. The server isn’t dead; it’s just too busy to answer your knock at the door. In the Reddit thread that sparked this post, the culprit was `kubelet`, but I’ve seen it be anything from a faulty monitoring agent to a poorly written application log shipper.

The goal isn’t just to fix it now, but to make sure it never holds your systems hostage again. Here’s how we do it, from the desperate battlefield triage to the permanent engineering solution.

Solution 1: The Quick Fix (The “Side Door”)

When SSH is down, you need another way in. Forget the front door; we’re using the maintenance hatch. Every major cloud provider has a tool for this exact situation.

• AWS: EC2 Instance Connect or Systems Manager (SSM) Session Manager
• GCP: Serial Console
• Azure: Serial Console

These tools give you shell access without relying on the instance’s networking or `sshd` daemon. Once you’re in, you’re back in business. The first thing you’ll do is find the offender.

# Run top to see what's eating all the CPU
top

You’ll immediately see the process at the top of the list with 99.9% CPU usage. Let’s say its Process ID (PID) is 12345 and the process is `kubelet`.

Warning: Be careful what you kill. Killing a critical system process can be just as bad as letting it run wild. But in this case, the system is already useless, so we’re in damage control mode.

It’s time for the hammer. Don’t be gentle.

# Kill the process with extreme prejudice
kill -9 12345

Your server’s CPU usage will plummet, and you’ll probably be able to SSH in again. You’ve stopped the bleeding. But make no mistake: this is not a fix. You’ve only treated the symptom. The underlying problem is still there, waiting to happen again on the next deploy or reboot.

Solution 2: The Permanent Fix (The “Resource Cage”)

The real, long-term solution is to prevent any single process from being able to take down the entire server. You do this by setting resource limits using Linux control groups (cgroups). If you’re using systemd (and you probably are), this is surprisingly easy.

We need to create or modify the systemd unit file for the service that caused the problem (in our example, `kubelet`). We’re going to tell systemd, “Hey, never let this `kubelet` thing use more than 80% of one CPU core or more than 4GB of RAM.”

You can create a drop-in configuration file to modify the existing service.

# Create a directory for the kubelet service override
sudo mkdir -p /etc/systemd/system/kubelet.service.d/

# Create a new configuration file
sudo nano /etc/systemd/system/kubelet.service.d/10-resources.conf

Now, add the following content to that file:

[Service]
# CPU limit: 80% of one CPU core.
CPUQuota=80%

# Memory limit: 4 Gigabytes.
MemoryMax=4G

# Optional: If the service is critical, make it restart automatically.
Restart=always
RestartSec=5

After saving the file, reload the systemd daemon and restart the service for the changes to take effect.

# Reload the configuration
sudo systemctl daemon-reload

# Restart the service
sudo systemctl restart kubelet

Now, `kubelet` is in a cage. If it tries to go rogue again, the kernel will throttle it, ensuring there’s always enough CPU and memory for `sshd` and other critical processes to run. You’ve fixed the root cause.

Solution 3: The ‘Nuclear’ Option (The “Cloud-Native Slap”)

Sometimes, you can’t get console access, or the issue is happening across hundreds of nodes at once. In a modern, ephemeral, cloud-native environment, the fastest path to resolution is sometimes just to… well, shoot the server in the head.

This approach treats instances as disposable cattle, not precious pets. The idea is to use your cloud provider’s automation tools to handle this for you.

Here’s the pattern:

1. Create an Alarm: Set up an alarm (e.g., AWS CloudWatch Alarm) that triggers when a server’s CPU utilization is at 100% for more than, say, 5 minutes.
2. Trigger an Action: Configure the alarm to trigger an action. This could be an AWS Lambda function, a GCP Cloud Function, or an Azure Function.
3. Execute the Reboot: The function’s code is brutally simple: it receives the instance ID from the alarm and uses the cloud provider’s API to force a hard reboot or termination of that instance.

PRO TIP: This is an incredibly powerful but blunt instrument. NEVER use this strategy on stateful servers like your primary database (`prod-db-01`). This is for stateless application servers, web servers, or workers in an auto-scaling group where a replacement can be spun up automatically without data loss.

This doesn’t fix the underlying bug, but in a large, distributed system, it maintains overall service availability while you work on the permanent fix (Solution 2) in the background. It’s the automated version of “have you tried turning it off and on again?” — and sometimes, that’s exactly what you need.

Which To Choose?

Solution	Best For	Pros	Cons
1. The Quick Fix	Immediate emergency response.	Fast, restores access quickly.	Doesn’t prevent recurrence.
2. The Permanent Fix	Any critical service on any server.	Solves the root cause, robust.	Requires config changes, testing.
3. The Nuclear Option	Large fleets of stateless servers.	Automated, scales well.	Brute-force, can hide bugs, risky for stateful services.

Seeing your whole infrastructure light up is stressful. But with a methodical approach, you can go from panicked and locked out to in control and implementing a permanent fix. Remember to breathe, use the side door when you have to, and then put a cage around the beast so it can’t hurt you again.

---

🤖 Frequently Asked Questions