Tired of a rogue process hogging CPU and crashing your servers? A Senior DevOps Engineer shares three battle-tested automation scripts to kill stuck tasks for good, from the quick and dirty to the architecturally sound.

That One Rogue Process: My Favorite ‘Oddly Specific’ Automation

I still remember the 3 AM pagers. It was always a Monday. A data processing job, kicked off by some ancient cron entry nobody dared to touch, would occasionally go completely off the rails. It would lock onto a CPU core on prod-reporting-db-01, spin it up to 100%, and hold it there like a dog with a bone. The whole reporting dashboard would grind to a halt, and the on-call engineer (usually me) had to wake up, SSH in, and manually `kill -9` the runaway `data_cruncher.py` process. After the third time, I said, “Never again.” We automate everything else; why not automate killing the stupid things that break?

The Root of the Problem: Why Processes Go Rogue

Before we jump into the fixes, let’s talk about why this happens. It’s rarely malice; it’s usually just… entropy. A junior dev pushes a query with a Cartesian join, a memory leak in a long-running service finally hits a critical threshold, or an external API it depends on starts timing out, causing an infinite retry loop. The application itself isn’t smart enough to self-terminate when it enters a bad state. The “right” fix is to solve the underlying code bug, but while you’re waiting for that ticket to get prioritized in a backlog three sprints away, production is on fire. Our job is to put out the fire and build a fire-suppression system for next time.

Three Ways to Slay the Beast

Here are three approaches I’ve used over the years, ranging from the quick-and-dirty band-aid to a more permanent, architectural solution.

Solution 1: The ‘Get Me Through The Night’ Cron Job

This is the classic “hacky but it works” solution. It’s a simple Bash script run by cron every five minutes. It’s not elegant, but it’s effective, and you can write it in less time than it takes to brew a pot of coffee.

The script finds a process by name, checks its CPU usage, and if it’s been over a certain threshold (say, 90%) for more than a few minutes, it gets the axe. Here’s what that might look like:

#!/bin/bash

PROCESS_NAME="data_cruncher.py"
CPU_THRESHOLD=90
LOG_FILE="/var/log/process_killer.log"

# Find the PID and CPU % of the process
PROC_INFO=$(ps -eo pid,pcpu,comm | grep $PROCESS_NAME | grep -v grep | awk '{print $1 " " $2}')

if [ -z "$PROC_INFO" ]; then
  exit 0 # Process not running, we're good.
fi

PID=$(echo $PROC_INFO | awk '{print $1}')
CPU_USAGE=$(echo $PROC_INFO | awk '{print $2}' | cut -d. -f1)

if [ "$CPU_USAGE" -gt "$CPU_THRESHOLD" ]; then
  echo "$(date): Rogue process detected! PID: $PID, CPU: $CPU_USAGE%. Terminating." >> $LOG_FILE
  kill $PID
  sleep 5
  # If it's still alive, bring out the big guns.
  if ps -p $PID > /dev/null; then
    echo "$(date): Process $PID survived SIGTERM. Sending SIGKILL." >> $LOG_FILE
    kill -9 $PID
  fi
else
  echo "$(date): Process $PID is behaving. CPU at $CPU_USAGE%." >> $LOG_FILE
fi

You’d just save this as `process_killer.sh`, make it executable, and add it to your crontab to run every 5 minutes: */5 * * * * /usr/local/bin/process_killer.sh.

A Word of Warning: Be very careful with kill -9 (SIGKILL). It doesn’t give the process a chance to clean up after itself, which can lead to corrupted files or orphaned child processes. It’s a last resort, which is why we always try a graceful kill (SIGTERM) first.

Solution 2: The ‘Grown-Up’ Systemd Approach

If this is a service you manage, a much cleaner and more robust solution is to use the tools your operating system already provides. Systemd, which is on pretty much every modern Linux distro, is fantastic for this. You can set resource limits directly in the service’s unit file.

This method doesn’t just kill a runaway process; it prevents it from running away in the first place by sandboxing it. Let’s say our `data_cruncher.py` is run by a service called `datacruncher.service`.

You would edit the unit file at /etc/systemd/system/datacruncher.service and add resource controls under the [Service] section:

[Unit]
Description=Data Cruncher Service

[Service]
ExecStart=/usr/bin/python3 /opt/scripts/data_cruncher.py
Restart=always

# --- Here's the magic ---
# Limit to 50% of one CPU core.
CPUQuota=50%

# Set a hard memory limit.
MemoryMax=2G

# If it exceeds limits, what action to take? 'kill' is the default.
MemoryAccounting=true
CPUAccounting=true

# If the main process dies, kill any lingering children.
KillMode=process

[Install]
WantedBy=multi-user.target

After saving, you just run systemctl daemon-reload and systemctl restart datacruncher.service. Now, if the process tries to exceed 2GB of RAM or hog more than half a CPU core, systemd’s cgroup management will either throttle it or kill it outright. This is the way to go for long-running services.

Solution 3: The ‘Just Nuke It From Orbit’ Reboot

Sometimes, a single process isn’t the problem; it’s a symptom of a deeper system state corruption. This is especially true for stateless application servers, like a web worker in a load-balanced pool (e.g., `prod-web-app-07`). If one node starts acting up, often the fastest and most reliable fix is to just shoot it in the head and let the auto-scaling group replace it with a fresh instance.

This “automation” is more about architecture. You’d use a health check. For example, a script that runs locally and checks if the application is responsive. If it fails a few times in a row, the script calls the cloud provider’s API to mark itself as unhealthy.

• In AWS: An EC2 instance can run a command like aws autoscaling set-instance-health --instance-id $(curl -s http://169.254.169.254/latest/meta-data/instance-id) --health-status Unhealthy. The Auto Scaling Group (ASG) sees this, terminates the instance, and spins up a new one.
• In Kubernetes: This is literally what Liveness Probes are for. You define a probe in your pod spec, and if it fails, Kubernetes automatically kills the container and restarts it. It’s the “nuke it” option, but built right into the orchestrator.

This sounds extreme, but for a stateless, horizontally-scaled service, it’s often the safest and most reliable recovery mechanism. You aren’t debugging a sick server; you’re replacing it with a healthy one.

Comparison of Approaches

In the end, we implemented the systemd limits on the reporting server. It stopped the 3 AM pages cold. But I’ll always have a soft spot for that first, ugly little Bash script. It was the oddly specific automation that let us all get a good night’s sleep.

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions