Stop wasting talent on manual, repetitive “treatment plans” for server issues. This guide shows DevOps engineers how to move from reactive scripting to building truly resilient, self-healing systems.

From Manual ‘Treatment Plans’ to Self-Healing Systems: A Senior DevOps Perspective

I remember it vividly. It was a Tuesday, around 2:30 AM, and my phone was blowing up. Not from PagerDuty—I was off-call—but from a junior engineer on my team, let’s call him Alex. The Slack messages were frantic. “Darian, sorry to wake you, but prod-session-db-01 is down again. The treatment plan isn’t working.” The “treatment plan” was a dog-eared Confluence page: SSH in, run a script to clear the /var/log cache, restart the service, and pray. It was the third time this month. Alex was doing exactly what he was told, but he was burning out, and the business was seeing recurring micro-outages. That night, I realized the problem wasn’t the server; it was our entire approach. We weren’t engineering solutions; we were performing digital CPR on a system with a chronic condition.

The “Why”: The Anatomy of a Flawed ‘Treatment Plan’

That Reddit thread about “automating treatment plans” hit close to home. It’s a classic sign of a team trapped in a reactive cycle. This usually happens for a few key reasons:

• Technical Debt: The service was likely rushed into production without proper log rotation, monitoring thresholds, or disk space planning. The “fix” was pushed to a later sprint that never came.
• Misaligned Incentives: To management, a 5-minute outage fixed by a junior engineer looks like a success. The “Mean Time to Resolution” (MTTR) is low, so there’s no perceived urgency to invest engineering hours into a permanent solution.
• The “Hero” Complex: Sometimes, we engineers love to swoop in and save the day. A “treatment plan” makes us feel needed. But being a hero at 3 AM gets old, and it doesn’t scale. Real engineering is about making sure you’re never needed for that problem again.

The core issue is that a manual plan treats the symptom (a full disk, a crashed process) but completely ignores the disease (poor log management, a memory leak, inadequate capacity).

The Fixes: Climbing Out of the Reactive Hole

So, how do we break the cycle? We evolve our approach. You don’t go from manual checklists to a self-healing utopia overnight. It’s a staged process.

Solution 1: The ‘Get Some Sleep’ Fix (Automating the Checklist)

First things first, let’s stop the bleeding and the 3 AM wake-up calls. If you have a manual set of steps, the very first thing you do is automate that exact set of steps. Yes, it’s a band-aid, but it’s an automated, smarter band-aid that can run without a human.

We’re basically turning the Confluence page into a cron job. Let’s say the problem is that application logs on prod-web-app-03 fill up the disk.

#!/bin/bash
#
# triage_disk_space.sh
# A "smarter band-aid" to keep prod-web-app-03 alive.
# Run this via cron every 15 minutes.

THRESHOLD=90
MOUNT_POINT="/"
SERVICE_NAME="webapp.service"
LOG_DIR="/var/log/webapp/"

# Get current disk usage percentage
CURRENT_USAGE=$(df ${MOUNT_POINT} | grep -v Filesystem | awk '{ print $5 }' | sed 's/%//g')

if [ "$CURRENT_USAGE" -gt "$THRESHOLD" ]; then
    echo "WARNING: Disk usage at ${CURRENT_USAGE}%, exceeding threshold of ${THRESHOLD}%."
    echo "Finding and removing log files older than 3 days..."
    
    # IMPORTANT: Use find with -delete carefully. Test this first!
    find ${LOG_DIR} -type f -name "*.log" -mtime +3 -delete
    
    echo "Restarting the application service to ensure it's healthy."
    systemctl restart ${SERVICE_NAME}

    # You could add a Slack notification here too
    echo "Triage complete. Service restarted."
else
    echo "OK: Disk usage at ${CURRENT_USAGE}%. No action needed."
fi

Warning: This is a hack, and you should treat it as such. It’s a temporary measure to give your team breathing room to work on a real fix. It introduces its own risks (what if you delete the wrong files? What if the service restart fails?). But sometimes, a hack that lets your team sleep is a necessary evil.

Solution 2: The Architectural Fix (Building for Resilience)

Once the fire is out, it’s time to become a fire marshal. The goal here is to fix the underlying problem so the script from Solution 1 becomes obsolete. This is where real systems design comes into play.

Using our disk space example, the architectural fix involves a multi-pronged approach:

1. Proper Log Management: Don’t just let logs pile up on the local disk. Configure your application or a log shipper (like Fluentd, Filebeat, or Promtail) to send logs to a centralized platform (like an ELK stack, Loki, or Splunk). Now the local disk is only used for a small buffer, not long-term storage.
2. Intelligent Monitoring & Alerting: Stop alerting on “disk is 99% full.” That’s too late. Use a tool like Prometheus with `node_exporter` to track disk usage and, more importantly, predict when it will be full. An alert that says, “Based on the last 4 hours, prod-web-app-03 will run out of disk space in 24 hours” is infinitely more valuable. It gives you time to react during business hours.
3. Configuration as Code: Ensure a tool like Ansible, Puppet, or Terraform provisions your `logrotate` configuration on every single server. This prevents “configuration drift” where one server was set up incorrectly. The fix should be codified and applied everywhere, automatically.

Solution 3: The ‘Cattle, Not Pets’ Option (Self-Healing Infrastructure)

This is the paradigm shift. In the old model, servers are “pets.” When `prod-session-db-01` gets sick, we nurse it back to health. In the new model, servers are “cattle.” When one is sick, you don’t treat it; you replace it with a healthy one from the herd.

This is the cloud-native way, and it requires a few key components:

Pro Tip: This approach only works if your application is stateless or if its state is externalized. You can’t just terminate a primary database server. Its state must live somewhere durable and independent, like an RDS instance, an S3 bucket, or a managed Redis cluster.

When you reach this level, the original problem of a full disk becomes a non-event. The unhealthy instance is simply and automatically replaced in minutes, with zero human intervention. That’s the end goal. Stop treating symptoms and start building systems that can heal themselves.

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions