Stop writing ‘happy path’ automation and start designing for failure. This shift in mindset from promising success to explaining (and handling) failure is the key to building resilient systems and ending those dreaded 3 AM pages.

The ‘Happy Path’ Deception: Why Your Automation Keeps Breaking

I still remember the pager going off at 3:17 AM. It was a “simple” deployment script I’d written to sync new assets to a fleet of web servers. It worked flawlessly a dozen times in staging. But in production, one of the servers, `prod-web-12`, had a momentary network blip right in the middle of an `scp` command. The script wasn’t built for that. It just errored out and stopped, leaving the fleet in a Frankenstein state—half updated, half not. The load balancer had a fit, the site went down, and my phone started screaming at me. I spent the next hour manually fixing my “automation.” I had promised a seamless deployment but delivered a middle-of-the-night catastrophe because I only planned for success.

The Core of the Problem: We Code for Sunny Days

As engineers, we love to solve the primary problem. We write the script to copy the file, configure the service, or deploy the container. We test the “happy path” where every network call succeeds, every disk has space, and every API returns a `200 OK`. We implicitly promise the system will work.

The problem is, production is a chaotic storm. Networks are unreliable, APIs get overloaded, and disks fill up. The “happy path” is a pleasant fiction. The real work of a senior engineer isn’t just making things work; it’s defining what happens when they inevitably break. Your automation’s value isn’t measured when things go right, but how gracefully it handles things going wrong.

The Fixes: From Patching Holes to Building a Fortress

Shifting your mindset from “promising results” to “explaining and handling failure” requires a change in tactics. Here are three levels of maturity I’ve seen in my career.

1. The Quick Fix: The Defensive Scripting Mindset

This is the first, most crucial step. Stop writing naive, optimistic scripts. Start treating every external call—every network request, every file write—as a potential point of failure. It’s the difference between telling a junior “just copy the file” and showing them how to do it safely.

Before: The ‘YOLO’ Script

# Hope for the best!
echo "Copying new config to prod-db-01..."
scp ./new.conf user@prod-db-01:/etc/myapp/app.conf
echo "Restarting service..."
ssh user@prod-db-01 "sudo systemctl restart myapp"
echo "Done!"

After: The Defensive Script

#!/bin/bash
set -e # Exit immediately if a command exits with a non-zero status.
set -o pipefail # The return value of a pipeline is the status of the last command to exit with a non-zero status.

CONFIG_FILE="./new.conf"
REMOTE_HOST="prod-db-01"
REMOTE_PATH="/etc/myapp/app.conf"

echo "Validating config file exists..."
if [ ! -f "$CONFIG_FILE" ]; then
    echo "ERROR: Config file $CONFIG_FILE not found."
    exit 1
fi

echo "Copying new config to $REMOTE_HOST..."
if ! scp "$CONFIG_FILE" "user@$REMOTE_HOST:$REMOTE_PATH"; then
    echo "FATAL: Failed to copy config. Aborting."
    # Maybe add a Slack notification here
    exit 1
fi

echo "Restarting service on $REMOTE_HOST..."
if ! ssh "user@$REMOTE_HOST" "sudo systemctl restart myapp"; then
    echo "FATAL: Service restart failed. The server may be in a bad state. MANUAL INTERVENTION REQUIRED."
    exit 1
fi

echo "Deployment successful."

This is still a bit “hacky,” but it’s a world of difference. It stops a partial failure from becoming a total outage. It acknowledges that things can, and will, go wrong.

2. The Permanent Fix: Design for Idempotency

The next level is to stop thinking in sequential steps and start thinking in desired states. An operation is idempotent if running it multiple times has the same effect as running it once. If my script from the war story had been idempotent, I could have just run it again after the network recovered, and it would have intelligently skipped the servers that were already updated and only fixed `prod-web-12`.

This is the entire philosophy behind tools like Terraform, Ansible, and Puppet. You don’t write a script to “create a server.” You write a definition of the server you want, and the tool figures out the steps to make reality match your definition.

Pro Tip: When you can’t use a declarative tool, you can still write idempotent scripts. Before creating a directory, check if it exists. Before adding a line to a config file, `grep` for it first. Assume your script might have failed midway through the last time it ran.

3. The ‘Nuclear’ Option: Document the Failure, Not the Success

This is the most important, and most overlooked, piece of the puzzle. Your documentation shouldn’t just be a “How to deploy” guide. That’s the happy path. Your most valuable documentation is the runbook titled: “What to do when the deployment fails.”

This is the ultimate form of “explaining why things fail.” You are literally explaining it to your future, panicked, 3-AM-on-call self. A good failure runbook includes:

• Known Failure Points: Where does this process usually break? (e.g., “Step 3, database schema migration, often fails if `prod-db-01` is under heavy load.”)
• How to Verify a Failure: What specific error message or log entry confirms this is the problem?
• The Immediate Fix: What is the command to roll back safely? How do you get the system back to a stable state right now?
• The Post-Mortem Contact: Who needs to be looped in tomorrow to investigate the root cause?

Stop promising your team a perfect process. Instead, give them a well-lit path for when things go dark. That builds more trust and resilience than any “it will just work” promise ever could.

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions