Inspired by a discussion on entrepreneurial success, I’m sharing the one pivotal change that transformed our infrastructure: stop ‘fixing’ servers and start replacing them. Learn how embracing Infrastructure as Code moves you from constant firefighting to scalable engineering.

That Reddit Thread Was Right: The One Change That Made Our Infrastructure Successful

I remember a Tuesday, probably around 3 AM. A pager alert dragged me out of bed for the third time that week. The error was familiar: `CRITICAL: Disk space low on prod-payment-gateway-02`. I groggily grabbed my laptop, SSH’d into the box, and ran my usual heroic command: `rm -rf /var/log/app/archive/*`. I was the hero, the problem was solved, and I could go back to sleep. But I wasn’t a hero. I was a janitor. And the same “fire” would just pop up somewhere else the next night. That Reddit thread about entrepreneurs hit me hard because we were making the same mistake: we were working *in* the business, not *on* the business. We were stuck in a loop of manual fixes, and it was killing us.

The “Why”: You’re Treating Servers Like Pets, Not Cattle

The root cause of our burnout wasn’t disk space, a memory leak, or a misconfigured nginx worker. The root cause was our philosophy. We treated every server like a unique, handcrafted pet. When `prod-payment-gateway-02` got sick, we nursed it back to health with special commands and manual tweaks. The problem is, after a few months, that server had a completely unique history. Its configuration had drifted so far from the original image that we couldn’t possibly reproduce it. We had created a “snowflake server”—delicate, unique, and impossible to replace. The real problem is configuration drift caused by manual, reactive changes. It’s the silent killer of scale.

The Fixes: From Firefighting to Engineering

Moving away from this model isn’t about buying a fancy new tool. It’s about changing your entire approach. Here’s how we did it, step-by-step.

1. The Quick Fix: The “Duct Tape” Playbook

You can’t boil the ocean overnight. The very first step is to stop the tribal knowledge. If you’re going to SSH in and fix something manually, you must document it. We started with a simple Git repo filled with Markdown files and simple bash scripts. It’s a hack, but it’s an effective one.

Instead of just running a command from memory, we’d write a small script. For that disk space issue, it looked something like this:

#!/bin/bash
# clear_archived_logs.sh
#
# WARNING: This is a temporary fix for prod-payment-gateway-02.
# The real fix is to implement proper log rotation via Ansible.
#
# Run: ./clear_archived_logs.sh

echo "Clearing archived logs from /var/log/app/archive..."
rm -rf /var/log/app/archive/*
echo "Done. Verifying disk space..."
df -h /

This isn’t automation, but it’s the first step toward it. It makes the process repeatable and visible to the whole team. It turns an individual’s “magic” into a documented procedure.

2. The Permanent Fix: Infrastructure as Code (IaC)

This is the real game-changer. This is the “one change” that mattered. We made a commitment to stop modifying running servers. Instead, we defined our entire infrastructure in code using tools like Terraform and Ansible. If a server had a problem, we didn’t fix it. We terminated it and deployed a new one from a known, good configuration.

Our mindset shifted from “How do I fix this machine?” to “How do I fix the code that builds this machine?”.

A Terraform configuration for a server isn’t just a script; it’s a declaration of the desired state. It’s the blueprint.

# main.tf - Example of an EC2 instance definition

resource "aws_instance" "payment_gateway" {
  ami           = "ami-0c55b159cbfafe1f0" # Latest Amazon Linux 2
  instance_type = "t3.medium"
  tags = {
    Name        = "prod-payment-gateway-new"
    Environment = "Production"
    ManagedBy   = "Terraform"
  }

  # This script runs on boot to install our app
  user_data = file("scripts/setup_payment_app.sh")

  # If this instance is ever changed manually, Terraform will detect it.
}

With this, deploying a fix for our logging issue meant updating an Ansible role for log rotation, building a new AMI, updating the `ami` variable in our Terraform code, and running `terraform apply`. The old, broken server was terminated, and a new, perfect one took its place with zero downtime thanks to our load balancer.

3. The ‘Nuclear’ Option: The “No SSH” Mandate

This is the change that cements the culture. It’s painful, and it requires buy-in, but it’s the most powerful move you can make. We instituted a policy: no more direct SSH access to production instances for making changes. Access was restricted to a “break-glass” bastion host that required a reason for access and triggered alerts to the whole team.

Heads Up: This will feel like you’re taking away your team’s power. They will complain. They’ll say it slows them down. And for the first couple of weeks, they might be right. But soon, they’ll realize that the time they used to spend firefighting at 3 AM is now spent improving the automation that prevents fires from ever starting. It’s a trade-off from short-term speed to long-term stability and sanity.

This mandate forces everyone to adopt the “Permanent Fix.” If you can’t log in to run `rm -rf`, you *have* to fix the Ansible playbook. It forces the right behavior. It’s the ultimate cure for configuration drift.

Ultimately, the secret isn’t about being a better firefighter. It’s about becoming an architect who designs a fireproof system. That’s the one change that lets you finally get a good night’s sleep.

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions