Running out of disk space on a production server is a classic, painful lesson. Learn why it happens and how to implement quick, permanent, and automated fixes to prevent catastrophic outages and late-night pages.

The Silent Killer: When a 100GB Log File Took Down Production

It was 2:17 AM. My phone screamed with a PagerDuty alert. The kind of alert that makes your heart sink: `CRITICAL – prod-db-01 – UNREACHABLE`. The primary database for our biggest client was down. Hard down. SSH timed out. The cloud provider’s console showed it was running, but it wasn’t responding to anything. After a painful 15 minutes wrestling with the emergency web console, I finally got a shell. The first command I ran, almost by muscle memory, was `df -h`. And there it was, glowing in red: `/dev/sda1 … 100% /`. The root partition was completely, utterly full. We weren’t hacked, the database hadn’t corrupted itself… we were killed by our own verbosity. A single, forgotten debug log file had eaten every last byte of space, and the OS simply gave up.

Why This Is More Than Just a “Full Disk”

Listen, a full disk isn’t just a storage problem; it’s a symptom of a deeper, systemic issue. When a critical service can’t write to disk, all hell breaks loose. It can’t create PID files, it can’t write to its own logs (ironic, huh?), and sometimes you can’t even log in to fix it because the system can’t create a new session file. The root cause is never just “the logs grew too big.” It’s usually a combination of failures:

• No Log Rotation: The most common culprit. A process is just endlessly appending to a file with no mechanism to cycle it.
• Lack of Monitoring: The problem didn’t happen in a split second. That disk was filling up for days, maybe weeks. An alert at 85% capacity would have turned a 2 AM emergency into a 2 PM routine ticket.
• Production Debugging: A developer left verbose debug logging on for a new feature. Great for staging, catastrophic for production.
• Poor Partitioning: Lumping `/var/log` in with the root partition (`/`) means that runaway logs can take down the entire operating system, not just the application.

Triage and Fixes: From 2 AM Panic to Permanent Solution

So you’re in the hot seat. The server is down and everyone’s looking at you. Here’s how you handle it, from the immediate panic to making sure this never, ever happens again.

Solution 1: The “Get It Back Online NOW” Quick Fix

Your only goal right now is to restore service. We don’t have time for elegance. We need space, and we need it five minutes ago. First, find the offender.

# Find the top 10 largest files/directories in /var
du -ah /var | sort -rh | head -10

You’ll probably see something like `/var/log/app/application.log` at a ridiculous size. Your first instinct is to run `rm application.log`. DO NOT DO THIS. If a process has that file open, `rm` will delete the file’s name, but the process will keep its file handle and continue writing to the now-nameless file. The disk space won’t be freed until you restart the service.

Pro Tip: You can see which process has a file open with `lsof /path/to/big/logfile.log`. This is a lifesaver for identifying the culprit application without guesswork.

The correct, immediate fix is to truncate the file. This command empties the file without deleting it, instantly freeing up the disk space.

# This empties the file without breaking the open file handle
> /var/log/app/application.log

The server will breathe again. Restart the application to be safe, and you’re back online. Now, go get some coffee and prepare for Phase 2.

Solution 2: The “Never Again” Permanent Fix

The next morning, you need to make sure this isn’t a recurring nightmare. The answer is simple, old-school, and beautiful: `logrotate`.

Most Linux distros have it installed. You just need to give it a configuration file. Create a file like `/etc/logrotate.d/my-awesome-app` and put something like this inside:

/var/log/app/*.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    copytruncate
}

This tells the system to, on a daily basis, rotate any `.log` files in that directory. It will keep 7 of them, compress the old ones, and use `copytruncate` (which copies the content then truncates the original) to play nice with running services. This, combined with setting up disk usage alerting in your monitoring tool (Prometheus, Datadog, etc.), will prevent 99% of these issues.

Solution 3: The “Architectural” Fix

This is the real, modern DevOps answer. Stop treating server disks as a permanent resting place for logs. Servers are cattle, not pets, and their local storage should be considered ephemeral.

The solution is to ship your logs off-platform immediately. The architecture looks like this:

1. Your application logs to standard output (`stdout`) and standard error (`stderr`), not to a file.
2. A lightweight log forwarder (like Fluentd, Vector, or Promtail) running on the server (or as a DaemonSet in Kubernetes) captures everything written to `stdout`/`stderr`.
3. This agent forwards the logs to a centralized logging platform like the ELK Stack (Elasticsearch, Logstash, Kibana), Splunk, Graylog, or a cloud service like AWS CloudWatch Logs.

In this model, local disk usage for logs becomes a non-issue. It’s a small, temporary buffer at most. This completely decouples your application’s health from the local disk state and makes your logs searchable and manageable across your entire fleet.

Comparing the Approaches

Final Thoughts

We’ve all been there, staring at a dead server at an ungodly hour. Learning this lesson the hard way is practically a rite of passage. But the real mark of a senior engineer isn’t just knowing how to fix it in a panic; it’s about having the discipline to implement the permanent or architectural solutions so that the 2 AM page never happens again. Automate, monitor, and for goodness sake, rotate your logs.

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions