Aggressive system pruning can inadvertently delete critical, untagged container images or volumes needed for rollbacks or debugging, leading to production failures. Here’s how to implement smarter, safer cleanup strategies without losing important data.

The Pruning Trap: Why Your CI/CD Keeps Deleting Things You Need

It was 2 AM on a Tuesday. I remember it because my pager, which is supposed to be for P1 production-down emergencies only, went off. The on-call engineer was in a panic. A hotfix deployment had failed catastrophically, and the automated rollback was also failing. The error? 'image not found'. I logged into the build server, a box named ci-runner-build-node-5, and my blood ran cold. The specific image hash needed for the rollback—the last known stable version—was just… gone. A well-meaning but overzealous cron job running docker system prune -af had wiped it out hours earlier because it was “dangling” and “unused.” We spent the next three hours rebuilding the old version from source control. That night, I swore I’d never let a simple cleanup script hold production hostage again.

The Root of the Problem: Context is Everything

Let’s be clear: pruning isn’t evil. Tools like docker system prune or Kubernetes garbage collection are essential for keeping our servers from drowning in gigabytes of old layers, dangling images, and stopped containers. The problem isn’t the tool; it’s the lack of context. These commands are brutally efficient. They see an image without a tag or a volume not attached to a running container and declare it “unused.”

But “unused” right now doesn’t mean “unimportant.” That untagged image might be:

• The exact version currently running in production (referenced by its hash/digest).
• The last known good build, critical for an emergency rollback.
• A base image for a build that only runs once a week.
• A cached data volume you need for post-mortem debugging.

Pruning tools don’t know your business logic. They just follow the rules, and sometimes, those rules are too simple for the complex reality of our CI/CD pipelines. This is how you lose data that seems invisible until it’s critically needed.

The Solutions: From Quick Fix to Permanent Strategy

So how do we fix it without letting our disks fill up? We get smarter. Here are three approaches I’ve used, ranging from a quick band-aid to a proper architectural change.

Solution 1: The Tactical Band-Aid (Using Filters)

This is the fastest way to stop the bleeding. The docker system prune command isn’t just an on/off switch; it has filters. You can tell it to ignore certain things. The most useful filter is based on labels.

First, start labeling the images or containers you want to keep. For example, when you build a critical image, add a label:

docker build -t myapp:1.2.3 --label "keep=true" .

Then, modify your pruning command to respect that label. Instead of a blunt prune -af, you use a filter:

# This will prune all dangling images EXCEPT those with the 'keep=true' label
docker image prune -a --filter "label!=keep=true"

This is a “hacky” but effective way to immediately protect your critical assets without rewriting your entire cleanup process. It puts the responsibility on the build process to “flag” what’s important.

Solution 2: The Strategic Fix (Better Tagging & a Real Registry)

Relying on images existing on a local build runner is a fragile strategy. The real permanent fix is to treat your container images like the first-class build artifacts they are. This means two things:

1. Immutable, Semantic Tagging: Stop using the :latest tag for anything other than local dev. Every single image pushed from CI should have a unique, immutable tag. Use the Git commit hash (e.g., myapp:1.2.3-a8c4efd) or a build number. This way, you are never guessing what version an image contains.
2. Use an Artifact Registry: Push every build to a proper container registry (Docker Hub, AWS ECR, GCP Artifact Registry, Harbor). These tools are built for this. You can then configure retention policies on the registry itself, like “keep the 10 most recent images for this repository” or “delete any image older than 90 days that isn’t tagged as a production release.”

This approach moves the source of truth from a volatile local disk to a managed, durable, and auditable service. The build runner can be pruned aggressively because the important images are safe elsewhere.

Darian’s Pro Tip: If you go this route, make sure your deployment scripts pull images by their unique hash or immutable tag (myapp@sha256:... or myapp:1.2.3-a8c4efd), not a floating tag like :stable. This guarantees you are deploying the exact artifact you tested.

Solution 3: The ‘Controlled Demolition’ (A Smarter Prune Script)

Sometimes, you can’t use a remote registry, or you have specific stateful data in volumes you need to manage locally. In this case, you replace the blunt docker system prune command with your own, smarter script. This is the “nuclear” option because you’re taking on the full responsibility for garbage collection.

Here’s a conceptual bash script you might run on a cron job:

#!/bin/bash

# Get a list of image hashes currently used by running containers on prod-db-01 and web-frontend-cluster
PROD_IMAGES=$(ssh user@prod-db-01 'docker ps -q | xargs docker inspect --format="{{.Image}}"')
WEB_IMAGES=$(ssh user@web-frontend-cluster 'docker ps -q | xargs docker inspect --format="{{.Image}}"')

# Get all local image IDs
ALL_IMAGES=$(docker images -q)

# Loop through all local images and check if they are in the 'prod' or 'web' list
for img in $ALL_IMAGES; do
    if [[ ! " ${PROD_IMAGES[@]} " =~ " ${img} " ]] && [[ ! " ${WEB_IMAGES[@]} " =~ " ${img} " ]]; then
        # This image is not running in our key environments.
        # Add more logic here: Check age? Check labels?
        # For now, let's assume it's safe to remove if it's not in use.
        echo "Pruning unused image: $img"
        # docker rmi --force $img
    fi
done

# Now, handle old, unattached volumes more carefully
# Prune volumes older than 7 days (24*7=168 hours)
docker volume prune -f --filter "until=168h"

This script is far more intelligent. It queries your production environment to build an “allow list” of images that absolutely cannot be deleted. It introduces logic and context that the default prune command lacks. It’s more work to set up and maintain, but it gives you total control.

Comparing the Approaches

Each solution has its place. Here’s how I think about them:

At the end of the day, that 2 AM outage taught me a valuable lesson: housekeeping is not just about freeing up disk space. It’s about risk management. By being deliberate and adding context to our cleanup processes, we can keep our systems lean without accidentally deleting the one thing that will save us during our next emergency.

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions