🚀 Executive Summary

TL;DR: Docker defaults to storing data in /var/lib/docker, often filling the root partition even after new NVMe storage is added. The primary solution involves explicitly reconfiguring Docker to use the new storage location, either via a quick symlink or, preferably, by setting the ‘data-root’ parameter in daemon.json for a permanent fix.

🎯 Key Takeaways

  • Docker’s default data directory at /var/lib/docker can exhaust root partition space, even with new storage, if not reconfigured.
  • Always stop the Docker daemon (sudo systemctl stop docker and sudo systemctl stop docker.socket) before attempting to move Docker data or modify its configuration.
  • The ‘daemon.json’ method, configuring the ‘data-root’ parameter in /etc/docker/daemon.json, is the officially supported and recommended approach for permanently relocating Docker’s data directory.

Added NVME Storage - Where's Docker?

Just added a massive new NVMe drive to your server but Docker is still filling up your tiny root partition? Here’s how to tell Docker where its new home is, from the quick symlink hack to the proper daemon configuration.

Added NVME Storage – So, Where’s Docker?

I remember one frantic Tuesday. We were getting alerts that `prod-db-01` was critically low on disk space. Impossible, I thought, we just added a 2TB NVMe to that box last quarter. I SSH’d in, ran `df -h`, and my heart sank. The root partition, `/`, was at 98% capacity. The new `/mnt/data` drive? Sitting there, empty and pristine. The culprit, as it so often is, was Docker. It had happily churned through gigabytes of images and volumes, completely oblivious to the spacious new drive we gave it. It’s a classic, infuriating “right hand doesn’t know what the left hand is doing” problem, and it can take a production server offline if you’re not careful.

So, What’s Actually Happening?

This isn’t a bug; it’s a feature of default configurations. When you install Docker, it doesn’t scan your system for the “best” place to store its data. It follows a simple rule: create a home at /var/lib/docker and put everything there. Your images, your volumes, your containers, all of it. It will stay there forever unless you explicitly tell it to move. Adding a new drive is great, but Docker is blissfully unaware of it until you connect the dots.

How to Fix It: From Quick & Dirty to Clean & Proper

You’ve got a few ways to tackle this, each with its own pros and cons. Let’s walk through them, starting with the fastest way to stop the bleeding and moving to the solution you’ll want for the long term.

Heads Up! Before you try any of these, STOP THE DOCKER DAEMON. Don’t just stop your containers, stop the service itself. If you try to move files while it’s running, you’re going to have a very, very bad day. Use sudo systemctl stop docker and sudo systemctl stop docker.socket.

Solution 1: The Quick Fix (The Symlink Shuffle)

This is the “it’s 3 AM and I need the server back online now” solution. It’s a bit of a hack, but it’s fast and effective. We’re going to move the existing Docker directory to the new drive and then create a symbolic link (symlink) from the old location to the new one. Docker won’t even know it’s been moved.

Let’s assume your new drive is mounted at /mnt/nvme-data.

  1. Stop Docker: As mentioned above, this is critical.
    sudo systemctl stop docker
    sudo systemctl stop docker.socket
  2. Move the directory: We’ll use rsync because it preserves permissions and gives you a nice progress bar.
    sudo rsync -avz /var/lib/docker /mnt/nvme-data/
  3. Rename the old directory: Don’t delete it yet! We keep it as a backup just in case something goes wrong.
    sudo mv /var/lib/docker /var/lib/docker.old
  4. Create the symlink: This is the magic trick.
    sudo ln -s /mnt/nvme-data/docker /var/lib/docker
  5. Restart and Verify: Start Docker back up and check that it’s happy.
    sudo systemctl start docker
    docker ps -a

Once you’re confident everything is working, you can safely delete the backup: sudo rm -rf /var/lib/docker.old.

Solution 2: The Permanent Fix (The `daemon.json` Method)

This is the “right” way to do it. It’s the officially supported method and avoids the potential weirdness of symlinks during system updates. You’re telling the Docker daemon itself where its root directory should be.

  1. Stop Docker: You know the drill.
    sudo systemctl stop docker
    sudo systemctl stop docker.socket
  2. Create/Edit the config file: The Docker daemon configuration lives in /etc/docker/daemon.json. It might not exist, so you may be creating it from scratch.
    sudo nano /etc/docker/daemon.json
  3. Add the `data-root` path: Add this JSON content to the file. Make sure your new path (e.g., /mnt/nvme-data/docker) exists.
    {
      "data-root": "/mnt/nvme-data/docker"
    }
  4. Move the data: If you have existing Docker data, you still need to move it.
    sudo rsync -avz /var/lib/docker/ /mnt/nvme-data/docker/
  5. Restart and Verify: Start the daemon. It will now read the config file and use the new path.
    sudo systemctl start docker
    docker info | grep "Docker Root Dir"

    The output of that last command should show your new path. If it does, you’ve succeeded! You can now safely remove the old /var/lib/docker directory.

Solution 3: The ‘Nuclear’ Option (The Clean Slate)

Sometimes, your Docker installation is already a mess of dangling images and unused volumes. Maybe you don’t need to preserve the old data. In this case, starting fresh can be the cleanest path forward. This is for non-production environments or when you have a solid backup/restore process for your persistent volumes.

  1. Back up what you need: Identify any critical volumes you need to save. Use docker volume ls and back up the data from within /var/lib/docker/volumes/.
  2. Purge Docker: Don’t just uninstall it, purge it. This removes the configuration files and the /var/lib/docker directory.
    sudo apt-get purge docker-ce docker-ce-cli containerd.io
    sudo rm -rf /var/lib/docker
    sudo rm -rf /var/lib/containerd
  3. Configure Before Installing: This is the key step. Before you even run `apt-get install` again, create the /etc/docker/daemon.json file from Solution 2 and point it to your new, empty directory on the NVMe drive.
    sudo mkdir -p /mnt/nvme-data/docker
    sudo nano /etc/docker/daemon.json
    
    # Add this content:
    # {
    #   "data-root": "/mnt/nvme-data/docker"
    # }
  4. Reinstall Docker: Now, install Docker as you normally would. When the service starts for the first time, it will see the daemon.json file and build its data directory in the correct location from the very beginning.

My Take: Which One Should You Choose?

Here’s how I break it down for my team.

Method Best For Downside
1. Symlink Emergencies. When you need to fix a full disk right now and can’t afford downtime to reconfigure. It’s a “hack”. It can sometimes cause issues with package managers or Docker upgrades that don’t expect a symlink.
2. `daemon.json` Almost all situations. This is the clean, supported, and professional way to handle it. Do this for any production server. Requires a bit more care (editing a JSON file) and a service restart. Minimal downside, really.
3. Nuke & Pave New server setups or when the existing Docker state is not worth saving. Great for dev environments. Data loss. You are intentionally deleting everything. Not for systems with stateful containers unless you have solid backups.

Let’s be real: we’ve all used the symlink method to get out of a jam. There’s no shame in it. But for any long-term, stable system, take the extra five minutes and use the daemon.json configuration. Your future self, and the junior engineer who inherits the server from you, will thank you for it.

Darian Vance - Lead Cloud Architect

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.


🤖 Frequently Asked Questions

âť“ Why is Docker filling my root partition despite having a new NVMe drive?

Docker defaults to storing all its data (images, volumes, containers) in /var/lib/docker on the root partition and is unaware of new storage unless explicitly configured to use it.

âť“ How does using a symlink compare to the daemon.json method for moving Docker’s data?

The symlink method is a quick, emergency fix that creates a symbolic link from /var/lib/docker to the new location. The daemon.json method is the permanent, officially supported solution where you configure the ‘data-root’ parameter in /etc/docker/daemon.json to point to the new path, offering better stability and compatibility with Docker upgrades.

âť“ What is a common implementation pitfall when relocating Docker’s data directory?

A critical pitfall is attempting to move Docker’s data while the daemon is still running, which can lead to data corruption. The solution is to always stop the Docker daemon completely using ‘sudo systemctl stop docker’ and ‘sudo systemctl stop docker.socket’ before any modifications.

Leave a Reply

Discover more from TechResolve - SaaS Troubleshooting & Software Alternatives

Subscribe now to keep reading and get access to the full archive.

Continue reading