🚀 Executive Summary

TL;DR: Docker image pulls that hang without a VPN but succeed with one typically point to underlying network issues like an MTU size mismatch or a DNS conflict. The problem can be diagnosed and resolved by adjusting Docker’s MTU, configuring the host network interface, or temporarily bypassing corporate DNS resolvers.

🎯 Key Takeaways

  • Docker pull failures that are resolved by a VPN connection are almost always network-related, primarily due to MTU mismatches or corporate DNS issues, not Docker itself.
  • An MTU mismatch occurs when network packets exceed a router’s or firewall’s maximum transmission unit, leading to packet drops; VPNs often fix this by automatically lowering the effective MTU.
  • Solutions range from a quick diagnostic fix by forcing a lower MTU in `daemon.json` for Docker, to a permanent host-level MTU adjustment, or diagnosing DNS problems by configuring Docker to use public DNS resolvers.

Stuck on a Docker pull that hangs forever, only to work magically when you connect to a VPN? The culprit is likely a network issue like an MTU size mismatch or a DNS conflict, not Docker itself. Here’s a field guide to diagnosing and fixing it for good.

Why Did I Have to Use a VPN to Pull Docker Images? A Senior Engineer’s Field Guide

It was 2 AM. A critical production deployment on prod-api-03 was failing. Not with a bang, but with a whimper. The logs just showed Pulling fs layer..., followed by an agonizingly slow progress bar that never moved, and then eventually a timeout. Everything else on the box worked—curl to google.com, ping, even ssh to other instances. For 30 painful minutes, we were stumped. Then, on a whim, a junior engineer on my team said, “I’m going to try connecting to the corporate VPN from the server… just to see.” We all scoffed. And then it worked. The pull finished in seconds. That was the night I learned that sometimes, the most baffling Docker problems have nothing to do with Docker at all.

The Root Cause: It’s (Almost) Always The Network

When you see this behavior, your gut should immediately scream “NETWORK!” ninety-nine percent of the time. Docker is just the victim. This weird “VPN fixes it” scenario almost always boils down to one of two things:

  1. MTU Mismatch (The usual suspect): MTU stands for Maximum Transmission Unit. Think of it as the maximum size of a package your network can send in one go. The default is typically 1500 bytes. However, somewhere between your server and Docker Hub, a router, firewall, or switch might enforce a smaller MTU. When your server sends a 1500-byte packet that hits a link expecting 1450, that packet gets dropped. Your server never gets a response and just sits there, retrying. When you connect to a VPN, the VPN client often automatically sets a lower MTU (e.g., 1400) to account for its own overhead. Suddenly, your packets are small enough to get through, and the Docker pull works.
  2. DNS Issues (The less common culprit): Sometimes, the corporate DNS servers you’re assigned via DHCP are flaky, slow, or are filtering requests to certain domains that Docker Hub’s CDN relies on. A VPN often forces the use of its own, more reliable DNS servers, bypassing your local network’s DNS resolver and solving the problem.

The Solutions: From Quick Hack to Permanent Fix

Okay, theory is great, but you’ve got a pipeline to fix. Here are the three ways I tackle this, in order of escalation.

1. The Quick & Dirty Fix: Force Docker’s MTU

This is my go-to for confirming the diagnosis. We’re going to tell the Docker daemon itself to use a smaller MTU for all its network traffic. This doesn’t fix the underlying network issue, but it gets Docker working right now.

First, create or edit the Docker daemon configuration file at /etc/docker/daemon.json.

{
  "mtu": 1400
}

Then, restart the Docker daemon:

sudo systemctl restart docker

Try your docker pull again. If it works, you’ve confirmed an MTU issue. You can play with the value—1450, 1400, 1300—to find what works, but 1400 is usually a safe bet.

Pro Tip: Don’t leave this as the permanent solution without documenting it. The next engineer who comes along will have no idea why you’ve set a custom MTU, and it can cause confusion down the line. Use it to get unblocked, then pursue the permanent fix.

2. The “Right Way”: Fix The Host Network

Forcing the MTU in Docker is a band-aid. The real problem is on the host’s network interface or somewhere upstream. The proper long-term solution is to fix the network path.

Option A: Set the MTU on the Host Interface

You can set the MTU directly on the server’s primary network interface. This is better than the Docker-specific fix because it makes the change transparent to all applications on the server. How you do this depends on your Linux distribution (e.g., editing /etc/network/interfaces on Debian/Ubuntu or using nmcli on RHEL/CentOS). The immediate command to test this would be:

# Find your primary interface, e.g., eth0
ip link set dev eth0 mtu 1400

If that works, make the change permanent in your netplan/ifcfg/network-scripts configuration.

Option B: Talk to Your Network Team

In a perfect world, you shouldn’t have to do this at all. A mechanism called Path MTU Discovery (PMTUD) is supposed to handle this automatically. Its failure often points to misconfigured firewalls that silently drop “ICMP Fragmentation Needed” packets. File a ticket with your network team. Tell them, “We’re experiencing packet loss to Docker Hub from prod-api-03, likely due to a PMTUD black hole. We suspect a firewall is dropping necessary ICMP packets. Can you investigate the path?” They’ll thank you for the detailed diagnosis.

3. The ‘Nuclear’ Option: Change Docker’s DNS

If the MTU trick didn’t work, it’s time to investigate DNS. Corporate DNS can be a real headache. To rule it out, you can tell the Docker daemon to ignore the host’s DNS and use public resolvers instead.

Edit your /etc/docker/daemon.json again:

{
  "dns": ["8.8.8.8", "1.1.1.1"]
}

And restart Docker. If this suddenly fixes your pull, you have a corporate DNS problem.

Warning! Bypassing corporate DNS can have serious security implications. It might prevent you from resolving internal-only hostnames and could violate your company’s security policy. Use this for diagnosis only. If it works, take your findings (e.g., “We can’t pull images unless we use Google’s DNS”) to your security and network teams to get the internal resolvers fixed.

Summary: Your Troubleshooting Cheat Sheet

Next time a Docker pull hangs, don’t blame Docker. Blame the network and follow these steps.

Solution When to Use It Caveat
1. Set MTU in daemon.json Quickest way to confirm an MTU issue and get unblocked. It’s a band-aid. The underlying network problem still exists.
2. Fix Host Network MTU The proper, permanent fix when you control the host configuration. Requires coordination with network admins for upstream issues (PMTUD).
3. Set DNS in daemon.json To diagnose if flaky or restrictive corporate DNS is the culprit. DANGER: Use for diagnosis only. Can break internal resolution and violate security policy.

At the end of the day, remember the golden rule of operations: It’s always the network. Until it’s DNS. Then it’s the network again.

Darian Vance - Lead Cloud Architect

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.


🤖 Frequently Asked Questions

âť“ Why do my Docker pulls only work when I’m on a VPN?

Docker pulls failing without a VPN but succeeding with one usually indicates a network issue, specifically an MTU mismatch or a DNS problem, rather than a Docker bug. The VPN often compensates for these network path issues.

âť“ What’s the difference between setting MTU in `daemon.json` versus on the host interface?

Setting MTU in `daemon.json` is a Docker-specific band-aid primarily for diagnosis, affecting only Docker’s network traffic. Setting the MTU on the host interface is a more transparent, permanent fix that applies to all applications on the server.

âť“ What are the risks of using public DNS resolvers in Docker’s configuration?

Bypassing corporate DNS with public resolvers (e.g., 8.8.8.8) can prevent resolution of internal-only hostnames and may violate company security policies. It should be used for diagnosis only, with findings reported to network and security teams.

Leave a Reply

Discover more from TechResolve - SaaS Troubleshooting & Software Alternatives

Subscribe now to keep reading and get access to the full archive.

Continue reading