Chasing down intermittent DNS resolution failures in a containerized environment can feel like a wild goose chase. Here’s a breakdown of why it happens and how to fix it, from a quick hack to a permanent architectural solution.

Why Can My Service Only Reach `streams99.com`? A Deep Dive into DNS Hell

I’ll never forget the 3 AM PagerDuty alert. A critical microservice, let’s call it `auth-service`, was failing all its health checks. The junior on-call was panicking. Logs showed “Unknown host” for every single downstream dependency: `prod-db-01.internal`, `redis-cache.internal`, you name it. He SSH’d into the box, `exec`’d into the container, and started trying to `curl` things. Everything failed. In a desperate, caffeine-deprived move, he tried to `curl streams99.com`—a random site from his browser history. It worked. Instantly. How can a container resolve some random external domain but not its own critical internal dependencies? If you’ve ever been in that situation, you know the feeling of pure confusion mixed with rage. It’s not magic; it’s almost always DNS.

The “Why”: Your Container is Living in a DNS Bubble

When your container starts, it doesn’t magically know how to find other services. The container runtime (like Docker or Kubernetes) hands it a configuration file, typically /etc/resolv.conf. This file tells your container, “Hey, when you need to look up a name, ask these servers.”

The problem arises when those designated DNS servers are:

• Overloaded: The internal DNS resolver is swamped and starts dropping requests.
• Misconfigured: The server itself has incorrect forwarding rules or is pointing to a dead upstream server.
• Blocked by a Firewall: A new security group rule was applied, and suddenly your container’s outbound UDP/53 traffic is getting dropped, except for traffic going to a specific public resolver that was whitelisted.

In our war story, the internal resolver was failing, but the host machine’s secondary resolver was a public one (like Google’s 8.8.8.8), which could resolve public domains but knew nothing about our internal hostnames. This creates the bizarre scenario where the outside world is reachable, but your own datacenter isn’t.

The Fixes: From Duct Tape to a New Engine

Depending on how much sleep you’ve had and how permanent you need the solution to be, here are three ways to tackle this.

1. The Quick Fix (The “It’s 3 AM and I Need to Sleep” Fix)

This is the digital equivalent of hitting it with a hammer. You force the misbehaving container to use a reliable public DNS server. It’s a temporary hack that will get the service back online, but it will be wiped out the next time the container restarts.

You can do this by getting a shell inside the running container and manually overwriting its resolver configuration.

# Find your container ID
docker ps

# Exec into the container (replace with your container ID)
docker exec -it a1b2c3d4e5f6 /bin/sh

# Inside the container, overwrite the resolv.conf file
# This tells it to use Google's DNS and Cloudflare's as a backup
echo "nameserver 8.8.8.8" > /etc/resolv.conf
echo "nameserver 1.1.1.1" >> /etc/resolv.conf

Why it’s a hack: This change is ephemeral. As soon as the container is recreated or the host reboots, it will revert to the old, broken configuration. It also won’t work for internal-only hostnames.

2. The Permanent Fix (The “Let’s Do This Right” Fix)

The right way to solve this is at the source: the container runtime configuration. You can configure the Docker daemon itself to provide specific DNS servers to every container it creates. This ensures all containers, now and in the future, start with a sane and reliable DNS configuration.

You do this by editing the Docker daemon’s configuration file, usually located at /etc/docker/daemon.json.

{
  "dns": ["10.0.0.2", "8.8.8.8"]
}

After saving this file, you need to restart the Docker daemon for it to take effect:

sudo systemctl restart docker

Now, every new container will have a resolv.conf file that points to your reliable internal resolver (10.0.0.2) and falls back to a public one if needed.

Pro Tip: Be careful about hardcoding public DNS servers. Some security policies forbid it. Always check with your security team and prefer to use your organization’s designated, highly-available internal resolvers first.

3. The ‘Nuclear’ Option (The “Architect for Resilience” Fix)

For mission-critical environments, relying on external or even central internal DNS can be a single point of failure. The most robust solution is to run a local DNS caching resolver on each container host or as a sidecar in your Kubernetes pods. Tools like CoreDNS or dnsmasq are perfect for this.

In this model, your containers are configured to talk to their local resolver (at an address like 127.0.0.1 or the pod’s local DNS service). This local cache handles the request. If it has the answer, it returns it instantly. If not, it forwards the request to the upstream resolvers. This gives you speed, resilience to network hiccups, and a great place to add logging and metrics for DNS performance.

Here’s a quick comparison of the approaches:

So next time a service can only reach a single, random website, don’t panic. Take a deep breath, check /etc/resolv.conf, and remember that you have options—from the quick and dirty to the architecturally sound.

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions