DNS in Kubernetes is a silent killer of application performance. Learn why the default settings can cause massive latency and how to fix it before it takes down your production environment at 3 AM.

What Actually Goes Wrong in Kubernetes? It’s Always DNS.

I still remember the night. 2:37 AM. PagerDuty was screaming bloody murder about cascading failures. Our main `billing-service` couldn’t reach the external payment processor. Seconds later, the `auth-api` started reporting timeouts trying to talk to our identity provider. Everything was healthy, pods were running, resources were fine, but services were just… timing out. My gut sank. It was that feeling every seasoned engineer knows: it smells like DNS. Again. We spent the next hour chasing ghosts in the network stack until we found the culprit, a subtle default that was silently crippling our performance. This isn’t just a war story; it’s a rite of passage for almost every team running Kubernetes at scale.

The “Why”: The Deceptive Default of `ndots:5`

So, what’s actually happening here? It boils down to how your pods resolve DNS names. Inside every pod, there’s a file, /etc/resolv.conf, that Kubernetes configures for you. By default, it looks something like this:

nameserver 10.96.0.10
search my-namespace.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

The search line is for convenience, letting you type my-service instead of the full my-service.my-namespace.svc.cluster.local. The real troublemaker is options ndots:5. This little line tells the resolver: “If a hostname has fewer than 5 dots in it, try appending the search domains before treating it as an absolute name.”

This is great for internal service-to-service communication. But what happens when your `billing-service` needs to talk to an external API, say, api.stripe.com?

1. The name api.stripe.com has 2 dots, which is less than 5.
2. So, instead of just looking it up, the resolver first tries: api.stripe.com.my-namespace.svc.cluster.local. (Fails)
3. Then it tries: api.stripe.com.svc.cluster.local. (Fails)
4. Then it tries: api.stripe.com.cluster.local. (Fails)
5. Finally, after all those pointless, time-consuming failures, it looks up api.stripe.com on the external resolver.

Under load, these extra, doomed-to-fail lookups create massive latency. Your application isn’t broken, Kubernetes isn’t broken, but the performance hit is real and can easily cause services to time out. This is one of the most common “mystery” issues I see hit teams in production.

The Fixes: From Duct Tape to Re-Architecture

Alright, you’re in the middle of an outage and you suspect this is the cause. What do you do? Here are three ways to handle it, from “get me out of this hole now” to “let’s make sure this never happens again.”

Solution 1: The Quick Fix (The “Restart It” Play)

Sometimes, CoreDNS (or its predecessor, kube-dns) can get into a weird state—a connection to an upstream resolver is stuck, or its cache is poisoned. The fastest, dirtiest way to see if this is the problem is to just give it a kick.

# Find your CoreDNS pods
kubectl get pods -n kube-system -l k8s-app=kube-dns

# Force a rollout which creates new pods with a fresh state
kubectl rollout restart deployment/coredns -n kube-system

Why it works: This forces Kubernetes to terminate the old CoreDNS pods and spin up new ones. The new pods start with a clean slate, re-establishing connections and clearing any bad cache. It’s the classic “turn it off and on again” and you’d be surprised how often it resolves temporary DNS hiccups.

Warning: This is a temporary fix. It doesn’t solve the underlying ndots:5 problem. If the latency is caused by search path lookups, this restart will do absolutely nothing except make you feel like you’re doing something.

Solution 2: The Permanent Fix (Tuning Your Pod’s `dnsConfig`)

The right way to solve the ndots:5 issue for a specific application that makes many external calls is to override the DNS configuration for that pod. You can tell it directly to be smarter about its lookups.

You do this by adding a dnsConfig section to your Deployment’s pod spec:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: billing-service
spec:
  template:
    spec:
      containers:
      - name: billing-service
        image: my-company/billing-service:1.2.3
      # --- Add this entire section ---
      dnsConfig:
        options:
          - name: ndots
            value: "2"
      # ------------------------------

Why it works: By setting ndots:2, you’re telling the pod’s resolver: “Only try the search path for names with less than 2 dots.” Now, when your app looks up api.stripe.com (2 dots), the resolver immediately treats it as an external, absolute domain and queries the upstream DNS. The pointless internal lookups are skipped, and the latency disappears. We typically set this to 2 or 1 for any service that is heavy on external API calls.

Solution 3: The ‘Nuclear’ Option (NodeLocal DNSCache)

If DNS is a constant source of pain across your entire cluster—causing latency, overwhelming CoreDNS, or creating single points of failure—it might be time to change your architecture. Enter NodeLocal DNSCache.

NodeLocal DNSCache runs a DNS caching agent as a DaemonSet on every single node in your cluster. Pods are configured to talk to the caching agent on their own node (at 169.254.20.10) instead of the central CoreDNS service IP.

Why it works:

• Performance: Most DNS queries are now resolved on the same node, avoiding a trip across the network to the CoreDNS pods. This dramatically reduces latency.
• Reliability: It avoids the networking complexities and potential bottlenecks of routing all traffic to a single ClusterIP (e.g., conntrack table exhaustion).
• Scalability: Your DNS caching capacity scales horizontally as you add more nodes to the cluster.

Setting it up is more involved (it’s a whole component to install and manage), but for large, high-traffic clusters, it’s an absolute game-changer for DNS stability. Think of it as moving from a central database to a distributed, read-replica model. It’s a fundamental shift, but one that pays massive dividends in resilience.

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions