Tired of services failing due to stale DNS entries or decommissioned hostnames? Here’s a senior engineer’s guide to fixing it, from the quick-and-dirty hack to the permanent architectural solution.

Why Is My Service Still Calling legacy-api.internal? A DevOps Guide to Ghost Hostnames

It was 2:47 AM. PagerDuty was screaming about our primary authentication service being down. Not degraded—down. I jump on the emergency bridge call, and a junior engineer, bless his heart, is frantically trying to restart the pods. “It’s not working,” he says, “the logs show `Connection refused` to `legacy-auth-db.internal`… but we decommissioned that database six months ago!” My blood ran cold. Somewhere, in some config file, in some long-forgotten deployment artifact, a ghost from our past was bringing our entire platform to its knees. This isn’t just a technical problem; it’s a symptom of technical debt, and if you don’t know how to handle it, you’re in for a long night.

The “Why”: More Than Just a Bad Config

When a service tries to connect to a hostname that no longer exists, it’s easy to blame a single missed environment variable. But the root cause is often deeper. It’s a ghost in the machine left behind by rushed migrations, incomplete documentation, and the developer mantra of “I’ll fix it later.”

This phantom hostname could be hiding anywhere:

• A hardcoded string in the application source code.
• An old value in a Kubernetes ConfigMap or Secret that wasn’t updated.
• A default value in a public container image you’re using.
• A stale entry in a service discovery tool like Consul that failed to de-register.

Simply finding the source can be a nightmare, especially when the service is down and everyone is looking at you. So, here’s how we, in the trenches, deal with it—from the emergency patch to the permanent fix.

Solution 1: The Quick Fix (The `/etc/hosts` Band-Aid)

This is your “get the site back up in the next five minutes” move. It’s dirty, it’s temporary, and you should feel a little bad about doing it, but it works. The goal is to trick the application on the server itself into resolving the old hostname to the new, correct IP address.

Let’s say the pod auth-service-pod-5f8d7c9c4c-xyz12 is trying to reach legacy-auth-db.internal, but the new database is at prod-auth-db-01.internal (IP: 10.50.2.101).

You shell into the running container:

kubectl exec -it auth-service-pod-5f8d7c9c4c-xyz12 -- /bin/bash

Then, you edit the hosts file. You’ll probably need to install a text editor if you’re using a minimal base image.

# First, update package list and install an editor
apt-get update && apt-get install -y vim

# Now, edit the hosts file
vim /etc/hosts

And you add this magic line at the bottom:

# TEMPORARY FIX - REDIRECT OLD DB HOSTNAME - REMOVE BY 9 AM
10.50.2.101    legacy-auth-db.internal

The service should immediately be able to connect, and the incident is resolved. But you’ve just created a ticking time bomb. What happens when the pod restarts? Your change is gone. You are now manually managing DNS on a single machine.

Warning: This is NOT a permanent solution. It’s a battlefield triage technique. If you leave this in place, you WILL cause a future, more confusing outage. Create a high-priority ticket to track the real fix and assign it to yourself.

Solution 2: The Permanent Fix (Configuration as Code)

Now that the fire is out, you need to do the real work. You need to find where that ghost hostname lives and exorcise it properly. This is about paying down that technical debt. Your investigation should follow the code and configuration deployment pipeline.

Step 1: Grep the Codebase

Search your application’s source code repository for the string `legacy-auth-db.internal`. If you find it hardcoded, that’s a code change, a pull request, and a chat with the dev team about best practices.

Step 2: Check Your IaC and Configs

More than likely, it’s in your infrastructure or deployment configuration. Search your Git repos for where you manage Kubernetes manifests, Helm charts, or Terraform configurations.

You’re looking for something like this in a Helm `values.yaml` file:

# values.yaml
replicaCount: 3
image: "auth-service:1.2.5"

config:
  database_host: "legacy-auth-db.internal" # <-- THERE IT IS!
  database_port: "5432"

You find it, you change it to `prod-auth-db-01.internal`, you commit the change, and you re-deploy the application through your CI/CD pipeline. The fix is now version-controlled, auditable, and permanent. Once the new version is deployed, you can remove that `/etc/hosts` entry you felt so guilty about.

Solution 3: The ‘Nuclear’ Option (DNS Hijacking)

Sometimes, you have dozens of services, all managed by different teams, all potentially calling this old hostname. Fixing every single one isn’t feasible in the short term. This is when you can set up a “DNS sinkhole” or a CNAME record to intercept all requests for the old hostname and redirect them at the network level.

This is the infrastructure equivalent of a wide-net search campaign. You’re capturing all the “traffic” intended for the old name and pointing it where you want it to go.

In your DNS provider (like AWS Route 53, or your internal CoreDNS config), you create a new record:

Record Name: legacy-auth-db.internal
Record Type: CNAME (Canonical Name)
Value/Points to: prod-auth-db-01.internal

Now, any client anywhere in your VPC that asks for legacy-auth-db.internal will be told by the DNS resolver, “Actually, the real name is prod-auth-db-01.internal, go talk to it.”

Pro Tip: This is a powerful tool, but it can also mask underlying problems. Services are still configured incorrectly, but they work. I recommend pointing the CNAME to the new service for stability, but also setting up logging on your DNS resolver to see WHICH clients are still making requests to the old name. This gives you a hit list for tracking down and fixing the configurations one by one.

Comparing the Approaches

Approach	Speed	Risk	Permanence
1. /etc/hosts Band-Aid	Immediate	High (Easy to forget, not scalable)	None (Disappears on restart)
2. Config as Code	Slow (Requires redeploy)	Low (The “correct” way)	High (Permanent, versioned)
3. DNS Hijacking	Fast (DNS propagation time)	Medium (Can hide root causes)	Medium (A crutch, not a true fix)

In the end, there’s no silver bullet. You’ll probably use the `/etc/hosts` trick at 3 AM. You’ll implement the DNS CNAME the next morning to provide a safety net. And you’ll spend the rest of the week hunting down those incorrect configurations to do it right. Welcome to DevOps.

---

🤖 Frequently Asked Questions