Services still misbehaving after a scheduled maintenance window? You’re likely fighting stale DNS caches or zombie connections. Here’s how to diagnose the real problem and fix it, from quick command-line tricks to permanent architectural solutions.

That’s a Cache! Why Your “Fixed” Service is Still Broken After Maintenance

I’ll never forget it. It was 3:00 AM, my second year on the job, and my first time leading a critical database failover. The maintenance window was from 2:00 to 4:00 AM. We swapped the primary database CNAME, `prod-db.techresolve.com`, to point to the new server, `prod-db-secondary-01`. All our monitoring dashboards lit up green. The database was up, responding to pings, and my direct queries worked perfectly. We called it a success at 3:45 AM. But by 8:00 AM, my inbox was on fire. Half the company was getting connection errors. I was sweating bullets, convinced I’d corrupted something, only to discover our web application servers were stubbornly holding onto the old database IP address in their connection pools. They never bothered to re-resolve the DNS name. We fixed it by restarting the app servers, but I learned a hard lesson that day: “fixed” on the backend doesn’t mean “fixed” for the user.

The “Why”: It’s Not Broken, It’s Just Remembering Wrong

When your service is still failing after you’ve switched an IP or failed over a resource, the root cause is almost always a layer of caching that’s doing its job *too well*. This is a feature, not a bug, designed to make things fast. But during a change, it becomes your biggest enemy. There are two main culprits here:

• DNS Caching: To avoid constantly asking a DNS server “Where is google.com?”, every part of the chain—from your browser to your OS to the application itself—keeps a local copy of the answer for a period of time (called Time To Live, or TTL). If your TTL was set to 24 hours and you make a change, some systems won’t know about it for a full day unless you force them to.
• Connection Pooling: Applications, especially those talking to databases, are lazy. Opening and closing connections is expensive. So, they open a bunch of connections at startup (a “pool”) and reuse them. If your app started up when `prod-db.techresolve.com` pointed to `10.0.1.10`, it will keep using its open connection to that IP, completely oblivious to the fact that the DNS name now points to `10.0.1.50`.

These two issues create a frustrating situation where your infrastructure is perfectly healthy, but your application is living in the past.

The Fixes: From a Quick Slap to a Full Redesign

I’ve seen teams panic and start rolling back changes when the real problem is just a stubborn cache. Before you do anything drastic, walk through these steps.

Solution 1: The “It’s Always DNS” Dance (Client-Side Flush)

This is the first thing to try when a single user or server is having a problem. It’s a quick, dirty fix that forces the local machine to dump its DNS cache and ask for a fresh address. This is a great troubleshooting step, but it’s not a real solution for an entire fleet of servers.

ipconfig /flushdns

sudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder

sudo systemd-resolve --flush-caches

Pro Tip: This only fixes the problem for the machine you run it on. If your web servers are the issue, you’d need to run this on them. But if they’re also using connection pooling, this might not be enough.

Solution 2: The Grown-Up Fix (Application & Infrastructure Hygiene)

The real, permanent solution is to make your application and infrastructure smarter. This prevents the problem from ever happening again. It’s about planning, not panicking.

• Plan Your TTLs: Hours before your scheduled maintenance, lower the TTL on the DNS record you plan to change. Drop it from, say, 1 hour (3600s) to 1 minute (60s). This tells the internet’s caches to check back for updates more frequently. After the maintenance is over and everything is stable, you can raise it back up.
• Fix Your Application’s Cache Settings: Some runtimes are notorious for caching DNS forever. The Java Virtual Machine (JVM) is a classic example. By default, it will cache a DNS lookup for the entire life of the application. You have to explicitly configure it to respect the DNS TTL. For a Java app, this means setting the `networkaddress.cache.ttl` security property.

  // For Java, set this on startup
  -Dsun.net.inetaddr.ttl=60

• Force a Graceful Connection Recycle: The best way to kill stale connections is to make your application get new ones. Don’t just `kill` the process. Use a graceful restart or reload. For example, in a Kubernetes environment, you trigger a rolling update of the deployment. For a simple web server like Nginx, you can reload the configuration, which often spawns new worker processes with fresh connections.

  # For Kubernetes, this forces pods to be recreated
  kubectl rollout restart deployment/my-webapp
  
  # For a systemd service
  sudo systemctl restart my-app.service

Solution 3: The ‘Nuclear’ Option (When You’re Bleeding)

Sometimes, you’re out of time. The service is down, the business is losing money, and you don’t have the luxury of a graceful, elegant fix. This is when you just need to turn it off and on again. I’m not proud of it, but I’ve had to do it.

This means restarting every layer of the application stack that might be holding a bad connection. Log into your web servers (`prod-web-01`, `prod-web-02`, etc.) and restart the application service. If that doesn’t work, restart the whole server. It’s brutal and causes a brief, complete outage for that node, but it forces every single cache and connection pool to be wiped clean.

Warning: This is a symptom of a larger architectural problem. If you find yourself having to do this after every maintenance window, you need to go back and implement Solution 2. The nuclear option is a tool for emergencies, not a standard operating procedure.

At the end of the day, these post-maintenance “ghost” issues are a rite of passage. They teach us that our systems are a complex web of interconnected, stateful components. Understanding how and where information is cached is just as critical as the change you’re making in the first place. Plan ahead, and you can keep your 3:00 AM maintenance window from turning into an 8:00 AM fire drill.

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions