Experiencing bizarre DNS failures on Ubuntu 22.04 and newer? This post digs into the likely culprit, systemd-resolved, and provides three battle-tested solutions to get your servers resolving hostnames again.

That ‘The Internet is Down’ Panic: Untangling the Ubuntu 22.04 DNS Mess

It’s 2 AM. My phone buzzes with a PagerDuty alert. ‘CRITICAL: ci-runner-03 is OFFLINE’. My heart sinks. I SSH in, and the server is up. I can ping 8.8.8.8 just fine. But a simple apt update? A cascade of ‘Failed to fetch’ and ‘Temporary failure resolving’ errors. For a split second, you question everything. Is our provider having an outage? Did someone fat-finger a firewall rule? No. It was something much more subtle and infuriating: a DNS resolver change deep in the OS that decided to take a vacation. I’ve seen this exact scenario play out for folks on various providers, including the recent chatter about Canadian Web Hosting, and it almost always comes back to the same root cause.

So, What’s Actually Breaking? The “Why” Behind the Pain

Since Ubuntu 18.04, the operating system has used systemd-resolved for managing DNS. By default, it sets up a local “stub resolver.” This means your /etc/resolv.conf file doesn’t point directly to your DNS servers (like 8.8.8.8). Instead, it points to a local service running on your machine at 127.0.0.53.

# This is what /etc/resolv.conf looks like by default
nameserver 127.0.0.53
options edns0 trust-ad
search .

This local service is supposed to be smart—it caches requests and forwards them to the “real” DNS servers configured on your network interface. In theory, it’s great. In practice, it can cause chaos. Certain applications, especially container runtimes like Docker, can get confused by this setup. They expect to interact with a simple resolv.conf file, not a local forwarding service. This can lead to intermittent or complete DNS resolution failure inside your containers or even on the host itself.

The Fixes: From a Quick Patch to a Permanent Solution

Alright, enough theory. You’ve got a broken server and you need it working five minutes ago. Here are three ways to tackle this, from the fast-and-dirty to the “do it right” approach.

Solution 1: The Quick Fix (The Symlink Shuffle)

This is my go-to when a server is on fire and I just need to get the build pipeline un-stuck. We’re going to bypass the local stub resolver by changing a symbolic link. systemd-resolved actually maintains two versions of the resolv.conf file. We’re just going to switch which one is active.

1. Check your current symlink. It’s likely pointing to stub-resolv.conf.

ls -l /etc/resolv.conf

2. Now, force the symlink to point to the “real” resolver config, which contains the actual DNS server IPs.

sudo ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf

Warning: This is a “hacky” but effective fix. A system update or a reboot might revert this change, so don’t rely on it as a permanent solution for your critical production servers. It’s a great temporary patch, though.

Solution 2: The Permanent Fix (The ‘Systemd’ Way)

If you want to fix this properly without completely disabling the service, you can configure systemd-resolved directly. This tells the service which DNS servers to use permanently.

1. Edit the main configuration file with your favorite editor.

sudo nano /etc/systemd/resolved.conf

2. Uncomment the DNS= line in the [Resolve] section and add the DNS servers you want to use. You can use your provider’s DNS, or public ones like Google’s or Cloudflare’s.

[Resolve]
DNS=8.8.8.8 1.1.1.1
#FallbackDNS=
#Domains=
#LLMNR=no
#MulticastDNS=no
#DNSSEC=no
#DNSOverTLS=no
#Cache=no-negative
#DNSStubListener=yes

3. Save the file and restart the service for the changes to take effect.

sudo systemctl restart systemd-resolved

4. You might also need to apply the symlink fix from Solution 1 to make this stick, as the stub listener can sometimes still cause issues.

Solution 3: The ‘Nuclear’ Option (Going Old School)

Sometimes you just want a simple, predictable system. If you’re fed up with systemd-resolved entirely, you can disable it and go back to managing /etc/resolv.conf manually, the way it was done for years.

Heads Up: This is a major change to how your system handles networking. Do this if you are confident you know what you’re doing. It’s not for the faint of heart, but it gives you total control.

1. First, stop and completely disable the service.

sudo systemctl stop systemd-resolved
sudo systemctl disable systemd-resolved

2. Next, remove the symlink that it created.

sudo rm /etc/resolv.conf

3. Finally, create a new, static /etc/resolv.conf file and put your nameservers directly into it.

# Create a new file with sudo nano /etc/resolv.conf
# And add these lines:
nameserver 8.8.8.8
nameserver 1.1.1.1

4. Now your system will use this file directly, with no local stub resolver in the way.

Which Solution Should You Choose?

Here’s a quick breakdown to help you decide.

DNS is one of those foundational pillars of infrastructure that you don’t think about until it breaks in the most spectacular way. Hopefully, this guide saves you from that 2 AM panic and gets your services back online fast.

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions