Struggling with DNS resolution failures after a system update? Learn the root cause of the infamous `systemd-resolved` issue and explore three practical solutions, from a quick temporary hack to a permanent configuration fix.

So, an Update Broke Your DNS. Let’s Talk About `systemd-resolved`.

I remember a 2 AM outage call. The monitoring dashboards were screaming red, claiming half our services were down. But I could SSH into the boxes just fine. `ping 8.8.8.8`? Worked like a charm. `ping prod-db-01.internal.mycorp`? Timeout. Every single time. The new deployment hadn’t touched DNS, so what gives? The culprit, as it often is these days, was a minor OS patch that reset our network config, and a “helpful” little service called `systemd-resolved` decided my carefully crafted `/etc/resolv.conf` was merely a suggestion. If that feeling of dread sounds familiar, you’re in the right place. Let’s get this sorted.

What’s Actually Happening Here? The “Why” Behind the Pain

In modern Linux distributions, you’re not writing to `/etc/resolv.conf` directly anymore. That file is often just a symlink managed by `systemd-resolved`, a local DNS stub resolver. The goal is noble: to manage DNS settings more dynamically, especially for things like VPNs. In reality, it means your server now points all its DNS queries to a local address, usually 127.0.0.53, and lets `systemd-resolved` handle the forwarding.

The problem arises when an update, a network manager change, or a cloud-init script overwrites the configuration that `systemd-resolved` itself uses. When that happens, your server can still talk to IPs, but it has no idea how to look up a name. It’s like having a phone that can call numbers but has lost its entire contact list.

Taming the Beast: Three Ways to Fix Your DNS

We’ve got options, ranging from a quick-and-dirty fix to get you back online to the “proper” long-term solution. I’ve used all three in different situations.

Solution 1: The “Get It Working NOW” Fix (chattr)

This is the classic “brute force” method. We’re going to manually create the `/etc/resolv.conf` file with the correct settings and then make it immutable, preventing any process (including `systemd-resolved`) from changing it.

Step 1: First, stop and disable the service that’s causing the problem.

sudo systemctl disable systemd-resolved
sudo systemctl stop systemd-resolved

Step 2: The real `/etc/resolv.conf` is often managed through a symlink. We need to break that link and create a real file.

sudo rm /etc/resolv.conf
sudo nano /etc/resolv.conf

Step 3: Add your nameservers to the new file. This could be your internal DNS, a public one like Google’s, or your cloud provider’s.

# Our internal DNS servers
nameserver 10.0.1.10
nameserver 10.0.1.11
# Fallback to Google
nameserver 8.8.8.8
search internal.mycorp

Step 4: This is the magic. We lock the file using `chattr`.

sudo chattr +i /etc/resolv.conf

Darian’s Take: Be warned, this is a hack. It works, and it’ll save you during an outage. But you’re fighting the system, and a future OS update might have unpredictable results. Someone (maybe you, six months from now) will be very confused when they can’t edit that file. Remember to use `chattr -i` before you try to change it.

Solution 2: The “Do It Right” Permanent Fix (Configure resolved.conf)

This is the method I recommend for production systems. Instead of fighting `systemd-resolved`, we’re going to tell it how to behave correctly. This way, we’re working *with* the operating system, not against it.

Step 1: Edit the `systemd-resolved` configuration file.

sudo nano /etc/systemd/resolved.conf

Step 2: Find the `[Resolve]` section. Uncomment and set the `DNS`, `FallbackDNS`, and `Domains` directives. This tells the service which upstream servers to use.

[Resolve]
DNS=10.0.1.10 10.0.1.11
FallbackDNS=8.8.8.8 8.8.4.4
Domains=internal.mycorp mycorp.com

Step 3: Ensure `/etc/resolv.conf` is pointing to the systemd stub file. This is the “correct” symlink setup.

sudo ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf

Step 4: Restart the service to apply the new configuration.

sudo systemctl restart systemd-resolved
sudo systemctl status systemd-resolved

You can verify it’s working with `resolvectl status`.

Solution 3: The “Nuclear Option” (Disable and Go Old-School)

Sometimes, you just don’t want to deal with it. Maybe you have legacy scripts or configuration management (like an old version of Puppet) that expects to manage `/etc/resolv.conf` directly. In this case, you can completely disable `systemd-resolved` and revert to a more traditional networking setup.

Step 1: Completely stop and disable the service.

sudo systemctl disable systemd-resolved
sudo systemctl stop systemd-resolved

Step 2: Mask the service to prevent any other process from starting it again, ever.

sudo systemctl mask systemd-resolved

Step 3: At this point, you’re on your own. You’ll need to configure your primary networking service (like `NetworkManager` or `systemd-networkd`) to manage `/etc/resolv.conf` for you, or you’ll have to manage it manually as we did in Solution 1 (but without needing `chattr`). For `NetworkManager`, you’d typically edit `/etc/NetworkManager/NetworkManager.conf` and set `dns=default` or `dns=none` to give you back control.

Warning: This is a major change. Disabling a core systemd component can have side effects, especially with complex networking setups involving VPNs or containers that might rely on its features. Only do this if you know what you’re doing and are prepared to manage DNS resolution yourself.

Which Path Should You Choose?

Here’s how I break it down for my team:

Solution	Best For	Pros	Cons
1. The Quick Fix (chattr)	Emergency outage mitigation.	Fast, simple, effective immediately.	Brittle, confusing for future troubleshooting, fights the OS.
2. The Permanent Fix (resolved.conf)	Most production systems.	Works with the system, stable, “correct” method.	Requires understanding `systemd` config.
3. The Nuclear Option (disable)	Systems with specific legacy requirements or for experts who prefer manual control.	Gives you full, direct control.	Can have unintended side effects; you lose modern features.

Ultimately, DNS is one of those foundational services that must be rock solid. While it’s frustrating when an update knocks it over, understanding the components at play is the key to building resilient systems. My advice? Take the time to learn Solution 2. Your future self, at 2 AM, will thank you.

---

🤖 Frequently Asked Questions