🚀 Executive Summary
TL;DR: The UK government’s use of QR codes to ‘stop hackers’ highlights a fundamental problem with broken internal DNS resolution, where systems fail to resolve internal hostnames to IP addresses. The core issue is often a misconfigured `/etc/resolv.conf` pointing to public DNS servers instead of internal ones, leading to service unreachability. Solutions range from temporary `/etc/hosts` entries to correctly configuring internal DNS resolvers, or implementing robust service discovery for dynamic cloud environments.
🎯 Key Takeaways
- When a service is unreachable by hostname but works via its IP address, the primary suspect is always DNS resolution failure from the source machine, often due to querying the wrong DNS server for internal names.
- The `/etc/hosts` file provides a quick, temporary fix for individual servers by manually mapping IP addresses to hostnames, but it introduces significant technical debt and does not scale.
- The permanent solution involves correctly configuring the `/etc/resolv.conf` file to point to internal DNS resolvers (e.g., `10.10.0.2` in an AWS VPC) and utilizing `search` directives, or adopting Service Discovery mechanisms like HashiCorp Consul for dynamic, scalable environments.
The UK government printing QR codes to stop hackers isn’t a joke; it’s a symptom of a deep-seated networking problem. I’ll show you why this happens and how to fix broken DNS resolution in your own environments for good.
We’re Printing QR Codes to Stop Hackers. Let’s Talk About DNS.
I remember a 3 AM PagerDuty alert like it was yesterday. A critical payment processing service, let’s call it `payment-gateway-prod-01`, was throwing 500 errors. A junior engineer was on the call, bless his heart, frantically trying to debug. “I don’t get it,” he said, “I can `curl` the fraud-check API by its IP address from the gateway server and it works fine, but the application is failing!” He was moments away from hardcoding the IP address into the application config. We’ve all been there. That feeling of desperation where you find a workaround that feels so wrong, but the system is down and management is breathing down your neck. The Reddit story about printing QR codes to get a working link? It’s the exact same problem, just on a national scale. It’s a visceral, painful symptom of a fundamental misunderstanding of how networks talk to each other.
The “Why”: It’s Always DNS
Let’s break down what’s happening, both for the poor soul in the UK government office and for our junior engineer at 3 AM. The server, or in their case, the work computer, needs to talk to another service. It knows the service by its friendly name, like `api.fraud-check.internal.techresolve.com`.
To turn that name into an IP address (like `10.10.2.55`), it asks a Domain Name System (DNS) server. The problem is, the computer is configured to ask the wrong DNS server. It might be asking a public DNS server (like Google’s `8.8.8.8`) that has no idea what `api.fraud-check.internal.techresolve.com` is. That’s an internal-only name.
When the user scans the QR code with their phone, their phone is on a different network (cellular) and uses a different, public DNS server. The QR code doesn’t contain a magical link; it just contains the publicly accessible URL. The phone’s DNS can resolve it, bypassing the broken internal network configuration entirely. It’s not a security measure; it’s a clumsy, embarrassing workaround for a broken DNS resolver chain.
Pro Tip: When a service is unreachable by its hostname but works via its IP address, your first suspect should always, always be DNS resolution from the source machine. Stop what you’re doing and run `dig hostname.com` or `nslookup hostname.com` from that box.
Three Ways to Fix This Mess
Okay, so you’ve diagnosed it. The server can’t resolve the name. Let’s walk through the solutions, from the quick and dirty to the architecturally sound.
1. The Quick Fix: The `/etc/hosts` Band-Aid
This is the one my junior engineer was about to do, just in a different form. You can manually tell a single server how to resolve a specific name by editing its local `hosts` file. This file is a simple mapping of IP addresses to hostnames that the system checks before it even tries to ask DNS.
You SSH into the problematic server, `payment-gateway-prod-01`, and edit the file:
sudo nano /etc/hosts
Then you add a line at the bottom:
# TEMPORARY FIX for Incident-2023-12-04 - Payment Gateway DNS Failure
10.10.2.55 api.fraud-check.internal.techresolve.com
Why it’s a bad idea for the long term: This is a hack. It fixes the problem on one server. What happens when you have 50 servers in that fleet? What happens when the IP address of the fraud-check API changes? You now have a static, manually configured dependency that will absolutely come back to bite you. This is technical debt of the highest order. Use it to get the system back online, but you MUST create a follow-up ticket to implement the permanent fix.
2. The Permanent Fix: Configure the Resolver Correctly
The real problem is that the server doesn’t know who to ask for directions. We need to tell it where our internal DNS server lives. This is almost always handled by the `/etc/resolv.conf` file. This file tells the system which DNS servers (nameservers) to query.
On your broken server, you might see a file that looks like this:
# Generated by some random network utility
nameserver 8.8.8.8
nameserver 8.8.4.4
This server is trying to ask Google’s public DNS about our internal services. That’s never going to work. What we need is for it to point to our internal resolvers. In an AWS VPC, for instance, the resolver is always at the `.2` address of your CIDR block (e.g., `10.10.0.2`).
A correctly configured file should look something like this:
# Configured by Cloud-Init/Ansible
search techresolve.com internal.techresolve.com
nameserver 10.10.0.2
nameserver 172.16.0.2
The `search` directive is also a lifesaver, allowing you to use short names like `api.fraud-check` and the system will automatically try appending the domains in the search path.
Warning: Be careful! In modern systems, `/etc/resolv.conf` is often managed by another service like `systemd-resolved` or DHCP clients. Manually editing it might be temporary. The real fix is to configure the tool that generates this file, which usually means adjusting your network interface configuration or your cloud instance’s DHCP options set.
| The Wrong Way (Broken) | The Right Way (Fixed) |
| Relies on public DNS for private names. | Uses internal DNS resolvers that understand the private network. |
| Fails unpredictably when internal services are deployed. | Resolves internal and, if configured, public names reliably. |
| Leads to hacks like `/etc/hosts` entries or printing QR codes. | Scales with your infrastructure. New services are discovered automatically. |
3. The ‘Nuclear’ Option: Service Discovery
If you find your teams are constantly fighting DNS issues, it might be time to stop treating your servers like pets and start treating them like cattle. In a modern, dynamic cloud environment, relying on traditional DNS for service-to-service communication is brittle. IPs change, services scale up and down. The “right” way forward is to stop relying on DNS resolution at the OS layer for this.
Instead, you implement a Service Discovery mechanism. Tools like HashiCorp Consul, CoreDNS, or cloud-native solutions like AWS Cloud Map or Azure Service Discovery create a central, dynamic registry of services.
When `payment-gateway` needs to talk to `fraud-check`, it doesn’t ask the OS’s DNS. It asks the Service Discovery agent: “Hey, where’s a healthy instance of the `fraud-check` service?” The agent provides back a list of current, healthy IP addresses and ports. This is often paired with a service mesh like Istio or Linkerd, which makes this entire process transparent to the application itself.
This is a major architectural shift, not a quick fix. But it’s the true solution to this entire class of problems. It makes your infrastructure more resilient, scalable, and frees you from the tyranny of stale DNS records and misconfigured resolvers.
So next time you see a workaround that feels like printing a QR code, don’t just laugh. Dig in and ask “why?”. The answer is probably DNS, and it’s a chance to build something better.
🤖 Frequently Asked Questions
âť“ Why would a system fail to resolve an internal hostname?
A system fails to resolve an internal hostname when it is configured to query the wrong DNS server, typically a public DNS server (like `8.8.8.8`) that has no knowledge of internal-only names, instead of the internal DNS server responsible for those records.
âť“ How do `/etc/hosts`, `/etc/resolv.conf`, and Service Discovery compare as solutions for DNS issues?
`/etc/hosts` is a local, static workaround for a single server. Correctly configuring `/etc/resolv.conf` to point to internal nameservers is a system-wide, permanent fix for traditional environments. Service Discovery (e.g., Consul, AWS Cloud Map) is an architectural shift for dynamic cloud environments, providing a central, dynamic registry for services independent of OS-level DNS.
âť“ What is a common pitfall when attempting to fix `/etc/resolv.conf`?
Manually editing `/etc/resolv.conf` can be temporary because modern systems often manage and overwrite this file via services like `systemd-resolved` or DHCP clients. The true fix is to configure the underlying network interface settings or DHCP options that generate this file.
Leave a Reply