Experiencing an ‘AKS outage’ with NotReady nodes? It’s likely a DNS issue with the konnectivity-agent, not a platform failure. Here’s a senior engineer’s trench guide to diagnosing and fixing it for good.

Your AKS Nodes are NotReady. It’s Probably DNS. (Again.)

It’s 2 AM on a Tuesday, and PagerDuty is screaming my name. Half the nodes in our main prod-aks-cluster-01 are flapping, showing the dreaded NotReady status. My first thought, like any sane engineer on call, was to check the Azure Status page. Nothing. The team is spinning up a war room, management is asking for an ETA, and my heart rate is climbing. It felt like a massive outage. It wasn’t. It was DNS, the silent killer of cloud infrastructure, and a problem I’ve seen take down more clusters than any real Azure outage.

So, Why Does This Keep Happening?

Before we jump into fixing things, let’s talk about the “why”. When your Azure dashboard is green but your cluster is red, it’s easy to feel lost. The problem usually isn’t the node itself; the VM is running just fine. The problem is communication.

In AKS, the kubelet on each worker node needs to constantly check in with the Kubernetes API server (the control plane). To do this securely across the Azure network, it uses a proxy pod called konnectivity-agent. This agent’s one job is to establish a tunnel to the API server. But to do that, it first needs to find the API server using its fully qualified domain name (FQDN), something like my-aks-cluster-dns-a1b2c3d4.hcp.eastus.azmk8s.io.

If the node’s internal DNS resolution is broken—often because of a misconfigured custom VNet—the konnectivity-agent can’t find its destination. It’s like trying to make a phone call without a phonebook. The agent fails, the kubelet can’t report its status, and the control plane, hearing only silence, marks the node as NotReady. The node is fine, it’s just lost and can’t phone home.

The Fixes: From Dirty Patch to Permanent Solution

I’ve got three ways to tackle this, depending on how much time you have and whether you want to fix it now or fix it for good.

Solution 1: The “Get It Working NOW” Battlefield Patch

This is the hacky, break-glass-in-case-of-emergency fix. It’s not permanent, but it will get your nodes back online in minutes while you figure out a long-term strategy. The goal is to manually give the node a working DNS server.

1. Get a shell on the broken node. You can use the kubectl-debug plugin or create a privileged pod that mounts the host’s filesystem.
2. Check the node’s current DNS configuration. You’ll likely see an internal, non-functional DNS server.

   cat /etc/resolv.conf

3. Manually edit this file to add a public DNS resolver like Google’s or Azure’s own. I prefer using Azure’s magic IP.

   # This command requires root privileges
   sudo vi /etc/resolv.conf

4. Add one of the following lines to the very top of the file:

   # Add this line for Azure's DNS
   nameserver 168.63.129.16
   
   # Or add this for Google's DNS
   nameserver 8.8.8.8
   
   # --- Existing content below ---
   search your-internal-domain.local
   nameserver 10.100.1.4

Within a minute or two, the konnectivity-agent should be able to resolve the API server, reconnect, and the node’s status will flip back to Ready.

Warning: This is a temporary fix! The resolv.conf file is managed by the system and will be overwritten on the next node reboot or network service restart. Use this to stop the bleeding, not to cure the disease.

Solution 2: The “Do It Right” VNet Configuration

This is the real, permanent fix. The problem exists because the Virtual Network (VNet) your AKS nodes live in is handing out bad DNS settings. You need to fix it at the source.

1. In the Azure Portal, navigate to the VNet that your AKS cluster is deployed in.
2. Go to the DNS servers blade in the VNet’s settings.
3. You will likely see it set to ‘Custom’ with one or more IP addresses of internal DNS servers that are not correctly configured for public name resolution.
4. Your choice:
   • Change the setting to Default (Azure-provided). This is the easiest and most reliable option if you don’t have a hard requirement for custom DNS.
   • If you must use custom DNS, ensure those servers (e.g., your Active Directory controllers) are configured with conditional forwarders so they can resolve public internet addresses like the AKS API server FQDN.

You can also do this via the Azure CLI:

# To switch to Azure-provided DNS
az network vnet update \
  --resource-group MyResourceGroup \
  --name MyVNet \
  --dns-servers ""

# To update to corrected custom DNS servers
az network vnet update \
  --resource-group MyResourceGroup \
  --name MyVNet \
  --dns-servers 10.100.1.4 10.100.1.5

Pro Tip: This change won’t apply to existing nodes instantly. For the fix to take effect, you need to either reboot the nodes or, for a cleaner approach, scale your node pools down and back up, or use the az aks nodepool upgrade command to force a re-image of all nodes.

Solution 3: The “I Give Up” Node Replacement

Sometimes, a single node is just… weird. Or maybe you don’t have permissions to change VNet settings. In this case, your best bet is to just replace the misbehaving node. AKS’s self-healing capabilities make this easy.

1. First, safely evict all running pods from the bad node.

   kubectl cordon <node-name>
   kubectl drain <node-name> --ignore-daemonsets --delete-local-data

2. Next, find the underlying Virtual Machine Scale Set (VMSS) for your node pool. It will be in the AKS-managed resource group (usually named MC_MyResourceGroup_MyAKSCluster_Region).
3. In the VMSS, go to the Instances blade, find the instance corresponding to your bad node, and simply click Delete.

The AKS control plane and the VMSS controller will see that an instance is missing and automatically provision a brand new, healthy one to take its place. This new node will pull the latest (and hopefully correct) VNet configuration.

Choosing Your Weapon

Here’s a quick breakdown of when to use each approach:

So next time PagerDuty wakes you up with a sea of NotReady nodes, take a breath before you declare a cloud-wide outage. Grab a coffee, check your VNet’s DNS settings, and remember: it’s almost always DNS.

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions