Unlock network sanity by understanding why a single subnet across multiple VLANs causes chaos. Learn three key strategies—from quick command-line hacks to proper network architecture—to fix communication failures between your nodes.

VLANs, Subnets, and Lies: Untangling a Network Nightmare

I remember a 2 AM page like it was yesterday. A critical database replication job between prod-db-01 and its replica prod-db-02 had failed. The monitoring was screaming. I SSH’d into the primary, and the first thing I did was a simple ping prod-db-02. Nothing. Timeout. My heart sank. But then, on a hunch, I checked the ARP table with arp -n. There it was. The IP for prod-db-02 had a MAC address. The server was *right there* on the same L2 network, but my packets were going into the void. That was my first encounter with this exact problem: a network that looked like one thing at Layer 3 (the IP subnet) but was a segmented mess at Layer 2 (the VLANs). It’s a configuration that feels logical to someone, somewhere, but in practice, it’s a ticking time bomb.

Why This Network Is a Lie

Let’s get straight to the point. The core principle of modern networking is usually one VLAN = one Subnet. This isn’t just a suggestion; it’s how network devices are designed to think. Your server, prod-app-01 (192.168.1.10), wants to talk to prod-db-01 (192.168.1.20). Since they’re on the same subnet, your server doesn’t bother sending the packet to the gateway. It just shouts an ARP request into the local network: “Who has 192.168.1.20?”

Here’s the problem: if prod-app-01 is on VLAN 10 and prod-db-01 is on VLAN 20, that ARP request (a Layer 2 broadcast) is trapped inside VLAN 10. The database server will never hear it. The switch, doing its job correctly, ensures that the broadcast traffic from one VLAN never spills into another. Your IP configuration is writing checks that your VLAN configuration can’t cash. The two servers are in the same room, logically, but they’re in soundproof booths.

Three Ways Out of This Mess

You’ve inherited this setup, and now you have to fix it. We’ve all been there. Here are your options, ranging from a quick patch to a complete overhaul.

1. The Battlefield Fix: Static ARP Entries

This is the “I need it working five minutes ago” solution. It’s a hack, but it’s an effective one. You’re going to manually tell each server where the other servers are, bypassing the need for ARP broadcasts entirely. You’re essentially hardcoding the MAC-to-IP address mapping on each node.

On prod-app-01, you’d find the MAC address of prod-db-01 (let’s say it’s 00:1A:2B:3C:4D:5E) and run:

sudo arp -s 192.168.1.20 00:1A:2B:3C:4D:5E

You have to do this on every single node for every other node it needs to talk to. It’s a management nightmare and incredibly brittle. A new server or a NIC replacement breaks everything. But if the alternative is a total outage, you do what you have to do.

Warning: This is a temporary patch, not a solution. It creates technical debt that will come back to bite you. Use this to restore service, then immediately start planning for a real fix.

2. The Architect’s Fix: Proper Subnetting

This is the right way. The only long-term, scalable, and sane solution. You need to work with your network team to align the Layer 3 and Layer 2 configurations. Each unique VLAN ID gets its own unique IP subnet. This allows the network’s routing hardware to do its job properly.

Your migration plan would look something like this:

When prod-app-01 (now 192.168.10.10) wants to talk to prod-db-01 (now 192.168.20.10), its networking stack sees they are on different subnets. It correctly sends the packet to its default gateway (the router/L3 switch), which knows how to route traffic between VLAN 10 and VLAN 20. This is how networks are supposed to work.

Pro Tip: This is a significant change. Plan it carefully. You can run both the old and new IP addresses on your interfaces for a transitional period to minimize downtime during the cutover.

3. The ‘We Have No Choice’ Fix: Host-Level VLAN Tagging

Sometimes, you don’t control the switches. You’re given a trunk port and told to “make it work.” In this scenario, you can make your server hosts VLAN-aware. This is often called creating a “sub-interface.” You configure the host’s operating system to handle the 802.1q VLAN tags itself.

Instead of the switch port being an “access” port on a single VLAN, the network team configures it as a “trunk” port that allows traffic for multiple VLANs. Then, on your Linux server, you configure virtual interfaces for each VLAN.

For example, using netplan on Ubuntu, your config for the main interface eno1 might look like this:

network:
  version: 2
  ethernets:
    eno1:
      dhcp4: no
  vlans:
    vlan10:
      id: 10
      link: eno1
      addresses: [192.168.10.10/24]
      gateway4: 192.168.10.1
    vlan20:
      id: 20
      link: eno1
      addresses: [192.168.20.10/24]
      # No gateway, this is just for local comms

This approach makes your server configuration much more complex, and it tightly couples your OS to the physical network topology. I consider this a last resort, but it’s a powerful tool when you’re painted into a corner.

My Two Cents

Look, we all inherit weird environments. If you’re facing this, don’t panic. The static ARP entry will get you through the night. But your goal should always be to push for the proper architectural fix. A network that lies to you at Layer 3 is a foundation of sand. Build your house on the rock of a sane, properly segmented network. Your future self, at 2 AM, will thank you for it.

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions