Port security is a fantastic Layer 2 feature until it silently sabotages your high-availability cluster failover at 3 AM. Learn how to diagnose and fix the conflict between sticky MAC addresses and redundant hardware.

That Time Port Security Took Down Our Failover (And How to Fix It)

It’s a Tuesday. 2:47 AM, to be exact. My phone buzzes with an intensity that can only mean one thing: a P1 incident. The primary database cluster, `prod-db-cluster-01`, is completely unreachable. The on-call SRE has confirmed the secondary node, `prod-db-02`, successfully took over after the primary, `prod-db-01`, failed its health check. The application logs show the failover was clean. The server is up, it has the floating IP, but not a single packet is getting to it.

For twenty minutes, we’re chasing ghosts in firewall rules, routing tables, and application configs. Then, the network engineer on the call says the five words that make my blood run cold: “The switch port is in err-disable.” And I knew. It was port security again, the well-intentioned security feature that had just turned our high-availability setup into a high-anxiety paperweight.

The “Why”: When Good Intentions Go Bad

Let’s get this straight. Port security isn’t evil. It’s designed to stop someone from unplugging a trusted device (like a server) and plugging in their own laptop to get access to the network. It does this by locking a switch port to the MAC address of the first device it sees. The most common configuration is `switchport port-security mac-address sticky`, which tells the switch, “Learn the first MAC address you see and remember it forever.”

Here’s the problem: Your HA pair (firewalls, load balancers, database nodes) has two different physical servers with two different physical MAC addresses. When `prod-db-01` is active, the switch port happily learns its MAC address (let’s say `AAAA.AAAA.AAAA`). But when it fails, `prod-db-02` takes over the virtual IP and starts sending traffic from its own, completely different MAC address (`BBBB.BBBB.BBBB`).

The switch sees this new MAC on a port it was told should only ever see `AAAA.AAAA.AAAA`. It assumes a security violation is in progress and, to protect the network, it does its job: it shuts the port down. The failover is now dead in the water.

The Fixes: From Battlefield Patch to Permanent Solution

You’re in the middle of an outage, so let’s talk about how to fix it—both right now and for good.

1. The Quick Fix: “The Power Cycle”

This is the “get it working now” solution. You need to tell the switch to reset the port and forget what it learned. You have two primary ways to do this. The fastest is to just bounce the interface.

! Log into the switch (e.g., a Cisco Catalyst)
enable
conf t
!
interface GigabitEthernet1/0/24
shutdown
! Wait a few seconds...
no shutdown
exit

When the port comes back up, it will see the new active node (`prod-db-02`) and learn its MAC address as the “sticky” one. Service will be restored. This is a temporary fix because if the cluster fails back to the original node, the same problem will happen all over again.

Pro Tip: If you’re dealing with a large number of sticky addresses or want to be more surgical, you can clear the sticky MAC address specifically instead of bouncing the whole port. Use `clear port-security sticky interface GigabitEthernet1/0/24`. It’s slightly less disruptive.

2. The Permanent Fix: “The Right Way”

The real solution is to tell the switch to expect more than one MAC address on this port. For a simple active/passive HA pair, you only need to allow two MAC addresses. This is the fix you should be applying during your next change window.

! Log into the switch
enable
conf t
!
interface GigabitEthernet1/0/24
description "Uplink for HA Database Cluster prod-db-cluster-01"
switchport mode access
switchport port-security
! This is the key line
switchport port-security maximum 2
switchport port-security mac-address sticky
switchport port-security violation shutdown
!
end
wr

With `maximum 2`, the switch will learn the MAC of `prod-db-01` first. When `prod-db-02` takes over, the switch will see the second MAC, say “Ah, this is within my configured limit,” and learn that one, too. No security violation, no shutdown. Your failover will now work as intended.

3. The ‘Nuclear’ Option: “Just Turn It Off”

I’m going to be honest. Sometimes, for critical infrastructure ports in a physically secure data center, the risk of an outage caused by port security is greater than the risk of someone plugging a rogue device into that specific port. In these limited, highly-controlled situations, you might make the call to disable port security entirely on that single interface.

This is not a decision to be taken lightly. You are intentionally reducing your security posture on that port. But if the port connects to, say, a dedicated HA firewall pair in a locked cage, the risk is often negligible.

! Log into the switch
enable
conf t
!
interface GigabitEthernet1/0/24
! This removes all port-security configuration from the interface
no switchport port-security
!
end
wr

Warning: Document this decision. When a security auditor comes around asking why port security is disabled on this interface, you need to have a clear, written justification based on risk assessment. Don’t just do it because it’s easier.

Solution Comparison

Here’s a quick cheat sheet to help you decide which path to take.

At the end of the day, technology that is supposed to provide resilience shouldn’t be undermined by other systems meant to provide security. Understanding how these layers interact is what separates a junior from a senior engineer. Hopefully, my 3 AM war story saves you from having one of your own.

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions