A critical zero-day in Cisco IOS XE has allowed hackers to create privileged local users since 2023. Here’s my in-the-trenches guide to finding the breach, fixing the immediate problem, and making sure it never happens again.

They’re Already Inside: That Cisco Zero-Day Is Worse Than You Think

I got the ping at 2 AM on a Tuesday. A high-priority alert from our SIEM about an anomalous login on core-rtr-01-sjc. My first thought? “Great, another false positive from the overnight team.” I rolled over, ready to dismiss it. But something gnawed at me. I dragged my laptop open, SSH’d into the box, and ran a quick show users. Nothing. Then I ran show run | include username. And there it was. A username I’d never seen before, with privilege level 15. My blood ran cold. It wasn’t a false positive; someone had a key to the kingdom, and they’d had it for who knows how long.

So, What’s Actually Happening Here? (The “Why”)

Let’s cut through the corporate-speak. This isn’t some hyper-complex, nation-state-level attack that required a team of geniuses to pull off. The vulnerability, officially CVE-2023-20198, is in the Web UI feature of Cisco’s IOS XE software. Essentially, an unauthenticated attacker could send a specially crafted HTTP request to your device and just… create a new local user with full root-level (privilege 15) access. No password needed, no login required. They just knock on the web server door and it lets them right in to create their own backdoor key.

The scariest part? This has been actively exploited in the wild since at least 2023. The user they create often has a generic name like cisco_tac, cisco_support, or just a random string to blend in. They get in, create their account, and then can come and go as they please.

Darian’s Pro Tip: If you have the Web UI (ip http server or ip http secure-server) enabled and exposed to the internet on any of your IOS XE devices, you should assume you are compromised. Period. Start your incident response now.

The Triage: How We Fix This Mess

Alright, you’re probably sweating a bit. Good. Let’s channel that into action. I’m going to give you three levels of fixes, from the immediate “stop the bleeding” patch to the “burn it all down” solution for when you can’t trust the hardware anymore.

Solution 1: The Quick & Dirty – Find and Disable

This is your first move. You need to identify if you’re a victim and immediately shut the door the attackers used. This is about containment, not a permanent fix.

Step 1: Check for the Attack Vector. Is your Web UI even enabled? SSH into your device and run this:

core-rtr-01-sjc# show run | include ip http
ip http server
ip http secure-server

If you see either of those lines, the door is open. If not, you’re likely safe from this specific attack vector, but you should still check for rogue users.

Step 2: Hunt for Unauthorized Users. Run this command to check for any users that you or your team didn’t create.

core-rtr-01-sjc# show run | include username

Look for anything suspicious. If you find a user you don’t recognize, you’ve been hit. You also need to check the system logs for a specific message pattern which indicates compromise:

show log | include %WEBUI-6-INSTALL_OPERATION_INFO

This log message, followed by a user being created, is the smoking gun.

Step 3: Shut It Down. Whether you found a user or not, if the HTTP server is on and you don’t need it, kill it. This is a hacky fix, but it stops the bleeding now.

core-rtr-01-sjc# configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
core-rtr-01-sjc(config)# no ip http server
core-rtr-01-sjc(config)# no ip http secure-server
core-rtr-01-sjc(config)# end
core-rtr-01-sjc# write memory

And of course, remove the unauthorized user account immediately.

Solution 2: The ‘Right Way’ – Patch, Patch, Patch

Disabling the web server is a band-aid. The vulnerability is still in the code. The only real permanent fix is to upgrade your IOS XE to a patched version provided by Cisco. This is non-negotiable for long-term security.

I won’t list all the specific versions here because they change, but your process should be:

1. Go to the official Cisco Security Advisory for CVE-2023-20198.
2. Find the “Fixed Software” section for your specific hardware model.
3. Download the recommended patched version of the IOS XE software.
4. Schedule a maintenance window (yes, I know, but it has to be done) and perform the upgrade.

Warning: Just deleting the rogue user is not enough. If you leave the vulnerable Web UI active, they will simply get back in and create another one. You MUST disable the service or patch the device.

For those of us managing more than a handful of devices, doing this manually is a nightmare. This is where your automation tooling comes in. We used Ansible to push the new IOS XE image to our fleet of affected routers and schedule a rolling reboot. It turned a week of pain into a single day’s work.

Solution 3: The ‘Nuclear’ Option – Nuke and Pave

This is the one nobody wants to do, but sometimes it’s the only way you can sleep at night. If you found a rogue user and you have any reason to believe they moved laterally or installed other implants, you cannot trust that device anymore. The OS itself might be compromised in ways you can’t see.

The process is painful but simple:

1. Backup: Take a fresh backup of the running configuration. Sanitize it—manually review every line to make sure there’s nothing malicious in it (e.g., weird crypto maps, ACLs, static routes).
2. Wipe: Format the flash memory on the device. Erase everything.
3. Reinstall: Install a fresh, patched version of IOS XE that you’ve downloaded directly from Cisco and verified the checksum on.
4. Restore: Apply your sanitized configuration backup to the freshly installed device.
5. Monitor: Watch the device like a hawk for any unusual activity.

This is your last resort, but if you’re a major financial institution or handle critical infrastructure, it’s the only way to be 100% sure you’ve eradicated the threat.

Summary of Actions

Here’s a quick cheat sheet for your next team meeting.

Approach	Action Items	Best For
The Quick Fix	Check for rogue users. Disable ip http server and ip http secure-server.	Immediate containment. Buying yourself time.
The Permanent Fix	Upgrade IOS XE to a patched version from Cisco. Automate with Ansible/Python if possible.	The required long-term solution for all production environments.
The Nuclear Option	Wipe the device flash, reinstall a patched OS from a verified image, and restore a sanitized config.	High-security environments or when you can’t guarantee the integrity of the device.

This isn’t the first zero-day and it won’t be the last. The key is to have a plan. Don’t panic. Triage, contain, and then remediate properly. Now go check your devices.

---

🤖 Frequently Asked Questions