SEO Summary: Should you replace SSH host keys during an OS upgrade? A senior engineer explains why it happens, why it breaks your automation, and offers three practical solutions, from the quick-and-dirty fix to the proper, scalable approach.

SSH Host Keys and OS Upgrades: The Annoying Problem Nobody Warns You About

I remember it like it was yesterday. It was 2 AM, and a junior engineer had just proudly announced the completion of a “routine” OS upgrade on `prod-db-01` from Ubuntu 18.04 to 20.04. The box was up, services were responding, and all our monitoring dashboards were green. High fives all around. Thirty minutes later, all hell broke loose. Our nightly backup automation, driven by Ansible, started screaming with authentication failures. I tried to SSH into the box from our bastion host and was greeted by that terrifying, all-caps message that still makes my stomach drop: WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! In that moment, I knew someone’s “routine” upgrade had just cost us our morning.

So, What Actually Happened? The “Why” Behind the Warning

This isn’t a bug; it’s a feature, albeit a deeply inconvenient one. When you perform a major OS upgrade (like a `do-release-upgrade`), the process often involves completely replacing the OpenSSH server package. As part of that fresh installation, the package manager’s post-install scripts dutifully generate a brand new set of unique host keys in /etc/ssh/.

Your local machine, or your automation server, has a file called ~/.ssh/known_hosts. This file is like a fingerprint database. The first time you connected to `prod-db-01`, you saved its public host key (its fingerprint). Now, after the upgrade, `prod-db-01` is presenting a *new* fingerprint. Your SSH client, being security-conscious, throws up its hands and yells:

“Hey! The server at this IP address used to be someone else! This could be a Man-in-the-Middle (MITM) attack! I’m refusing to connect to protect you.”

While it’s doing its job, in our case, it’s just a massive headache. The server’s identity changed, and now we have to tell all its clients that it’s okay.

How to Tame the Beast: Three Approaches

You’re in the middle of this mess. What do you do? Depending on the situation, you have a few options, ranging from a quick fix to a proper architectural solution.

Solution 1: The Quick Fix (The `ssh-keygen -R` Hammer)

This is the “It’s 3 AM and I just want to go to bed” solution. You need to manually remove the old, incorrect key from your local known_hosts file. The easiest way is with ssh-keygen.

On your client machine (e.g., your laptop or the Ansible controller), run this:

ssh-keygen -R prod-db-01.techresolve.com

This command removes all keys belonging to `prod-db-01.techresolve.com` from your `known_hosts` file. The next time you SSH, it will ask you to verify and add the new key, and you’ll be back in business.

Warning: This is a manual, one-off fix. You have to do this on every single client that connects to the upgraded server. It’s a hacky whack-a-mole game in a large environment, but for a single user, it’s the fastest way to solve the immediate problem.

Solution 2: The Permanent Fix (The “Right” Way)

If you have the chance to plan ahead, this is the professional approach. The goal is to preserve the server’s identity across the upgrade. The server’s identity is its set of private and public host keys in /etc/ssh/.

Before the OS upgrade, on the server you’re about to upgrade (`prod-db-01`):

# Create a backup directory
sudo mkdir /root/ssh_keys_backup/

# Copy the existing keys to the backup location
sudo cp /etc/ssh/ssh_host_*_key* /root/ssh_keys_backup/

Now, perform your OS upgrade. Once it’s complete and the new SSH server is installed, it will have generated new, unwanted keys.

After the OS upgrade, simply restore your old keys:

# Restore the original keys
sudo cp /root/ssh_keys_backup/ssh_host_*_key* /etc/ssh/

# Restart the SSH service to load the old keys
sudo systemctl restart sshd

Boom. The server now presents its original identity, and none of your clients will even know an upgrade happened. This should be a standard step in your upgrade runbook.

Solution 3: The ‘Nuclear’ Option (The Automation-First Approach)

In large, dynamic, or ephemeral environments (think cloud, CI/CD runners), managing `known_hosts` is a fool’s errand. Instead, we can tell our SSH clients to be a bit more… trusting. We can disable host key checking.

Pro Tip & Security Warning: This is a significant security trade-off. You are disabling a core protection against MITM attacks. NEVER do this for connections over the public internet. This is only acceptable for trusted, internal, and tightly firewalled networks where you control the entire environment.

You can configure this on a per-client basis by editing the ~/.ssh/config file on your Ansible controller or bastion host:

Host *.internal.techresolve.com
   StrictHostKeyChecking no
   UserKnownHostsFile /dev/null

This configuration tells your SSH client: “For any host in our internal domain, don’t bother checking its key against the `known_hosts` file, and don’t even bother writing new keys to a file.” This makes your automation resilient to host key changes, but again, use this power wisely.

Comparison of Approaches

Let’s break it down in a table.

Approach	Speed	Security	Scalability
1. The Quick Fix (`ssh-keygen -R`)	Fastest for one person	High (Manual verification)	Terrible
2. The Permanent Fix (Backup/Restore)	Slow (Requires planning)	Highest (Preserves identity)	Excellent (The correct way)
3. The Nuclear Option (Disable Checking)	Instant (Once configured)	Low (Significant trade-off)	Good (For controlled environments)

My Final Take

For any planned work on production systems, Solution 2 is the only answer. Bake the host key backup and restore process into your automation and checklists. It’s the mark of a seasoned engineer. The other options are tools for your belt—the `ssh-keygen -R` hammer for when something is already broken and you’re in a firefight, and the config-based approach for very specific, high-velocity internal systems. Don’t let a routine upgrade turn into a 2 AM crisis. Plan ahead, save your keys, and save your sleep.

---

🤖 Frequently Asked Questions