Tired of SSH’ing to a bastion host only to get a “Permission denied” error on the next hop? This guide explains the root cause and provides three practical solutions, from a quick command-line fix to the permanent, secure configuration every engineer should use.

So You Can’t SSH From Your Bastion Host? A Senior Engineer’s Guide.

I still remember it. 2:37 AM. A PagerDuty alert screams about a database connection pool exhaustion on `prod-db-01`. A junior engineer, sharp but still green, is on call. I see him in the team chat: “I’m on it.” Ten minutes later: “I can’t get into the DB server from the bastion. It’s saying ‘permission denied (publickey)’.” The clock is ticking. The site is down. He’s stuck in that classic DevOps trap: he has the key, but it’s on his laptop, not on the jump box. We’ve all been there, and it’s a terrible feeling.

That frantic, late-night scramble is why understanding how SSH authentication flows through multiple servers isn’t just “good to know” — it’s fundamental. Let’s break down why this happens and how to fix it for good.

The “Why”: SSH Doesn’t Read Your Mind

The root of the problem is simple: SSH is designed to be secure by default. When you connect from your laptop to the bastion host (`prod-bastion-01`), you authenticate using the private key on your machine. That connection is now secure. But when you try to jump from `prod-bastion-01` to `prod-db-01`, you’re initiating a brand new SSH connection. The bastion host itself doesn’t have your private key (and it absolutely shouldn’t!), so it has no way to prove to the database server that you are you. The chain of trust is broken.

The goal is to securely “forward” your authentication credentials from your laptop, through the bastion, to the final destination. Here are three ways to do it, ranging from the quick-and-dirty to the professional standard.

Solution 1: The Quick Fix (The `-A` flag)

This is the “I need to get in *right now*” solution. The `-A` flag enables agent forwarding, which basically tells SSH to let the subsequent connection attempt to the DB server “borrow” the authentication from your laptop.

# On your local machine
ssh -A dev-user@prod-bastion-01.techresolve.com

# Now, once you're on the bastion...
ssh prod-db-01

This works, and it’ll get you out of a jam. However, it’s not something you want to rely on. It’s a temporary, manual step you have to remember every single time.

Warning: Agent forwarding comes with a security risk. If the bastion host is compromised, an attacker could potentially hijack your forwarded connection to access other servers you have keys for. Use it with caution, especially in untrusted environments.

Solution 2: The Permanent Fix (The `~/.ssh/config` File)

This is the way. This is how senior engineers and sane DevOps teams operate. By setting up your local SSH config file, you create aliases and rules so that connecting to a firewalled server becomes a single, seamless command. We’ll use the modern `ProxyJump` directive, which is more secure than agent forwarding.

Open or create the file `~/.ssh/config` on your local laptop and add the following:

# ~/.ssh/config

# First, define the bastion host so we can reference it
Host bastion
  HostName prod-bastion-01.techresolve.com
  User dev-user
  # Make sure you point to the correct key for the bastion
  IdentityFile ~/.ssh/id_rsa_techresolve

# Now, define the target server and tell it to use the bastion as a proxy
Host prod-db
  HostName prod-db-01.internal.techresolve.com
  User ec2-user
  # This is the magic line
  ProxyJump bastion
  # And the key for the internal server
  IdentityFile ~/.ssh/id_rsa_internal_prod

Now, to connect directly to the database server from your laptop’s terminal, you just type:

ssh prod-db

SSH reads your config file, sees that to get to `prod-db` it first needs to connect to `bastion`, and handles the multi-hop connection automatically and securely in the background. No `-A` flag, no two-step login. It just works.

Solution 3: The “Power User” Option (Using `ssh-agent`)

What if you have multiple keys, and some are protected by passphrases? Typing that passphrase every time is a pain. `ssh-agent` is a background program that holds your private keys in memory so you don’t have to re-enter your passphrase constantly.

Step 1: Start the agent (if it’s not already running)

On most modern systems, it starts automatically. If not, you can run:

eval "$(ssh-agent -s)"

Step 2: Add your key to the agent

This is the key step. You’re loading your identity into the agent for the duration of your session.

ssh-add ~/.ssh/id_rsa_techresolve

If the key has a passphrase, you’ll be prompted to enter it once. Now the agent has it unlocked. From here, you can combine this with either Solution 1 (the `-A` flag) or Solution 2 (`ProxyJump`) and you won’t be prompted for your passphrase again.

Pro Tip: On macOS, you can store your passphrase in the Keychain by running `ssh-add -K ~/.ssh/your_key`. This is a huge quality-of-life improvement.

Which Solution Should You Choose?

Here’s a quick breakdown to help you decide.

My advice? Take the 15 minutes to learn and set up your `~/.ssh/config` file properly. It’s a one-time investment that will pay for itself a hundred times over. It streamlines your workflow, reduces cognitive load, and is the most secure and scalable way to manage server access. Don’t be the engineer fumbling with keys at 3 AM. Be the one who types `ssh prod-db` and is already fixing the problem.

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions