Quick Summary: Struggling to hop from a public bastion to a private instance? Stop copying your private keys to the server—use SSH Agent Forwarding, ProxyJump, or better yet, ditch SSH entirely for SSM.

SSH from Public to Private Subnets: Stop Copying Your Keys to the Bastion

It was 2 AM on a Tuesday in 2018 when my pager went off. A junior dev, bless his heart, had just triggered a high-severity security alert. In a desperate attempt to debug a failing migration on prod-db-01 (our private RDS proxy instance), he had scp‘d our root production PEM file onto a public-facing Nginx server. His logic? “Well, I need the key to SSH from the web server to the database server.”

I nearly had a heart attack. I revoked the keys, rotated the credentials, and we had a very long chat the next morning.

If you are reading this because you are stuck trying to SSH from your public EC2 instance to a private one, and you are tempted to upload your private key to that public server: Stop. Put the mouse down. There is a better way to handle the “jump” without compromising your entire security posture.

The “Why”: The Chain of Trust

Here is the scenario: You are on your laptop. You have my-key.pem. You can SSH into bastion-public. But when you get onto bastion-public and try to SSH into backend-private, you get a Permission denied (publickey) error.

This happens because SSH keys are not transitive by default. bastion-public does not have your private key. It only checks the public key authorized in its ~/.ssh/authorized_keys file. When you try to hop to the next server, the chain breaks because the bastion cannot prove to the private instance that it is you.

Here are three ways to fix this, ranging from “The Classic” to “The Architect’s Choice.”

---

Solution 1: The Quick Fix (Agent Forwarding)

This is the classic method we’ve been using for decades. Instead of copying the key to the server, we tell the SSH client on your laptop to act as a middleman. The bastion simply “forwards” the signature request back to your laptop.

Step 1: Add the key to your local agent
On your local machine (Mac/Linux), load the key into memory:

ssh-add -K ~/.ssh/my-key.pem

Step 2: SSH with the -A flag
This flag enables agent forwarding.

ssh -A ec2-user@bastion-public-ip

Step 3: Jump
Once you are on the bastion, just SSH normally. It will magically work.

[ec2-user@bastion-public]$ ssh ec2-user@10.0.1.55

Pro Tip: While convenient, Agent Forwarding has a slight security risk. If the bastion is compromised by a sophisticated attacker while you are connected with -A, they can technically hijack that socket to authenticate as you elsewhere. Use with caution on untrusted networks.

Solution 2: The Elegant Fix (ProxyJump)

If you are using a modern SSH client (OpenSSH 7.3+), you don’t even need to log into the bastion shell. You can use ProxyJump (or -J) to tunnel straight through.

This is my preferred method for quick debugging because it feels like you are connecting directly to the private instance.

ssh -J ec2-user@bastion-public-ip ec2-user@10.0.1.55

If you want to look like a senior engineer, add this to your local ~/.ssh/config file so you never have to type IPs again:

Host bastion
    HostName 34.202.xx.xx
    User ec2-user
    IdentityFile ~/.ssh/my-key.pem

Host private-db
    HostName 10.0.1.55
    User ec2-user
    ProxyJump bastion

Now you just type ssh private-db, and it handles the hop automatically. Clean. Efficient.

Solution 3: The “Nuclear” Option (AWS SSM)

This is the “War Story” prevention method. As a Cloud Architect, I try to kill SSH usage entirely. Managing keys, rotating them, and maintaining bastion hosts is operational toil.

The solution is AWS Systems Manager (SSM) Session Manager.

This allows you to get a shell on your private instances via the AWS Console or AWS CLI without opening port 22 and without SSH keys. It uses IAM roles to authenticate.

1. Attach the AmazonSSMManagedInstanceCore IAM policy to the EC2 instance role.
2. Ensure the SSM Agent is running (installed by default on Amazon Linux 2/2023).
3. Connect via CLI:

aws ssm start-session --target i-0123456789abcdef0

No keys to lose. No ports open to the internet. Full audit logs of every command run.

Comparison: Which one should you use?

Method	Pros	Cons
Agent Forwarding (-A)	Easy to remember, works on old clients.	Minor security risks if bastion is compromised.
ProxyJump (-J)	Cleaner, scriptable, no key exposure on intermediate hop.	Requires configuring .ssh/config for sanity.
AWS SSM	Most secure. No keys. No Port 22.	Requires IAM setup and agent installation.

If you’re in a rush? Use ProxyJump. If you’re building the infrastructure for the long term? Set up SSM and delete your SSH keys. Just please, whatever you do, delete that .pem file from the public server.

---

🤖 Frequently Asked Questions