Struggling with SSH agent forwarding failing in scripts or sudo sessions? Unravel the mystery of why your keys “disappear” and learn three battle-tested fixes, from the quick hack to the permanent architectural solution.

Your SSH Keys Aren’t Disappearing, Your Session Is. A Deep Dive into “Weird” Agent Forwarding Issues.

It was 2 AM, and a critical hotfix deployment to our `prod-db-01` cluster was failing. The Ansible playbook, which I could run flawlessly from my local machine, was bombing out when triggered by our Jenkins runner on the bastion host. The error? The classic, infuriating `Permission denied (publickey)`. I SSH’d into the bastion, ran the exact same command manually, and it worked perfectly. It felt like the server was gaslighting me. If you’ve ever stared at a screen, knowing a command works when you type it but fails in a script, this one’s for you. You’re not going crazy; you’ve just run into one of the most common “gotchas” in the world of SSH.

So, What’s Actually Happening Here? The “Why” Behind the Weirdness

This isn’t black magic. The problem almost always boils down to one little thing: an environment variable called SSH_AUTH_SOCK. When you connect to a server (let’s call it `bastion-host-01`) with agent forwarding enabled (ssh -A), your local SSH agent creates a secure socket on that server for authentication requests. The path to this socket is stored in the SSH_AUTH_SOCK environment variable. Your interactive shell session knows all about it.

The “weirdness” begins when you introduce a new, non-interactive session. This could be:

• A script executed by cron.
• A `sudo su – someuser` command that completely resets the environment.
• A build job kicked off by a CI/CD tool like Jenkins or GitLab CI.

These new sessions don’t automatically inherit the environment of your interactive login shell. They get a fresh, clean environment, which means no SSH_AUTH_SOCK variable. Without that variable, the `ssh` client on the bastion has no idea how to contact your agent back on your laptop. To the next server in the chain (`prod-db-01`), it looks like you’re trying to connect without a key. Hence, “Permission denied.”

The Fixes: From Duct Tape to a New Engine

I’ve seen this issue trip up everyone from interns to senior engineers. Here are the three ways we handle it at TechResolve, depending on the situation.

Solution 1: The Quick and Dirty Fix (The Duct Tape)

Let’s be honest, sometimes you just need to get the deployment out the door. This method is a bit of a hack, but it works. The goal is to find the active agent socket from your login session and manually “inject” it into your script’s environment.

First, find the socket path in your active, working SSH session:

# On bastion-host-01, in the shell where agent forwarding works
echo $SSH_AUTH_SOCK
# Output might be: /tmp/ssh-a1B2c3D4e5/agent.12345

Now, you can hardcode this value into the top of your script:

#!/bin/bash
# WARNING: This is a brittle fix!
export SSH_AUTH_SOCK="/tmp/ssh-a1B2c3D4e5/agent.12345"

# Now your ssh, scp, or ansible commands will work
ssh user@prod-db-01 "hostname"

Heads Up! This is brittle because that socket path can change every time you log in. It’s a temporary fix to get you out of a jam, not a permanent solution. Use it to make your deadline, then schedule time to implement a real fix.

Solution 2: The ‘Right Way’ for Users (The Permanent Fix)

A much more robust approach is to stop chaining `ssh` commands and start using SSH’s built-in proxying capabilities. The `ProxyJump` directive (or the `-J` command-line flag) is your best friend here. It creates a seamless, tunneled connection straight through the bastion to the final destination, forwarding your agent correctly in one go.

You can configure this in your local `~/.ssh/config` file (the one on your laptop, not the server).

# In ~/.ssh/config on YOUR local machine

Host bastion
    HostName bastion.techresolve.com
    User darian
    ForwardAgent yes

# This host will automatically 'jump' through the bastion
Host prod-db-*
    HostName %h.internal.techresolve.com
    User service_account
    ProxyJump bastion

With this config, running a command from your laptop is now incredibly simple:

# This command connects directly to the DB host THROUGH the bastion
# Your local agent is forwarded securely all the way.
ssh prod-db-01 "whoami"

This is the ideal solution for human operators. It’s clean, secure, and completely avoids the “disappearing agent” problem because you never actually run a second `ssh` command on the bastion host itself.

Solution 3: The Architectural Fix (The ‘Nuclear’ DevOps Option)

Agent forwarding is convenient, but it’s also a security risk. A compromised bastion host could potentially hijack your forwarded agent socket. For automated systems (CI/CD), forwarding a developer’s personal, highly-privileged key is a huge anti-pattern.

The truly robust, enterprise-grade solution is to take user keys out of the equation entirely for automation.

This approach decouples deployment from individual user accounts. It’s more secure, more auditable, and completely sidesteps the agent forwarding problem because the process has its own dedicated identity. This is the path we take for all of our production automation at TechResolve.

So next time a script mysteriously fails with a permission error, take a breath. It’s probably not you, and it’s definitely not the server gaslighting you. It’s just a missing environment variable, and now you have a full toolkit to fix it—from the quick hack to the right architecture.

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions