Frustrated with persistent SSH key or dependency issues in your CI/CD pipeline? A senior DevOps engineer shares three real-world fixes, from the quick-and-dirty to the architecturally sound, to get your builds running again.

So, Your CI/CD Pipeline Broke… Again. An Old-Timer’s Guide to Fixing It For Good.

I remember it like it was yesterday. 2:17 AM. The PagerDuty alert blared, shaking me out of a dead sleep. A critical security patch for our main `prod-auth-service` was failing in the deployment pipeline. I jump on the call, and a panicked junior engineer is sharing his screen. The error staring back at us was the one that gives us all nightmares: Host key verification failed. fatal: Could not read from remote repository. He’d been trying to regenerate keys for 45 minutes, convinced he’d messed something up. We’ve all been there. That feeling of dread when the thing that’s supposed to be automated and reliable suddenly isn’t. That Reddit thread title? “Is anyone else considering alternatives?” Yeah, I felt that in my bones at 2 AM.

The Root of the Problem: Ephemeral Trust

Before we dive into the fixes, let’s talk about why this keeps happening. Your CI/CD runner—whether it’s a GitLab Runner, a GitHub Action, or a Jenkins agent—is usually an ephemeral, stateless container. It spins up, does its job, and vanishes. It has no memory of the past. The core issue is establishing trust for that short-lived environment so it can securely access other resources, like a private Git repository.

When you use SSH keys, you’re managing that trust manually. The runner needs the private key, and the remote server needs to know about the runner’s host key (the whole known_hosts dance). When the base image for your runner changes, or a key expires, or permissions get tweaked, that trust breaks. The pipeline fails, and you’re left scrambling.

Fix #1: The “3 AM Hotfix” (Quick & Dirty)

This is the band-aid. The duct tape. The thing you do to get the critical patch out the door while the building is on fire. It’s not pretty, and you’re absolutely creating technical debt, but sometimes, you just need the build to pass right now.

The idea is to inject the private SSH key directly into the CI job as a variable and use ssh-agent to load it at runtime.

1. Go to your project’s CI/CD settings (e.g., GitLab CI/CD Variables or GitHub Actions Secrets).
2. Create a new variable, let’s call it SSH_PRIVATE_KEY.
3. Paste the entire contents of your private key file (including -----BEGIN OPENSSH PRIVATE KEY----- and the end line) into the value field. Make sure it’s set as “Masked” and “Protected” if the platform supports it.

Then, add this boilerplate to the top of your build script in your .gitlab-ci.yml or equivalent:

before_script:
  - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
  - eval $(ssh-agent -s)
  - echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
  - mkdir -p ~/.ssh
  - chmod 700 ~/.ssh
  - ssh-keyscan gitlab.com >> ~/.ssh/known_hosts
  - chmod 644 ~/.ssh/known_hosts

Warning: This is a hack! You are passing a sensitive secret as a variable and relying on the CI system’s masking. This key is often tied to a user account, which is a bad practice for automation. Use this to solve an outage, then immediately schedule time to implement a better fix.

Fix #2: The “Do It Right” Fix (Permanent & Scoped)

Okay, the fire is out. Now let’s rebuild the house properly. Instead of using a user’s personal SSH key, we should use a key that is specifically designed for machine-to-machine communication. Most Git platforms call these Deploy Keys or Deploy Tokens.

A Deploy Key is an SSH key you generate and add to a single repository. It grants read-only (or sometimes read-write) access to just that one repo. This is the perfect solution for when your pipeline needs to pull in a specific private library, like `shared-utils-repo`.

Deploy Tokens are even better. They are typically HTTP-based credentials (a username and a password/token) that can be scoped to grant access to multiple repositories within a group or organization.

Here’s a quick comparison:

By using a dedicated Deploy Key/Token, you’ve decoupled your CI process from a human user account. It’s more secure, easier to audit, and you can rotate it without breaking someone’s local development setup.

Fix #3: The “Rethink Everything” Option (Modern & Token-Based)

This is my personal favorite and what we use for all new services at TechResolve. We get rid of SSH keys for this workflow entirely.

Modern CI/CD platforms provide short-lived, auto-generated, OIDC-compliant tokens for each and every job. In GitLab, this is the CI_JOB_TOKEN. In GitHub, it’s the GITHUB_TOKEN. These tokens are magical—they are automatically created when the job starts and automatically expire when it finishes. They have permissions scoped to the project the pipeline is running in.

Instead of cloning with SSH (git@gitlab.com:group/project.git), you configure Git to clone with HTTPS using this token.

Here’s how you’d do it in a pipeline script:

# This command tells Git that for any HTTPS operations to gitlab.com,
# it should use the username 'gitlab-ci-token' and the password from the job token.
git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.com/".insteadOf "https://gitlab.com/"

# Now, when your build tool (npm, pip, go) tries to fetch a private repo,
# it will automatically use the job token for authentication.
# For example, your package.json might have: "my-private-lib": "git+https://gitlab.com/my-group/my-private-lib.git"
# The git config above will transparently handle authentication.
npm install

Pro Tip: This approach completely eliminates the need to manage SSH keys for dependency cloning. No more ssh-agent, no more known_hosts, no more expiring keys. The identity and trust are handled for you by the platform. It’s a bigger change to your dependency definitions, but it solves an entire class of frustrating pipeline failures.

So next time you see that dreaded permission error, take a breath. Get the system back online with the hotfix if you must, but promise yourself you’ll come back and do it right. Your future self (and your sleep schedule) will thank you.

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions