🚀 Executive Summary
TL;DR: The ‘Permission denied (publickey)’ error, even with correct SSH key ownership, often stems from overly permissive file modes on the ~/.ssh directory or authorized_keys file. The SSH daemon (sshd) enforces strict security, rejecting keys if these files are group or world-writable, requiring specific chmod 700 and 600 permissions respectively, ideally managed through Infrastructure as Code.
🎯 Key Takeaways
- The SSH daemon (sshd) enforces ‘StrictModes’, checking not only file ownership but also the permissions (modes) of the ~/.ssh directory and ~/.ssh/authorized_keys file.
- Overly permissive file modes, such as drwxrwxr-x (775) for ~/.ssh or -rw-rw-r– (664) for authorized_keys, will cause ‘Permission denied (publickey)’ errors due to security concerns.
- The secure and functional permissions are drwx—— (700) for the ~/.ssh directory and -rw——- (600) for the ~/.ssh/authorized_keys file.
- Immediate fixes involve using `chmod 700 ~/.ssh` and `chmod 600 ~/.ssh/authorized_keys` on the affected server.
- For a permanent solution, integrate these secure permissions into Infrastructure as Code (e.g., Ansible’s `file` and `authorized_key` modules) to prevent recurrence.
- Avoid using `chmod 777` on sensitive files or directories as it creates a severe security vulnerability rather than solving the permission issue.
Summary: Struggling with infuriating ‘Permission denied’ errors even when file ownership is correct? Your file modes are likely too permissive. I’ll show you why this happens and how to fix the silent killer that’s breaking your deployments.
That Time ‘chmod 777’ Wrecked Our Friday Night Deployment
I remember it like it was yesterday. It was 7 PM on a Friday, and we were trying to push a critical hotfix to our production authentication service, `prod-auth-svc-01`. The deployment pipeline, a battle-tested Ansible playbook, suddenly started failing. The error? A cryptic and infuriating `Permission denied (publickey)`. We checked everything. The SSH key was on the server. The `deploy-user` owned the key file. We could even `su` to the user and see the files. For an hour, we stared at the screen, baffled. This, my friends, is a classic DevOps rite of passage—a problem where everything looks right, but is fundamentally wrong.
The “Why”: More Than Just Ownership
Here’s the piece of tribal knowledge that gets passed down in the trenches: for some services, especially security-sensitive ones like the SSH daemon (`sshd`), ownership isn’t enough. The service also checks the permissions of the file and its entire parent directory path.
If `sshd` sees that your `~/.ssh` directory or your `~/.ssh/authorized_keys` file is writable by anyone else (i.e., the group or “the world”), it makes a logical security assumption: “This file could have been modified by another user. I cannot trust its contents.” It then silently rejects the key and gives you that generic “Permission denied” error, never telling you that the file permissions are too open. It’s a feature, not a bug, designed to protect you from yourself.
Here’s a quick breakdown of what goes wrong:
| File/Directory | Insecure (Will Fail) | Secure (Will Work) |
/home/deploy-user/.ssh |
drwxrwxr-x (775) |
drwx------ (700) |
/home/deploy-user/.ssh/authorized_keys |
-rw-rw-r-- (664) |
-rw------- (600) |
The Fixes: From Triage to Architecture
When you’re hit with this, you have a few ways out. Which one you choose depends on whether you’re bleeding out on the floor or designing the hospital.
1. The Quick Fix: Command-Line Triage
It’s Friday night, the site is down, and you just need it to work. This is the emergency fix you apply directly on the affected server, like `prod-auth-svc-01`. You need to lock down the permissions to the bare minimum required.
Log in to the server and run these two commands for the user whose key is failing (e.g., `deploy-user`):
# Lock down the .ssh directory itself. Only the owner should have access.
chmod 700 /home/deploy-user/.ssh
# Lock down the authorized keys file. Only the owner can read and write.
chmod 600 /home/deploy-user/.ssh/authorized_keysThis almost always solves the immediate problem. Your pipeline will light up green, and you can go back to your weekend. But remember, this is a band-aid. The problem will happen again the next time a new server is built or a user is configured incorrectly.
Warning: The path of least resistance often leads to `chmod 777`. Never do this on a home directory or anything remotely sensitive. You’re not fixing a permission problem; you’re creating a security vulnerability. If you feel the urge, take a walk first.
2. The Permanent Fix: Bake It Into Your IaC
A senior engineer doesn’t just fix problems; they prevent them from ever happening again. The real solution is to enforce the correct permissions using your configuration management tool (like Ansible, Puppet, or Salt) or your provisioning scripts.
If you’re using Ansible, your user setup or SSH key deployment task should look something like this:
- name: Ensure .ssh directory has secure permissions
ansible.builtin.file:
path: "/home/deploy-user/.ssh"
state: directory
owner: deploy-user
group: deploy-user
mode: '0700'
- name: Deploy the authorized key with secure permissions
ansible.posix.authorized_key:
user: deploy-user
key: "{{ lookup('file', 'deploy-user.pub') }}"
state: present
manage_dir: no # We are managing the directory permissions above
# As an alternative to the authorized_key module for existing files:
- name: Ensure authorized_keys file has secure permissions
ansible.builtin.file:
path: "/home/deploy-user/.ssh/authorized_keys"
owner: deploy-user
group: deploy-user
mode: '0600'
state: touch # Creates the file if it doesn't exist
By defining the permissions as code, every server you build will be configured correctly from the start. This is how you move from firefighting to engineering.
3. The ‘Nuclear’ Option: The Filesystem Audit
Sometimes, the problem isn’t in `~/.ssh` at all. Maybe a parent directory like `/home/deploy-user` has loose permissions, which `sshd` can also complain about in `StrictModes` mode. Or maybe some other sensitive config file is the culprit.
When you’re completely lost, you can go hunting for any file with dangerously open permissions. This is a bit of a “hail mary” but can uncover issues you didn’t know you had.
Use the `find` command to search for files and directories in a user’s home that are writable by people who shouldn’t be writing to them.
# Find all directories in deploy-user's home that are group- or world-writable
find /home/deploy-user -type d -perm /g+w,o+w
# Find all files that are group- or world-writable
find /home/deploy-user -type f -perm /g+w,o+w
This won’t fix anything, but it will give you a list of suspects. From there, you can investigate each one and decide on the appropriate `chmod` value. Use this when you’re debugging a truly weird issue, not as a first resort.
So next time you see `Permission denied`, take a breath. Don’t just check `who` owns the file; check who else can touch it. Your sanity (and your Friday nights) will thank you.
🤖 Frequently Asked Questions
❓ Why am I getting ‘Permission denied (publickey)’ even though my SSH key is present and owned by the correct user?
This error typically occurs because the SSH daemon (sshd) enforces ‘StrictModes’ and finds the permissions on your ~/.ssh directory or ~/.ssh/authorized_keys file to be too permissive (e.g., group or world-writable). sshd will silently reject the key for security reasons.
❓ How does fixing SSH key permissions compare to other common SSH authentication issues?
Unlike issues such as an incorrect key path or the key not being loaded, which are about existence or configuration, permission issues specifically relate to the security posture of the key files themselves. While other problems might prevent the key from being found, permission issues prevent it from being trusted by the sshd service even if found.
❓ What is a common implementation pitfall when setting up SSH key permissions, and how can it be avoided?
A common pitfall is resorting to `chmod 777` to ‘fix’ permission errors, which creates a significant security vulnerability. This can be avoided by understanding the specific secure permissions required (700 for ~/.ssh, 600 for authorized_keys) and enforcing them consistently through Infrastructure as Code tools like Ansible, rather than manual, insecure overrides.
Leave a Reply