🚀 Executive Summary
TL;DR: Modern Linux systems, primarily due to `systemd-logind`’s default behavior, can leave user processes running after logout, consuming vital server resources. This issue can be resolved through immediate manual cleanup using `loginctl`, a permanent systemd configuration change (`KillUserProcesses=yes`), or by hardening SSHD to proactively terminate unresponsive connections.
🎯 Key Takeaways
- `systemd-logind` is the root cause of lingering user processes after logout on modern Linux, as it’s configured by default to allow processes to persist, supporting tools like `screen` and `tmux`.
- For immediate incident response, `loginctl terminate-user
` is the recommended command to cleanly send a SIGTERM signal to all processes within a user’s systemd scope. - Setting `KillUserProcesses=yes` in `/etc/systemd/logind.conf` provides a permanent solution by automatically terminating all user processes upon logout, though it requires users to adopt `nohup` or `systemd-run` for long-running tasks.
Lingering user sessions and processes after logout can cripple servers by consuming vital resources. Learn the root cause in modern Linux systems and explore three practical solutions, from a quick manual cleanup to a permanent systemd configuration fix.
The Ghosts in the Machine: Why Your SSH Sessions Won’t Die (And How to Kill Them for Good)
I remember a 2 AM page. prod-db-01 was unresponsive. Not down, just… sluggish. After finally getting a shell, a quick ps aux | grep 'someuser' revealed hundreds of sleeping Python processes. A junior dev had been running a data migration script, disconnected their laptop, and went home. But their processes didn’t get the memo. They were ghost sessions, silently eating up PIDs and memory, and they almost brought our primary database to its knees. This isn’t a rare problem; it’s a rite of passage. If you’ve seen a server mysteriously run out of resources, there’s a good chance you’ve been haunted by these digital ghosts.
So, What’s Actually Happening Here? The “Why”
In the old days, when you logged out, the parent process of your shell would die, and the kernel would send a SIGHUP (hang-up signal) to all its children. Most well-behaved programs would exit gracefully. But things are different now, mostly thanks to systemd.
On most modern Linux distributions, systemd-logind manages user sessions. By default, it’s configured to let your processes keep running even after you log out. This is actually a feature, not a bug! It’s what allows tools like screen and tmux to work their magic, letting you detach from a session and come back to it later. The problem is, this “feature” doesn’t distinguish between a legitimate, long-running job in tmux and a buggy script that forked itself into oblivion before you disconnected.
When you log out, your last session ends, but the user’s “slice” in systemd’s cgroup hierarchy remains, keeping all those stray processes alive and well. Let’s fix that.
Solution 1: The Quick Fix (Manual Cleanup)
You’re in the middle of an incident. The server is on fire. You don’t have time for config changes and service reloads. You just need to kill the offending processes now. This is your tactical, get-it-done approach.
First, identify the lingering sessions using loginctl:
$ loginctl list-sessions
SESSION UID USER SEAT TTY
c2 1001 someuser /dev/pts/0
1 sessions listed.
You see the session, but the user isn’t actually connected. They’re a ghost. You can get more details, but what you really want is to terminate everything associated with that user cleanly. Don’t use a messy pkill; use the tool designed for the job.
# As root
loginctl terminate-user someuser
This command tells systemd-logind to find all processes belonging to someuser and send a SIGTERM signal, effectively cleaning up everything in their scope. It’s fast, clean, and much safer than a blind kill -9.
Pro Tip: This is a great command to have in your back pocket for emergencies. It’s a scalpel, not a sledgehammer. But remember, this is purely reactive. You’ll be doing this again next week if you don’t apply a permanent fix.
Solution 2: The Permanent Fix (The ‘Right’ Way)
If you’re tired of playing whack-a-mole with ghost processes, it’s time to tell systemd how you want it to behave. We can do this by changing one line in the logind.conf file.
The setting we care about is KillUserProcesses. By default, it’s set to no. We’re going to change that.
Step 1: Edit the configuration file.
# As root
nano /etc/systemd/logind.conf
Step 2: Find and change the setting.
Look for the commented-out line #KillUserProcesses=no. Uncomment it and change it to yes:
[Login]
...
KillUserProcesses=yes
...
Step 3: Restart the service to apply the change.
# As root
systemctl restart systemd-logind.service
Now, when the last session for a user ends (i.e., they log out of their last SSH connection), systemd-logind will automatically terminate all processes running in that user’s slice. The ghosts are busted, for good.
Warning! This is a powerful change with consequences. This will also kill any legitimate background processes, including
screenandtmuxsessions, when the user logs out. You must communicate this change to your team. They’ll need to use commands likenohupor learn to start their long-running tasks using systemd’ssystemd-runif they want them to persist.
Solution 3: The ‘Network Layer’ Fix (Hardening SSHD)
Sometimes the problem isn’t just backgrounded processes, but SSH connections that simply hang due to network issues, never properly terminating. They sit in a zombie-like state, holding a TTY and other resources. This is where we can harden the SSH daemon itself.
We’ll edit the sshd_config file to proactively check if a client is still alive and drop it if it’s not responding.
Step 1: Edit the SSHD configuration file.
# As root
nano /etc/ssh/sshd_config
Step 2: Add or modify these two settings.
ClientAliveInterval 300
ClientAliveCountMax 2
Here’s what this does:
ClientAliveInterval 300: The server will send a keep-alive message to the client every 300 seconds (5 minutes).ClientAliveCountMax 2: If the server sends a keep-alive message and gets no response, it will try again. After 2 failed attempts, it will kill the connection.
Effectively, this means a non-responsive SSH session will be terminated after about 10 minutes (300 seconds * 2). You can tune these values to be more or less aggressive.
Step 3: Restart the SSHD service.
# As root
systemctl restart sshd.service
This fix is complementary to Solution 2. It cleans up dead connections, while the logind.conf change cleans up orphaned processes after a successful logout. Using both gives you a robust, two-layer defense against ghost sessions.
Which one is for you?
| Solution | Best For | Caveat |
| 1. Manual Cleanup | Immediate, emergency situations. | Doesn’t prevent the problem from recurring. |
| 2. The Permanent Fix | Servers where users should not have long-running processes after logout (e.g., web servers, app servers). | Breaks tools like screen/tmux. Requires team buy-in. |
| 3. Hardening SSHD | All servers, as a general hardening best practice to clean up dead network connections. | Won’t stop intentionally backgrounded processes. |
There’s no one-size-fits-all answer, but for 90% of the production servers I manage, a combination of #2 and #3 is the gold standard. It establishes a clear, predictable environment where logging out means your processes are cleaned up, period. Stop letting ghosts run your machines. Take control back.
🤖 Frequently Asked Questions
âť“ Why do processes continue to run after a user logs out on modern Linux systems?
On modern Linux, `systemd-logind` manages user sessions and, by default, allows processes to persist after logout. This is a feature designed to support tools like `screen` and `tmux` for detaching and reattaching to sessions.
âť“ How does setting `KillUserProcesses=yes` compare to using `screen` or `tmux` for persistent sessions?
Setting `KillUserProcesses=yes` in `logind.conf` will cause `systemd-logind` to terminate all user processes, including active `screen` or `tmux` sessions, upon logout. Users wishing for persistent tasks must then utilize `nohup` or `systemd-run` instead.
âť“ What is a common implementation pitfall when configuring `KillUserProcesses=yes` and how can it be addressed?
A common pitfall is inadvertently terminating legitimate background processes or `screen`/`tmux` sessions that users expect to persist. This can be addressed by clearly communicating the change to the team and instructing them to use `nohup` or `systemd-run` for tasks intended to run after logout.
Leave a Reply