🚀 Executive Summary

TL;DR: User processes and screen sessions vanish on modern Linux due to systemd-logind’s aggressive cleanup, specifically `RemoveIPC=yes` by default. This can be solved by enabling linger for specific users, modifying `logind.conf` system-wide, or using `systemd-run` for robust service management.

🎯 Key Takeaways

  • Modern Linux systems utilizing `systemd-logind` are configured by default to aggressively clean up user session scopes, including background processes and IPC objects, when the last user session logs out, driven by the `RemoveIPC=yes` setting.
  • The `loginctl enable-linger ` command is the officially recommended and targeted method to allow specific users’ processes and services to persist indefinitely after logout.
  • For ad-hoc critical tasks requiring proper service management, `systemd-run –user –scope -p “KillMode=process” /path/to/script.sh` provides a robust, systemd-native way to run processes detached from login sessions.

What happened to Ingram?

Uncover the reason your user processes and screen sessions are vanishing on modern Linux distributions. We’ll explore systemd-logind’s aggressive cleanup and provide actionable fixes to ensure your long-running jobs are safe.

What Happened to ‘Ingram’? Unmasking the Silent Killer of Your Linux Sessions

I got the panicked Slack message at 2:17 AM. It was from Mark, one of our sharpest junior engineers. “Darian, the migration script on prod-db-01 is gone. The screen session just… vanished. No errors in the script logs. I think I broke something.” My heart rate spiked. This was a multi-terabyte data sync that had been running for 18 hours. Starting over wasn’t an option. I logged in, checked `ps`, checked `screen -ls`… nothing. It was like the user ‘ingram’, which ran the script, had been wiped clean by a ghost. But I’d seen this ghost before. It wasn’t Mark’s fault. It was the system trying to be “helpful”, and in the process, costing us hours of work and a whole lot of stress.

The “Why”: It’s Not a Bug, It’s a (Frustrating) Feature

So, what’s really going on here? This isn’t a random kernel bug or a hardware fault. This is a deliberate design choice in modern Linux systems that use systemd. Specifically, we’re talking about the `systemd-logind` service.

Its job is to manage user login sessions. By default, it’s configured to be a very tidy housekeeper. When the last session for a user logs out (e.g., your final SSH connection closes), `systemd-logind` assumes the user is done. It then proceeds to clean up everything associated with that user’s “session scope.” This includes background processes, running scripts, and even inter-process communication (IPC) objects. Your carefully detached screen or tmux session? To systemd, that’s just another mess to sweep up.

The specific culprit is a setting in /etc/systemd/logind.conf called RemoveIPC. By default, it’s set to yes, which enables this aggressive cleanup. The system logs a message like “Stopping User Manager for UID 1001,” but if you don’t know to look for it in the journal, it feels like your process just disappeared into thin air.

The Fixes: From Quick Patch to Proper Solution

You’ve got a few ways to tackle this, depending on your needs and your philosophy. I’ll lay out the three main approaches we use at TechResolve.

Solution 1: The Quick Fix – Enable Linger

This is my go-to for service accounts or any user that needs to run background tasks indefinitely. You’re essentially telling systemd, “Hey, for this specific user, leave their stuff alone even after they log out.” It’s targeted, clean, and the officially recommended method.

You do it with a single command:

loginctl enable-linger ingram

This creates a file at /var/lib/systemd/linger/ingram, marking the user ‘ingram’ as a “lingering” user. Their services and processes will now persist after logout. It’s the perfect scalpel for the job.

Solution 2: The Permanent Fix – System-Wide Config Change

If you’re running a multi-user development box where everyone needs to run long-running tasks, changing the system-wide behavior might be easier than enabling linger for every new user. This is the “blunt instrument” approach.

You’ll edit the systemd login manager configuration file:

# Edit this file:
sudo nano /etc/systemd/logind.conf

# Find this line (it might be commented out with a '#'):
#RemoveIPC=yes

# Change it to 'no' and remove the '#':
RemoveIPC=no

After saving the file, you need to tell systemd to reload the configuration.

sudo systemctl restart systemd-logind

Warning: Restarting systemd-logind can sometimes terminate current user sessions. It’s best to do this during a maintenance window or ensure nobody is performing critical work on the machine.

This approach disables the process cleanup for all users on the system. It’s effective, but less precise. Use it when the machine’s primary purpose is long-running, user-initiated jobs.

Solution 3: The ‘Nuclear’ Option – Use systemd-run

This is for the purists. The idea here is: instead of fighting systemd, use it properly. If you want a process to run as a service, detached from your login session… then run it as a service! The systemd-run command lets you do this on the fly without writing a full-blown service file.

Instead of running your script inside screen, you’d do this:

systemd-run --user --scope -p "KillMode=process" /path/to/your/long_running_script.sh

This command tells systemd to run your script in its own “scope,” completely independent of your login. It’s now a first-class citizen in the systemd world. You can check its status with systemctl --user status and see its logs with journalctl. It’s more complex and has a steeper learning curve, but it’s arguably the “correct” way to manage daemons and long-running tasks in a modern Linux environment.

Choosing Your Weapon

There’s no single right answer, only the right answer for your context. Here’s how I break it down for my team:

Method Best For Pros Cons
Enable Linger Dedicated service accounts (e.g., ‘jenkins’, ‘deployer’). Targeted, clean, reversible per-user. Must be run for each user account.
Edit logind.conf Multi-user dev servers where everyone needs this behavior. Set it and forget it. Affects all users. System-wide change, less secure, might leave clutter.
systemd-run Ad-hoc critical tasks that need proper service management. Integrates with systemd tooling, very robust. Complex syntax, higher learning curve.

The next time a process mysteriously dies, don’t blame the junior dev (or yourself). Take a deep breath, check the system journal for `systemd-logind` messages, and remember the ghost in the machine. Then, pick your fix and get back to building cool stuff.

Darian Vance - Lead Cloud Architect

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.


🤖 Frequently Asked Questions

❓ Why do my Linux processes disappear after I log out?

Your Linux processes disappear after logout because `systemd-logind`, by default, cleans up user session scopes, including background processes and IPC objects, when the last session for a user closes. This behavior is controlled by the `RemoveIPC=yes` setting in `/etc/systemd/logind.conf`.

❓ How do `loginctl enable-linger` and editing `logind.conf` compare for process persistence?

`loginctl enable-linger` is a targeted, per-user solution, ideal for service accounts, allowing their processes to persist without affecting other users. Editing `logind.conf` to set `RemoveIPC=no` is a system-wide change that affects all users, suitable for multi-user development servers but is less precise and potentially less secure.

❓ What is a common pitfall when modifying `systemd-logind` configuration?

A common pitfall is that restarting `systemd-logind` after modifying `/etc/systemd/logind.conf` (using `sudo systemctl restart systemd-logind`) can terminate current user sessions. It is best to perform this action during a maintenance window to avoid disrupting critical work.

Leave a Reply

Discover more from TechResolve - SaaS Troubleshooting & Software Alternatives

Subscribe now to keep reading and get access to the full archive.

Continue reading