Monitoring Systemd Services: Auto-restart and Email Alert Script

Hey team, Darian here. I wanted to share a workflow that’s saved me a ton of headache and reclaimed a few hours of my week. I used to spend the first 30 minutes of my day manually checking the status of our critical services across different servers. It was tedious, and if something failed overnight, I wouldn’t know until the next morning after a client already found it. This simple setup fixed all that. Now, systemd handles the instant restart, and I get an email alert right away. Let’s walk through it so you can get that time back, too.

Prerequisites

• A Linux server running systemd.
• Python 3 installed on the server.
• Access to an SMTP server for sending email alerts (e.g., Gmail with an App Password, SendGrid, etc.).
• A specific systemd service you need to monitor (e.g., `nginx.service` or `my_custom_app.service`).

The Step-by-Step Guide

Step 1: Configure Your Systemd Service for Auto-Restart

First things first, let’s make sure systemd itself is doing the heavy lifting. The goal is self-healing. Our script is just for notification, not intervention. You’ll need to edit the unit file for the service you want to monitor. These files are typically located in the standard systemd service directories.

Open your `your_app.service` file and add or modify the following lines in the `[Service]` section:

[Service]
ExecStart=/path/to/your/application
Restart=on-failure
RestartSec=5s

• Restart=on-failure: This is the key. It tells systemd to restart the service if it exits with a non-zero exit code (which typically indicates an error).
• RestartSec=5s: This tells systemd to wait 5 seconds before attempting a restart. It prevents a rapid-fire restart loop if the service is failing immediately on startup.

After editing the file, remember to reload the systemd daemon to apply the changes. You can do this with `systemctl daemon-reload`.

Pro Tip: For services that are absolutely critical and should always be running, I sometimes use Restart=always. But be careful. If a service is failing due to a bad configuration, this can cause a non-stop restart loop and spam your logs. Use `on-failure` for most cases.

Step 2: Create the Python Monitoring Script

Now for the notification part. This script will check if a service is active. If it’s not, it will send us an email. I’ll skip the standard virtualenv setup since you likely have your own workflow for that. Let’s jump straight to the Python logic.

First, we need a configuration file to store our sensitive email credentials. Create a file named config.env in the same directory as your script:

SMTP_SERVER="smtp.your-email-provider.com"
SMTP_PORT=587
EMAIL_SENDER="your-email@example.com"
EMAIL_PASSWORD="your-app-password"
EMAIL_RECEIVER="your-alert-email@example.com"

Next, create the Python script, let’s call it monitor_service.py:

import os
import smtplib
import subprocess
from email.message import EmailMessage
from dotenv import load_dotenv

# --- Configuration ---
# Name of the systemd service to check
SERVICE_TO_CHECK = "nginx.service" 

def check_service_status(service_name):
    """Checks if a systemd service is active. Returns True if active, False otherwise."""
    try:
        # The '--quiet' flag makes it return exit code 0 if active, non-zero otherwise.
        # This is more reliable than parsing text output.
        subprocess.run(
            ["systemctl", "is-active", "--quiet", service_name], 
            check=True
        )
        print(f"Service '{service_name}' is active.")
        return True
    except subprocess.CalledProcessError:
        # This exception is raised for a non-zero exit code
        print(f"Service '{service_name}' is not active or has failed.")
        return False
    except FileNotFoundError:
        # This happens if 'systemctl' isn't found
        print("Error: 'systemctl' command not found. Is this a systemd-based system?")
        return False

def send_alert_email(service_name):
    """Sends an email notification that the service is down."""
    load_dotenv("config.env")
    
    sender = os.getenv("EMAIL_SENDER")
    password = os.getenv("EMAIL_PASSWORD")
    receiver = os.getenv("EMAIL_RECEIVER")
    smtp_server = os.getenv("SMTP_SERVER")
    smtp_port = int(os.getenv("SMTP_PORT", 587))

    if not all([sender, password, receiver, smtp_server]):
        print("Email configuration is incomplete. Check your config.env file.")
        return

    msg = EmailMessage()
    msg.set_content(f"The service '{service_name}' on your server has failed and could not be automatically restarted by systemd.\n\nPlease investigate immediately.")
    msg['Subject'] = f"ALERT: Systemd Service '{service_name}' is Down!"
    msg['From'] = sender
    msg['To'] = receiver

    try:
        with smtplib.SMTP(smtp_server, smtp_port) as server:
            server.starttls()
            server.login(sender, password)
            server.send_message(msg)
            print("Alert email sent successfully.")
    except Exception as e:
        print(f"Failed to send email: {e}")

def main():
    if not check_service_status(SERVICE_TO_CHECK):
        send_alert_email(SERVICE_TO_CHECK)

if __name__ == "__main__":
    main()

Step 3: Prepare the Environment

To run the script, you’ll need one external library, `python-dotenv`, to load our `config.env` file. In your activated virtual environment, you would typically run `pip install python-dotenv` to get it installed. Make sure both `monitor_service.py` and `config.env` are in the same project directory.

Step 4: Automate the Script with a Cron Job

A monitoring script is only useful if it runs automatically. We’ll use cron for this. To edit your cron jobs, you’ll use your system’s standard scheduler command. Add a line like this to run our script every 5 minutes:

*/5 * * * * python3 /path/to/your/project/monitor_service.py

For example, if you wanted it to run once a week on Monday at 2 AM, the format would be `0 2 * * 1 python3 script.py`. The key is to provide the full path to your script so cron knows exactly what to execute.

Common Pitfalls (Where I Usually Mess Up)

• Permissions: The user running the cron job must have permission to execute the Python script and, more importantly, permission to run `systemctl` commands. If your script works manually but fails via cron, permissions are the first thing I check.
• Email Authentication: I’ve spent way too long debugging scripts only to realize my SMTP password was wrong or I’d triggered a security lockout on my email provider (like Gmail requiring an “App Password” instead of your main password). Always test the email function separately.
• Cron’s Minimal Environment: Cron jobs run with a very limited set of environment variables. That’s why providing an absolute path to your script is crucial. Also, ensure the `python3` command in your cron job points to the correct interpreter, especially if you’re using a virtual environment.

Conclusion

And that’s the setup. It’s a simple, robust way to let systemd handle the immediate recovery while ensuring you’re kept in the loop if something goes seriously wrong. In my production setups, I’ve expanded this script to take the service name as an argument, allowing me to monitor multiple services with the same logic. Feel free to adapt it to your needs. Now you can have peace of mind knowing you’ll be the first to know, not the last.

– Darian

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions