Visualizing Systemd Boot Time Performance across Fleet

Hey there, Darian Vance here. If you’re managing more than a handful of Linux servers, you’ve probably had that sinking feeling when one of them takes an eternity to reboot. Is it a network mount? A misconfigured service? Hunting down the cause by SSHing into box after box is a massive time sink. I used to spend a couple of hours a week just spot-checking slow boots after maintenance windows. It was tedious, reactive, and frankly, a waste of engineering time.

That’s why I put together this workflow. We’re going to build a simple, automated system to pull `systemd-analyze` data from every server in your fleet, aggregate it, and generate a single chart showing the worst offenders. This turns a manual, multi-hour chore into a 5-minute coffee break review. Let’s get it done.

Prerequisites

Before we start, make sure you have the following ready:

• Central Admin Server: A machine where our script will run. This machine needs network access to your fleet.
• Python 3: Installed on your central server.
• Required Python Packages: You’ll need `paramiko` for SSH, `pandas` for data manipulation, and `matplotlib` for plotting. I’ll skip the standard virtualenv setup since you likely have your own workflow for that. Just make sure you install those three packages into your project’s environment.
• SSH Access: SSH key-based authentication from your central server to the fleet servers. Password-based works, but key-based is more secure and what I use in production.
• Server List: A simple text file with one hostname or IP address per line.

The Guide: From Raw Logs to Insight

Step 1: Understand the Source – `systemd-analyze blame`

First, let’s look at our data source. When you run `systemd-analyze blame` on a server, you get a list of all the units that started during boot, sorted by the time they took. The output looks something like this:

          25.871s networkd-wait-online.service
          15.234s postgresql.service
           8.110s some-custom-app.service
           ...

Our goal is to run this command on every server, collect all this text, and find the services that consistently take the longest across the entire fleet.

Step 2: Project Setup and Configuration

On your central server, create a project directory. Inside, we’ll need three things:

1. `servers.txt`: A plain text file listing your target server hostnames or IPs, one per line.
2. `config.env`: A configuration file to hold our credentials securely. Never hardcode secrets!
3. `boot_analyzer.py`: Our main Python script.

Your `config.env` file should be simple:

SSH_USER=your_ssh_username
SSH_KEY_PATH=/home/your_ssh_username/.ssh/id_rsa

Pro Tip: Make sure your `config.env` file is in your `.gitignore` file. You do not want to commit credentials to source control. It’s a classic mistake that can have serious consequences.

Step 3: The Python Script – Fetching the Data

Now for the fun part. Let’s start building `boot_analyzer.py`. The first piece of logic will be to read our server list, connect to each one via SSH, and run our command. We’ll use the `paramiko` library for the heavy lifting.

Here’s the initial code to handle the connections and data fetching. I’ve added comments to explain what each part does.

import os
import paramiko
import pandas as pd
import matplotlib.pyplot as plt

# WARNING: This is a simplified example. In production, use a proper config library.
def load_config():
    config = {}
    with open('config.env') as f:
        for line in f:
            if '=' in line:
                key, value = line.strip().split('=', 1)
                config[key] = value
    return config

def fetch_boot_data(server, user, key_path):
    """Connects to a server and fetches systemd-analyze blame output."""
    try:
        ssh = paramiko.SSHClient()
        ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
        
        print(f"Connecting to {server}...")
        ssh.connect(server, username=user, key_filename=key_path, timeout=10)
        
        stdin, stdout, stderr = ssh.exec_command('systemd-analyze blame')
        
        # Handle potential errors from the command itself
        err = stderr.read().decode()
        if err:
            print(f"Error on {server}: {err}")
            return None

        output = stdout.read().decode()
        ssh.close()
        print(f"Successfully fetched data from {server}.")
        return output
    except Exception as e:
        print(f"Failed to connect or run command on {server}: {e}")
        return None

def main():
    config = load_config()
    SSH_USER = config.get('SSH_USER')
    SSH_KEY_PATH = config.get('SSH_KEY_PATH')

    if not all([SSH_USER, SSH_KEY_PATH]):
        print("Error: SSH_USER or SSH_KEY_PATH not found in config.env")
        return

    all_data = []
    with open('servers.txt') as f:
        servers = [line.strip() for line in f if line.strip()]

    for server in servers:
        data = fetch_boot_data(server, SSH_USER, SSH_KEY_PATH)
        if data:
            # We add the server name to each line for later processing
            for line in data.strip().split('\n'):
                all_data.append(f"{server} {line.strip()}")
    
    # We will add parsing and plotting logic here in the next steps.
    print("\nData collection complete.")
    # For now, let's just see what we got
    for item in all_data[:10]: # Print first 10 lines
         print(item)

if __name__ == "__main__":
    main()

Step 4: Parsing and Aggregating with Pandas

Great, we have a list of strings. Each string is a raw line from `systemd-analyze blame`, prefixed with the server it came from. This is messy. Let’s use the power of `pandas` to structure this data.

We’ll convert our list into a DataFrame, which is essentially a powerful, in-memory spreadsheet. We’ll then clean it up, extract the service name and boot time, and aggregate the results.

Add this function to your script and call it from `main()`.

def parse_and_aggregate_data(raw_data):
    """Parses raw text lines into a structured pandas DataFrame and aggregates it."""
    # Create a DataFrame from our list of strings
    df = pd.DataFrame(raw_data, columns=['raw'])

    # Split the raw string into columns. This regex is a bit tricky but robust.
    # It splits on whitespace, but groups time values (e.g., '1min 2.345s') together.
    parts = df['raw'].str.extract(r'(\S+)\s+([\dms\.]+)\s+(.*)')
    df['server'] = parts[0]
    df['time_str'] = parts[1]
    df['service'] = parts[2]

    # Drop rows that didn't parse correctly
    df.dropna(inplace=True)

    # A simple function to convert time strings (e.g., '25.871s', '1min', '3.123ms') to seconds
    def convert_time_to_seconds(time_str):
        try:
            if 'min' in time_str:
                # This is a simplification; a more robust solution would handle minutes and seconds
                return float(time_str.replace('min', '')) * 60
            elif 'ms' in time_str:
                return float(time_str.replace('ms', '')) / 1000.0
            else:
                return float(time_str.replace('s', ''))
        except (ValueError, AttributeError):
            return 0.0 # Return 0 if conversion fails

    df['time_sec'] = df['time_str'].apply(convert_time_to_seconds)
    
    # Now for the magic: group by service name and sum the boot times across all servers
    # We are calculating the *total* impact of each service on the fleet's boot time.
    aggregated_data = df.groupby('service')['time_sec'].sum().sort_values(ascending=False)
    
    return aggregated_data

# In your main() function, replace the print loop with:
# ...
#    if not all_data:
#        print("No data collected. Exiting.")
#        return
#
#    aggregated_results = parse_and_aggregate_data(all_data)
#    print("\nTop 15 Slowest Services Across Fleet (Total Time):")
#    print(aggregated_results.head(15))

Pro Tip: Another valid approach is to find the *average* boot time instead of the sum (`.mean()` instead of `.sum()`). I prefer the sum because it highlights services that might be okay on one server but are a widespread, moderate problem, which adds up to a significant fleet-wide delay.

Step 5: Visualization with Matplotlib

A table of numbers is good, but a chart is better. It immediately draws your eye to the biggest problems. Let’s use `matplotlib` to create a simple bar chart of the top 15 worst offenders and save it to a file.

Add this final function and call it from `main()`.

def create_plot(data, top_n=15):
    """Generates and saves a bar chart of the top N slowest services."""
    top_data = data.head(top_n)
    
    plt.figure(figsize=(12, 8))
    top_data.sort_values(ascending=True).plot(kind='barh') # Horizontal bar chart is easier to read
    
    plt.xlabel('Total Time (Seconds) Across Fleet')
    plt.ylabel('Systemd Service')
    plt.title(f'Top {top_n} Slowest Boot Services Across Fleet')
    plt.tight_layout() # Adjusts plot to ensure everything fits without overlapping
    
    output_filename = 'boot_performance_report.png'
    plt.savefig(output_filename)
    print(f"\nChart saved to {output_filename}")

# Finally, in your main() function, add the call:
# ...
#    aggregated_results = parse_and_aggregate_data(all_data)
#    print("\nTop 15 Slowest Services Across Fleet (Total Time):")
#    print(aggregated_results.head(15))
#    create_plot(aggregated_results)

Step 6: Automation

The whole point of this is to save time, so let’s automate it. We can run this script weekly using a simple cron job. This command will run the script every Monday at 2 AM.

0 2 * * 1 python3 script.py

You can then have another process pick up the generated `boot_performance_report.png` and email it to the team or post it to a Slack channel. Now you have a weekly, automated health report on your fleet’s boot performance.

Common Pitfalls

Here are a few places where I’ve stumbled in the past, so you can avoid them:

• SSH Key Permissions: If you’re using SSH keys, make sure the private key file on your central server has strict permissions (e.g., `600`). SSH will refuse to use a key that is too openly accessible.
• Firewall Rules: The most common issue is a network firewall blocking port 22 from your central server to the fleet. Always check connectivity to one server manually first.
• Varying `systemd-analyze` Output: Different Linux distributions or even different versions of systemd might have slightly different output formats. The parsing logic I provided is fairly standard, but you might need to tweak the regular expression if you have a very diverse fleet.
• Paramiko Timeouts: If a server is offline or unresponsive, the script can hang. The `timeout=10` parameter in the `ssh.connect()` call is crucial for preventing the entire script from stalling on one bad host.

Conclusion

And that’s it. We’ve gone from a mountain of disorganized text logs to a clean, actionable bar chart that visualizes fleet-wide performance. This isn’t just a technical exercise; it’s about reclaiming your time. By automating this kind of analysis, you shift from being a reactive firefighter to a proactive engineer who spots trends before they become critical issues. Now, when someone asks “why was the reboot slow last night?”, you’ll already have the answer.

---

🤖 Frequently Asked Questions