Track Backup Restic Repository Health and Snapshot stats

Hey there, Darian here. Let’s talk about something that used to keep me up at night: silent backup failures. For a while, I had my Restic backups running smoothly, but the only way I knew they were *actually* working was by SSH’ing into a server and manually running `restic check` and `restic snapshots`. It was a tedious, repetitive task that I’d often forget. After a near-miss where a repository had some index issues I didn’t catch for three days, I knew I had to automate this. I built a simple Python script to do the heavy lifting, and it’s saved me hours of manual checks and given me immense peace of mind. Today, I’m going to walk you through setting it up.

Prerequisites

• A working Restic repository (local or remote, like S3, B2, etc.).
• Python 3 installed on a machine that can access the repository.
• Restic binary installed and available in the system’s PATH.
• A webhook URL for your notification service (I’m using Slack here, but the logic can be adapted for Teams, Discord, etc.).

The Guide: Step-by-Step

Step 1: Project Setup and Configuration

First, you’ll want to set up a dedicated directory for this project. I’m not going to walk you through the basic shell commands for creating a directory or a Python virtual environment; you’ve got your own workflow for that. The key is to keep things organized.

Once your environment is active, you’ll need a few Python packages. You can install them using pip: `python-dotenv` for managing our credentials, `requests` for sending the web-hook notification, and `tabulate` to make our data look clean.

Next, create a file named config.env in your project directory. This is where we’ll store our secrets so they aren’t hardcoded in the script. It’s much cleaner and more secure.

# Restic Repository Details
RESTIC_REPOSITORY="s3:s3.amazonaws.com/your-bucket-name/your-repo-path"
RESTIC_PASSWORD="your-super-secret-password"

# Notification Webhook
SLACK_WEBHOOK_URL="https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX"

Pro Tip: In my production setups, I manage these secrets using a tool like HashiCorp Vault or AWS Secrets Manager and inject them into the environment at runtime. For this tutorial, a `config.env` file is perfectly fine to get started.

Step 2: The Python Script – The Brains of the Operation

Now, let’s create our Python script. I’ll call mine `monitor_restic.py`. We’ll build this piece by piece, and I’ll explain the logic as we go.

First, we need to import our libraries and load the environment variables from our `config.env` file. We’ll also define a helper function to run Restic commands. This keeps our code DRY (Don’t Repeat Yourself).

import os
import subprocess
import json
from datetime import datetime
import requests
from dotenv import load_dotenv
from tabulate import tabulate

# Load environment variables from config.env
load_dotenv('config.env')

# --- Helper Function to run Restic commands ---
def run_restic_command(args):
    """A wrapper to execute Restic commands and capture output."""
    try:
        # We add '--json' to commands that support it for easy parsing.
        # The 'check' and 'stats' commands are good candidates.
        base_command = ['restic']
        command = base_command + args
        
        # Using subprocess.run is the modern way to do this.
        result = subprocess.run(
            command,
            capture_output=True,
            text=True,
            check=True,  # This will raise an exception on non-zero exit codes
            encoding='utf-8'
        )
        
        # If the command supports JSON output, parse it. Otherwise, return raw text.
        if '--json' in args:
            return json.loads(result.stdout)
        return result.stdout

    except FileNotFoundError:
        print("Error: 'restic' command not found. Is it in your PATH?")
        return None
    except subprocess.CalledProcessError as e:
        print(f"Error executing Restic command: {' '.join(args)}")
        print(f"Stderr: {e.stderr}")
        return None

This helper function is robust. It captures output, handles errors, and automatically parses JSON for us, which will be incredibly useful in the next steps.

Step 3: Gathering Restic Data

Now let’s create functions to get the specific data we need: the health check, repository stats, and a list of the latest snapshots.

def get_repository_health():
    """Runs 'restic check' and returns a boolean for health status."""
    print("Running repository health check...")
    output = run_restic_command(['check'])
    if output and "no errors were found" in output:
        return True
    return False

def get_snapshot_stats():
    """Runs 'restic stats' in JSON mode to get total size and file count."""
    print("Fetching repository stats...")
    stats = run_restic_command(['stats', '--json'])
    return stats

def get_latest_snapshots(count=5):
    """Gets the latest 'count' snapshots in JSON format."""
    print(f"Fetching latest {count} snapshots...")
    snapshots = run_restic_command(['snapshots', '--latest', str(count), '--json'])
    return snapshots

By separating these into functions, our code becomes modular and easy to read. Each function has one clear job.

Step 4: Formatting and Sending the Notification

Raw data is useful, but a well-formatted notification is what provides real value. We’ll create a function to build a Slack message using their Block Kit format for a nice, clean look. The `tabulate` library will help us create a professional-looking table for our snapshots.

def format_and_send_notification(health_ok, stats, snapshots):
    """Formats the data into a Slack message and sends it."""
    webhook_url = os.getenv('SLACK_WEBHOOK_URL')
    if not webhook_url:
        print("Error: SLACK_WEBHOOK_URL not set in config.env")
        return

    # Determine status and color for the message
    status_icon = ":white_check_mark:" if health_ok else ":x:"
    status_text = "Healthy" if health_ok else "Check Required!"
    color = "#36a64f" if health_ok else "#d50000"

    # Format stats
    total_size_gb = stats.get('total_size', 0) / (1024**3)
    total_files = stats.get('total_file_count', 0)
    stats_summary = f"*Total Size:* {total_size_gb:.2f} GB\n*Total Files:* {total_files:,}"

    # Format latest snapshots into a table
    snapshot_table = []
    headers = ["Timestamp", "Host", "Tags"]
    if snapshots:
        for snap in snapshots:
            # Parse timestamp and make it human-readable
            ts = datetime.fromisoformat(snap['time'].replace('Z', '+00:00'))
            snapshot_table.append([
                ts.strftime('%Y-%m-%d %H:%M:%S'),
                snap.get('hostname', 'N/A'),
                ', '.join(snap.get('tags', []))
            ])
    
    table_str = tabulate(snapshot_table, headers=headers, tablefmt="pipe")

    # Build the Slack Block Kit payload
    payload = {
        "attachments": [
            {
                "color": color,
                "blocks": [
                    {
                        "type": "header",
                        "text": {
                            "type": "plain_text",
                            "text": f"{status_icon} Restic Repository Report"
                        }
                    },
                    {
                        "type": "section",
                        "fields": [
                            {"type": "mrkdwn", "text": f"*Status:*\n{status_text}"},
                            {"type": "mrkdwn", "text": f"*Repository:*\n`{os.getenv('RESTIC_REPOSITORY')}`"}
                        ]
                    },
                    {"type": "divider"},
                    {
                        "type": "section",
                        "text": {"type": "mrkdwn", "text": "*Repository Stats*"}
                    },
                    {
                        "type": "section",
                        "text": {"type": "mrkdwn", "text": stats_summary}
                    },
                    {"type": "divider"},
                    {
                        "type": "section",
                        "text": {"type": "mrkdwn", "text": "*Latest Snapshots*"}
                    },
                    {
                        "type": "section",
                        "text": {
                            "type": "mrkdwn",
                            "text": f"```\n{table_str}\n```"
                        }
                    },
                    {
                        "type": "context",
                        "elements": [
                            {
                                "type": "mrkdwn",
                                "text": f"Report generated on {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}"
                            }
                        ]
                    }
                ]
            }
        ]
    }

    # Send the request
    try:
        print("Sending notification to Slack...")
        response = requests.post(webhook_url, json=payload)
        response.raise_for_status()  # Raises an exception for 4XX/5XX responses
        print("Notification sent successfully!")
    except requests.exceptions.RequestException as e:
        print(f"Error sending Slack notification: {e}")

Pro Tip: Using Slack’s Block Kit instead of a simple text message allows for much richer notifications. You can add colors, headers, dividers, and formatted text, which makes the report significantly easier to read at a glance.

Step 5: Putting It All Together

Finally, we’ll create the main execution block that calls our functions in the correct order. This is standard practice in Python to ensure the script only runs the logic when executed directly.

if __name__ == "__main__":
    health_ok = get_repository_health()
    
    # We proceed even if the health check fails, as we still want the report.
    # The notification will clearly show the failure.
    stats = get_snapshot_stats()
    snapshots = get_latest_snapshots()
    
    if stats and snapshots is not None:
        format_and_send_notification(health_ok, stats, snapshots)
    else:
        print("Failed to retrieve repository data. Aborting notification.")

Step 6: Automate with a Cron Job

The final step is to automate the script. A simple cron job is perfect for this. You can schedule it to run daily or weekly depending on your needs. For a weekly report every Monday at 2 AM, the cron entry would look like this:

0 2 * * 1 python3 script.py

Remember to run this from the directory containing your script and `config.env` file, or use absolute paths in your cron job (while avoiding the forbidden paths in the rules).

Common Pitfalls

Here are a few places where I’ve tripped up in the past:

• Incorrect Credentials: Double-check your `RESTIC_REPOSITORY` and `RESTIC_PASSWORD` in the `config.env` file. A typo here is the most common source of failure.
• Permissions: The user running the script (or the cron job) needs read permissions for the script, the `config.env` file, and network access to both the Restic repository and the Slack webhook URL.
• PATH Issues: If cron jobs run with a minimal environment, they might not find the `restic` binary. It’s sometimes safer to provide the full path to `restic` inside the script, but I’ve avoided that here for simplicity. Just ensure `restic` is in the cron user’s PATH.
• Invalid Webhook: Make sure your Slack webhook URL is correct and active. These can be revoked, which will cause the notification step to fail.

Conclusion

And that’s it! You now have a powerful, automated monitoring system for your Restic backups. It checks for corruption, gives you a summary of repository size, and lists the latest snapshots to confirm backups are running. This simple script turns a manual, error-prone task into a reliable, automated process that delivers a clear report right to your team’s chat. It’s a small investment of time that pays huge dividends in reliability and peace of mind.

---

🤖 Frequently Asked Questions