Monitor MongoDB Replica Set Status and Failover Events

Alright, team. Darian Vance here. Let’s talk about something that used to eat up way too much of my time: manually checking the health of our MongoDB replica sets. I’d SSH into a box, run rs.status() in the mongo shell, and try to parse that dense JSON output to make sure everything was okay. After a failover scare one weekend where the primary flipped and no one noticed for an hour, I knew this reactive approach wasn’t going to cut it.

So, I automated it. This guide will walk you through the Python script I now use across our production environments. It not only checks the status of each member but also explicitly detects and logs failover events. It’s about getting ahead of the problem, and trust me, it’s a huge peace-of-mind upgrade.

Prerequisites

• Python 3.6 or higher installed.
• Access to a running MongoDB replica set.
• Basic familiarity with Python and MongoDB concepts.
• The MongoDB user for this script needs the clusterMonitor role.

The Step-by-Step Guide

Step 1: Setting Up Your Project

First, get your project directory organized. You’ll need two files to start: our Python script and a configuration file. I’ll skip the standard virtualenv setup since you likely have your own workflow for that. Just make sure you have the necessary Python libraries installed. You can do this by running a command like pip install pymongo python-dotenv in your terminal.

Our file structure will look like this:

• monitor_mongo.py: This is where our monitoring logic will live.
• config.env: This file will store our database connection string securely.
• mongo_primary.state: This file will be created by the script to track the current primary node.

Step 2: The Configuration File

Let’s create the config.env file. This keeps our credentials out of the source code, which is a critical security practice.

# config.env
MONGO_URI="mongodb://your_user:your_password@host1:27017,host2:27017/?replicaSet=yourRepSetName"

Pro Tip: Your MongoDB connection string for a replica set must include the ?replicaSet=yourRepSetName parameter. The PyMongo driver uses this to discover all members of the set, even if you only list one or two hosts. This is the most common point of failure I see when people are setting this up for the first time.

Step 3: The Monitoring Script (`monitor_mongo.py`)

Now for the core logic. This script will connect to the replica set, fetch its status, check for a valid primary, and log the state of each member. I’ve added plenty of comments to explain what each part does.

import os
import logging
from pymongo import MongoClient
from pymongo.errors import ConnectionFailure
from dotenv import load_dotenv

# --- Basic Configuration ---
load_dotenv('config.env')
MONGO_URI = os.getenv("MONGO_URI")
STATE_FILE = "mongo_primary.state"

# --- Setup Logging ---
logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(levelname)s - %(message)s',
    handlers=[
        logging.FileHandler("mongo_monitor.log"),
        logging.StreamHandler()
    ]
)

def get_previous_primary():
    """Reads the last known primary from the state file."""
    try:
        if os.path.exists(STATE_FILE):
            with open(STATE_FILE, 'r') as f:
                return f.read().strip()
    except IOError as e:
        logging.error(f"Could not read state file: {e}")
    return None

def set_current_primary(primary_host):
    """Writes the current primary to the state file."""
    try:
        with open(STATE_FILE, 'w') as f:
            f.write(primary_host)
    except IOError as e:
        logging.error(f"Could not write to state file: {e}")

def check_replica_set_status():
    """Connects to MongoDB, checks replica set status, and logs failover events."""
    if not MONGO_URI:
        logging.critical("MONGO_URI not found in config.env. Cannot proceed.")
        return

    try:
        logging.info("Connecting to MongoDB replica set...")
        client = MongoClient(MONGO_URI, serverSelectionTimeoutMS=5000)
        # The ismaster command is cheap and does not require auth.
        client.admin.command('ismaster')
        logging.info("Connection successful.")
    except ConnectionFailure as e:
        logging.critical(f"MongoDB connection failed: {e}")
        return

    try:
        # Get replica set status
        status = client.admin.command('replSetGetStatus')
        members = status.get('members', [])
        
        current_primary = None
        primary_count = 0
        
        # Find the primary and log member states
        for member in members:
            logging.info(f"Member: {member['name']} - State: {member['stateStr']} - Health: {'UP' if member['health'] == 1 else 'DOWN'}")
            if member['stateStr'] == 'PRIMARY':
                primary_count += 1
                current_primary = member['name']

        # Validate the replica set state
        if primary_count == 0:
            logging.critical("CRITICAL: No PRIMARY member found in the replica set!")
        elif primary_count > 1:
            logging.critical("CRITICAL: Multiple PRIMARY members found! The replica set is in a broken state.")
        else:
            logging.info(f"Found PRIMARY member: {current_primary}")
            previous_primary = get_previous_primary()
            
            if previous_primary and previous_primary != current_primary:
                logging.warning(f"FAILOVER DETECTED: Primary changed from {previous_primary} to {current_primary}")
            elif not previous_primary:
                logging.info("First run, setting initial primary state.")

            set_current_primary(current_primary)

    except Exception as e:
        logging.error(f"An error occurred while checking replica set status: {e}")
    finally:
        client.close()
        logging.info("Connection closed.")

if __name__ == "__main__":
    check_replica_set_status()

Step 4: Scheduling with Cron

This script is most useful when it runs automatically. On my Linux setups, I use a simple cron job. You don’t want to run it too frequently to avoid unnecessary load, but often enough to catch issues quickly. Every 5 or 10 minutes is a reasonable starting point.

Here is an example cron entry to run the script every 10 minutes. Note that I am not using absolute paths, as your environment may differ. You’ll want to ensure the script is run from the correct directory so it can find the config.env file.

*/10 * * * * python3 monitor_mongo.py

Pro Tip: For true production-grade monitoring, you’ll want to forward these logs to a centralized logging system like an ELK stack or Splunk. Even better, extend the script to send alerts directly to a service like PagerDuty or a Slack channel when a `CRITICAL` or `WARNING` event is logged. The Python `logging` library is highly extensible and can be configured with custom handlers for this purpose.

Common Pitfalls (Where I Usually Mess Up)

• The Connection String: This is the number one culprit. A typo in a hostname, wrong credentials, or forgetting the ?replicaSet=... parameter will cause an immediate `ConnectionFailure`. Double- and triple-check it.
• Firewall and Network ACLs: The machine running this script must be able to reach all members of the replica set on the MongoDB port (typically 27017). A common mistake is to only allow access to the primary, but the driver needs to talk to all nodes to determine the set’s health.
• Insufficient Permissions: The MongoDB user specified in your URI needs permissions to run the replSetGetStatus command. The built-in clusterMonitor role is perfect for this. Don’t use a highly privileged account; always follow the principle of least privilege.

Conclusion

And there you have it. A robust, automated way to keep an eye on your MongoDB replica sets. This simple script forms the foundation of a solid monitoring strategy. It provides clear logs, tracks the most critical state changes (failovers), and gives you the data you need to act quickly when things go wrong. It’s a small investment that pays huge dividends in system stability and, more importantly, your own peace of mind. Now you can get back to building, knowing your database is being watched.

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions