Migrate MongoDB Atlas Collections to Self-Hosted MongoDB

Hey everyone, Darian Vance here from TechResolve. A while back, our team was working on a project that required a local, air-gapped analytics environment. The data lived in MongoDB Atlas, but we needed it on-premise. The off-the-shelf migration tools felt like overkill and introduced complexities we didn’t need.

So, I did what any of us would do: I wrote a Python script. It gave us full control, was easy to debug, and got the job done in an afternoon. I’ve since polished this script for various projects, and it’s become my go-to for moving collections. It saves time and avoids the headache of vendor-specific tooling. Let’s walk through how you can build it.

Prerequisites

Before we jump in, make sure you have the following ready to go:

• Your MongoDB Atlas cluster connection string (with appropriate read permissions).
• Your self-hosted MongoDB instance connection string (with read/write permissions).
• A Python 3 environment. I’ll skip the standard virtualenv setup since you likely have your own workflow for that. Let’s jump straight to the logic.
• A couple of Python libraries. You’ll need to install pymongo for database interaction and python-dotenv for managing our credentials. You can add these to your project using your standard package manager.

The Step-by-Step Guide

Step 1: Project Setup and Configuration

First, let’s organize our work. In your project directory, create two files: `migrate_collections.py` for our script and `config.env` to securely store our connection strings. Using a config file like this keeps sensitive credentials out of our source code.

In your `config.env` file, add your connection strings. It should look like this:

ATLAS_URI="mongodb+srv://your_user:your_password@your_atlas_cluster..."
LOCAL_URI="mongodb://your_user:your_password@localhost:27017/"

Pro Tip: In my production setups, I never commit files like `config.env` to version control. I use a `.gitignore` entry for them and manage the production secrets through a secure vault or environment variables provided by the CI/CD system. It’s a critical security practice.

Step 2: The Python Script – Establishing Connections

Now, let’s open `migrate_collections.py`. We’ll start by importing the necessary libraries and loading the environment variables from our `config.env` file. This code sets the stage by creating client connections to both our source (Atlas) and destination (self-hosted) databases.

import os
from dotenv import load_dotenv
from pymongo import MongoClient
from pymongo.errors import ConnectionFailure

def migrate_collections(db_name, collections_to_migrate):
    """
    Connects to source and destination MongoDB and migrates specified collections.
    """
    load_dotenv('config.env')

    atlas_uri = os.getenv('ATLAS_URI')
    local_uri = os.getenv('LOCAL_URI')

    if not atlas_uri or not local_uri:
        print("Error: ATLAS_URI or LOCAL_URI not found in config.env")
        return

    print("Connecting to source (Atlas) and destination (Local) databases...")

    try:
        source_client = MongoClient(atlas_uri)
        dest_client = MongoClient(local_uri)
        
        # The ismaster command is cheap and does not require auth.
        source_client.admin.command('ismaster')
        dest_client.admin.command('ismaster')
        
        print("Successfully connected to both databases.")
    except ConnectionFailure as e:
        print(f"Connection failed: {e}")
        return

    source_db = source_client[db_name]
    dest_db = dest_client[db_name]

    # The migration logic will go here in the next step
    print("Starting migration process...")

    # Close connections
    source_client.close()
    dest_client.close()
    print("Migration complete. Connections closed.")

if __name__ == "__main__":
    DATABASE_NAME = "your_database_name"
    COLLECTIONS = ["users", "products", "orders"] # Add your collection names here
    migrate_collections(DATABASE_NAME, COLLECTIONS)

This boilerplate handles loading our secrets, establishing the client connections, and includes some basic error handling if a connection fails. Notice I’m verifying the connection with a simple command before proceeding.

Step 3: The Core Migration Logic

This is where the magic happens. We’ll iterate through our list of collections, pull all the documents from the source, and insert them into the destination. To ensure a clean slate, I’m including a step to delete existing documents in the destination collection before inserting the new ones.

Let’s add this logic inside our `migrate_collections` function, right after we define `source_db` and `dest_db`.

    # (Inside the migrate_collections function)
    source_db = source_client[db_name]
    dest_db = dest_client[db_name]

    for collection_name in collections_to_migrate:
        print(f"\nProcessing collection: {collection_name}")
        
        source_collection = source_db[collection_name]
        dest_collection = dest_db[collection_name]

        # Fetch all documents from the source collection
        print(f"  -> Fetching documents from source...")
        documents = list(source_collection.find({}))
        
        if not documents:
            print(f"  -> No documents found in source collection '{collection_name}'. Skipping.")
            continue
            
        print(f"  -> Found {len(documents)} documents.")

        # Clear the destination collection before inserting
        print(f"  -> Clearing destination collection '{collection_name}'...")
        dest_collection.delete_many({})

        # Insert documents into the destination collection
        print(f"  -> Inserting documents into destination...")
        result = dest_collection.insert_many(documents)
        print(f"  -> Successfully inserted {len(result.inserted_ids)} documents.")

    # (The rest of the function remains the same)

We’re using `find({})` to get all documents, `delete_many({})` to wipe the destination collection, and `insert_many()` for a bulk write, which is much more efficient than inserting one document at a time.

Pro Tip: For massive collections (think millions of documents), loading everything into a list with `list(source_collection.find({}))` can consume a lot of memory. A more robust approach for production is to process the data in batches. You can iterate through the cursor returned by `find()` and insert documents in chunks of, say, 10,000 at a time. This keeps the memory footprint nice and low.

Here’s Where I Usually Mess Up (Common Pitfalls)

1. **Firewall and IP Access Rules:** This is the #1 cause of connection failures. Remember that your machine running the script needs network access to *both* the Atlas cluster and your self-hosted instance. In Atlas, make sure the IP of the machine running the script is on the IP Access List.
2. **Forgetting to Migrate Indexes:** This script only moves data. It does *not* migrate indexes, user roles, or validation rules. After you run the script, your queries on the self-hosted instance might be incredibly slow. You’ll need to re-create the indexes manually or script it using PyMongo’s `index_information()` and `create_index()` methods.
3. **Typos in Connection Strings:** It sounds simple, but a wrong password, username, or hostname in the `config.env` file will stop you in your tracks. Double-check them!

Conclusion

And there you have it. A clean, controllable, and understandable Python script for migrating MongoDB collections. This approach gives you a solid foundation you can build on. You could extend it to handle index creation, add more sophisticated error logging, or even schedule it to run periodically for data synchronization tasks using a simple cron job like `0 2 * * 1 python3 migrate_collections.py`.

Hopefully, this walkthrough saves you some time and gives you a powerful tool for your DevOps toolkit.

– Darian Vance

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions