🚀 Executive Summary
TL;DR: Config drift between network devices and NetBox can lead to critical outages by making the ‘single source of truth’ unreliable. This guide offers three solutions: a quick audit script, a robust Git-driven workflow making Git the definitive source, and an advanced event-driven watcher to detect and alert on manual changes.
🎯 Key Takeaways
- The ‘Audit & Sync’ script provides a quick, one-way sync using NAPALM and pynetbox to pull running configs from devices and store them in NetBox custom fields for baselining.
- A Git-driven workflow establishes Git as the true source of truth, leveraging CI/CD pipelines to render configurations from NetBox data and push them to devices, effectively eliminating config drift.
- The ‘Event-Driven Watcher’ uses tools like Oxidized or LibreNMS to detect manual configuration changes on devices, triggering webhooks to alert engineers and update NetBox status to ‘Stale’ for reconciliation.
Tired of config drift between your network devices and your single source of truth? This guide breaks down three real-world methods, from quick scripts to robust GitOps workflows, to finally get your network device configurations synced with NetBox.
So, Your Network Gear and NetBox Are Lying to Each Other? Let’s Fix That.
I remember it like it was yesterday. 2 AM, the on-call phone screaming, and half of our services for a major client were dark. The culprit? A junior engineer, bless his heart, had pushed a “routine” firewall change using an Ansible playbook that pulled its variables from NetBox. The problem was, someone—a senior network admin, it turned out—had made a “quick manual fix” directly on the core router an hour earlier during a different incident and hadn’t updated our source of truth. The playbook dutifully overwrote his emergency fix, and everything went sideways. NetBox said one thing, the router said another, and our automation turned that discrepancy into a production outage. That’s not a NetBox problem; that’s a workflow problem. And it’s one we need to solve.
The “Why”: The Myth of the Single Source of Truth
We preach “NetBox is the single source of truth” until we’re blue in the face. And it should be. But a source of truth is only useful if it’s actually, you know, true. The root of this problem isn’t the tool; it’s the reality of network operations. People make manual changes. Emergencies happen. Config drift is as inevitable as gravity. Your source of truth becomes a “source of lies” the moment its state deviates from the actual production state. The goal isn’t just to have a database; it’s to have a reliable, closed-loop system where reality and documentation are one and the same.
So how do we bridge that gap? Let’s look at a few ways, from the quick-and-dirty to the enterprise-grade.
Solution 1: The ‘Audit & Sync’ Script (The Quick Fix)
This is my go-to when I’m parachuted into a new environment and need to quickly assess the gap between documentation and reality. It’s a one-way sync from the device to NetBox. It’s manual, it’s a bit clunky, but it gets the job done for an audit.
The idea is simple: write a Python script using a library like NAPALM to connect to the device and pull its running config, then use the pynetbox library to push that data into a custom field in NetBox. It’s not elegant, but it works.
Here’s a conceptual snippet of what that might look like:
import pynetbox
from napalm import get_network_driver
# --- Device & NetBox Details ---
NETBOX_URL = 'https://netbox-prod.techresolve.io'
NETBOX_TOKEN = 'YOUR_SUPER_SECRET_TOKEN'
DEVICE_IP = '10.20.30.1'
DEVICE_TYPE = 'junos' # or 'ios', 'eos', etc.
DEVICE_USER = 'darian'
DEVICE_PASS = 'password123' # Use a vault in real life!
# --- Connect to NetBox ---
nb = pynetbox.api(url=NETBOX_URL, token=NETBOX_TOKEN)
device_in_netbox = nb.dcim.devices.get(name='core-switch-01.phx-dc')
# --- Connect to the Device and get config ---
driver = get_network_driver(DEVICE_TYPE)
with driver(DEVICE_IP, DEVICE_USER, DEVICE_PASS) as device:
running_config = device.get_config()['running']
# --- Shove it into a custom field in NetBox ---
device_in_netbox.custom_fields['running_config_backup'] = running_config
if device_in_netbox.save():
print(f"Successfully backed up config for {device_in_netbox.name} to NetBox!")
else:
print(f"ERROR: Failed to save config for {device_in_netbox.name}.")
Warning: The Hacky Reality
This is a bandage, not a cure. Storing a full, multi-line config in a single custom field is messy. It doesn’t version control the changes, and it doesn’t prevent future drift. Use this to get a baseline, not to run your daily operations.
Solution 2: The Git-Driven Workflow (The Permanent Fix)
This is how we do it at TechResolve. This is the “right” way. We shift our thinking entirely. The network device is NOT the source of truth. Git is. NetBox holds the structured data (IPs, VLANs, interfaces), and a Git repository holds the declarative configuration files (Ansible templates, Jinja2 files, etc.).
The workflow looks like this:
- An engineer needs to make a change. They create a new branch in a Git repo.
- They modify a YAML file or a template that defines the desired state.
- They submit a Pull Request (PR).
- A CI/CD pipeline (GitLab CI, Jenkins) automatically triggers. It runs linters, validation tests, and a ‘dry-run’ of the configuration push (e.g., `ansible-playbook –check`).
- A senior engineer reviews and approves the PR.
- On merge to the `main` branch, the pipeline triggers for real. It uses Ansible (or a similar tool) to pull data from NetBox’s API, render the final configuration, and push it to the network device.
- As a final step, the pipeline can make a callback to NetBox to update a “last_provisioned” timestamp or config hash.
Here’s a breakdown of the pros and cons of this approach:
| Pros | Cons |
|---|---|
|
|
Solution 3: The Event-Driven Watcher (The ‘Nuclear’ Option)
Okay, so what if you need the safety of the GitOps model but the reality of your org is that manual changes are still going to happen? You can build a system that watches for them and yells at you.
This is a more advanced, event-driven architecture. It’s powerful, but it has a lot of moving parts.
- You use a tool like Oxidized or LibreNMS to continuously poll your network devices for configuration changes. These tools are great at this and already store versioned configs.
- When Oxidized detects a change, you configure it to fire a webhook.
- The webhook’s destination is a lightweight application—an AWS Lambda function, a simple Flask app, or even an AWX/Tower workflow listener.
- This application receives the webhook payload (which includes the device name and the config diff). It then does a few things:
- It sends a high-priority alert to a Slack channel (e.g., `#network-alerts`) with the diff, tagging the on-call engineer.
@on-call: Manual change detected on core-switch-01.phx-dc! - It uses the pynetbox API to update the device’s status in NetBox to “Stale” or “Needs Review”.
- In a very advanced setup, it could even attempt to create a new Git branch and commit the detected change automatically, creating a PR for a human to review and reconcile.
- It sends a high-priority alert to a Slack channel (e.g., `#network-alerts`) with the diff, tagging the on-call engineer.
Pro Tip: Start Small
This approach can get complicated fast. Don’t try to build the whole auto-remediation system at once. Start with the detection and alerting part (Oxidized -> Webhook -> Slack). That alone provides immense value by shining a light on unauthorized or undocumented changes. Once that’s stable, you can add the NetBox integration.
Ultimately, there’s no single magic bullet. The “Quick Fix” is great for getting a handle on a messy environment. The “Nuclear Option” is fantastic for high-compliance environments where every change must be tracked. But for most of us, building a robust, Git-driven workflow is the most sustainable path. It forces good habits, creates a resilient and auditable system, and finally lets NetBox be the source of truth you always wanted it to be.
🤖 Frequently Asked Questions
âť“ What is network configuration drift and why is it problematic for NetBox as a single source of truth?
Config drift occurs when manual changes on network devices deviate from NetBox’s recorded state, making NetBox a ‘source of lies.’ This discrepancy can lead to automation failures, unexpected behavior, and production outages.
âť“ How do the three proposed solutions for config sync differ in their approach and complexity?
The ‘Audit & Sync’ script is a manual, quick fix for baselining. The ‘Git-Driven Workflow’ is an enterprise-grade, permanent solution establishing Git as the source of truth via CI/CD. The ‘Event-Driven Watcher’ is an advanced, event-driven system that detects and alerts on unauthorized manual changes.
âť“ What is a significant challenge when adopting a Git-driven workflow for network configuration management?
A major challenge is the cultural shift required for network engineers to embrace Git and a DevOps mindset, alongside the high upfront cost of setting up Git repositories and robust CI/CD pipelines. It also fundamentally breaks the traditional ’emergency fix’ model of direct device access.
Leave a Reply