Checking NTP Time Drift across Distributed Servers

Hey everyone, Darian Vance here. Let’s talk about time. Specifically, why a chunk of my morning used to be wasted SSH’ing into a dozen servers, one by one, to check their NTP sync status. It’s one of those “death by a thousand cuts” tasks that feels productive but is really just a time sink.

If a server’s clock drifts by even a few seconds, it can wreak havoc on time-sensitive logs, Kerberos authentication, and complex distributed transactions. I finally got fed up with the manual checks and wrote a simple script to automate the whole process. This little bit of Python saved me a couple of hours a week and gives me a daily report before I’ve even had my first coffee. Today, I’m going to walk you through it.

Prerequisites

Before we dive in, make sure you have the following ready:

• Python 3 installed on your local machine or a dedicated management server.
• SSH key-based access to all the target servers you want to monitor. Using password authentication is possible but much less secure and harder to automate.
• A list of your server hostnames or IP addresses.
• Network access (typically port 22) from your management machine to the target servers.

The Guide: Step-by-Step

Step 1: Your Project Environment

First, get your project folder ready. I’ll skip the standard virtualenv setup since you likely have your own workflow for that. The only external Python library we’ll need is Paramiko, which is my go-to for handling SSH connections in Python. In your activated virtual environment, you can typically add it by running pip install paramiko. This command handles the installation for you.

Step 2: Create the Server List

This is as simple as it gets. In your project directory, create a plain text file named servers.txt. List each server’s IP address or fully qualified domain name (FQDN) on a new line. For example:

web-prod-01.techresolve.corp
web-prod-02.techresolve.corp
db-primary-01.techresolve.corp
api-worker-01.techresolve.corp

Step 3: The Python Monitoring Script

Alright, this is the core of our solution. We’ll create a script named check_ntp_drift.py. The logic is straightforward:

1. Read the list of servers from servers.txt.
2. Loop through each server.
3. Establish an SSH connection using Paramiko.
4. Execute the ntpq -p command, which gives us a detailed report of NTP peers.
5. Parse the output to find the active time source (usually marked with an asterisk) and check its “offset”. The offset is the measured time difference in milliseconds.
6. Flag any server where the absolute offset is greater than our defined threshold.

Pro Tip: In my production setups, I set the drift threshold to around 50 milliseconds. For most web applications and databases, anything under 100ms is generally acceptable, but tighter is always better. If you’re running something extremely sensitive like a high-frequency trading platform, you’d want this value to be much, much lower.

Here is the complete script. Just save it as check_ntp_drift.py in the same directory as your server list.

import paramiko
import os

# --- Configuration ---
SERVER_LIST_FILE = 'servers.txt'
SSH_USER = 'your_ssh_user'  # Use a dedicated read-only user for monitoring
SSH_KEY_FILE = os.path.expanduser('~/.ssh/id_rsa_monitoring')
DRIFT_THRESHOLD_MS = 50.0  # Alert if drift is over 50 milliseconds

def check_server_drift(hostname):
    """Connects to a server, runs 'ntpq -p', and checks the time drift."""
    print(f"--- Checking {hostname} ---")
    client = paramiko.SSHClient()
    client.set_missing_host_key_policy(paramiko.AutoAddPolicy())

    try:
        client.connect(hostname, username=SSH_USER, key_filename=SSH_KEY_FILE, timeout=10)
        
        stdin, stdout, stderr = client.exec_command('ntpq -p')
        output = stdout.read().decode()
        error = stderr.read().decode()

        if error:
            print(f"Error executing command: {error.strip()}")
            return

        active_peer_found = False
        for line in output.splitlines():
            # The active peer is marked with a '*' at the start of the line
            if line.startswith('*'):
                active_peer_found = True
                parts = line.split()
                # The offset is typically the second to last column
                offset_str = parts[-2]
                
                try:
                    offset = float(offset_str)
                    print(f"Active peer found. Offset: {offset} ms")
                    
                    if abs(offset) > DRIFT_THRESHOLD_MS:
                        print(f"ALERT! Time drift on {hostname} is {offset} ms, which exceeds the threshold of {DRIFT_THRESHOLD_MS} ms!")
                    else:
                        print(f"OK. Time drift is within acceptable limits.")
                except ValueError:
                    print(f"Could not parse offset value: '{offset_str}'")
                break # We only care about the primary active peer
        
        if not active_peer_found:
            print("WARNING: No active NTP peer ('*') found. The server may not be syncing correctly.")

    except Exception as e:
        print(f"Failed to connect or execute on {hostname}: {e}")
    finally:
        client.close()

def main():
    """Main function to read servers from file and check each one."""
    try:
        with open(SERVER_LIST_FILE, 'r') as f:
            servers = [line.strip() for line in f if line.strip()]
    except FileNotFoundError:
        print(f"Error: The server list file '{SERVER_LIST_FILE}' was not found.")
        return

    if not servers:
        print("Server list is empty. Exiting.")
        return

    print("Starting NTP Drift Check across all servers...")
    for server in servers:
        check_server_drift(server)
        print("-" * 25)

if __name__ == "__main__":
    main()

Step 4: Scheduling the Check

Running this on-demand is useful, but the real value comes from automation. I have this running as a cron job every morning at 2 AM. This gives me a fresh report to review when I start my day. A simple cron entry looks like this:

0 2 * * * python3 check_ntp_drift.py > /path/to/your/logs/ntp_check.log 2>&1

Just make sure to replace the paths with your own. This will run the script and save the output to a log file for later review.

Common Pitfalls

Here are a few places where I usually mess things up the first time I set this up on a new system:

• SSH Key Problems: This is number one. Paramiko can be particular about key permissions and formats. Make sure the SSH private key file specified in the script (`~/.ssh/id_rsa_monitoring` in the example) is readable by your user and that the corresponding public key is in the `authorized_keys` file on all target servers for the `your_ssh_user` account.
• Firewall Rules: If the script hangs on “Connecting…”, it’s almost always a firewall blocking port 22. Double-check any network ACLs or local firewalls (like ufw or firewalld) on the target machines.
• Parsing `ntpq` Output: The script assumes a standard output format from `ntpq -p`. While it’s fairly consistent across modern Linux distributions, a very old or unusual OS might format it differently, causing the script to fail. If you run into issues, a good first step is to manually run `ntpq -p` on the problematic server and compare its output to what the script expects.

Conclusion

And that’s really all there is to it. With a simple text file and a Python script, you’ve replaced a tedious, error-prone manual task with a reliable, automated check. This ensures a foundational aspect of your distributed system—accurate timekeeping—is consistently monitored. It’s a small investment of time that pays huge dividends in stability and peace of mind. Now you can focus on bigger, more interesting problems.

– Darian Vance

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions