Tired of chasing phantom bugs caused by system clock drift? A Senior DevOps Engineer breaks down why your server clocks are out of sync and provides three real-world solutions to fix NTP issues for good.

Your Servers Are Lying to You: Fixing the “Crooked Monitors” of System Time

I remember a Tuesday from hell. A critical deployment failed right after lunch. Kerberos tickets were suddenly being rejected between our new application servers and the domain controller. We spent three hours tearing through firewall rules, checking service principal names, and suspecting a botched security update. The junior engineer on my team was about to roll back the entire release when I saw it. A five-minute and three-second time difference between web-app-12 and auth-kdc-01. That was it. A tiny, infuriating time skew. This is what I call a “crooked monitor” problem—everything technically ‘works,’ but something is just off-kilter enough to drive you insane and cause chaos downstream. It’s not a crash; it’s a subtle lie that corrupts everything.

The “Why”: What’s Really Going On With Your Clocks?

Let’s get one thing straight: the hardware clock inside your server is not a Swiss watch. It drifts. It’s imprecise. Over time, seconds are lost or gained. In a virtualized environment, this problem is even worse due to hypervisor scheduling and CPU state saving. The solution for decades has been the Network Time Protocol (NTP), a simple system where your servers periodically ask a more authoritative source, “Hey, what time is it, really?” and adjust themselves. The problem—the “crookedness”—creeps in when this process fails. Maybe the firewall is blocking UDP port 123, maybe your cloud provider’s default NTP sources are unreachable, or maybe the NTP service itself just isn’t configured or running correctly. The result is a server that is confidently, and quietly, wrong about the current time.

The Fixes: From Duct Tape to a Proper Foundation

Depending on how much trouble you’re in, there are a few ways to tackle this. Let’s walk through the options, from getting the system running again *now* to making sure this never bites you again.

Solution 1: The Quick Fix (The “Hammer Sync”)

You’re in the middle of an outage. Logs are impossible to correlate, and authentication is failing. You don’t have time for a root cause analysis; you just need the bleeding to stop. This is where you manually force a sync. It’s a blunt instrument, but it’s effective.

On most modern Linux systems using chrony, you can force an immediate step correction:

sudo chronyc makestep

This command tells the chrony daemon to immediately correct any offset, ignoring the usual gradual slew settings. It’s the digital equivalent of smacking the side of the TV to fix the picture.

Warning: This is a temporary fix, not a solution. It’s a bandage. If the underlying configuration is broken, the clock will just drift out of sync again. Use this to get your services back online, then immediately move on to Solution 2.

Solution 2: The Permanent Fix (The “Engineer’s Way”)

Okay, the fire is out. Now, let’s do our job properly. The goal here is to ensure the NTP service is configured correctly and resiliently. On RHEL/CentOS 8+ or Ubuntu 20.04+, that service is almost always chronyd.

First, check your configuration file at /etc/chrony/chrony.conf (or /etc/chrony.conf). It should look something like this, pointing to a pool of reliable servers. Using a pool means you’re not dependent on a single source.

# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
pool 2.pool.ntp.org iburst

# Record the rate at which the system clock gains/loses time.
driftfile /var/lib/chrony/drift

# Allow the system clock to be stepped in the first three updates
# if its offset is larger than 1 second.
makestep 1.0 3

# Enable kernel synchronization of the real-time clock (RTC).
rtcsync

# Log measurements, statistics, and tracking information.
logdir /var/log/chrony

After confirming your config, restart the service (sudo systemctl restart chronyd) and, most importantly, verify its status. Don’t just fire and forget.

Use chronyc sources to see who you’re talking to:

The ^* indicates your primary sync source. A ‘Reach’ of 377 is perfect—it means the last 8 attempts were successful. If you see 0, you’ve got a networking problem. This command tells you the real story and confirms your fix is actually working.

Solution 3: The ‘Nuclear’ Option (The “Architect’s Mandate”)

You’ve been burned by this more than once across your fleet. Individual fixes are no longer enough. It’s time to solve this at an architectural level. This is how you stop crooked monitors from ever appearing in your lab again.

The strategy has two parts:

1. Deploy Internal NTP Servers: Stop relying entirely on public internet time servers. Set up three dedicated instances (for redundancy) inside your VPC or data center. Configure them to sync from reliable external sources (like the NTP pool project), and then point all your other internal servers (like prod-db-01) to these three internal sources. This reduces latency, minimizes firewall complexity, and gives you a stable, authoritative time source you control.
2. Enforce Configuration with IaC: Use your configuration management tool of choice (Ansible, Puppet, Salt, etc.) to manage the chrony.conf file on every single server. Create a standard template and enforce it. Run a periodic check to detect any server that has drifted from this configuration. This makes the correct setup the default and deviation impossible without deliberate action.

Pro Tip: When using cloud providers, take advantage of their internal time sync services. AWS has the Amazon Time Sync Service, and Google has its own internal NTP service. These are highly available, ridiculously accurate, and require no internet access from your instances. Make these your primary internal sources.

At the end of the day, a crooked monitor looks sloppy and unprofessional. A server with a crooked clock is a liability. It’s a silent killer that introduces subtle, non-deterministic bugs that waste your most valuable resource: your team’s time. So go straighten your monitors. Fix your clocks. You’ll thank yourself during the next outage.

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions