Struggling with External Secrets Operator (ESO) failing to sync in production? Learn the practical tradeoffs between reconciliation intervals and authentication, and discover three proven fixes to prevent stale secrets from ever taking down your services again.

External Secrets in Production: What to Do When Reconciliation Fails

I remember the PagerDuty alert like it was yesterday. 3:17 AM. CRITICAL: prod-billing-api cannot connect to prod-db-01. My heart sank. We’d just rolled out our automated database password rotation policy, a security win we’d celebrated two days prior. The logs were clear: “Authentication failed”. But… how? The new password was in AWS Secrets Manager, and our External Secrets Operator (ESO) was supposed to sync it into our Kubernetes cluster automatically. It turned out, it *was* supposed to, but a perfect storm of bad timing with temporary credentials and a long reconciliation interval meant our app was using a password that had expired 20 minutes earlier. That’s when I learned that “eventual consistency” can be a very painful concept at 3 AM.

So, What’s Actually Going Wrong? The “Why” Behind Stale Secrets

At its core, the External Secrets Operator is a polling machine. It’s not magic; it doesn’t get a “push” notification from your secrets store (like AWS Secrets Manager, GCP Secret Manager, or HashiCorp Vault) by default. It simply wakes up on a schedule, defined by the refreshInterval in your ExternalSecret manifest, and asks, “Hey, anything new for me?”

This creates two primary failure modes:

1. The Reconciliation Gap: If your secret is rotated in the vault right after a successful check, your cluster is stuck with a stale secret until the next refresh interval. If your interval is 15 minutes, you have a 14 minute, 59 second window of potential outage.
2. The Authentication Hiccup: This is the sneakier one. Most of us use temporary, short-lived credentials for our pods to talk to the cloud provider (like IAM Roles for Service Accounts in AWS). If the operator’s reconciliation loop tries to run *just* as those credentials expire, the API call to the secret store fails. ESO doesn’t panic; it just logs an error and waits for the next interval, hoping the credentials will be valid then. The result is the same: a stale secret and a dead application.

Let’s walk through the real-world fixes, from the emergency patch to the permanent architectural solution.

Solution 1: The “Get Me Out of This PagerDuty Hell” Quick Fix

It’s 3 AM, your service is down, and you don’t have time to re-architect anything. You just need the secret to sync. Now.

The fastest way to force ESO to reconcile an ExternalSecret is to change its metadata. The operator watches for changes on the custom resource itself, and any change will trigger an immediate re-sync. The easiest, non-destructive way to do this is with an annotation.

Run this command, replacing the placeholders with your details:

kubectl annotate externalsecret <your-external-secret-name> -n <your-namespace> \
  eso.external-secrets.io/force-sync=$(date +%s) --overwrite

This command adds a simple annotation with the current timestamp. Because the timestamp is always different, it guarantees a change to the object’s spec, which tells the ESO controller, “Hey, look at me! I’ve changed!” The controller then immediately runs its reconciliation logic, fetches the fresh secret from your vault, and updates the corresponding Kubernetes Secret. Your app recovers, and you can go back to sleep.

Darian’s Take: This is a fantastic incident response tool. It’s a “hack,” for sure, but it’s a clean, effective one. We have this command saved in our runbooks for exactly this scenario.

Solution 2: The “Let’s Fix This Properly” Event-Driven Approach

Polling is inefficient and prone to the timing issues we discussed. The architecturally superior solution is to move to a push model. Instead of ESO asking “is it ready yet?”, we want the secret store to tell ESO when a secret has changed.

This is typically done with webhooks. ESO has built-in support for a generic webhook receiver that can trigger reconciliations on demand. Here’s the high-level flow for AWS Secrets Manager:

1. A secret is updated in AWS Secrets Manager.
2. This action triggers an AWS EventBridge rule that you’ve configured to watch for PutSecretValue or UpdateSecret API calls.
3. The EventBridge rule’s target is an Amazon SNS topic or a Lambda function.
4. The target then sends a specially crafted payload to the ESO webhook receiver endpoint, which you expose via a Kubernetes Ingress or Service.
5. ESO receives the webhook and immediately reconciles the specific ExternalSecret mentioned in the payload.

Your ExternalSecret manifest would need to include the secret key that the webhook will use to find it:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: prod-db-credentials
  # This annotation is what the webhook uses to find this resource
  annotations:
    "reconcile.external-secrets.io/webhook-id": "aws-sm-12345"
spec:
  # ... your spec here

This completely eliminates the reconciliation gap. The moment the secret is updated in the vault, the sync process begins. It’s more complex to set up, but it’s the most robust and reliable solution.

Solution 3: The “Brute Force” Auth and Interval Tuning

If setting up a full event-driven pipeline isn’t feasible right now, you can mitigate both the auth and timing issues by carefully tuning your refresh interval. The core idea is to make the interval significantly shorter than the lifetime of the pod’s temporary credentials.

For example, in AWS EKS, the default expiry for an IAM Role for Service Account (IRSA) token is 1 hour (3600 seconds). If your refreshInterval is set to 15m, there’s a good chance that a reconciliation attempt will happen on an expiring token.

By shortening the interval, you increase the probability of a successful sync. A shorter interval also closes the “reconciliation gap” we talked about earlier.

For most of our critical services, we’ve settled on a refreshInterval of 5m. It’s a pragmatic balance between reliability and resource consumption.

Warning: Before you set your interval to 1m across the board, check the API pricing and rate limits for your secret store! A few hundred secrets polling every minute can add up surprisingly quickly on your monthly bill and potentially get your cluster’s IP throttled.

Ultimately, there’s no single magic bullet. We use a combination of these techniques at TechResolve. Our most critical services use the event-driven webhook model, while less sensitive applications use a well-tuned refresh interval. And everyone on the team knows the `kubectl annotate` command for when things inevitably go bump in the night.

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions