When an external system of record conflicts with your internal state, you can’t just sue the API. Here are three battle-tested strategies for forcing data reconciliation, from a quick manual fix to a full architectural redesign.

Our Database Says They Paid. Stripe Says They Didn’t. Who Do You Believe?

I remember a launch day from about five years ago like it was yesterday. We’d just rolled out a major feature tied to user subscription tiers. Everything looked green on our end. Then the support tickets started flooding in. “I can’t access the new feature,” “It says my account is ‘Past Due’.” We checked our main `prod-db-01` replica, and the `users.subscription_status` column clearly said ‘active’ for these customers. We were stumped. It took us two hours of frantic debugging to realize our payment processor’s webhooks had been silently failing for the last 18 hours. Our system thought everyone was paid up, but the actual source of truth—the bank, the processor—had a very different story. We were living a lie, and our customers were paying the price.

That feeling of helplessness, of your own system’s state being overruled by an external, authoritative source you don’t control, is infuriating. It’s the technical equivalent of a bank siding with a fraudulent chargeback. You have the receipt, the proof, but their system says “no,” and their decision is final. So what do you do when you can’t just “sue the API” to get it to listen?

The “Why”: The Dangerous Myth of a Single Source of Truth

The root of this problem isn’t just a failed webhook; it’s a flawed architectural assumption. In any distributed system, especially one relying on third-party services (like payment gateways, identity providers, or shipping APIs), you don’t have a single source of truth. You have your truth and their truth. The problem arises when these two truths diverge and you don’t have an automated process for reconciliation. Relying solely on event-driven updates (like webhooks) is optimistic. You have to plan for the pessimistic scenario: the events will, at some point, fail to arrive.

Solution 1: The Quick Fix (And Why You Shouldn’t Get Used to It)

This is the “get the system working again at 3 AM” approach. It’s manual, it’s ugly, but it stops the bleeding. You manually pull a report from the external system and force your internal state to match.

Let’s say you need to fix the subscription status for users whose payments failed in Stripe. You’d go into Stripe, export a CSV of all failed payments for the day. Then, you (or a junior engineer you trust) would write a one-off SQL script to update your database.

-- WARNING: Run this inside a transaction. ALWAYS.
-- update_failed_subs.sql

BEGIN;

UPDATE customer_accounts
SET subscription_status = 'past_due',
    updated_at = NOW()
WHERE
    customer_email IN (
        'user1@example.com',
        'user5@example.com',
        'user12@example.com'
        -- ...paste the list of 50 emails from your CSV here
    );

COMMIT;

This is technical debt, plain and simple. It works once, but it doesn’t solve the underlying problem. It’s the equivalent of calling the bank, yelling at a manager, and getting a single chargeback overturned. You won, but you’ll have to fight the same battle tomorrow.

Solution 2: The Permanent Fix (The Reconciliation Service)

This is where we put on our architect hats. Instead of waiting for the external system to tell us about a change, we build a service to proactively ask it. This is typically a cron job or a scheduled lambda that runs periodically (e.g., every hour).

The process looks like this:

• Fetch: The service calls the third-party API to get a list of all objects updated within the last X hours. (e.g., `stripe.subscriptions.list({updated: {gte: one_hour_ago}})`).
• Compare: It iterates through the results and compares the status of each object with the corresponding record in your local `prod-db-01` database.
• Reconcile: If there’s a mismatch, it updates your local record to match the external source of truth.
• Alert: It logs every change it makes and sends an alert to a Slack channel (`#billing-reconciliation-alerts`) so the team is aware of what’s being fixed automatically.

Pro Tip: Don’t just update silently. Log every single discrepancy and fix. When your reconciliation service changes a user from ‘active’ to ‘cancelled’ without a corresponding webhook log, that’s a signal that your primary event pipeline is broken and needs investigation.

This is the responsible, scalable solution. It assumes failure will happen and cleans it up automatically. It’s the equivalent of having a finance team member whose job is to audit every transaction against bank statements every single day.

Solution 3: The ‘Nuclear’ Option (Architectural Surrender)

Sometimes, the external system is so critical and your internal copy of its data is so frequently wrong that you have to take a more drastic step: Stop copying the data altogether.

Instead of storing `subscription_status` in your `customer_accounts` table, you treat the external API as the one and only source of truth. Every time a user tries to access a protected feature, your application makes a real-time API call to the payment processor to check their status.

The Obvious Problem: This is slow and can be expensive. Hitting an external API on every request is a recipe for latency.

The Fix: A caching layer. When you need to check a user’s status, your logic looks like this:

1. Check for the user’s status in Redis/Memcached with a key like `user:123:sub_status`.
2. If it’s a cache hit, use that value. Great.
3. If it’s a cache miss, make the live API call to the external service.
4. Store the result in the cache with a short TTL (Time To Live), like 5-10 minutes.

This approach completely eliminates data drift between you and the third party, but it introduces a dependency. If the external API goes down, your feature goes down. It’s the ultimate “sue the bank” option—you’re not just correcting their decision, you’re fundamentally changing your relationship with them to make them directly responsible for the outcome on every single transaction. It forces honesty but comes at a high price in performance and availability risk.

Choosing Your Battle

Here’s a quick breakdown of when to use each approach:

In the end, you can’t sue an API. You can’t force a bank’s ledger to match your books through litigation. What you can do is build resilient systems that assume the outside world is chaotic and that your truth will be challenged. Plan for it, build for it, and you’ll sleep a lot better during your next on-call shift.

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions