Discover three battle-tested methods for deterministic API record and replay, from quick local fixes to robust, team-wide solutions that eliminate flaky integration tests for good.

Tired of Flaky Tests? Let’s Talk Deterministic Record & Replay

I remember a Tuesday night, 2 AM, staring at a failing CI pipeline. The integration tests for our new checkout service were flaking out. Sometimes they passed, sometimes they failed with a 429 “Too Many Requests” from our payment provider’s sandbox API. Was our code bad? Was their API having a moment? We burned half a day trying to debug a ghost. That’s the exact kind of expensive, soul-crushing hell that deterministic record-and-replay tools are designed to prevent.

If you’re writing code that talks to any third-party service—a payment gateway, a shipping calculator, a social media API—you’re dealing with non-determinism. The network can be slow, the service can go down, or the data it returns can change. Your tests become a lottery. We can do better.

The “Why”: You’re Testing a Moving Target

The root of the problem is simple: your tests have a dependency on something outside your control. When you run an integration test against a live third-party sandbox, you’re not just testing your code; you’re also testing:

• Your network connection to their server.
• Their server’s current uptime and load.
• Their rate limits.
• The exact state of the data on their end.

That’s four extra points of failure that have nothing to do with whether your logic is correct. The goal is to isolate your code by replacing that unpredictable external service with a predictable, local fake that behaves exactly the same way every single time.

Here are the three ways we handle this at TechResolve, from a quick fix to a full-blown architectural pattern.

The Fixes: From Duct Tape to Dedicated Service

Solution 1: The “Get Me Unstuck Now” Fix (Local Proxy)

This is for the developer who’s stuck right now and just needs to get their feature built. You use a simple “Man-in-the-Middle” (MITM) proxy that sits between your local app and the real API. You run it in ‘record’ mode once, make your API calls, and it saves the requests and responses to a file. Then, you flip it to ‘replay’ mode, and it serves those saved responses back to you instantly, no network required.

I saw a developer on my team use a tool like mitmproxy for this. It’s a quick and dirty, but incredibly effective, way to get moving.

Step 1: Record the interaction

# This command starts a proxy and saves all traffic to a file named 'fedex_api.flow'
mitmproxy -w fedex_api.flow --listen-port 8080

Step 2: Point your app to the proxy
In your local .env file or config, you’d change the API endpoint.

# Before
FEDEX_API_BASE_URL=https://api.realfedex.com/v1

# After
FEDEX_API_BASE_URL=http://localhost:8080

Now run your app and make the API call you want to capture.

Step 3: Replay the interaction
Stop the recorder and start it in replay mode.

# Now it serves the saved responses from the file, no real network call is made
mitmproxy --server-replay fedex_api.flow --listen-port 8080

Warning: This is a great solo tool, but it doesn’t scale for a team. The saved recording file (`fedex_api.flow`) lives on your machine and can get out of date if the real API changes. Use this to unblock yourself, not for your main CI pipeline.

Solution 2: The “Let’s Not Do This Again” Fix (Shared Mocking Service)

This is the grown-up version. Instead of a local proxy, we run a dedicated mocking service like WireMock or Mountebank that the whole team can use. We define our expected API interactions as JSON or code, check them into source control, and run the mock server as part of our `docker-compose` setup.

Now, every developer and the CI server (e.g., ci-test-runner-03) gets the exact same predictable API behavior. No more “it works on my machine!”

Here’s a snippet of what a docker-compose.yml might look like:

services:
  my_app:
    build: .
    ports:
      - "3000:3000"
    environment:
      # Point the app to our mock service container
      PAYMENT_API_URL: http://payment-mock:1080

  payment-mock:
    image: wiremock/wiremock:latest
    ports:
      - "1080:1080"
    volumes:
      # Mount the saved API responses into the container
      - ./mocks/payment-api:/home/wiremock

The definitions in ./mocks/payment-api are explicit. You’re no longer just recording traffic; you’re defining behavior. For example, a WireMock mapping to simulate a successful payment might look like this:

{
  "request": {
    "method": "POST",
    "url": "/v1/charges"
  },
  "response": {
    "status": 201,
    "jsonBody": {
      "id": "ch_12345",
      "status": "succeeded"
    },
    "headers": {
      "Content-Type": "application/json"
    }
  }
}

This is the sweet spot for most teams. It’s version-controlled, repeatable, and fast.

Solution 3: The “We Own The Universe” Option (Full Simulator)

Sometimes, a simple mock isn’t enough. I’m talking about stateful, complex, mission-critical APIs like a full payment gateway or a stock trading platform. The API doesn’t just return data; it changes state over time. A payment might go from `pending` to `succeeded` to `refunded`, often triggered by webhooks.

In this scenario, you bite the bullet and build your own high-fidelity simulator. This is a dedicated internal service that mimics the real API’s behavior, state machine and all. It’s a massive undertaking, but for some systems, it’s the only way.

We had to do this for a project that integrated with a legacy banking mainframe. We built a “Mainframe Simulator” service that not only responded to requests but also simulated end-of-day batch processing. It was a huge effort, but it allowed us to run a full regression test suite in 15 minutes instead of waiting 24 hours for the real mainframe cycle.

Pro Tip: Don’t even think about this option unless the external service is core to your business and its sandbox environment is notoriously unreliable or expensive. This is a product in itself, not a weekend hack.

Comparison: Which Should You Choose?

Let’s be real, it’s all about trade-offs. Here’s how I break it down for my team:

Solution	Effort	Fidelity (Realism)	Best For…
1. Local Proxy	Low	High (it’s a real recording)	Individual developers trying to unblock themselves quickly.
2. Shared Mocking Service	Medium	Medium (as real as you define it)	Most teams. The standard for reliable CI/CD and local development.
3. Full Simulator	Very High	High (simulates state and behavior)	Mission-critical, complex, stateful integrations where mocks fall short.

Stop letting flaky external APIs dictate your release schedule. Start with a simple proxy, graduate to a shared mocking service, and get your test environments back under your control. Your on-call self will thank you.

---

🤖 Frequently Asked Questions