Learn why building a distributed queue on a single S3 JSON file is a recipe for disaster due to race conditions, and explore three practical solutions—from a quick hack to a robust architectural fix—to handle concurrent writes safely.

You Can’t Build a Distributed Queue in a Single S3 JSON File. Here’s Why (And What to Do Instead).

I remember the PagerDuty alert like it was yesterday. It was 2:30 AM, and our critical end-of-day billing report job was going haywire. Customers were getting duplicate invoices, and our finance team was about to have a very, very bad morning. After a frantic hour of digging through logs from our fleet of `batch-worker` instances, we found the culprit: a seemingly innocent S3 object named `job-queue.json`. We had designed a “simple” system where workers would read this file, grab a task, remove it from the list, and write the file back to S3. Simple, right? Wrong. It was a ticking time bomb of a race condition, and it had just detonated.

The “Why”: S3 Isn’t a Database, and Last Writer Wins

Before we dive into the fixes, let’s get on the same page about why this blew up. When you’re dealing with a distributed system—multiple workers, multiple servers, all acting at once—you need a way to manage shared state safely. The core problem with the single JSON file approach is that S3 object operations are not atomic.

Here’s the sequence of events that was killing us:

1. Worker A performs a GET on job-queue.json. The list has tasks [1, 2, 3].
2. Worker B performs a GET on job-queue.json. It also sees tasks [1, 2, 3].
3. Worker A claims task 1, updates its in-memory list to [2, 3], and performs a PUT to S3.
4. Worker B, completely unaware of Worker A’s actions, claims task 2, updates its in-memory list to [1, 3], and performs a PUT to S3.

Worker B’s PUT just overwrote Worker A’s change. The file on S3 now contains [1, 3]. Task 1, which Worker A is happily processing, is now back in the queue, ready to be picked up again. Task 2, which Worker B processed, is gone. It’s a classic “last writer wins” scenario, and it’s a guaranteed way to lose data and create chaos.

The Fixes: From Hacky to High-End

Look, I get it. Sometimes you’re trying to build something quickly without spinning up a whole new piece of infrastructure. But you have to respect the laws of distributed computing. Here are three ways to solve this problem, ranging from a quick fix to the “right” way.

Solution 1: The ‘Get-Me-Through-The-Night’ Fix (Object-per-Task)

Instead of one monolithic JSON file, treat each task as its own object in S3. You create a “directory” (or prefix, in S3 terms) that acts as your queue.

How it works:

• Your producer writes each task as a new, unique object, e.g., s3://my-bucket/queue/task-uuid-1.json.
• Your worker lists objects under the /queue/ prefix.
• To “claim” a task, the worker performs a RENAME operation on the object, which in S3 is a COPY to a new key (e.g., /processing/task-uuid-1.json) followed by a DELETE of the original.
• If the rename succeeds, you’ve got the lock. If it fails (because another worker got there first), you try the next object.

Warning: This is still a bit hacky. Listing objects with s3:ListBucket can get slow and expensive if you have thousands of tasks. The RENAME operation isn’t atomic either, so you need careful error handling to avoid a task getting stuck if the worker dies between the COPY and DELETE. But for a low-volume queue, it’ll stop the bleeding.

Solution 2: The Pragmatic Architect’s Fix (S3 Events + SQS)

This is my personal favorite because it uses S3 for what it’s good at—storing data—and a real queue for what it’s good at—managing tasks. This pattern is robust, scalable, and surprisingly easy to set up.

How it works:

1. You configure S3 Event Notifications on your bucket. Specifically, you want to fire an event on s3:ObjectCreated:*.
2. Set the destination for that event to be an Amazon SQS (Simple Queue Service) queue.
3. Now, when a producer uploads a file like s3://my-bucket/tasks-to-process/some-big-file.csv, S3 automatically sends a message to your SQS queue. The message body contains details about the object, including its bucket and key.
4. Your workers don’t poll S3 anymore. They poll the SQS queue, which is built for this. SQS guarantees that once a message is pulled by a worker, it’s hidden from other workers for a configurable “visibility timeout”. If the worker processes it and deletes the message, great. If it crashes, the message reappears on the queue to be tried again.

This decouples your system beautifully. S3 holds the data, and SQS manages the work. No more race conditions.

Solution 3: The ‘Right Tool for the Job’ Fix (DynamoDB Atomic Counters & Conditional Writes)

If your “task” is less about a file and more about a piece of data that needs its state managed, you’ve outgrown an object store as your primary coordination mechanism. It’s time for a database built for this kind of concurrent access, like DynamoDB.

How it works:

You can model your queue in a DynamoDB table. Each item represents a task, with attributes like task_id, status (e.g., PENDING, PROCESSING, COMPLETE), and worker_id.

The magic is in DynamoDB’s conditional writes. A worker can try to claim a task like this:

# Pseudocode for a DynamoDB UpdateItem call
aws dynamodb update-item \
    --table-name my-job-queue \
    --key '{"task_id": {"S": "task-uuid-1"}}' \
    --update-expression "SET #s = :s, worker_id = :w" \
    --condition-expression "#s = :p" \
    --expression-attribute-names '{"#s": "status"}' \
    --expression-attribute-values '{
        ":s": {"S": "PROCESSING"},
        ":w": {"S": "worker-hostname-42"},
        ":p": {"S": "PENDING"}
    }'

This operation says: “Update the status of this task to PROCESSING, but only if its current status is PENDING.” This operation is atomic on the server side. If two workers try this at the same time, only one will succeed. The other will get a ConditionalCheckFailedException, and it will know to simply try another task.

Here’s a quick comparison of the approaches:

So, the next time someone on your team suggests building a queue in a single file on S3, you can gently guide them away from the 2:30 AM PagerDuty alert I had to live through. It might seem simple, but the fundamental primitives of a queue—atomic claims and visibility—just aren’t there. Use the right tool for the job.

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions