A quick guide to taming the SQS and Lambda firehose. We’ll explore three battle-tested methods for rate-limiting message processing to protect your downstream services, from the quick-and-dirty fix to the architecturally sound solution.

I Called Our Payment Processor and Apologized: A Guide to Rate Limiting SQS and Lambda

I still remember the “Black Friday Incident of ’21.” We’d just launched a shiny new microservice to process orders. Simple stuff: an SQS queue feeds a Lambda function, which then calls our third-party payment processor’s API. We load-tested it, it scaled beautifully, and we all patted ourselves on the back. Then the sale hit. The SQS queue filled up with thousands of messages in seconds, and Lambda did exactly what it was designed to do: it scaled out to hundreds of concurrent executions to burn through that queue. The only problem? Our payment processor’s API had a rate limit of 100 requests per second. We hit them with a tsunami. They IP-banned our entire VPC. My morning involved a very humbling phone call to their tech support, lots of apologies, and a promise that it would never happen again. This post is for anyone who’s made that call, or wants to avoid ever having to.

So, Why Does This Even Happen?

It’s not a bug; it’s a feature, and a dangerous one if you’re not prepared. A standard SQS queue is built for maximum throughput. A Lambda function is built to scale massively to meet demand. When you connect the two, you create a system that will try to process every message as fast as humanly possible. It has no concept of the “feelings” or rate limits of whatever poor downstream service, database, or API you’re calling. The SQS-Lambda trigger is a firehose, and if you point it at a garden sprinkler, you’re going to have a bad time.

Let’s look at three ways to control the flow, from the emergency stop-gap to the “let’s architect this properly” solution.

Solution 1: The Quick Fix (a.k.a. “The Panic Button”) – Reserved Concurrency

This is the fastest way to stop the bleeding. You’re essentially telling the Lambda function, “I don’t care how many messages are in that queue, you are never allowed to run more than ‘X’ concurrent instances of yourself.” It’s a hard ceiling.

You can set this in the Lambda console under Configuration > Concurrency, or via the AWS CLI.

aws lambda put-function-concurrency \
    --function-name my-order-processor-lambda \
    --reserved-concurrent-executions 10

In this example, no matter if there are 10 or 10,000 messages in the SQS queue, a maximum of 10 Lambda functions will be running at any given time.

Warning: This is a blunt instrument. Messages will simply back up in the SQS queue while they wait for a free Lambda slot. If they wait too long (past your message retention period), they’ll be lost or sent to a Dead-Letter Queue (DLQ). Use this to prevent an outage, but don’t rely on it as your primary, long-term strategy.

Solution 2: The Right Way – SQS FIFO Queues & Message Groups

If you need more granular control and want an architecturally sound solution, this is it. Unlike a standard queue, a FIFO (First-In, First-Out) queue processes messages in order, but it has a killer feature for our use case: Message Group IDs.

Here’s the magic: Lambda will only ever process one batch of messages for a given MessageGroupId at a time. This means the number of concurrent Lambda executions is effectively limited by the number of unique, active message group IDs in your queue.

If you want to limit processing to, say, 5 messages per second, you can have your producers send messages with one of five rotating Message Group IDs (e.g., group-1, group-2, … group-5).

When your producer sends a message, it looks like this:

# Example using Python's boto3
import boto3

sqs = boto3.client('sqs')

sqs.send_message(
    QueueUrl='YOUR_FIFO_QUEUE.fifo',
    MessageBody='Your message body here',
    MessageGroupId='processor-group-3',  # The key to rate limiting!
    MessageDeduplicationId='some-unique-id-for-the-message' # Required for FIFO
)

By controlling how many unique MessageGroupId values you use, you directly control the maximum concurrency. This is a much more elegant way to throttle the system because it’s managed at the source, not by artificially strangling the processor.

Pro Tip: Always, and I mean always, configure a Dead-Letter Queue (DLQ) on your SQS queue. When a message fails processing multiple times, it will be sent to the DLQ instead of being lost forever. This has saved my team from data loss more times than I can count. It’s your ultimate safety net.

Solution 3: The “Nuclear” Option – AWS Step Functions

Sometimes, you need more than simple rate-limiting. You need complex orchestration, long waits, or dynamic throttling based on API feedback. This is where you bring in the heavy machinery: AWS Step Functions.

Instead of SQS -> Lambda, the flow becomes SQS -> Lambda (that just starts the workflow) -> Step Function. The state machine can then control the entire process.

A common pattern is a loop that:

1. Calls a Lambda to pull a batch of messages from the queue.
2. Processes them.
3. Enters a “Wait” state for a defined period (e.g., 10 seconds).
4. Repeats.

Here’s a simplified Amazon States Language (ASL) definition:

{
  "Comment": "A rate-limited SQS processor.",
  "StartAt": "ProcessMessages",
  "States": {
    "ProcessMessages": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789012:function:MyBatchProcessorLambda",
      "Next": "WaitForNextInterval"
    },
    "WaitForNextInterval": {
      "Type": "Wait",
      "Seconds": 60,
      "Next": "ProcessMessages"
    }
  }
}

This creates an infinite loop that processes messages and then explicitly waits for 60 seconds before trying again. It’s incredibly powerful but also more complex and costly. Use this when you have a truly complex workflow, not just for basic rate limiting.

Summary Table: Which One Should I Use?

Hopefully, this saves you from having to make that awkward apology call. The SQS and Lambda combo is one of the most powerful patterns in the serverless world, but as we all learn, with great power comes the great responsibility of not DDoSing your partners.

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions