Frustrated by your long-running AI agents timing out through your web stack? I’ll break down why it’s not your code’s fault and give you three real-world solutions, from the quick-and-dirty fix to the proper asynchronous architecture.

Your AI Agent Timed Out. Again. Let’s Fix That For Good.

I still remember the 2 AM PagerDuty alert. We had just pushed a new “AI Insight Generator” for a major client. It worked flawlessly in staging. But in production, every single request that took longer than 60 seconds would die with a `504 Gateway Time-out`. The on-call engineer was blaming the AI model, the data team was blaming the database `prod-db-01`, and management was just blaming everyone. After an hour of frantic log diving, we found the culprit: a default, undocumented 60-second idle timeout on the Application Load Balancer. A single setting, buried deep in the AWS console, was bringing our entire feature launch to its knees. This problem is infuriatingly common, and it’s almost never the agent’s fault.

Why This Keeps Happening: You’re Treating a Marathon Runner Like a Sprinter

Here’s the hard truth: your entire web infrastructure is built for speed. It’s designed for quick, stateless HTTP requests that last milliseconds, not seconds or minutes. Think of it as a chain of sprinters passing a baton. The problem is, your AI agent is a marathon runner.

When a user makes a request, it travels through a chain, and each link in that chain has its own stopwatch:

• The User’s Browser
• The Cloud Load Balancer (like an AWS ALB or Google Cloud Load Balancer)
• The Reverse Proxy / Ingress (like Nginx or Traefik)
• The Application Server (like Gunicorn or uWSGI for Python)
• Finally, your AI application code.

If any single link in that chain decides the wait has been too long, it drops the connection. The shortest timeout in the chain always wins, and that’s why your agent gets cut off mid-thought.

Three Ways to Tame the Timeout Beast

Alright, enough theory. Let’s get our hands dirty. I’ve used all three of these approaches in production. They range from “get me through the night” to “build it to last a decade.”

Solution 1: The “Duct Tape” – Crank Up the Timeouts

This is the first thing everyone tries, and for good reason: it’s fast. The idea is to find every stopwatch in the chain and set it to a much higher value. If your agent needs 5 minutes, you set all the timeouts to 300 seconds or more.

Step 1: Your Load Balancer
In AWS, this is the “Idle timeout” on your Application Load Balancer’s target group. The default is 60 seconds. Bump it up.

Step 2: Your Reverse Proxy (e.g., Nginx)
You need to tell Nginx it’s okay to wait longer for the application to respond. You’ll add these to your `nginx.conf` in the relevant `location` or `server` block.

# In your nginx.conf
proxy_connect_timeout 300s;
proxy_send_timeout    300s;
proxy_read_timeout    300s;
send_timeout          300s;

Step 3: Your Application Server (e.g., Gunicorn)
If you’re running a Python app with Gunicorn, it has its own worker timeout. The default is a measly 30 seconds.

# When you start your gunicorn process
gunicorn --workers 3 --bind 0.0.0.0:8000 --timeout 300 my_app.wsgi:application

Warning: This is a dangerous path. While it provides immediate relief, holding HTTP connections open for minutes is a terrible use of resources. If 10 users run a 5-minute task, you now have 10 web workers on `prod-web-api-01` completely tied up, unable to serve any other traffic. You’re basically inviting a Denial-of-Service attack on yourself.

Solution 2: The “Right Way” – Go Asynchronous with a Task Queue

This is the pattern we use for all heavy lifting at TechResolve. Instead of making the user wait, you change the contract. You decouple the initial request from the actual work.

The flow looks like this:

1. The user sends a request to your API to start the AI job.
2. Your API server does zero AI work. It simply validates the request, creates a job ticket, puts it onto a message queue (like RabbitMQ, Redis, or AWS SQS), and immediately returns a `202 Accepted` response to the user with a `job_id`. This whole process takes 50ms.
3. A completely separate fleet of worker processes (running on different servers, maybe `prod-ai-worker-pool-01`) are listening to the queue. One of them picks up the job and starts the long-running AI task.
4. The user’s frontend can now poll a separate endpoint, like `/api/v1/jobs/<job_id>`, to check the status (“pending”, “running”, “success”, “failed”).
5. When the status is “success”, the frontend makes one last call to that endpoint to retrieve the results.

Here’s a quick comparison of the two models:

This is more work to set up, but it’s the only truly scalable and resilient solution. Your API stays snappy, your workers can be scaled independently, and a failed job can be retried automatically from the queue.

Solution 3: The “Real-Time” Option – WebSockets or Server-Sent Events (SSE)

Sometimes, polling isn’t enough. What if you’re building a chatbot and want to stream the response back token-by-token, like ChatGPT does? Making hundreds of polling requests would be wildly inefficient. This is where you need a persistent connection.

WebSockets: This is a full, two-way communication channel between the client and server. It’s great for interactive, back-and-forth communication. It’s powerful but can be complex to manage at scale, especially with load balancers that need to support “sticky sessions.”

Server-Sent Events (SSE): This is a simpler, one-way street. The server can push messages to the client, but the client can’t send messages back over the same connection (it would have to use a separate HTTP request). For streaming an AI response, SSE is often the perfect tool—it’s simpler than WebSockets and built on standard HTTP.

Pro Tip: Don’t reach for WebSockets just because they sound cool. If you only need to stream data from the server to the client without a constant back-and-forth, SSE is your best friend. It’s easier to implement and much easier to scale behind a standard load balancer.

So, Which Path Do You Take?

My advice is almost always the same. If you’re on fire and need to fix production right now, use Solution 1. Increase the timeouts, get the system stable, and apologize to your users. But the very next morning, you need to have a meeting and start planning your migration to Solution 2. Building a proper asynchronous task queue system is the mark of a mature engineering organization. It’s how you go from fighting fires to building resilient, scalable systems that don’t wake you up at 2 AM.

Stop fighting the infrastructure. Start designing for it.

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions