Struggling with network connectivity despite having ‘allow all’ firewall rules and a massive budget? You’re probably looking in the wrong place. We break down the real culprit behind these maddening issues and show you how to fix it for good.

I Have ‘Unlimited Spend’ and My Services Still Can’t Talk. Help.

I remember it vividly. It was 2 AM, the coffee was stale, and a brand new `auth-service-v3` deployment was stubbornly refusing to connect to our primary database, `prod-db-01`. The on-call engineer was frantic. We had triple-checked the AWS Security Groups. The database SG explicitly allowed ingress from the auth service’s SG on port 5432. The auth service’s egress was wide open (`0.0.0.0/0`). We had unlimited spend, we could have spun up a gold-plated load balancer to route the traffic if we wanted. But money doesn’t fix logic, and for two hours, we were stumped by a problem that felt like a ghost in the machine. It turns out, we were looking at the wrong firewall all along.

The “Why”: You’re Fighting a Stateless Bouncer

When you’re dealing with cloud networking, especially in AWS, you have two main layers of network firewalls: Security Groups (SGs) and Network Access Control Lists (NACLs). Most of us live and breathe Security Groups, and we forget about their grumpy, less-forgiving older brother.

Here’s the difference that causes 99% of these headaches:

• Security Groups are STATEFUL. If you allow inbound traffic on port 80, the SG automatically allows the return traffic for that connection back out, regardless of your outbound rules. It’s like a bouncer at a club who remembers your face and lets you leave without checking your ID again.
• Network ACLs are STATELESS. A NACL doesn’t remember anything. If you allow inbound traffic on a port, you must also create a separate, explicit outbound rule to allow the return traffic. It’s like having two bouncers—one at the entrance and one at the exit—who don’t communicate. You need to show your ID to both of them.

Your “unlimited spend” doesn’t matter if your return packet is being silently dropped by a NACL rule you forgot existed. The connection just times out, leaving you with zero useful error messages.

Solution 1: The Quick Fix (The “Sledgehammer”)

This is my go-to for sanity checking. Is it the NACL? Let’s find out, fast. The goal here isn’t security; it’s diagnosis. We’re going to temporarily blast the door wide open to see if the traffic flows. If it does, we’ve found our culprit.

Go to the NACL associated with your subnet and add these two rules with a low rule number (like 90) so they’re evaluated first:

Warning: This is the networking equivalent of leaving your front door wide open with a sign that says “Free Stuff Inside.” Do this to confirm the problem, prove your theory, and then IMMEDIATELY remove these rules. Do not leave this in production for more than five minutes.

Now, try your connection test again. If it works, you’ve confirmed the NACL was the problem. Now, let’s fix it properly.

Solution 2: The Permanent Fix (The “Scalpel”)

The right way to fix this is to account for the stateless nature of NACLs by allowing the return traffic. When your `auth-service` tries to connect to `prod-db-01` on port 5432, it sends the request from a random, high-numbered port on its own end. This is called an ephemeral port. The database server then tries to send its response back to that specific ephemeral port.

Your NACL needs rules to allow for this two-way conversation. Let’s say your auth service has an IP of `10.0.1.50` and the database is `10.0.2.100`.

Inbound Rules (for the Database Subnet):

# Allows the initial request from the auth service to the database
Rule #: 100
Type: Inbound
Protocol: TCP (6)
Port Range: 5432
Source: 10.0.1.50/32
Allow/Deny: Allow

Outbound Rules (for the Database Subnet):

# THIS IS THE PART EVERYONE FORGETS
# Allows the return traffic from the DB back to the auth service's ephemeral port
Rule #: 100
Type: Outbound
Protocol: TCP (6)
Port Range: 1024-65535
Destination: 10.0.1.50/32
Allow/Deny: Allow

By adding that outbound rule for the ephemeral port range (1024-65535), you complete the circuit. The database’s response is now allowed to leave the subnet and get back to the service that requested it.

Solution 3: The ‘Nuclear’ Option (Redesign the Architecture)

Sometimes, you inherit a network topology so complex that fighting with NACLs is a daily battle. If this is a recurring nightmare, throwing money at the problem is actually the right call—just not by hiring more consultants to stare at NACL rules.

Instead, simplify the architecture to make these problems impossible.

• Co-location: Can the services live in the same subnet? Traffic within a subnet doesn’t pass through the NACL. This is the simplest solution, though not always feasible.
• Use VPC Endpoints (PrivateLink): This is the grown-up solution. Instead of routing traffic over the VPC network, you can create a private endpoint for services like RDS, S3, or even your own custom services. It effectively creates a private, secure wormhole for your traffic that bypasses NACLs and route tables entirely. It costs money, but it deletes this entire class of problems.
• Implement a Service Mesh: Tools like Istio or Linkerd can handle service-to-service communication policies at a higher level of abstraction. While complex to set up, they can make network routing rules much more intelligent and dynamic, moving policy away from rigid, low-level constructs like NACLs.

Pro Tip: My personal rule is this: if a team spends more than two hours a month debugging a NACL issue, we schedule an architecture review to see if we can eliminate the NACL from the equation with a better design. Your engineers’ time is more expensive than a VPC Endpoint.

So next time you’re stuck, remember that “unlimited spend” is useless if you’re looking in the wrong place. Take a step back from the instance firewall, check the layer below it, and I bet you’ll find your culprit silently dropping packets in the dark.

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions