Tired of staging environments breaking production? This guide breaks down three real-world strategies for structuring your cloud infrastructure and deployments, from quick emergency fixes to a proper, scalable IaC setup. Stop the chaos and build resilient systems.

From Shared VPCs to Total Chaos: A Senior Engineer’s Guide to Fixing Your Deployment Structure

I remember a Tuesday morning, 10 AM. The pager goes off. Not just a blip, the real one that means revenue is actively being lost. All our user profiles were suddenly showing gibberish names from a fantasy novel generator. After a frantic 15 minutes, we found the culprit. A junior engineer, let’s call him Alex, was testing a new user data migration script. He thought he was hitting the staging database. He wasn’t. Because our staging and prod environments were living in the same VPC with overly permissive security group rules, his local .env file, which was missing a variable, defaulted to the production DB connection string. We spent the next six hours restoring from a backup. Alex felt awful, but it wasn’t his fault. It was ours. We had set him up to fail with a tangled, fragile infrastructure.

The “Why”: The Perils of a Flat Network

This story isn’t unique. Most teams don’t set out to build a house of cards. It happens gradually. You start with a single AWS account or a single VPC because it’s fast. You need to ship. “We’ll fix it later,” you say. But “later” never comes. The root of the problem is a lack of blast radius isolation. When your non-production environments can even *see* your production resources on the network, you’re not dealing with an “if” but a “when” for a catastrophic human error. The goal is to make it impossible, or at least incredibly difficult, to make the kind of mistake Alex made.

So how do we fix it? We’ve got a few options, ranging from a quick fix to a full-blown rebuild.

Solution 1: The “It’s 3 AM and Prod is Down” Fix

Tactic: Network ACLs and Security Group Jiu-Jitsu

This is your immediate damage control. You can’t rebuild the world during an outage, but you can stop the bleeding. The idea is to create hard network boundaries inside your existing, messy VPC. This is a hacky, manual solution, but it’s a thousand times better than nothing.

1. Isolate Subnets: Ensure your staging resources (e.g., staging-api-01) are in different subnets from your production resources (e.g., prod-api-01).
2. Use Network ACLs (NACLs): Think of these as a firewall for your subnets. Create rules that explicitly DENY traffic between the staging subnet CIDR range and the production subnet CIDR range. This is your iron curtain.
3. Tighten Security Groups (SGs): Your production database SG (prod-db-sg) shouldn’t allow traffic from 0.0.0.0/0 or even the whole VPC CIDR. It should ONLY allow traffic from the specific security group of your production API (prod-api-sg).

# Example: An insecure Security Group Ingress Rule
# This allows ANYONE inside the VPC to access the database on port 5432. BAD.
- Ingress:
    Protocol: TCP
    PortRange: 5432
    Source: 10.0.0.0/16

# A much better, more secure rule
# This ONLY allows resources with the 'prod-api-sg' group attached. GOOD.
- Ingress:
    Protocol: TCP
    PortRange: 5432
    SourceSecurityGroupId: sg-012345abcdefg # (ID for prod-api-sg)

Warning: This is a band-aid. It relies on manual configuration and is prone to drift. One wrong click in the console and your “iron curtain” has a hole in it. It’s a good first step, but don’t stop here.

Solution 2: The Permanent Fix

Tactic: IaC with Separate Environment Accounts

This is the way. The gold standard. You treat each environment—dev, staging, prod—as its own sovereign nation. The best way to do this in the cloud is with separate accounts (AWS), projects (GCP), or subscriptions (Azure). This provides the ultimate blast radius isolation for billing, IAM, and networking.

To manage this without going insane, you must use Infrastructure as Code (IaC). My weapon of choice is Terraform.

• Directory Structure: You create a clear, repeatable structure for your code. Configuration for production lives entirely separate from staging.
• CI/CD Pipelines: Your deployment pipeline for the `main` branch can only assume an IAM role in the Production account. A pull request from a feature branch can only deploy to the Dev account. It becomes structurally impossible to deploy staging code to prod.
• State Management: Terraform state files for each environment are kept completely separate (e.g., in different S3 buckets), preventing a change in staging from ever impacting production resources.

Here’s what a sane Terraform project structure looks like:

/infrastructure
├── /environments
│   ├── /prod
│   │   ├── main.tf
│   │   └── terraform.tfvars  # Contains prod-specific variables like instance sizes
│   └── /staging
│       ├── main.tf
│       └── terraform.tfvars  # Contains staging-specific variables
└── /modules
    ├── /vpc
    │   ├── main.tf
    │   └── variables.tf
    └── /database
        ├── main.tf
        └── variables.tf

In this model, the `main.tf` in both `prod` and `staging` calls the same reusable modules (`vpc`, `database`), but passes in different variables from their respective `terraform.tfvars` files. You get consistency without cross-contamination.

Solution 3: The ‘Nuclear’ Option

Tactic: The Strangler Fig Pattern Migration

Sometimes, the existing infrastructure is so tangled and undocumented—a single account with hundreds of manually-created resources named things like `test-final-PROD`—that fixing it in place is more dangerous than starting over. In this case, you don’t try to untangle the knot; you just cut it off.

The “Strangler Fig” pattern is a gradual migration strategy:

1. Build the New World: Using the IaC approach from Solution 2, build out a brand new, pristine set of `prod-v2` and `staging-v2` accounts.
2. Introduce a Proxy/Router: Place a reverse proxy, load balancer, or API gateway in front of your *old* production environment. This will be your traffic cop.
3. Redirect Traffic, One Piece at a Time: When you build a new microservice or migrate an existing one, you deploy it to your new `prod-v2` infrastructure. Then, you change the routing rule in the proxy to send traffic for that specific endpoint (e.g., `/api/v2/users`) to the new service. All other traffic continues to flow to the old system.
4. Strangle and Decommission: Over time, more and more traffic is routed to the new system. Eventually, the old system is no longer receiving any traffic. You’ve effectively “strangled” it. Now, you can finally decommission it with confidence.

Here’s a quick comparison of the approaches:

Pro Tip: Don’t let perfect be the enemy of good. If you’re in a bad spot, implement Solution 1 *tonight*. Then, start planning for Solution 2 tomorrow. Don’t wait for another outage to force your hand. Your future self—and your junior engineers—will thank you.

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions