A Senior DevOps Engineer breaks down three battle-tested strategies for structuring Terraform configurations. Move from monolithic state files to a scalable, multi-repo architecture to prevent production outages and regain your sanity.

How I Escaped ‘Module Hell’: A Senior Engineer’s Guide to Architecting Terraform

I still remember the feeling in the pit of my stomach. It was 2 PM on a Tuesday. A junior engineer, bless his heart, was tasked with a simple change: add a new egress rule to a security group for our staging environment’s new analytics service. He ran terraform apply from what he thought was the right directory. A few minutes later, Slack exploded. Our entire production customer-facing application was down. Hard down. It turned out our “one size fits all” Terraform repo, with its complex web of shared modules and a monstrous state file, had allowed his staging change to rip out a critical security group rule from our production ELB. We spent the next three hours in a war room, manually rebuilding AWS resources while the CTO breathed down my neck. That was the day I declared war on our Terraform structure.

The ‘Why’: Your Real Enemy is Blast Radius

Let’s be clear: the problem isn’t Terraform. It’s a fantastic tool. The problem is complexity and blast radius. When your entire infrastructure—from the dev sandbox VPC to the production database—lives in the same configuration root or state file, a single typo can become a company-ending event. Every `apply` becomes a high-stakes gamble. The fundamental goal of a good Terraform architecture is to build bulkheads, just like on a ship. If one compartment floods (a staging environment breaks), it shouldn’t sink the entire vessel (production).

We’re going to look at three ways to structure your code, moving from a quick fix for small projects to the battle-hardened setup we use at TechResolve today.

Solution 1: The Workspace Shuffle (The Quick Fix)

When you’re just starting out or working on a small project, you don’t need a massive, multi-repo setup. The quickest way to get some separation is using Terraform Workspaces. Think of them as different `tfstate` files within the same directory.

You manage them with simple commands:

# Create a new environment
terraform workspace new staging

# Switch to it
terraform workspace select staging

# See which one you're on
terraform workspace show

Inside your HCL, you can then reference the current workspace to make decisions. For example, you might want fewer servers in staging than in production.

variable "instance_count" {
  type    = map(number)
  default = {
    default = 1
    prod    = 5
  }
}

resource "aws_instance" "web_server" {
  # Use 5 for prod, 1 for anything else (like staging)
  count = lookup(var.instance_count, terraform.workspace, var.instance_count.default)

  ami           = "ami-0c55b159cbfafe1f0"
  instance_type = "t2.micro"
}

This is great for getting off the ground, but it’s a leaky abstraction. The code is still shared, and it’s dangerously easy for a developer to forget which workspace they’re in or for a shared module change to have unintended consequences. It’s a band-aid, not a cure.

War Story Warning: Be very careful with this approach. While workspaces create separate state files, you’re still working from the same set of .tf files. We once saw a developer add a lifecycle { prevent_destroy = true } block to a production database module, but since he was in the dev workspace, he didn’t apply it to prod. A week later, another engineer ran a cleanup job in prod and nearly deleted the database. The code said one thing, but the state said another.

Solution 2: The ‘Live vs. Modules’ Architecture (The Scalable Fix)

This is where we are today, and it’s the pattern I recommend for any serious team or company. The core idea is to separate the *what* from the *how*. We maintain two separate Git repositories:

1. The Modules Repo: This contains our reusable, battle-tested building blocks. A VPC module, an EKS cluster module, an RDS Aurora module. They are generic and highly configurable.
2. The Live Repo: This repository describes our *actual* infrastructure. It contains no complex logic; it just consumes the modules from the other repo and stitches them together with environment-specific variables.

The directory structure of our `live-infra` repo is the real magic. We structure it by environment, region, and component:

live-infra/
└── aws/
    └── us-east-1/
        ├── prod/
        │   ├── vpc/
        │   │   ├── main.tf
        │   │   └── terraform.tfvars
        │   ├── services/
        │   │   ├── customer-api/
        │   │   │   ├── main.tf
        │   │   │   └── terraform.tfvars
        │   └── data/
        │       ├── prod-db-01/
        │       │   ├── main.tf
        │       │   └── terraform.tfvars
        └── staging/
            ├── vpc/
            │   ├── main.tf
            │   └── terraform.tfvars
            └── services/
                ├── customer-api/
                │   ├── main.tf
                │   └── terraform.tfvars

Each bottom-level directory (e.g., `prod/vpc/`) is a self-contained Terraform root. It has its own provider configuration, its own backend for state, and its own `plan` and `apply` lifecycle. The blast radius is now microscopic. If I’m working in `staging/services/customer-api`, there is absolutely no way I can accidentally damage `prod/data/prod-db-01`.

A typical `main.tf` in one of these directories looks beautifully simple:

# live-infra/aws/us-east-1/prod/vpc/main.tf

terraform {
  # Remote state configuration for THIS component
  backend "s3" {
    bucket = "techresolve-tfstate-prod"
    key    = "us-east-1/vpc/terraform.tfstate"
    region = "us-east-1"
  }
}

provider "aws" {
  region = "us-east-1"
}

# Consuming our generic VPC module
module "vpc" {
  # We pin to a specific version for stability!
  source = "git::https://github.com/TechResolve/terraform-modules.git//aws/vpc?ref=v1.4.1"

  name       = "prod-vpc"
  cidr_block = "10.100.0.0/16"
  # ... other variables defined in prod/vpc/terraform.tfvars
}

Here’s a quick comparison of the approaches:

Solution 3: The ‘Hands-Off’ GitOps Pipeline (The ‘Nuclear’ Option)

Once you have the ‘Live vs. Modules’ structure, you can ascend to the final level: full GitOps automation. The principle here is simple: no human ever runs `terraform apply` from their laptop. Ever.

Our workflow is now entirely managed through pull requests in the `live-infra` repo:

1. PR Opened: A developer needs to change the instance size for the staging API. They change one line in a `.tfvars` file and open a Pull Request.
2. Automated Plan: Our CI/CD system (we use Atlantis, but GitHub Actions or Spacelift work too) automatically detects the change, runs `terraform plan` in the correct directory, and posts the output as a comment on the PR.
3. Peer Review: I, or another senior engineer, can review the code *and* the plan. I see exactly what will be added, changed, or destroyed. No surprises.
4. Apply on Merge: Once the PR is approved and merged into the `main` branch, the CI/CD system automatically runs `terraform apply`.

This enforces a perfect audit trail, mandatory peer review, and removes the “it worked on my machine” class of errors entirely. It’s a cultural shift, for sure. It forces everyone to be more deliberate, but the safety and stability it provides are unmatched.

Pro Tip: This approach is the ultimate form of infrastructure as code. Your Git history becomes the source of truth for your infrastructure’s history. Need to know why a firewall rule was opened on March 15th? Find the PR. It will have the code, the plan, the approval, and a link to the Jira ticket. It’s self-documenting.

Start Small, Iterate

If you’re staring at a monolithic Terraform repo right now, don’t panic. You don’t have to boil the ocean. Start by carving off one non-critical piece of infrastructure—maybe a monitoring tool or a dev VPC. Move it into a separate directory with its own state. Get comfortable with the process. Then, over time, keep chipping away at the monolith. Your future self, and your sleep schedule, will thank you.

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions