A well-intentioned cost optimization on a CI/CD server backfired, slowing down development and costing more in lost productivity. Here’s how to avoid this common but painful mistake and fix it properly.

The Silent Killer of Productivity: When Cost Optimization Goes Horribly Wrong

I still remember the 7 AM Slack message. It was from our lead front-end dev, and it just said, “Darian, builds are broken.” That’s the kind of message that makes a DevOps engineer’s stomach drop. I grabbed my coffee, logged in, and saw a sea of red in our GitLab pipeline. Every single merge request was stuck, jobs timing out after an hour. I shelled into our primary build runner, `ci-runner-prod-01`, and ran `htop`. The CPU cores were completely pegged, a solid wall of 100% utilization. I didn’t even have to check the audit logs. I knew, with the certainty of a man who has seen this a dozen times, that someone had tried to be a hero. Someone saw a “low average CPU” metric in CloudWatch and decided to “optimize” our `c5.2xlarge` instance down to a `t3.large` to save a few hundred bucks a month. A classic, painful, and completely avoidable own-goal.

The Real Problem: Optimizing Without Context

Look, I get it. The pressure from finance to cut the cloud bill is relentless. A junior engineer sees a dashboard showing an average CPU utilization of 15% over a 24-hour period and thinks, “That’s wasted money!” They aren’t wrong, but they’re not right, either. They’re missing the most critical piece of the puzzle: context.

That build server isn’t meant to have high *average* utilization. Its job is to be mostly idle, then burst to 100% CPU and memory for 10-15 minutes to compile a massive React project or run a complex integration test suite, and do it *fast*. When you average those intense bursts over a full day of near-idleness, you get a misleadingly low number. Optimizing for the average kills the peak performance, and for a CI/CD system, peak performance is the only thing that matters. You’re not saving money; you’re trading cheap compute for expensive developer time. A team of ten engineers waiting an extra 30 minutes for a build costs the company way more in salary than that `c5.2xlarge` instance ever will.

Pro Tip: Never trust average utilization metrics for bursty workloads like CI/CD runners, batch processing jobs, or even some databases. Look at peak metrics (p95, p99, or max) to understand the real performance requirements.

How to Fix This Mess (And Never Make It Again)

So you’re here. The builds are broken, developers are grabbing torches and pitchforks, and you need to fix it. Here’s my playbook, from the immediate triage to the long-term architectural fix.

Solution 1: The Quick Fix (The ‘Oh Crap’ Button)

Stop the bleeding. Right now. Your goal is not to be clever; it’s to get the team working again. Go into the AWS console, find that instance, and resize it back to its original, more powerful instance type. Yes, it feels like admitting defeat, but it’s the right call.

1. Log in to the AWS Console.
2. Navigate to EC2 > Instances.
3. Find the instance (e.g., `ci-runner-prod-01`).
4. Stop the instance.
5. Go to Actions > Instance Settings > Change instance type.
6. Select the original, larger instance type (e.g., `c5.2xlarge`).
7. Start the instance.

Send a message to the engineering team: “Apologies everyone, a misconfigured change impacted the build runners. I’ve reverted it, and pipelines should be returning to normal now.” Take the heat, get the team unblocked, and then focus on the real solution.

Solution 2: The Permanent Fix (The Architect’s Approach)

A single, oversized, static build server is a relic of the on-prem world. In the cloud, we can do better. The right way to handle bursty CI workloads is with ephemeral, auto-scaling runners. You pay for compute *only when a build is actually running*.

The idea is to have a small manager node that farms out jobs to containerized runners or spot instances that are created on-demand and destroyed when the job is done. This gives you the best of both worlds: insane scalability for peak demand and near-zero cost when idle.

Here’s a conceptual Terraform snippet for creating an Auto Scaling Group for GitLab runners on AWS. This isn’t production-ready, but it shows the core concept:

resource "aws_launch_template" "gitlab_runner_template" {
  name_prefix   = "gitlab-runner-"
  image_id      = "ami-0c55b159cbfafe1f0" # Amazon Linux 2 AMI
  instance_type = "c5.large" # Start with a reasonable size, spot is great here

  user_data = base64encode(<<EOF
#!/bin/bash
# Script to install Docker, GitLab Runner, and register it
# ... (your runner installation script here)
EOF
  )
}

resource "aws_autoscaling_group" "gitlab_runners" {
  name                = "gitlab-runner-asg"
  desired_capacity    = 1
  max_size            = 10 # Scale up to 10 runners during peak times
  min_size            = 0  # Scale down to ZERO at night/weekends
  launch_template {
    id      = aws_launch_template.gitlab_runner_template.id
    version = "$Latest"
  }

  # Add your VPC subnets, security groups, etc.
}

This approach (whether with GitLab, Jenkins, GitHub Actions, or CodeBuild) is the definitive solution. It aligns cost directly with usage.

Solution 3: The ‘Nuclear’ Option (The Process Fix)

The root cause here wasn’t just a technical mistake; it was a process failure. A junior engineer shouldn’t be able to unilaterally change the instance type of a critical piece of infrastructure from a web console.

This is where you lock it down.

• Infrastructure as Code (IaC) is Mandatory: All changes must go through a pull request. No more console cowboys. Use Terraform, Pulumi, or CloudFormation. The PR process provides visibility, requires peer review, and creates an audit trail.
• Implement Guardrails: Use AWS Service Control Policies (SCPs) at the Organization level or strict IAM policies to prevent direct modification of resources tagged as `critical` or `ci-cd`. For example, you can create a policy that denies the `ec2:ModifyInstanceAttribute` action on specific resources unless the request comes from a specific IaC role.
• Cost Tooling in CI: Integrate tools like Infracost into your CI pipeline. When an engineer opens a PR to change an instance from a `c5.2xlarge` to a `t3.large`, Infracost will post a comment saying, “This change will save you $250/month.” This forces a conscious decision and discussion about the trade-off.

Warning: Be careful not to create a culture of fear. The goal of process is not to punish people for making mistakes, but to make it harder for those mistakes to happen in the first place. Mentor the junior engineer, explain the *why* behind the process, and empower them to make better decisions next time.

Comparing the Solutions

In the end, we used all three. We reverted the change immediately (Fix 1). Then, over the next sprint, we planned and executed a migration to a Kubernetes-based runner system (Fix 2). And finally, we tightened our IAM policies and made IaC PRs the only way to modify core infrastructure (Fix 3). The junior who made the change? He helped build the new system, and now he’s one of the sharpest people I know when it comes to balancing performance and cost.

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions