When an AWS Availability Zone goes down, your multi-AZ architecture is put to the test. Here’s a Senior DevOps Engineer’s real-world playbook for surviving an outage, from quick hacks to long-term resilience.

The Pager Woke Me at 3 AM: A DevOps Playbook for When an AWS AZ Goes Dark

I remember it like it was yesterday. The alert was for our primary PostgreSQL database, prod-db-master-01. Unreachable. My heart sank. That instance handled all our user authentication and billing transactions. I couldn’t SSH in, the AWS console was hanging trying to get its status, and reboot commands were timing out. A junior engineer on my team was frantically checking security groups, convinced we’d pushed a bad rule. But then the second pager alert hit. Then the third. A Redis cache, a fleet of web servers… they were all dark. And they all had one thing in common: they were all running in ap-southeast-2a. That’s when you realize this isn’t your bug. This is bigger. An entire slice of the cloud just vanished, and it’s your job to put the pieces back together before the CEO starts calling.

So, What Actually Happened?

The recent incident in me-central-1, where the mec1-az2 Availability Zone went down due to a power outage and fire, is a perfect, terrifying reminder of a fundamental truth: an Availability Zone (AZ) is just one or more data centers. And data centers, for all their redundancy, are still physical buildings. They can lose power, they can have network links severed, and yes, they can catch on fire. When you place a resource in a single AZ, you are accepting the risk of that entire physical location failing.

AWS’s Shared Responsibility Model is key here. They are responsible for the resilience of the cloud (the physical data centers). You are responsible for building resilient applications in the cloud. Simply launching an EC2 instance and hoping for the best isn’t a strategy. Let’s walk through how we handle this in the real world, from the desperate scramble to the architecturally sound solution.

Solution 1: The ‘Get It Working NOW’ Hack

Let’s say you have a single, critical legacy server, prod-legacy-reports-01, running in the failed AZ. It’s not in an Auto Scaling Group, and it has a dedicated EBS volume with critical state. The business is screaming. This is the break-glass procedure.

The Goal: Manually resurrect the server’s disk in a healthy AZ.

1. Acknowledge the Failure: First, accept that the instance is probably gone. The console will likely show it as “running” but with status checks failing. Trying to stop or reboot it will probably time out. You might have to use a “Force Stop”.
2. Snapshot the Disk: Go to the EBS Volumes section. Find the volume attached to your dead instance. Even if the instance is non-responsive, you can usually still create a snapshot. This is your lifeline.
3. Create a New Volume: Once the snapshot is complete, create a new volume from that snapshot. Crucially, make sure you create it in a healthy AZ, like mec1-az1.
4. Launch a Replacement Instance: Spin up a new EC2 instance (of the same type) in that same healthy AZ.
5. Attach and Mount: Once the new instance is running, detach the root volume if needed and attach your newly created EBS volume. SSH into the new instance and mount the drive. You may need to deal with network interface or IP address changes.
6. Update DNS: If you were pointing a DNS record at the old IP, update it to point to the new instance’s IP address.

Warning: This is a dirty, manual process. You will have downtime, and any data written to the disk in the minutes before the failure that hadn’t been flushed from memory might be lost. This is a last resort, not a strategy.

Solution 2: The Permanent Fix (The Way It Should Be)

This isn’t a fix for an ongoing outage; this is the architecture you should have had in the first place to ensure the outage is a non-event. This is how we build all our modern, critical services at TechResolve.

The Goal: Build an application that can automatically withstand an entire AZ failure with minimal to no impact.

• Use an Application Load Balancer (ALB): Your ALB should be configured to distribute traffic across multiple AZs (e.g., mec1-az1, mec1-az2, and mec1-az3).
• Use an Auto Scaling Group (ASG): Your EC2 instances should be managed by an ASG, also configured to span the same multiple AZs. The ASG’s job is to maintain a desired number of healthy instances.
• Use Multi-AZ Databases: For your database (like RDS), enable the “Multi-AZ” option. This creates a synchronous standby replica in a different AZ. If the primary AZ fails, RDS automatically fails over to the standby. The endpoint address remains the same; your application doesn’t even need to know it happened.

When mec1-az2 goes down in this scenario:

1. The ALB’s health checks fail for all instances in mec1-az2. It immediately stops sending traffic to them. Your users don’t notice.
2. The ASG sees that the instances in mec1-az2 are unhealthy. It terminates them and launches new replacement instances in the remaining healthy AZs (mec1-az1 and mec1-az3) to meet its capacity requirements.
3. Your RDS instance automatically fails over to its standby in another AZ in a minute or two.

The result? A blip. A few elevated error metrics, an automatic recovery, and you get to go back to sleep.

Solution 3: The ‘Nuclear’ Option (Region Evacuation)

Sometimes an AZ failure can have knock-on effects, or, in a truly catastrophic scenario, an entire region could become impaired. This is rare, but for tier-0 services, you need a plan. This is your Disaster Recovery (DR) strategy.

The Goal: Fail over your entire service to a different AWS region.

This is complex and expensive, but the core tool is Amazon Route 53. Here’s a simplified view:

# Simplified Terraform-like concept for Route 53 Failover

resource "aws_route53_record" "primary" {
  name    = "api.mycompany.com"
  type    = "A"
  alias {
    name    = aws_lb.me_central_1_alb.dns_name
    zone_id = aws_lb.me_central_1_alb.zone_id
  }
  failover_routing_policy {
    type = "PRIMARY"
  }
  health_check_id = aws_route53_health_check.primary_health_check.id
}

resource "aws_route53_record" "secondary_dr" {
  name    = "api.mycompany.com"
  type    = "A"
  alias {
    name    = aws_lb.eu_west_1_alb.dns_name
    zone_id = aws_lb.eu_west_1_alb.zone_id
  }
  failover_routing_policy {
    type = "SECONDARY"
  }
}

In this setup, a Route 53 Health Check is monitoring an endpoint in your primary region (me-central-1). If that health check fails for a sustained period, Route 53 will automatically stop resolving your DNS to the primary load balancer and start sending all traffic to your DR stack in another region (e.g., eu-west-1). This requires you to have already replicated your data (using S3 Cross-Region Replication, Aurora Global Databases, etc.) and have at least a “pilot light” version of your infrastructure ready to be scaled up in the DR region.

Pro Tip: Don’t try to build a regional failover plan during a real disaster. This needs to be planned, architected, and tested regularly. It’s a significant undertaking.

Comparing The Approaches

Final Thoughts

The me-central-1 outage is a lesson that we in DevOps and Cloud Architecture need to constantly relearn: hope is not a strategy. We can’t just assume the cloud will take care of us. We have to use the tools AWS gives us to build for failure. If you’re running single-AZ workloads for anything important, your pager is going to go off at 3 AM one day. It’s not a matter of if, but when.

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions