🚀 Executive Summary
TL;DR: Organizational silos between development, operations, and marketing lead to features and campaigns that are out of touch with users and cause production outages. The core problem is a lack of empathy and understanding of downstream impact, which can be solved by building systems that force cross-functional conversations and foster shared ownership.
🎯 Key Takeaways
- Implementing a Pre-Flight Checklist acts as a forcing function to make invisible operational concerns (e.g., database index performance, monitoring dashboards) visible and prompt necessary conversations before deployment.
- The Operational Readiness Review (ORR) is a formal, recurring meeting that systematically builds organizational empathy by requiring teams to present changes to diverse stakeholders, addressing monitoring, resiliency, and cost early in the development cycle.
- The ‘You Build It, You Run It’ (YBIYRI) model dissolves traditional dev/ops separation, making the code-writing team responsible for carrying the pager, thereby creating a powerful, immediate feedback loop for writing resilient, observable, and efficient code.
A breakdown of why teams ship features (or marketing campaigns) that are completely out of touch with their audience, and three practical, in-the-trenches fixes to bridge the gap between building and running software.
That McDonalds Ad, A Database Fire, and Why Your Team Is Shipping Blind
I remember getting the page at 2:17 AM. High latency, database connections maxed out, the whole nine yards. A new “user profile recommendations” feature had just been rolled out. On paper, it was great. In staging, it worked fine. But in production, the query it ran was so horribly inefficient at scale that it was effectively a denial-of-service attack against our primary database, prod-db-01. The dev team that built it was brilliant, but they saw the world in terms of features and code. We in Ops saw it in terms of query plans and I/O. They shipped a feature; we received an outage. It’s the exact same disconnect I saw when I read that Reddit thread about the McDonalds “product” post. A team, completely siloed, built something without a shred of empathy or understanding for the people who would actually consume it.
The Root Cause: The Silo and The Wall
This isn’t about blaming marketing, or developers, or anyone else. The problem is organizational. It’s the “Wall of Confusion” we’ve been talking about in DevOps for over a decade. One team’s “finished product” is another team’s “problem to operate.”
When a product team works in isolation, they optimize for their own metrics: features shipped, user stories closed, marketing copy approved. They don’t see the downstream impact:
- Increased CPU load on the API gateways.
- Log spam that makes debugging a nightmare.
- A marketing message that makes the brand look foolish and disconnected.
The McDonalds team thought they were being clever and meta. But the audience, the actual “end users,” just saw a confusing, low-effort post that didn’t offer them anything. The team shipped their “product” but failed to deliver any value. It’s the same as my 2 AM database fire. A feature was shipped, but the value (a working application) was destroyed.
Fixing The Disconnect: Three Levels of Intervention
So, how do we fix it? You can’t just tell people to “collaborate more.” You need to build systems that force the right conversations to happen. Here are three ways to do it, from a quick patch to a full re-org.
1. The Quick Fix: The Pre-Flight Checklist
This is the fastest, albeit most superficial, way to start. You create a mandatory checklist that must be completed and signed off on before any new feature, service, or major marketing campaign goes live. It’s a forcing function for a conversation.
It’s not about bureaucracy; it’s about making the invisible visible. The dev team might not even know they should be thinking about database index performance. This makes them ask.
A simple version might look something like this in a pull request template:
# Pre-Launch Go/No-Go Checklist
## Operational Readiness
- [ ] Have we load-tested the new database queries? (Link to report)
- [ ] Is there a Grafana dashboard for this new feature? (Link to dashboard)
- [ ] Has the on-call engineer reviewed the runbook for this service? (@darian.vance)
## Dependency Review
- [ ] Does this change impact any downstream services? (e.g., billing, auth)
- [ ] Have the owners of those services signed off? (@jane.doe)
## Rollback Plan
- [ ] Is there a documented, one-click rollback procedure?
- [ ] Under what conditions will we trigger a rollback? (e.g., p99 latency > 500ms)
Warning: Be careful this doesn’t become a “check the box” exercise. The goal is the conversation that the checklist forces, not the checklist itself. If people are just ticking boxes without thinking, the fix has failed.
2. The Permanent Fix: The Operational Readiness Review (ORR)
This is a more formal, scalable version of the checklist. The ORR is a recurring meeting where teams present upcoming changes to a board of stakeholders from across the company—SRE/Ops, Security, Product, even Legal or Marketing where appropriate. It’s not a gate; it’s a review.
The presenting team has to answer the hard questions:
- How will we know this is working? (Monitoring & Alerting)
- What happens when it breaks? (Resiliency & Rollback)
- How much will this cost to run in the cloud? (FinOps)
- What’s the blast radius if this fails? (System Architecture)
This process systematically builds organizational empathy. After a few ORRs, developers start building monitoring into their features from day one because they know they’ll be asked about it. They start thinking about failure modes because they had to explain them last time. It moves the operational thinking “left,” earlier in the development cycle.
3. The ‘Nuclear’ Option: You Build It, You Run It (YBIYRI)
This is the most culturally disruptive but ultimately most effective fix. You dissolve the traditional separation between “dev” and “ops.” The team that writes the code is the same team that carries the pager for it at 3 AM.
There is no more powerful incentive for writing resilient, observable, and efficient code than knowing you’re the one who will be woken up when it breaks. The feedback loop is immediate and painful. That team that shipped the bad query in my story? If they had been on call for it, I guarantee you they never would have made that same mistake again.
This requires a massive investment in tooling, training, and automation. You can’t just hand a pager to a developer and say “good luck.” You need to give them platforms, paved roads, and the autonomy to actually fix their own problems.
| Solution | Implementation Effort | Impact | Risk |
|---|---|---|---|
| Pre-Flight Checklist | Low | Medium | Can become a meaningless tick-box exercise. |
| Operational Readiness Review | Medium | High | Can become a bureaucratic bottleneck if not run well. |
| You Build It, You Run It | Very High | Transformational | High risk of team burnout if implemented poorly without proper support. |
That McDonalds post was a symptom of a team that didn’t know its customer. The outages I’ve fought are symptoms of teams that don’t know their systems. The root cause is the same: a lack of ownership and empathy created by organizational silos. Whether you use a checklist, a meeting, or a re-org, the goal is to tear down that wall and get people to understand the real-world impact of what they’re building.
🤖 Frequently Asked Questions
❓ Why do development teams often ship features that cause production issues or fail to meet user expectations?
The root cause is organizational silos, creating a ‘Wall of Confusion’ where teams optimize for their own metrics (features shipped) without understanding downstream impacts like increased CPU load, log spam, or disconnected marketing messages, leading to a lack of empathy for the end-user or operational burden.
❓ How do the Pre-Flight Checklist, Operational Readiness Review (ORR), and You Build It, You Run It (YBIYRI) approaches compare?
The Pre-Flight Checklist is a low-effort, quick fix with medium impact, risking becoming a tick-box exercise. The ORR is a medium-effort, high-impact formal review that can become a bureaucratic bottleneck if poorly run. YBIYRI is a very high-effort, transformational re-org with the highest impact but also high risk of team burnout if implemented without proper support.
❓ What is a common implementation pitfall for a Pre-Flight Checklist and how can it be avoided?
A common pitfall is the checklist becoming a ‘check the box’ exercise without genuine thought or conversation. To avoid this, the focus must remain on the conversation the checklist forces, ensuring teams actively engage with the questions and understand the implications, rather than just ticking boxes for compliance.
Leave a Reply