🚀 Executive Summary
TL;DR: Ghost features, which are unused services consuming resources and causing alerts, represent significant operational dead weight. The solution involves a structured approach to identify these features using observability data, temporarily mitigate their impact, and then formally decommission them through a phased plan with stakeholder buy-in.
🎯 Key Takeaways
- Immediately mitigate active ghost features by scaling deployments to zero/one or wrapping core logic in a feature flag to stop resource drain and alerts.
- Permanently decommission services by gathering inarguable observability data, initiating conversations with Product for buy-in, and executing a documented multi-phase deprecation plan.
- For undocumented or politically challenging dependencies, employ a ‘brownout’ strategy by temporarily disabling the service in production during low-traffic periods to identify critical, uncommunicated reliance.
Are you supporting a ‘ghost feature’ that consumes resources but provides no value? Learn how to identify, triage, and responsibly decommission unused services before they cause a production outage.
That “Critical” Feature Nobody Uses is Your Next PagerDuty Alert
It was 2:37 AM on a Tuesday. The on-call alert screamed about high latency in our “Dynamic Asset Optimization” service. This thing was my predecessor’s magnum opus—a beautiful, over-engineered beast that was supposed to resize images on the fly based on user bandwidth. I dove in, checking the pods on our EKS cluster, tracing dependencies, and finally found the culprit: a downstream metadata service, `meta-tag-api-prod-04`, was timing out. As I was digging through the logs to restart the pod, I noticed something weird. The only traffic hitting the Dynamic Asset Optimizer… was our own health checks. I Grep’d the access logs for the last 90 days. Nothing. Zero real user requests. We were spending thousands a month on compute and multi-region storage for a feature nobody was even calling. The alert wasn’t from a customer impact; it was from our own tooling telling us that the expensive thing nobody used was broken.
The “Why”: How We End Up Here
This isn’t about blaming anyone. It’s a classic story. A feature gets built based on a solid idea from the Product team. Engineering, proud of their work, delivers it. But then, the go-to-market strategy pivots, the client-side implementation gets delayed, or users just… don’t adopt it. The feature becomes a “ghost in the machine.” It still consumes CPU, memory, and database connections. It’s still part of our CI/CD pipeline, and it still has the power to wake you up in the middle of the night. It’s not just technical debt; it’s operational dead weight.
The root cause is almost always a broken feedback loop between Product, Engineering, and actual user behavior. We get so focused on shipping that we forget to validate if anyone is receiving.
The Fixes: From Band-Aid to Surgery
So, you’ve found a ghost feature. It’s costing you money and sleep. What do you do? Shouting “who approved this?!” into a Slack channel isn’t the answer. Here’s how we handle it at TechResolve.
1. The Quick Fix: The Mute Button
Your immediate goal is to stop the bleeding. You don’t want to rip the service out yet because you might break some obscure, undocumented dependency. The safest first step is to put it into a dormant state.
- Scale to Zero (or One): Take the service’s deployment configuration and scale its replica count down to the absolute minimum, maybe even zero if your service mesh can handle the routing gracefully. In Kubernetes, it’s a one-liner.
- Feature Flag It:** If you can, wrap the service’s core logic in a feature flag and turn it off. This is the least disruptive option. If one doesn’t exist, now’s a good time to add one.
# A quick 'kubectl' command to stop the resource drain
# Don't do this without telling anyone, but it's your emergency stop.
kubectl scale deployment/dynamic-asset-optimizer --replicas=0 -n prod
Warning: This is a temporary measure. You’ve stopped the immediate cost, but the dead code is still in your repository, waiting to cause confusion for the next junior engineer who stumbles upon it.
2. The Permanent Fix: The Detective Work
This is the “right” way to do it. It requires data, communication, and a plan. You need to prove the feature is unused and get formal buy-in to remove it. This isn’t just about deleting code; it’s about institutional memory and shared responsibility.
Step 1: Gather Inarguable Data. Use your observability tools. Build a dashboard in Grafana or Datadog showing traffic, error rates, and CPU/memory usage for the service over the last 6-12 months. An empty graph is the most powerful evidence you can present.
Step 2: Start the Conversation. Go to the Product Manager with your dashboard. Don’t be accusatory. Frame it as a cost-saving and risk-reduction effort. My go-to line is: “Hey, I was investigating our cloud spend and noticed the Dynamic Asset Optimizer service seems to be idle. Is this still part of our active roadmap, or can we plan a formal deprecation to simplify our stack?”
Step 3: Create a Decommissioning Plan. Once you have buy-in, document the plan. It’s not as scary as it sounds.
| Phase | Action | Timeline |
| Phase 1: Announce | Send an email/Slack message to engineering & product mailing lists announcing the deprecation. | Week 1 |
| Phase 2: Disable | Use a feature flag to disable the service. The code is still there, but unreachable. Monitor for side effects. | Weeks 2-3 |
| Phase 3: Remove | Delete the service code, CI/CD pipelines, and IaC definitions (Terraform/CloudFormation). | Week 4 |
3. The ‘Nuclear’ Option: The Controlled Demolition
Sometimes, the original stakeholders are gone. The Product Manager is new. No one knows what the service does, but everyone is too scared to approve its removal. In this rare case, you have to force the issue. This is risky and can burn political capital, so use it sparingly.
You schedule a “brownout.” You publicly announce that you will be disabling the service endpoint in a lower environment (like staging) for 48 hours. Then, a week later, you announce you’ll be doing the same in production during a low-traffic maintenance window. The message is simple:
Team,
As part of our ongoing efforts to optimize our infrastructure, we have identified the 'Dynamic Asset Optimizer' (endpoint: api.techresolve.com/assets/v1) as a candidate for decommissioning due to low usage.
The service will be temporarily disabled in production on Friday, Oct 27th, from 11:00 PM to 11:30 PM UTC.
If your service or workflow relies on this endpoint, please contact the Cloud Engineering team in #cloud-eng-support immediately. If no dependencies are reported, we will proceed with a full decommissioning plan.
Pro Tip: 99% of the time, you will hear absolute silence. That silence is your green light. The 1% of the time someone actually screams, you’ve found a critical undocumented dependency and just saved the company from a future self-inflicted outage. It’s a win either way, but be prepared for a tense conversation if that happens.
Ultimately, our job isn’t just to build and run systems. It’s to be good stewards of the infrastructure. That means cleaning up after ourselves, questioning assumptions, and having the courage to delete code. Your future on-call self will thank you for it.
🤖 Frequently Asked Questions
âť“ What is a ‘ghost feature’ in a technical context?
A ‘ghost feature’ is an over-engineered or unused service that consumes compute, memory, and database resources, remains in CI/CD pipelines, and can cause alerts, despite having zero real user traffic or adoption.
âť“ How does this approach compare to simply leaving unused services running?
Leaving unused services running incurs significant operational dead weight, including unnecessary cloud spend, increased complexity, and the risk of future production outages from non-critical components. This approach actively identifies and removes such services, reducing costs and improving system reliability.
âť“ What is a common pitfall when attempting to decommission a service?
A common pitfall is failing to gather sufficient data or communicate effectively with stakeholders, leading to resistance or unknowingly breaking undocumented dependencies. The solution is to use observability tools to prove non-usage and secure formal buy-in before proceeding with a structured decommissioning plan.
Leave a Reply