🚀 Executive Summary
TL;DR: A “passive income” script on a staging cluster consumed excessive egress bandwidth and CPU, disrupting internal services. The issue was resolved by immediately terminating the rogue process and then implementing strict Kubernetes Network Policies and Resource Quotas to prevent future unauthorized external communication and resource starvation.
🎯 Key Takeaways
- Unrestricted egress and loose RBAC in staging environments facilitate “shadow IT” and resource abuse.
- Immediate remediation for rogue Kubernetes pods involves using `kubectl top pods` to identify and `kubectl delete pod –grace-period=0 –force` to terminate.
- Preventative measures include Kubernetes Network Policies to deny external egress by default and Resource Quotas to cap CPU/memory usage per deployment.
- For compromised nodes, the “cattle not pets” approach dictates cordoning, draining, and terminating the instance to ensure a clean, fresh replacement.
A rogue background process spawned from a “passive income” script nearly capped our egress bandwidth—here is how we identified the shadow workload and locked down the cluster constraints to prevent it from happening again.
Post-Mortem: When a “Passive Income” Script Took Down stage-cluster-02
I was halfway through my second espresso when the PagerDuty alert screamed at me. It wasn’t the usual memory leak in prod-db-01 or a hung process in the payment gateway. No, this was an egress bandwidth spike on stage-worker-04 that looked like we were streaming 4K video to half of the planet. I logged in, ran top, and saw a headless Chrome instance chewing through CPU cycles like they were free candy. Turns out, one of our juniors read a “Complete Beginner-Friendly Guide to Survey Affiliates” on Reddit and decided our idle staging environment was the perfect place to host his Selenium automation farm. He thought he was being clever optimizing his side-hustle; instead, he accidentally DDoSed our internal monitoring service with referral pings.
The “Why”: Unrestricted Egress and Shadow IT
The root cause wasn’t just a junior engineer trying to make a quick buck. It was a failure in our Network Policies. We treat Staging like the Wild West—loose RBAC, unrestricted internet access for “npm installs,” and zero egress filtering.
The script he ran wasn’t malicious malware, but it behaved like it. It spun up dozens of threads to poll affiliate APIs, checking for new surveys every 500ms. Without resource quotas, his “get rich quick” container starved the actual business logic services (like auth-service-01) of CPU, causing a cascade of timeouts.
The Quick Fix: The “Kill -9” Approach
When the house is on fire, you don’t debate fire safety codes—you grab the hose. The immediate solution was to identify the rogue namespace and terminate the pods. We used kubectl to find the highest consumers and hard-killed them to recover bandwidth for the team.
# Identify the resource hog
kubectl top pods --all-namespaces --sort-by=cpu
# Output revealed the culprit:
# default survey-bot-x9z2 3800m 512Mi
# Terminate immediately
kubectl delete pod survey-bot-x9z2 --grace-period=0 --forcePro Tip: Always use
--grace-period=0 --forcewhen you suspect a process is hung or actively hostile to the node stability. Don’t wait for it to clean up.
The Permanent Fix: Network Policies & Resource Quotas
We can’t rely on trust alone. To prevent stage-db-01 or any worker node from being used as a personal botnet again, we need to apply strict NetworkPolicies that deny internet access by default, whitelisting only necessary package repositories and APIs.
We also implemented a ResourceQuota on the default namespace to cap how much CPU a single deployment can request.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: deny-external-egress
namespace: default
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
name: kube-system
# Allow internal DNS only, block everything else by default
ports:
- protocol: UDP
port: 53The “Nuclear” Option: Tainting and Cordoning
Sometimes a node is just too far gone. In this case, stage-worker-04 had cached so much garbage data in the ephemeral storage that disk pressure warnings were flashing. The cleanest way to ensure no residual scripts or cached credentials remained was to drain the node and terminate the instance entirely.
| Step 1: Cordon | Mark the node as unschedulable to prevent new pods from landing there. |
| Step 2: Drain | Evict all existing valid workloads to other healthy nodes. |
| Step 3: Terminate | Kill the instance at the cloud provider level (AWS/GCP). |
# Evacuate the node
kubectl drain stage-worker-04 --ignore-daemonsets --delete-emptydir-data
# Delete the node object (Let the Auto Scaling Group replace it with a fresh server)
kubectl delete node stage-worker-04It might seem harsh to nuke a server because of a survey bot, but in DevOps, cattle—not pets. If a node acts weird, replace it. And maybe tell your juniors to keep their affiliate schemes on their home Raspberry Pi.
🤖 Frequently Asked Questions
âť“ How can I prevent unauthorized egress traffic from my Kubernetes pods?
Implement a Kubernetes NetworkPolicy like `deny-external-egress` that denies all external egress by default, only whitelisting essential services such as internal DNS (UDP port 53 to `kube-system` namespace).
âť“ How does immediate pod termination (kill -9) compare to other methods?
The “kill -9” approach (`kubectl delete pod –grace-period=0 –force`) is for immediate, forceful termination of actively hostile or hung processes. More graceful alternatives, like scaling down deployments or updating images, allow for proper shutdown but are unsuitable for critical, immediate resource recovery.
âť“ What is a common pitfall when implementing Network Policies for egress?
A common pitfall is over-restricting egress, which can inadvertently block legitimate services requiring external access (e.g., package repositories, external APIs). Careful whitelisting of necessary endpoints is crucial to avoid breaking functionality.
Leave a Reply