Stop running infrastructure agents as root. A senior DevOps lead breaks down battle-tested methods—from quick sudo fixes to container isolation—for deploying monitoring agents safely and effectively.

Don’t Give Your Agents Root: A Guide to Building and Running Infrastructure Agents Safely

I still get a cold sweat thinking about it. It was a Tuesday, around 2 PM. A junior engineer pushed an update to our fleet-wide monitoring agent config. A simple change, just adding a new log file path to watch. The agent, of course, was running as root because… well, because it always had. Ten minutes later, alerts started screaming. Latency was through the roof on our primary database cluster. Turns out, the new config had a regex typo. Instead of reading a log file, the agent’s log processor went into a manic loop, consuming 100% CPU on every single database node, effectively knocking them all offline. We spent the next four hours recovering. All because an agent, a simple little monitoring tool, had too much power. That day, we changed everything.

The “Why”: The Deadly Sin of Convenience

Let’s be honest. Why do we run agents as root? Because it’s easy. It solves all the “permission denied” errors when the agent needs to read /proc, inspect network sockets, or access protected log files. But giving an agent—especially a third-party, open-source one—unfettered root access is like giving a houseguest a master key to your entire life. It violates the Principle of Least Privilege, the absolute bedrock of sane infrastructure security. A single bug, a supply chain attack on the agent’s source, or a simple misconfiguration can lead to catastrophic failure, data exfiltration, or a complete system compromise.

The core problem is granting broad, powerful permissions to solve a few, very specific access needs. So, how do we give the agent what it needs without handing over the keys to the kingdom? Here are three approaches, from a quick patch to a full architectural shift.

Solution 1: The Quick Fix (The “Sudo-and-Pray” Method)

This is the first step away from running the process directly as root. It’s not perfect, but it’s a massive improvement and can be implemented quickly to stop the immediate bleeding.

First, we create a dedicated, unprivileged service user for our agent.

# Create a system user with no password and no shell
sudo useradd --system --no-create-home --shell /bin/false infra-agent

Next, we use the visudo command to give this new user passwordless access to only the specific commands it needs to run with elevated privileges. This is the critical part. Don’t give it full root; be surgical.

# /etc/sudoers.d/90-infra-agent
# Allow the infra-agent user to run specific diagnostic commands as root
infra-agent ALL=(root) NOPASSWD: /usr/sbin/smartctl, /usr/bin/read-kernel-log.sh

Your agent’s startup script or systemd service file will then run as the infra-agent user, calling sudo for the specific commands you’ve allowed. It’s hacky, and a vulnerability in the agent could still allow an attacker to run those whitelisted commands, but you’ve dramatically shrunk the attack surface.

Warning: The ‘Quick Fix’ is a gateway drug. It feels easy, but it can lead to a messy and hard-to-maintain sudoers file. Use it to stabilize the situation, but immediately start planning your move to a more permanent solution.

Solution 2: The Permanent Fix (The “Linux Capabilities” Approach)

This is the “grown-up” solution for running on bare-metal or traditional VMs. Instead of the all-or-nothing power of root, Linux Capabilities allow you to grant specific kernel-level privileges to an executable file. Think of it as a set of fine-grained permissions.

For example, if your agent needs to sniff network packets (like tcpdump does), it needs the CAP_NET_RAW capability. If it needs to bind to a privileged port below 1024, it needs CAP_NET_BIND_SERVICE. You grant these directly to the agent’s binary.

# Grant the agent binary the ability to capture network traffic and view admin stats
sudo setcap cap_net_raw,cap_net_admin+eip /usr/local/bin/my-awesome-agent

Now, you can run the agent process as your unprivileged infra-agent user, and it will magically have the specific kernel permissions it needs, without having any other root-level powers. It can’t write to arbitrary files or install software. It can only do what its capabilities allow.

Pro Tip: Combine this with an AppArmor or SELinux profile for maximum security. This lets you define exactly which files the agent is allowed to read and write, providing an extra layer of defense in case the binary itself is compromised.

Solution 3: The ‘Nuclear’ Option (The “Containerized & Isolated” Method)

When you need the highest level of isolation, especially in a cloud-native environment, run the agent in its own hardened, minimal container. This treats the agent as a potentially hostile piece of software and locks it down accordingly.

The idea is to use container security features to limit the agent’s blast radius. Even if the agent is fully compromised, the attacker is trapped inside a heavily restricted container.

Here’s a conceptual docker run command showing some of the key principles:

docker run \
    --name my-secure-agent \
    --detach \
    --pid=host \
    --network=host \
    --read-only \
    --user 1001:1001 \
    --security-opt no-new-privileges \
    --cap-drop ALL \
    --cap-add SYS_TIME \
    --cap-add NET_RAW \
    -v /proc:/host/proc:ro \
    -v /sys:/host/sys:ro \
    your-agent-image:latest

What’s happening here? We are:

• Dropping all Linux capabilities and then adding back only the few that are absolutely necessary.
• Running the container with a read-only root filesystem (--read-only).
• Mounting necessary host paths like /proc and /sys as read-only so the agent can gather metrics but not change anything.
• Preventing privilege escalation inside the container (--security-opt no-new-privileges).

This method is more complex to set up and manage, but it provides unparalleled security and isolation. It’s the standard for modern, security-conscious environments.

Which Solution Is Right For You?

There’s no excuse. The days of casually running things as root are over. It’s not about being a security paranoid; it’s about being a professional. Start with the quick fix if you have to, but make a plan to implement a proper, secure solution. Your future self—at 2 AM during an outage—will thank you.

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions