Azure Application Insights is powerful, but its default settings can lead to surprise bills and security holes by logging sensitive data. This guide provides a senior engineer’s real-world fixes for taming costs and securing your telemetry data.

That “Tiny” Service Just Cost Us $5k: A Senior Engineer’s Guide to Application Insights Risks

I still remember the PagerDuty alert that woke me up at 2 AM on a Saturday. My heart sank, expecting a full-blown production outage. But the alert wasn’t for `prod-api-gateway` being down. It was a custom Azure budget alert: “Project Phoenix Cost Anomaly – 500% over budget”. My first thought was a crypto-mining breach. The reality was almost worse. A junior engineer had deployed a new, seemingly harmless microservice—the `user-auth-token-refresher`—with default Application Insights settings. A misconfigured upstream dependency was causing it to fail and retry in a tight loop. By Saturday morning, it had logged several hundred gigabytes of exception telemetry, and our projected Azure bill for the month was heading into orbit. That Monday morning meeting with my director was… tense.

The “Why”: App Insights is an Overeager Intern

The problem isn’t that Application Insights is bad; it’s that it’s too good out of the box. By default, its goal is to capture everything. Every single request, every dependency call, every trace, every exception. For a simple “Hello World” app, that’s fine. For a high-traffic production service like our `auth-api-prod-01`, it’s a recipe for disaster. This “collect-it-all” firehose approach leads to two core risks:

• Cost Overruns: You pay per gigabyte ingested. A chatty service, or one stuck in an exception loop, can generate terabytes of data unexpectedly. The costs are not linear; they’re exponential when things go wrong.
• Security Leaks: What if your URLs contain sensitive data? Think `GET /api/v1/users/reset?token=SUPER_SECRET_TOKEN` or `POST /api/v1/orders/123?customer_email=darian.vance@techresolve.com`. By default, App Insights happily logs that entire URL, sending potential PII or secrets straight into your logs where they can live for 90+ days.

The root cause is simple: the default configuration prioritizes data completeness over cost and security. It’s on us, the engineers, to rein it in.

The Fixes: From Band-Aids to Surgery

Over the years, we’ve developed a three-tiered approach to taming Application Insights. We start with the quickest fix to stop the bleeding and move toward a robust, permanent solution.

1. The Quick Fix: Turn on Adaptive Sampling

This is the first thing you should do when you see costs spiraling. Adaptive sampling is an intelligent feature that automatically adjusts the volume of telemetry it sends from the SDK. It tries to maintain a target rate of events per second, dropping telemetry if your app gets too noisy. It’s smart enough to preserve important events like exceptions while dropping repetitive ones.

How to do it (ASP.NET):
You can enable it right in your ApplicationInsights.config file. It’s often there but commented out.

<TelemetryProcessors>
  <Add Type="Microsoft.ApplicationInsights.WindowsServer.TelemetryChannel.AdaptiveSamplingTelemetryProcessor, Microsoft.AI.ServerTelemetryChannel">
    <MaxTelemetryItemsPerSecond>5</MaxTelemetryItemsPerSecond>
    <ExcludedTypes>Event</ExcludedTypes>
  </Add>
</TelemetryProcessors>

Pro Tip: Adaptive Sampling is great for stopping a hemorrhage, but it can be unpredictable. The volume of data you get will fluctuate with your traffic, which can make long-term cost prediction tricky. It’s a fantastic first step, but not the final one.

2. The Permanent Fix: Fixed-Rate Sampling & Telemetry Initializers

This is the approach we enforce for all production services at TechResolve. It’s a two-part strategy that gives you predictable costs and fine-grained control over data security.

Part A: Fixed-Rate Sampling
Instead of letting an algorithm decide, we decide. We set a hard percentage of telemetry to keep. For most services, sampling at 10% or even 5% is more than enough to get a statistically significant view of performance. The key is that the data volume becomes a predictable function of your traffic.

Part B: Telemetry Initializers (The Scalpel)
This is where we solve the security problem. A telemetry initializer is a piece of code that intercepts every single telemetry item right before it’s sent. Here, we can inspect it and modify it. It’s the perfect place to scrub PII from URLs, remove sensitive headers, or redact custom properties.

Here’s a simple example in C# that removes an “email” query string parameter from all logged requests:

using Microsoft.ApplicationInsights.Channel;
using Microsoft.ApplicationInsights.DataContracts;
using Microsoft.ApplicationInsights.Extensibility;
using System.Web;

public class PiiScrubberTelemetryInitializer : ITelemetryInitializer
{
    public void Initialize(ITelemetry telemetry)
    {
        if (telemetry is RequestTelemetry requestTelemetry)
        {
            var uriBuilder = new UriBuilder(requestTelemetry.Url);
            var query = HttpUtility.ParseQueryString(uriBuilder.Query);

            if (query["email"] != null)
            {
                query["email"] = "[REDACTED]";
                uriBuilder.Query = query.ToString();
                requestTelemetry.Url = uriBuilder.Uri;
            }
        }
    }
}

Warning: Be careful what you scrub. Accidentally removing a critical transaction ID can make debugging impossible. We once had a team redact a `session_id` field that was crucial for tracing user journeys. Always scrub with a scalpel, not a sledgehammer.

3. The ‘Nuclear’ Option: Daily Cap & Ingestion Filtering

Sometimes, you need a failsafe. This is your last line of defense against a catastrophic bill.

The Daily Cap:
In the Azure Portal, under your Application Insights resource > Usage and estimated costs > Daily cap, you can set a hard limit on the amount of data ingested per day (e.g., 100 GB/day). Once that limit is hit, Azure stops processing any more data until the next UTC day. It’s a blunt instrument, but it guarantees your bill will never exceed a certain amount.

Ingestion-Time Filtering with Data Collection Rules (DCRs):
This is a more modern and powerful approach. Instead of filtering at the SDK (client-side), you can use a Data Collection Rule in Azure to apply a KQL transformation to filter data *as it arrives at the ingestion endpoint*. This means the unwanted data is dropped before you even pay to ingest it. For example, you could write a KQL query to drop all trace messages with a “Verbose” severity level coming from your dev environments.

Choosing Your Strategy

Here’s a quick breakdown of when to use each approach:

In the end, Application Insights is an indispensable tool, but it’s not a “set it and forget it” service. Like any powerful instrument, it requires skill and discipline to use effectively. Take the time to configure it properly. Your on-call self, and your finance department, will thank you for it.

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions