Monitor Datadog Custom Metrics Count to avoid Overage

Hey team, Darian here. Let’s talk about something that used to give me a headache at the end of every month: the Datadog bill. Specifically, the part related to custom metrics. We all love detailed observability, but an accidental metric tag explosion can lead to some serious, and surprising, overage charges.

For a while, I was manually checking the usage page, but that’s reactive and, frankly, a waste of time. I realized I could use the Datadog API itself to monitor our usage proactively. This little script I’m about to show you saved me from that manual chore and gave our team an early warning system. It takes about 20 minutes to set up and provides peace of mind that’s well worth it.

Prerequisites

Before we dive in, make sure you have the following ready:

• A Datadog account with Administrator or a role with permissions to manage API/App keys and create monitors.
• Your Datadog API Key and Application Key.
• Python 3.x installed on a machine where you can run scheduled scripts.

The Guide: Step-by-Step

Step 1: Setting up your Environment

First, you’ll need a place for our script to live. I’ll skip the standard virtualenv setup since you likely have your own workflow for that. The important part is to create an isolated environment for our project.

Once your environment is active, you’ll need a couple of Python libraries. You can install them using pip: `pip install datadog-api-client python-dotenv`. The `datadog-api-client` is for interacting with the API, and `python-dotenv` is my preferred way to handle credentials without hardcoding them.

Next, create two files in your project directory: `monitor_metrics.py` for our script and `config.env` for our keys.

Your `config.env` file should look like this:

DATADOG_API_KEY="your_datadog_api_key_here"
DATADOG_APP_KEY="your_datadog_application_key_here"

Pro Tip: Always add your `config.env` file to `.gitignore`. Accidentally committing credentials to a repository is a security risk you don’t want to deal with. Trust me on that one.

Step 2: The Python Script

Now for the fun part. Open up `monitor_metrics.py` and let’s build this out. The script will do two things:

1. Query the Datadog API to get the current count of active custom metrics.
2. Submit that count back to Datadog as a new gauge metric that we can monitor.

Here is the complete script. I’ll break down the logic below it.

import os
import time
from dotenv import load_dotenv
from datadog_api_client.v1 import ApiClient, ApiException, Configuration
from datadog_api_client.v1.api import metrics_api
from datadog_api_client.v1.models import MetricsPayload, Series, Point

def get_active_custom_metrics_count():
    """
    Queries the Datadog API for the count of active custom metrics.
    """
    load_dotenv(dotenv_path='config.env')
    
    configuration = Configuration()
    # No need to pass keys here if DD_API_KEY and DD_APP_KEY are set
    # but we load them from config.env for clarity.
    configuration.api_key['apiKeyAuth'] = os.getenv('DATADOG_API_KEY')
    configuration.api_key['appKeyAuth'] = os.getenv('DATADOG_APP_KEY')

    with ApiClient(configuration) as api_client:
        api_instance = metrics_api.MetricsApi(api_client)
        try:
            # This API call gets a list of all active metrics
            response = api_instance.list_active_metrics()
            metric_count = len(response.get('metrics', []))
            print(f"Found {metric_count} active custom metrics.")
            return metric_count
        except ApiException as e:
            print(f"Exception when calling MetricsApi->list_active_metrics: {e}\n")
            return None

def send_count_to_datadog(metric_count):
    """
    Submits the metric count back to Datadog as a gauge.
    """
    if metric_count is None:
        print("Metric count is None, skipping submission.")
        return

    load_dotenv(dotenv_path='config.env')
    configuration = Configuration()
    
    body = MetricsPayload(
        series=[
            Series(
                metric="techresolve.custom_metrics.count",
                type="gauge",
                points=[Point([int(time.time()), float(metric_count)])],
                tags=["env:monitoring", "app:datadog-utils"]
            ),
        ]
    )

    with ApiClient(configuration) as api_client:
        api_instance = metrics_api.MetricsApi(api_client)
        try:
            response = api_instance.submit_metrics(body=body)
            print("Successfully submitted metric count to Datadog.")
            print(response)
        except ApiException as e:
            print(f"Exception when calling MetricsApi->submit_metrics: {e}\n")

if __name__ == "__main__":
    print("Starting Datadog custom metric count check...")
    count = get_active_custom_metrics_count()
    send_count_to_datadog(count)
    print("Script finished.")

**Breaking it down:**

• `get_active_custom_metrics_count()`: This is the core function. It initializes the Datadog API client using our keys from `config.env`. It then calls the `list_active_metrics` endpoint. This endpoint returns a list of all currently active custom metric names. We simply get the length of that list to find our total count.
• `send_count_to_datadog(metric_count)`: Once we have the count, we need to store it. This function constructs a `MetricsPayload`. We define a new metric named `techresolve.custom_metrics.count`, set its type to `gauge` (since it’s a single value at a point in time), and submit it with the current timestamp.
• `if __name__ == “__main__”:` This standard Python construct ensures the code only runs when the script is executed directly. It calls our two functions in order.

Step 3: Scheduling the Script

A script is only useful if it runs automatically. In my production setups, I use a simple cron job for this. You could also use a systemd timer, a Jenkins job, or even a serverless function. For simplicity, let’s use cron.

You would add a line like this to your crontab to run the script every Monday at 2 AM:

`0 2 * * 1 python3 script.py`

Make sure to adjust the schedule and the command to match your environment and desired frequency. Running it once a day or even once a week is usually sufficient.

Step 4: Creating the Datadog Monitor

Now that we’re sending the `techresolve.custom_metrics.count` metric to Datadog, we can build a monitor on top of it.

1. In your Datadog account, go to **Monitors > New Monitor**.
2. Select **Metric** as the monitor type.
3. Define the metric: select `techresolve.custom_metrics.count` from the dropdown. You shouldn’t need any complex filtering here.
4. Set Alert Conditions:

• Set the trigger to `above` a certain threshold. This number depends on your plan. Let’s say your limit is 5,000 custom metrics; you might set the warning threshold at `4500` and the critical alert at `4800`.
• Configure the evaluation window. For a metric that reports daily, `over the last 1 day` is fine.

5. Configure Notifications: This is the crucial part. Set it to notify your DevOps team’s Slack channel or PagerDuty. The message should be clear, something like: `{{#is_alert}}CRITICAL: Datadog custom metric count is at {{value}}. We are approaching our billing limit!{{/is_alert}}`
6. Save the monitor.

Common Pitfalls

Here are a few places where I’ve stumbled in the past, so you can avoid them:

• API Key Permissions: The first time I did this, I used a read-only API key. The script could fetch the count but failed silently when trying to submit the new gauge metric. Make sure your API key has the `metrics_post` permission.
• Timezone Troubles: Cron jobs run in the server’s timezone. If your server is in UTC and your team is in PST, that 2 AM job might run at an unexpected time for your team. Be mindful of this when setting the schedule.
• Not Filtering in the UI: When you first go to create the monitor, Datadog might show you multiple sources if the script runs in different places. Using the tags we added (`env:monitoring`) helps you narrow it down if needed.

Conclusion

And that’s it. With one Python script and a Datadog monitor, you’ve created an automated system to prevent billing surprises. It’s a classic DevOps win: automating a manual task to improve reliability and reduce operational overhead. Now you can get a Slack notification well before you hit your metric limit, giving you time to investigate and clean up any runaway tags or unnecessary metrics.

Stay proactive, and happy monitoring!

– Darian

---

🤖 Frequently Asked Questions