Syncing CloudWatch Logs to Datadog for Retention

Hey team, Darian Vance here. Let’s talk about something that used to be a real time-sink for me: log spelunking. I’d jump between CloudWatch and Datadog, trying to piece together an incident report. The biggest headache was CloudWatch’s default retention. A critical log from a month ago? Gone. Extending retention in AWS is an option, but it gets pricey fast. My solution, which I now use in all my production setups, is to sync the important logs over to Datadog. You get centralized visibility and cost-effective long-term storage. This simple script probably saves me a few hours every month, and I want to walk you through how to set it up.

Prerequisites

Before we dive in, make sure you have the following ready to go:

• An AWS account with IAM credentials that have permissions for logs:FilterLogEvents.
• A Datadog account with your API Key and Application Key handy.
• Python 3 installed on the machine where this script will run.
• Access to a terminal or command line to install a few Python packages.

The Guide: A Step-by-Step Walkthrough

Step 1: Setting Up Your Environment

First, we need to get our Python environment ready. I’ll skip the standard virtual environment setup since you likely have your own workflow for that. The main thing is to install the libraries we’ll need. You can do this by running a command like ‘pip install boto3 datadog python-dotenv’ in your terminal.

Next, let’s handle our secrets. Create a file in your project directory named config.env. This is where we’ll store our keys so they aren’t hardcoded in the script, which is a major security no-no. Your file should look like this:

AWS_ACCESS_KEY_ID="YOUR_AWS_ACCESS_KEY"
AWS_SECRET_ACCESS_KEY="YOUR_AWS_SECRET_KEY"
AWS_REGION="us-east-1"
DATADOG_API_KEY="YOUR_DATADOG_API_KEY"
DATADOG_APP_KEY="YOUR_DATADOG_APP_KEY"

Pro Tip: For any real production environment, I strongly recommend using an IAM Role attached to the EC2 instance or Lambda function running the script. This is far more secure than storing IAM user credentials in a file. The `boto3` library will automatically pick up credentials from an IAM role, so you wouldn’t even need the AWS keys in your `config.env` file.

Step 2: The Python Sync Script

Now for the core logic. The goal here is to create a reusable script that can pull logs from a specified CloudWatch Log Group from the last ‘X’ hours and forward them to Datadog as events.

Here’s the full script. I’ll break down what each part does below.

import os
import boto3
import time
from datetime import datetime, timedelta, timezone
from datadog import initialize, api
from dotenv import load_dotenv

def main():
    # --- Configuration ---
    load_dotenv('config.env')

    # Datadog credentials
    dd_options = {
        'api_key': os.getenv('DATADOG_API_KEY'),
        'app_key': os.getenv('DATADOG_APP_KEY')
    }
    initialize(**dd_options)

    # AWS credentials and log group
    aws_region = os.getenv('AWS_REGION')
    log_group_name = '/aws/lambda/your-function-name' # <-- IMPORTANT: Change this

    # --- Logic ---
    print(f"Starting sync for log group: {log_group_name}")

    # Define the time window (last 24 hours)
    end_time = datetime.now(timezone.utc)
    start_time = end_time - timedelta(hours=24)

    # Convert to milliseconds for AWS API
    start_time_ms = int(start_time.timestamp() * 1000)
    end_time_ms = int(end_time.timestamp() * 1000)

    try:
        # Connect to CloudWatch Logs
        client = boto3.client(
            'logs',
            region_name=aws_region,
            aws_access_key_id=os.getenv('AWS_ACCESS_KEY_ID'),
            aws_secret_access_key=os.getenv('AWS_SECRET_ACCESS_KEY')
        )

        # Use a paginator to handle large log volumes
        paginator = client.get_paginator('filter_log_events')
        page_iterator = paginator.paginate(
            logGroupName=log_group_name,
            startTime=start_time_ms,
            endTime=end_time_ms
        )

        log_count = 0
        for page in page_iterator:
            for event in page['events']:
                log_message = event['message']
                # Send to Datadog
                api.Event.create(
                    title=f"CloudWatch Log from {log_group_name}",
                    text=log_message,
                    tags=[f"source:cloudwatch", f"log_group:{log_group_name}"],
                    source_type_name="aws"
                )
                log_count += 1
                # Small delay to avoid hitting API rate limits
                time.sleep(0.1) 
        
        print(f"Successfully synced {log_count} log events to Datadog.")

    except Exception as e:
        print(f"An error occurred: {e}")
        return 1
    
    return 0

if __name__ == "__main__":
    main()

Code Breakdown:

1. Configuration: We use load_dotenv to pull in our secrets from the config.env file. We then initialize the Datadog API with our keys.
2. Time Window: We calculate the start and end times for our log query. It’s critical to use timezone-aware datetime objects (timezone.utc) to avoid any ambiguity, as AWS operates in UTC.
3. Boto3 Paginator: Instead of a simple filter_log_events call, I’m using a paginator. This is a robust way to handle situations where a query returns more logs than can fit in a single API response. It automatically handles fetching subsequent pages of results for you.
4. Forwarding to Datadog: We loop through each log event found. For each one, we call api.Event.create. Notice I’m adding tags like the source and log group name. This is incredibly useful for filtering and creating dashboards in Datadog later.

Step 3: Automating the Sync

Running this manually isn’t practical. We need to automate it. A simple cron job is perfect for this. For example, to run this script every day at 2 AM, you would set up a cron job like this. Don’t use a command like ‘crontab -e’, but rather add this line to your system’s crontab configuration.

0 2 * * * python3 script.py

Make sure you’re running this from the directory containing your script and `config.env` file, or adjust the paths accordingly. Remember, don’t use absolute paths starting with a slash in the cron command itself for security reasons.

Common Pitfalls (Where I Usually Mess Up)

• IAM Permissions: The most common error is an AccessDeniedException. Double-check that your IAM user or role has the logs:FilterLogEvents permission for the specific log group you are targeting.
• API Rate Limiting: If you’re syncing a very chatty log group, you might hit AWS or Datadog API rate limits. The small time.sleep(0.1) in the loop helps, but for massive volumes, you might need to batch your logs and send them to Datadog in fewer, larger requests.
• Incorrect Log Group Name: It sounds simple, but a typo in the log_group_name variable is a frequent source of frustration. Always copy and paste it directly from the AWS console.

Conclusion

And that’s it. With a simple Python script and a cron job, you’ve built a reliable pipeline to move your important CloudWatch logs into Datadog for long-term retention and analysis. This not only saves you from potential AWS costs but also consolidates your observability stack. It’s a small investment of time that pays huge dividends when you’re troubleshooting an issue months down the line. Hope this helps you out.

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions