Tired of manual conversion tracking scripts breaking at the worst possible moment? Discover three battle-tested methods, from quick fixes to permanent architectural solutions, to automate your marketing data pipeline and finally get some sleep.

I See Your Broken Python Script, and I Raise You a 3 AM Pager Alert

I still remember the call. It was 2:47 AM on a Tuesday, the morning of our big Q3 product launch. The PagerDuty alert was as cryptic as it was terrifying: CRITICAL: ConversionUpload_Job_FAIL. I rolled out of bed, logged in, and found the source of the chaos: a junior marketer, bless his heart, had been manually running a Python script every night from his laptop to upload conversion data to Google Ads. Tonight, his laptop was off. The entire launch day’s performance metrics were currently a big, fat zero, and the CMO was already sending panicked Slack messages.

This scene, or some variation of it, is a rite of passage. That’s why when I saw a Reddit thread titled “Conversion tracking automation,” I felt a deep, spiritual connection. We’ve all been there. Let’s talk about why this keeps happening and how to fix it for good.

The Real Problem: Your “Automation” is Just a Scheduled Chore

The core issue isn’t the script itself. The script probably works fine. The problem is treating a critical business process like a household chore. When your “automation” relies on a single person’s machine, a cron job on a non-redundant server (I’m looking at you, dev-utility-box-01), or manual intervention of any kind, it’s not automation—it’s a time bomb. It’s fragile. It has no monitoring, no error handling, and it’s completely invisible until it breaks spectacularly.

So, let’s walk through the evolution of solving this, from the “get it working by sunrise” fix to the architectural solution that lets you sleep through the night.

Solution 1: The “Duct Tape and a Prayer” Method

This is the quick and dirty fix. It’s not great, but it’s a step up from someone’s laptop. The goal here is to get the script off a personal machine and onto a server that’s at least monitored and always on. We’re talking about a bastion host or a dedicated utility EC2 instance.

Step 1: Get the script and its dependencies onto a stable, shared server. Let’s call it util-prod-01.

Step 2: Wrap the script in a simple shell script that handles basic logging.

#!/bin/bash
# run_conversion_upload.sh

LOG_FILE="/var/log/conversion_uploads.log"
SCRIPT_PATH="/opt/scripts/marketing/upload_conversions.py"
VENV_PATH="/opt/scripts/marketing/venv/bin/activate"

echo "====== Starting Upload Job: $(date) ======" >> $LOG_FILE

source $VENV_PATH
python3 $SCRIPT_PATH >> $LOG_FILE 2>&1

if [ $? -eq 0 ]; then
  echo "Job Succeeded" >> $LOG_FILE
else
  echo "!!!!!!!! JOB FAILED !!!!!!!!" >> $LOG_FILE
  # You could add a simple curl command here to ping a health check or Slack webhook
fi

echo "====== Finished Upload Job: $(date) ======" >> $LOG_FILE

Step 3: Schedule it with a systemd timer instead of a user’s crontab. It’s more robust and has better logging integration with `journalctl`.

Pro Tip: This is a band-aid, not a cure. You’ve centralized the point of failure, which is better, but it’s still a single point of failure. If util-prod-01 goes down, you’re back in the dark. Use this to stop the immediate bleeding, then plan for Solution 2.

Solution 2: The “Cloud Native” Permanent Fix

This is how we do it properly in 2024. We move away from scheduled polling and embrace an event-driven architecture. The process should trigger automatically when the data is ready, not on a blind schedule. For this example, let’s assume your application drops a CSV of daily conversions into an S3 bucket.

The Architecture: S3 Event → Lambda Function

1. Data Drop: Your application finishes its daily batch and uploads conversions-YYYY-MM-DD.csv to an S3 bucket named techresolve-conversion-data.
2. The Trigger: An S3 Event Notification is configured on that bucket to fire whenever a new .csv file is created.
3. The Worker: The event triggers an AWS Lambda function. This function contains your Python logic to read the CSV, process it, and push it to the Google Ads API.

Your Lambda function’s Python code might look something like this conceptually:

import boto3
import pandas as pd
from google_ads_api import GoogleAdsClient # Fictional library

s3 = boto3.client('s3')

def lambda_handler(event, context):
    # Get the bucket and file name from the event
    bucket_name = event['Records'][0]['s3']['bucket']['name']
    s3_file_key = event['Records'][0]['s3']['object']['key']
    
    # Get the file from S3
    response = s3.get_object(Bucket=bucket_name, Key=s3_file_key)
    csv_content = response['Body'].read().decode('utf-8')
    
    # Process the data
    df = pd.read_csv(io.StringIO(csv_content))
    # ... your data transformation logic here ...
    
    # Upload to Google Ads
    try:
        ads_client = GoogleAdsClient(api_key="...")
        ads_client.upload_conversions(df)
        print(f"Successfully processed and uploaded {s3_file_key}")
        return {'status': 'success'}
    except Exception as e:
        print(f"ERROR: Failed to upload {s3_file_key}. Reason: {e}")
        # This will automatically log to CloudWatch
        raise e # Re-raise exception to mark Lambda invocation as failed

Warning: The most critical part of this is IAM. Your Lambda function needs a tightly-scoped IAM role that ONLY gives it permission to read from that specific S3 bucket and whatever permissions are needed for the Google Ads API. Never use broad permissions like s3:*.

Solution 3: The “Stop Hacking, Start Architecting” Option

Sometimes, the problem isn’t the script; it’s the source. If you’re running complex queries against a read replica of your production database (e.g., prod-db-01-replica) to generate a CSV, you’re still creating a massive dependency and a point of failure. This is where you zoom out and fix the data flow itself.

This option involves using dedicated tools built for this exact purpose—a Customer Data Platform (CDP) or an ETL/ELT service.

This is the “nuclear” option because it often requires a bigger budget and buy-in from multiple teams (Engineering, Data, Marketing). But it’s also the one that truly solves the problem at its root by creating a resilient, observable, and scalable data infrastructure instead of a collection of fragile scripts.

So next time you find a critical process running on a dusty old cron job, don’t just move it to another server. Take a moment, think about the real business need, and build something that won’t wake you up at 3 AM. Your future self will thank you.

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions