`), and then email an alert if it finds something new.

The Code

Here’s a bare-bones Python example. Notice the `headers` dictionary—this is the absolute minimum you need to do to pretend you’re a browser.

import requests
from bs4 import BeautifulSoup

# --- CONFIG ---
TARGET_URL = "https://competitor-site.com"
PROMO_SELECTOR = "div.promo-banner-text"
HEADERS = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'
}

# --- SCRIPT ---
try:
    response = requests.get(TARGET_URL, headers=HEADERS, timeout=10)
    response.raise_for_status() # Will raise an exception for 4xx/5xx errors

    soup = BeautifulSoup(response.text, 'html.parser')
    promo_element = soup.select_one(PROMO_SELECTOR)

    if promo_element:
        promo_text = promo_element.get_text().strip()
        print(f"SUCCESS: Found promo text: '{promo_text}'")
        # Here you would add your logic to send an email or Slack alert
    else:
        print("INFO: No promo banner found with the specified selector.")

except requests.exceptions.RequestException as e:
    print(f"ERROR: Could not fetch the page. Error: {e}")

You’d save this as `check_promo.py` and set up a cron job to run it every hour: `0 * * * * /usr/bin/python3 /path/to/check_promo.py`.

Warning: This method is incredibly brittle. If the competitor changes their site’s layout (e.g., renames `div.promo-banner-text` to `div.sale-header-text`), your script breaks silently. It also won’t work on sites that load content with JavaScript.

Solution 2: The “We’re Serious Now” Serverless Approach

Okay, the cron job got you blocked after a week. It’s time for a real solution. We’re going to build a resilient, scalable, and much harder-to-detect system using serverless tools. This is the architecture I’d propose for a long-term, reliable monitor.

The Architecture

The core idea is to use a headless browser to render the page just like a real user would, and to rotate our IP address for every run so we don’t build up a suspicious request pattern.

1. Scheduler: An Amazon EventBridge (formerly CloudWatch Events) rule triggers our process on a schedule (e.g., every 15 minutes).
2. Executor: An AWS Lambda function contains our scraping logic. We’ll use a language like Node.js or Python.
3. Headless Browser: We’ll package a tool like Puppeteer (for Node.js) or Playwright (for Python) into our Lambda function using a Lambda Layer. This will execute the page’s JavaScript and give us the final, rendered HTML.
4. IP Rotation: Instead of making requests from the Lambda’s IP, we’ll route our traffic through a residential proxy service. This makes each request look like it’s coming from a different home user, making it nearly impossible to block based on IP.
5. Storage & Alerting: The Lambda function saves the results (promo text, screenshot) to an S3 bucket for historical tracking and sends an alert to a Slack channel via an SNS topic if a new promotion is detected.

This setup is far more robust. The headless browser defeats client-side rendering issues, and the proxy rotation handles IP blocking. Because it’s serverless, you only pay for the few seconds it runs, making it incredibly cost-effective.

Solution 3: The ‘Nuclear’ Option – Pay Someone Else

Let’s be realistic. Building and maintaining even a serverless scraping pipeline takes engineering time. You have to handle anti-bot countermeasures, CAPTCHAs, and framework updates. Sometimes, the most cost-effective solution is to offload the problem entirely.

This is where Scraping-as-a-Service platforms come in. Companies like Bright Data, ScrapingBee, or Apify specialize in this arms race. You essentially give their API a URL, and they handle everything: headless browsers, proxy rotation, CAPTCHA solving, and parsing. You just get clean JSON data back.

When to Choose This

I recommend this path when the targets are notoriously difficult to scrape (think major e-commerce or travel sites) or when my team’s backlog is already overflowing. If the cost of a monthly subscription is less than the cost of 10-20 hours of my time per month maintaining a custom solution, it’s a no-brainer. You’re paying for expertise and a system that just works.

Comparison at a Glance

Solution	Initial Cost	Maintenance Effort	Reliability
1. Cron Job	Very Low (~$5/mo VM)	High (Breaks often)	Low
2. Serverless	Low (Pay-per-use, pennies per run)	Medium (Initial setup, occasional tweaks)	High
3. Third-Party Service	Medium (Monthly subscription)	Very Low (It’s their problem)	Very High

A Final Pro Tip: Before you do any of this, check the website’s `robots.txt` file and Terms of Service. While scraping publicly available data is generally a legal gray area, being a good internet citizen is always the right call. Never overload a competitor’s server and always identify your bot in the User-Agent if you can. Don’t be the reason they have to beef up their security.

---

🤖 Frequently Asked Questions