Check SEO Meta Tags presence automatically for new deployments

Hey team, Darian here. Let me tell you a quick story. A few years back, we pushed a major site update, and everything looked perfect. The problem? A configuration hiccup meant an entire product section went live without meta titles or descriptions. Our SEO rankings took a nosedive, and marketing was, to put it mildly, not pleased. We spent the next week in frantic damage control. That’s when I decided: never again.

I built a simple, automated check that runs with every single deployment. It’s a five-minute setup that has saved us countless hours of headaches and prevented similar disasters. Today, I’m going to walk you through how to build it yourself.

Prerequisites

Before we dive in, make sure you have the following ready:

• Python 3 installed on your system or CI runner.
• A list of critical URLs from your site to check (e.g., homepage, product page, blog post).
• Access to your CI/CD pipeline configuration (like Jenkins, GitLab CI, GitHub Actions) to add a new step.

You’ll also need a couple of Python libraries, namely requests and beautifulsoup4. You can install them using pip from your terminal. I find they are essential for any kind of web scraping or validation task.

The Guide: Step-by-Step

Step 1: Project Setup

First things first, get your project folder ready. I’ll skip the standard virtualenv setup since you likely have your own workflow for that. Let’s jump straight to the logic. Inside your project, create two files:

• urls_to_check.txt: This will hold the list of URLs, one per line.
• check_meta_tags.py: This is where our Python script will live.

Populating urls_to_check.txt is easy. Just add your key pages:

https://your-staging-site.com/
https://your-staging-site.com/about-us
https://your-staging-site.com/products/main-product

Step 2: The Python Script

Now for the fun part. Open check_meta_tags.py. The script will perform a few key actions: read the URLs from our text file, loop through each one, fetch the page’s HTML, parse it, and then check for the presence of our required meta tags.

Here’s the complete script. I’ve added comments to explain what each part does.

import requests
from bs4 import BeautifulSoup

# Define the meta tags we consider essential.
# For 'name' attributes: 'description', 'viewport'
# For 'property' attributes (Open Graph): 'og:title', 'og:description'
REQUIRED_TAGS = {
    'name': ['description', 'viewport'],
    'property': ['og:title', 'og:description']
}

# Standard headers to mimic a browser and avoid getting blocked.
HEADERS = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'
}

def check_single_url(url):
    """Checks a single URL for required meta tags and returns a list of missing tags."""
    missing_tags = []
    print(f"--- Checking: {url} ---")

    try:
        response = requests.get(url, headers=HEADERS, timeout=10)
        # Raise an exception for bad status codes (4xx or 5xx)
        response.raise_for_status()
    except requests.exceptions.RequestException as e:
        print(f"Error fetching URL {url}: {e}")
        return [f"FAILED_TO_FETCH: {e}"]

    soup = BeautifulSoup(response.text, 'html.parser')

    # 1. Check for the <title> tag (it's special)
    if not soup.find('title') or not soup.find('title').string.strip():
        missing_tags.append("html_title")

    # 2. Check for meta tags with 'name' attribute
    for tag_name in REQUIRED_TAGS['name']:
        meta_tag = soup.find('meta', attrs={'name': tag_name})
        if not meta_tag:
            missing_tags.append(f"name='{tag_name}'")

    # 3. Check for meta tags with 'property' attribute (Open Graph)
    for tag_property in REQUIRED_TAGS['property']:
        meta_tag = soup.find('meta', attrs={'property': tag_property})
        if not meta_tag:
            missing_tags.append(f"property='{tag_property}'")

    return missing_tags

def main():
    """Main function to read URLs and orchestrate checks."""
    all_issues_found = False
    try:
        with open('urls_to_check.txt', 'r') as f:
            urls = [line.strip() for line in f if line.strip()]
    except FileNotFoundError:
        print("Error: `urls_to_check.txt` not found.")
        return 1 # Indicate failure

    for url in urls:
        missing = check_single_url(url)
        if missing:
            all_issues_found = True
            print(f"  [ISSUES FOUND] Missing tags for {url}: {', '.join(missing)}\n")
        else:
            print(f"  [OK] All required tags present for {url}\n")

    if all_issues_found:
        print("SCRIPT FAILED: One or more pages are missing required meta tags.")
        return 1 # Non-zero exit code signals failure to CI/CD pipelines
    else:
        print("SCRIPT PASSED: All checked pages have the required meta tags.")
        return 0 # Zero exit code signals success

if __name__ == "__main__":
    main()

Pro Tip: In my production setups, I don’t just check for a tag’s presence; I also check its content length. An empty `content=””` attribute is just as bad as a missing tag. You can easily add a check like `if not meta_tag.get(‘content’) or len(meta_tag.get(‘content’)) < 10:` to catch placeholder content.

Step 3: Integrate into Your Workflow

With the script ready, it’s time to automate it. The goal is to make it a mandatory check after every deployment to your staging or pre-production environment.

In your CI/CD tool (e.g., GitLab CI, GitHub Actions), add a new “Validation” or “Testing” stage that runs after the “Deploy” stage. This stage will execute a simple command:

python3 check_meta_tags.py

The script is designed to return a non-zero status code if it finds any issues. Most CI/CD systems will automatically interpret this as a build failure, stopping the pipeline from proceeding to production. This is your safety net in action!

Here’s Where I Usually Mess Up (Common Pitfalls)

• Firewall and IP Whitelisting: The first time I ran this, I got nothing but timeout errors. I forgot that our staging environment was locked down. Make sure the IP address of your CI runner is whitelisted to access the environment you’re testing.
• Handling JavaScript-Rendered Tags: This script reads the initial HTML source from the server. If your site is a Single Page Application (SPA) that injects meta tags using JavaScript, this script won’t see them. For that, you’d need a more advanced tool like Selenium or Puppeteer that can render the page in a full browser.
• Forgetting the User-Agent: Some web servers or WAFs are configured to block requests from default Python scripts. Setting a common browser `User-Agent` header, like I did in the script, helps you avoid a lot of mysterious “403 Forbidden” errors.

Conclusion

And that’s it. It’s a straightforward script, but it formalizes a crucial pre-flight check that’s too easy to forget during a hectic release cycle. By automating this, you catch SEO issues before they ever see the light of day, keeping your marketing team happy and your site’s health intact. It’s a small investment of time that pays for itself with the first bug it catches.

Let me know if you run into any issues. Happy deploying.

– Darian

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions