Check Broken Links in XML Sitemaps automatically

Alright, let’s talk about something that used to be a real time-sink for me: manually checking sitemaps. I remember spending a couple of hours every week just clicking through links or wrestling with clunky desktop tools after a big deployment. It was tedious and, frankly, a terrible use of engineering time. That’s when I decided to automate the whole process.

This little Python script I’m about to show you saved me that time, caught critical 404 errors before our SEO team did, and gave me peace of mind. It’s a simple, set-and-forget solution that every DevOps toolkit should have.

Prerequisites

Before we dive in, make sure you have the following:

• Python 3 installed on your system.
• Basic familiarity with running Python scripts from your terminal.
• A target XML sitemap URL you want to check.

The Guide: Step-by-Step

Step 1: Setting Up Your Environment

First, you’ll want to set up a new project directory for this script. I’ll skip the standard virtualenv setup since you likely have your own workflow for that. The key is to work in an isolated Python environment to keep dependencies clean.

Once your virtual environment is created and activated, you’ll need two excellent Python libraries. Go ahead and install them using pip from your terminal:

pip install requests beautifulsoup4

requests is the gold standard for making HTTP requests, and beautifulsoup4 is a lifesaver for parsing HTML and XML documents.

Step 2: The Script – Fetching and Parsing the Sitemap

Let’s start by writing the Python code to grab the sitemap and pull out all the URLs. Create a new file, let’s call it sitemap_checker.py, and add the following code.

The logic here is straightforward: We use `requests` to download the raw XML content of the sitemap. Then, we pass that content to BeautifulSoup, telling it to parse the content as XML. Finally, we use `find_all(‘loc’)` to create a list of all the URL locations defined in the sitemap.

import requests
from bs4 import BeautifulSoup

def get_sitemap_urls(sitemap_url):
    """Fetches and parses an XML sitemap, returning a list of URLs."""
    urls = []
    headers = {
        'User-Agent': 'TechResolve-Sitemap-Checker/1.0'
    }
    try:
        response = requests.get(sitemap_url, headers=headers, timeout=10)
        response.raise_for_status()  # Raises an exception for bad status codes (4xx or 5xx)

        soup = BeautifulSoup(response.content, 'xml')
        loc_tags = soup.find_all('loc')
        urls = [tag.text for tag in loc_tags]
        print(f"Found {len(urls)} URLs in the sitemap.")
    except requests.exceptions.RequestException as e:
        print(f"Error fetching sitemap: {e}")
    
    return urls

# --- Main execution part will go here later ---

Step 3: Checking Each URL for Errors

Now that we have a list of URLs, the next step is to check each one. We’ll loop through our list and send a request to each URL. A crucial optimization here is to use a `HEAD` request instead of a `GET` request.

Pro Tip: I always use a HEAD request for link checking. It just fetches the headers, not the entire page content. This is way faster and uses far less bandwidth, especially when you’re checking hundreds or thousands of links. You still get the crucial HTTP status code, which is all we need.

Let’s add the link-checking function to our script.

def check_urls(urls):
    """Checks a list of URLs and returns a list of broken ones."""
    broken_links = []
    headers = {
        'User-Agent': 'TechResolve-Sitemap-Checker/1.0'
    }
    
    for i, url in enumerate(urls):
        try:
            # Using a HEAD request for efficiency
            response = requests.head(url, headers=headers, timeout=10, allow_redirects=True)
            
            # We consider anything that isn't a success (2xx) or redirect (3xx) as a potential issue.
            # 400 and 500 level codes are definitely broken.
            if response.status_code >= 400:
                print(f"BROKEN: {url} (Status: {response.status_code})")
                broken_links.append({'url': url, 'status': response.status_code})
            else:
                # Optional: log progress for large sitemaps
                if (i + 1) % 50 == 0:
                    print(f"Checked {i + 1}/{len(urls)} URLs...")

        except requests.exceptions.RequestException as e:
            print(f"ERROR: {url} (Error: {e})")
            broken_links.append({'url': url, 'status': 'Request Error'})

    print("URL check complete.")
    return broken_links

Step 4: Putting It All Together and Reporting

Finally, let’s create the main part of the script that calls our functions and prints a final report. This is where you could later integrate email alerts, Slack notifications, or whatever your team uses.

if __name__ == "__main__":
    # IMPORTANT: Replace this with your actual sitemap URL
    SITEMAP_TO_CHECK = "https://your-website.com/sitemap.xml"
    
    print(f"Starting check for sitemap: {SITEMAP_TO_CHECK}")
    
    all_urls = get_sitemap_urls(SITEMAP_TO_CHECK)
    
    if all_urls:
        failed_links = check_urls(all_urls)
        
        if failed_links:
            print("\n--- BROKEN LINKS REPORT ---")
            for link in failed_links:
                print(f"URL: {link['url']}, Status: {link['status']}")
            print(f"--- Total broken links found: {len(failed_links)} ---")
        else:
            print("\n--- SUCCESS: No broken links found! ---")
    else:
        print("Could not retrieve any URLs to check.")

Step 5: Automating with Cron

A script is only useful if it runs consistently. On any Linux-based server, I use a simple cron job for this. You can schedule the script to run automatically at a set interval. For example, to run it every Monday at 2:00 AM, you would set up a cron job like this:

0 2 * * 1 python3 script.py

This “set it and forget it” approach ensures you’re always on top of broken links without lifting a finger.

Common Pitfalls (Where I Usually Mess Up)

• Sitemap Indexes: My script is designed for a single sitemap file. Many large sites use a sitemap index file that points to other sitemap files. If your target is an index, you’ll need to extend the script to first parse the index, get the list of sitemap URLs, and then loop through each one. It’s a simple recursive addition but easy to forget.
• User-Agent: Some web servers or firewalls are configured to block requests that don’t have a `User-Agent` header. I always include one to make my script look more like a legitimate browser and avoid getting blocked with a 403 Forbidden error.
• Request Timeouts: A server might hang and never respond. Without a `timeout` in your `requests` call, your script could be stuck forever. I always add a reasonable timeout (e.g., 10 seconds) to keep things moving.

Conclusion

And there you have it. A straightforward, automated way to keep your sitemaps clean and your SEO team happy. This small investment in automation pays off big time by preventing 404s from hurting your site’s reputation with search engines and user experience. You can easily extend this script to send an email or a Slack message with the report. Tweak it, integrate it, and enjoy the extra time back in your week.

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions