An XML sitemap is crucial for SEO, as it helps search engines crawl and index a website efficiently. However, broken URLs in sitemaps can lead to poor search engine rankings and user experience. Detecting and fixing these broken links manually is time-consuming, especially for large websites.

In this tech concept, we will automate the process using Python. We will parse an XML sitemap, check URLs for availability, and generate a cleaned version of the sitemap with only working links. For over two decades, I’ve been igniting change and delivering scalable tech solutions that elevate organisations to new heights. My expertise transforms challenges into opportunities—fueling business success for the organizations I’ve worked with and empowering me to strengthen my brand presence at NextStruggle in the digital age.

Why Fix Broken URLs in XML Sitemaps?

• Improved SEO: Search engines penalize sites with excessive broken links.
• Better User Experience: Visitors encounter fewer 404 errors.
• Efficient Crawling: Ensures search engines focus on valid pages.

By automating the cleanup process, you can keep your sitemap updated without manual intervention.

Step 1: Install Required Libraries

Before running the script, install the necessary Python libraries:

pip install requests lxml

• requests: To check URL status.
• lxml: To parse XML sitemaps.

Step 2: Load and Parse the XML Sitemap

First, we need to read the sitemap and extract the URLs.

from lxml import etree

def load_sitemap(file_path):
    tree = etree.parse(file_path)
    root = tree.getroot()
    return root.findall("{http://www.sitemaps.org/schemas/sitemap/0.9}url")

sitemap_urls = load_sitemap("sitemap.xml")
print(f"Total URLs found: {len(sitemap_urls)}")

This function extracts all URLs from the sitemap XML file.

Step 3: Check URLs for Validity

We need to verify which URLs return a valid response (status code 200).

import requests

def check_url_status(url):
    try:
        response = requests.head(url, allow_redirects=True, timeout=5)
        return response.status_code == 200
    except requests.RequestException:
        return False

This function checks if a URL is accessible. If the status code isn’t 200, it is considered broken.

Step 4: Filter Working URLs

We now filter only the working URLs.

working_urls = [url for url in sitemap_urls if check_url_status(url.find("{http://www.sitemaps.org/schemas/sitemap/0.9}loc").text)]
print(f"Valid URLs: {len(working_urls)}")

This step ensures only working URLs remain in our sitemap.

Step 5: Generate a Clean Sitemap

Once we have a list of working URLs, we generate a new XML sitemap.

def create_clean_sitemap(working_urls, output_file):
    root = etree.Element("urlset", xmlns="http://www.sitemaps.org/schemas/sitemap/0.9")
    for url in working_urls:
        url_element = etree.SubElement(root, "url")
        loc = etree.SubElement(url_element, "loc")
        loc.text = url.find("{http://www.sitemaps.org/schemas/sitemap/0.9}loc").text
    
    tree = etree.ElementTree(root)
    tree.write(output_file, pretty_print=True, encoding="utf-8", xml_declaration=True)

create_clean_sitemap(working_urls, "clean_sitemap.xml")
print("Clean sitemap generated successfully!")

This script generates a new sitemap with only valid URLs.

My Tech Advice: Keeping an updated and error-free XML sitemap is essential for SEO. Automating the process with Python saves time and ensures search engines can efficiently crawl your website. By applying above tech concepts, professionals can automatically eliminate broken links, maintain a healthy sitemap, and enhance SEO by ensuring only functional pages are indexed. Implementing this automated solution keeps your website optimised for search engines at all times.

#AskDushyant

Note: The example and pseudo code is for illustration only. You must modify and experiment with the concept to meet your specific needs.

#TechConcept #TechAdvice #SEO #SearchEngine #Python