SEO & Redirects
What is an XML sitemap?
An XML sitemap is a structured file in XML format that lists the URLs on a website, providing search engines with a comprehensive map of the site’s content and helping crawlers discover pages that might be difficult to find through link following alone. The sitemap communicates directly with search engines, telling them which URLs exist on the site, when they were last modified, how frequently they change, and in some cases their relative priority compared to other pages on the same site.
The XML format, Extensible Markup Language, provides a standardised, machine-readable structure that search engine crawlers parse programmatically. Unlike HTML which is designed for human reading XML sitemaps are designed exclusively for crawler consumption, the format is precise, predictable, and validated against a defined schema that all major search engines support.
XML sitemaps were introduced in 2005 through a collaboration between Google, Yahoo, and Microsoft, who jointly developed the Sitemap Protocol that defines the format. ICANN subsequently standardised the protocol. All major search engines, Google, Bing, Yandex, Baidu, support the Sitemap Protocol and use sitemaps as part of their content discovery processes.
Sitemaps are complementary to but not a replacement for crawling through links. A search engine that finds a URL in a sitemap still needs to crawl the URL to assess its content and decide whether and how to index it. The sitemap does not guarantee indexation, it guarantees discovery. The actual indexation decision depends on content quality, technical factors, and the search engine’s assessment of the page’s relevance and value.
Why XML sitemaps matter
The web’s link graph, the network of hyperlinks connecting pages, is the primary mechanism through which search engine crawlers discover content. Following links from page to page crawlers can theoretically discover all publicly linked content. In practice however the link graph has significant gaps, content that exists but is not well-linked may be difficult or slow to discover through links alone.
Discovery of deep or orphaned pages: pages many clicks from the homepage with few internal links pointing to them are rarely crawled without explicit sitemap guidance. An article published years ago that has lost its position in the site’s navigation, a product page with no category links pointing to it, or a landing page accessible only through paid advertising, these pages may never be discovered by crawlers following links. An XML sitemap ensures their URLs are communicated directly to search engines regardless of their link accessibility.
Faster indexation of new content: new pages added to a site may not be discovered quickly through link crawling, crawlers may not revisit linking pages frequently enough to discover new links promptly. Submitting an updated sitemap to Google Search Console signals new content for immediate crawl consideration, accelerating indexation for time-sensitive content like news articles, product launches, and campaign pages.
Large site crawl efficiency: for sites with millions of pages crawlers cannot follow every link on every crawl visit. XML sitemaps provide a curated list of important URLs, helping crawlers prioritise which pages to crawl rather than discovering pages through random link following. The priority and changefreq attributes provide additional signals about which pages deserve more frequent crawling.
Search engine communication: sitemaps establish a direct communication channel between site owners and search engine crawlers, supplementing the implicit communication that happens through links and content structure. This direct channel is particularly valuable for large, complex sites where the link structure alone may not adequately communicate content discovery priorities.
XML sitemap format and elements
The XML sitemap format follows a defined schema, each element serving a specific purpose in communicating URL information to search engines.
Sitemap declaration and namespace: every XML sitemap begins with the XML declaration and the sitemap namespace:
The urlset element wraps all URL entries. The xmlns attribute specifies the Sitemap Protocol namespace.
URL element: each page is represented by a url element containing the URL’s information:
loc: the only required element. The canonical URL of the page, absolute, fully qualified, and including the protocol. The URL must match the canonical version of the page, HTTPS, consistent subdomain form, without unnecessary parameters. URLs in the sitemap should never be redirect sources, only canonical destination URLs.
lastmod: the date the page was last significantly modified, in W3C datetime format. 2024-03-15 for date only or 2024-03-15T14:30:00+00:00 for date and time with timezone. The lastmod date helps search engines prioritise recrawling, pages with recent modifications are recrawled sooner than pages with old modification dates.
lastmod should reflect genuine content changes, not superficial page touches or automated date updates. A sitemap where every page shows today’s date as the lastmod, because a template automatically updates it, provides false signals that may reduce the value search engines place on the lastmod attribute.
changefreq: an estimate of how frequently the page’s content changes. Valid values are always, hourly, daily, weekly, monthly, yearly, and never. This is a hint rather than a directive, search engines use it as one input into recrawl frequency decisions alongside their own assessment of content change patterns.
always indicates the page changes on every access, appropriate for real-time content. never indicates the page is archived content that will not change, appropriate for historical articles. Most pages fall between these extremes. changefreq is less influential than lastmod for most search engine implementations.
priority: a relative priority for this URL compared to other URLs on the same site. Values range from 0.0 to 1.0, the default is 0.5. Priority communicates the site owner’s assessment of relative page importance, homepage might be 1.0, main category pages 0.8, individual articles 0.6, supporting pages 0.4.
Priority only affects crawl priority within the same site, it does not affect ranking. A page with priority 1.0 is not boosted in rankings, only signalled as relatively important for crawling. Search engines may use or ignore priority values at their discretion.
Sitemap index files
Individual XML sitemaps are limited to 50,000 URLs and 50MB uncompressed, constraints that large sites exceed with a single sitemap. The Sitemap Protocol addresses this through sitemap index files, sitemaps of sitemaps.
Sitemap index structure: a sitemap index file lists multiple individual sitemap files:
The sitemap index file itself can contain up to 50,000 sitemap entries, allowing up to 2.5 billion URLs across all sitemaps in an index.
Logical sitemap organisation: dividing URLs across multiple sitemap files by content type, products, articles, pages, images, provides several benefits. Crawlers can access specific content type sitemaps independently. Sitemap submission in Google Search Console shows separate indexation statistics for each sitemap file, providing visibility into how different content types are being crawled and indexed. Updating only the sitemap file for recently changed content types, rather than regenerating a monolithic sitemap, is more efficient.
Sitemap types and extensions
Beyond the basic URL sitemap the Sitemap Protocol has been extended to support additional content types with their own specific metadata.
Image sitemaps: an extension that provides image-specific metadata for images embedded in pages. Image sitemaps include image URL, caption, title, geographic location, and licence information, helping search engines discover and index images more effectively. Images within pages are included as extensions within the standard url element for the page that contains them.
Video sitemaps: an extension for video content providing metadata including video title, description, thumbnail URL, content URL, duration, expiry date, and rating. Video sitemaps help Google’s video search discover and index video content on pages.
News sitemaps: a specialised sitemap extension for news publishers. News sitemaps include publication name, language, publication date, and article title alongside the URL. News sitemaps support time-sensitive content discovery, Google News uses news sitemaps to discover and index articles for inclusion in news search results. News sitemaps should include only articles published within the last 48 hours, they are not a mechanism for submitting the site’s full article archive.
hreflang sitemaps: for international sites with multiple language and regional versions hreflang annotations can be included in sitemaps rather than in HTML head elements. Sitemap-based hreflang provides the same language and regional targeting signals as HTML hreflang tags, communicating to search engines which language and regional version of a page to serve to users in different markets.
XML sitemaps and redirects
The relationship between XML sitemaps and redirects is straightforward but frequently mismanaged, leading to indexation inefficiency and canonicalization signal confusion.
Sitemaps should contain only canonical URLs: the most important principle for sitemap-redirect interaction. Every URL in an XML sitemap should be the canonical destination URL, the HTTPS, correctly formed, non-redirect URL that search engines should index. Redirect source URLs, old domain URLs, HTTP variants, www/non-www non-canonical forms, should never appear in sitemaps.
Including redirect sources in sitemaps signals that the site owner considers those redirect sources to be important indexable URLs, contradicting the redirect that tells search engines those URLs are not canonical. The conflicting signals create indexation confusion and waste crawl budget on redirect processing that the sitemap itself triggered.
Post-migration sitemap updates: after a domain migration the sitemap must be updated immediately to reflect the new domain’s canonical URLs. A sitemap still listing old domain URLs after migration, when those URLs now redirect to the new domain, creates conflicting signals. The sitemap says old domain URLs are important. The redirects say new domain URLs are canonical. Update sitemaps to the new canonical URLs as part of the migration process, not as an afterthought.
Sitemap errors revealing missing redirects: submitting a sitemap to Google Search Console and reviewing the Coverage report reveals which sitemap URLs are returning errors, 404s, 500s, or other non-200 responses. Sitemap URLs returning errors indicate either that redirect configurations are missing for those URLs or that the sitemap itself contains incorrect URLs. Auditing sitemap URLs against actual redirect configurations identifies gaps in redirect coverage.
Redirect-only domain sitemaps: domains configured solely as redirect sources, brand protection domains, legacy domains maintained for link juice, do not need XML sitemaps. No content is served at these domains, only redirects. A sitemap on a redirect-only domain would list URLs that all redirect to the primary domain, serving no useful discovery purpose and potentially confusing search engines about the redirect domain’s canonical status.
Submitting and managing sitemaps
Sitemap submission communicates sitemaps to search engines through established mechanisms, ensuring search engines know where to find them.
Google Search Console submission: the primary mechanism for communicating sitemaps to Google. In the Sitemaps section of Google Search Console enter the sitemap URL, sitemap.xml or the full URL, and submit. Google acknowledges the submission and begins processing the sitemap. The Search Console interface shows the number of URLs submitted, the number indexed, and any errors found.
Bing Webmaster Tools submission: Bing has its own webmaster tools that accept sitemap submissions, https://www.bing.com/indexnow or through the Bing Webmaster portal. Submitting to Bing ensures Bing’s crawler discovers the sitemap and processes its URLs.
robots.txt sitemap declaration: adding a Sitemap: directive to robots.txt communicates the sitemap location to all crawlers that read robots.txt, not just those for which manual submission was performed:
This passive discovery mechanism ensures crawlers that read robots.txt before beginning to crawl the site immediately know where to find the sitemap.
Auto-discovery via crawling: Google crawls sites and may discover sitemaps through links within the site or through common sitemap locations, /sitemap.xml, /sitemap_index.xml. Auto-discovery is less reliable than explicit submission, particularly for new sites or after significant site changes. Explicit submission through Google Search Console is the recommended approach for ensuring Google is aware of the current sitemap.
Ping endpoints: the Sitemap Protocol defines ping endpoints that notify search engines of sitemap updates without requiring manual resubmission:
Pinging Google when sitemaps are updated notifies the search engine to re-fetch the sitemap and process any new or changed URLs, accelerating the discovery of fresh content.
Common XML sitemap mistakes
Including non-canonical URLs: the most impactful sitemap error. Redirects, parameter variants, HTTP versions, and non-www URLs included in sitemaps create conflicting canonicalization signals. Sitemaps must contain only the exact canonical HTTPS URLs that should be indexed.
Including noindex pages: pages with <meta name="robots" content="noindex"> tags or noindex HTTP headers should not be included in sitemaps. The noindex directive tells search engines not to index the page while the sitemap says it is important, conflicting signals. Exclude all noindex pages from sitemaps.
Stale or inaccurate lastmod dates: lastmod dates that do not reflect actual content modification dates, automatically set to today’s date for all pages, or never updated from initial values. Inaccurate lastmod dates reduce the value search engines place on the attribute for recrawl prioritisation.
Exceeding size limits: individual sitemaps exceeding 50,000 URLs or 50MB before compression cause parsing errors. Use sitemap index files to split large URL sets across multiple sitemaps.
Including broken URLs: sitemap URLs that return 404 or other error responses. All URLs in a sitemap should return 200 status. Including broken URLs wastes crawl budget and signals poor site maintenance. Audit sitemap URLs regularly to identify and remove broken entries.
Not updating sitemaps after URL changes: leaving sitemaps with old URL structures after site migrations, URL restructuring, or content reorganisation. Updated sitemaps reflecting current canonical URLs are essential after any URL change.