SEO & Redirects
What is a robots.txt file?
A robots.txt file is a plain text file placed at the root of a website that instructs web crawlers, search engine bots, automated scrapers, and other automated agents, which parts of the site they are permitted to access and which parts they should not crawl. It is the primary mechanism through which website owners communicate crawling permissions to automated agents, providing a voluntary, convention-based protocol for managing how bots interact with web content.
The robots.txt file implements the Robots Exclusion Protocol, a standard established in 1994 that defines how websites and crawlers communicate about crawling permissions. The protocol is voluntary, crawlers are not technically prevented from accessing disallowed content, they are only instructed not to. Well-behaved crawlers, including Googlebot, Bingbot, and other reputable search engine crawlers, respect robots.txt instructions. Malicious scrapers and poorly implemented crawlers may ignore them entirely.
The name robots.txt reflects its original purpose, a text file for communicating with web robots. The file lives at a specific, standardised location, https://example.com/robots.txt: the root of the domain. Every crawler checks this location before beginning to crawl a site, the file is one of the very first things a crawler fetches when it begins indexing a domain. Any crawler that does not check robots.txt before crawling is considered poorly implemented by web standards.
For redirect management and technical SEO robots.txt is an important tool for controlling crawl budget allocation, directing crawler attention toward valuable content and away from low-value, duplicate, or sensitive URL spaces. Understanding robots.txt correctly, including its capabilities and its important limitations, is essential for anyone managing technical SEO for a website.
How robots.txt works
A robots.txt file consists of one or more blocks, each block specifying a set of rules for a specific crawler or group of crawlers. The file uses a simple key-value syntax that has been extended over the years with additional directives beyond the original specification.
File location: robots.txt must be placed at the root of the domain, https://example.com/robots.txt. It cannot be placed in subdirectories, https://example.com/subdirectory/robots.txt has no special meaning to crawlers. The root-level robots.txt file applies to the entire domain. Subdomains have their own robots.txt files, https://blog.example.com/robots.txt governs crawling on the blog subdomain independently of https://example.com/robots.txt.
User-agent: the User-agent directive specifies which crawler the following rules apply to. Crawlers identify themselves through user agent strings, Googlebot uses the user agent Googlebot, Bingbot uses Bingbot, and so on. The wildcard * applies rules to all crawlers not explicitly named. Multiple user agent directives can appear in a file, each followed by the rules specific to that crawler.
Disallow: the Disallow directive specifies URL paths that the specified crawler should not access. Disallow: /admin/ prevents crawling of the /admin/ directory and all URLs beneath it. Disallow: / prevents crawling of the entire site. Disallow: with no value is a no-op, it does not disallow anything.
Allow: the Allow directive, a widely supported extension to the original specification, specifies URL paths that should be accessible even within a disallowed parent directory. Allow: /public/ within a block that disallows a parent directory explicitly permits access to the /public/ subdirectory. Allow directives override Disallow when both apply to the same URL, the more specific rule takes precedence.
Sitemap: the Sitemap directive specifies the location of the site’s XML sitemap, a separate document listing all URLs the site wants crawled. Including the sitemap URL in robots.txt helps crawlers discover the sitemap without needing to find it through other means:
Crawl-delay: the Crawl-delay directive, supported by some crawlers but not Googlebot, specifies a minimum delay in seconds between successive crawl requests. Crawl-delay: 10 instructs crawlers to wait 10 seconds between requests. For sites where crawler activity affects server performance crawl-delay can reduce the crawling rate, though for Google this is better controlled through Google Search Console’s crawl rate settings.
Example robots.txt configurations
Understanding robots.txt through practical examples clarifies how the directives work together to create crawling policies.
Allow all crawlers access to everything: an empty robots.txt file or a file with no disallow directives permits all crawlers to access all content:
This explicit allow-all configuration confirms that all crawlers may access all content, equivalent to having no robots.txt file but more explicit.
Block all crawlers from everything: used for staging environments or sites not ready for public indexing:
This prevents all crawlers from accessing any content on the site. Search engines that have previously indexed the site will eventually deindex it when they find that all crawls return disallowed responses.
Block specific directories: prevent crawling of admin, internal search, and duplicate content areas:
Block URL parameter patterns: prevent crawling of parameter variants that create duplicate content:
The asterisk in URL paths matches any characters, these rules block any URL containing the specified parameter.
Crawler-specific rules: different rules for different crawlers:
Googlebot is given specific rules while all other crawlers receive a broader set of restrictions.
What robots.txt cannot do
Understanding the limitations of robots.txt is as important as understanding what it can do, several common misconceptions about robots.txt lead to incorrect implementations.
robots.txt does not prevent indexation: the most critical limitation. Disallowing a URL in robots.txt prevents crawlers from accessing that URL but does not prevent the URL from being indexed. If other pages link to a disallowed URL search engines can discover and index that URL from the links, without ever crawling its content. The indexed version may show only the URL with no snippet, because the content was never crawled, but the URL can still appear in search results.
To prevent a page from appearing in search results a noindex meta tag on the page itself or in an HTTP response header is required, not a robots.txt disallow directive. Robots.txt and noindex serve different purposes, robots.txt controls crawling, noindex controls indexation.
robots.txt is not enforced: well-behaved crawlers respect robots.txt by convention. Malicious scrapers, poorly implemented bots, and crawlers that choose to ignore the protocol can access disallowed content freely. robots.txt provides no technical access restriction, it is a request, not a barrier. Sensitive content that should not be publicly accessible should be protected through authentication or access control, not relying on robots.txt.
robots.txt does not affect all crawlers equally: robots.txt rules apply to crawlers that read and respect the file. Different search engines interpret robots.txt rules differently, Google supports certain extensions like Allow and wildcard matching that other search engines may not implement identically. Crawler behaviour for less common crawlers may differ from Googlebot behaviour.
Crawl-delay is not supported by Googlebot: the Crawl-delay directive is ignored by Googlebot. Google’s crawl rate is controlled through Google Search Console settings, not robots.txt. Other major crawlers may support Crawl-delay but Googlebot does not.
robots.txt and crawl budget
Robots.txt is a significant tool for crawl budget optimisation, directing crawler attention away from low-value URL spaces and toward valuable content. This application is particularly relevant for large sites where crawl budget is a meaningful constraint.
Blocking URL parameter spaces: URL parameters that create duplicate content: tracking parameters, sort parameters, filter parameters, session IDs, consume crawl budget without contributing unique indexable content. Blocking these parameter patterns in robots.txt prevents crawlers from wasting budget on duplicate URL variants.
For example blocking all URLs containing UTM tracking parameters:
This prevents crawling of any URL with a UTM parameter, freeing that crawl budget for unique content.
Blocking internal search results: internal site search creates potentially infinite URL spaces, every search query generates a unique URL. These search result pages rarely have external backlinks and often contain thin or near-duplicate content. Blocking internal search from crawling conserves budget:
Blocking admin and utility pages: login pages, admin interfaces, shopping cart pages, and other utility URLs have no place in search indexes. Blocking them prevents crawl waste:
Blocking staging and development paths: staging subdirectories or paths used for testing should not be indexed or unnecessarily crawled:
Crawl budget reallocation: blocking low-value URL spaces does not directly increase the crawl rate for other parts of the site, the total crawl budget is determined by the site’s overall authority and server performance. However reducing wasted crawl requests on blocked areas means more of the available crawl budget is spent on valuable content, improving indexation coverage for pages that matter.
robots.txt and redirects
The interaction between robots.txt and redirects has important implications for how crawlers discover and process redirect configurations.
Crawlers follow redirects regardless of robots.txt: if a URL is disallowed in robots.txt but that URL serves a redirect Googlebot follows the redirect to its destination. The disallowed URL is not crawled, the redirect is processed, and the destination URL is crawled and potentially indexed. This means disallowing redirect source URLs in robots.txt does not prevent crawlers from following the redirects and indexing the destination.
Do not block redirect source URLs: a common mistake is blocking redirect source URLs, old URLs that now redirect, in robots.txt to prevent them from being crawled. This is unnecessary, crawlers follow the redirect regardless of the disallow directive and discover the destination. Blocking redirect sources in robots.txt serves no useful purpose and may interfere with how search engines process the redirect signals.
robots.txt for redirect infrastructure: redirect management domains, domains that exist solely to redirect traffic to a primary domain, may have robots.txt files that disallow all crawling since there is no content on the redirect domain to index:
This prevents crawl budget waste on redirect-only domains that serve no unique content. The redirects function regardless of robots.txt, crawlers follow redirects without needing to crawl the redirect source domain extensively.
Sitemap location in robots.txt: including the XML sitemap URL in robots.txt ensures crawlers can easily find the sitemap, helping them discover canonical URLs efficiently:
The sitemap guides crawlers to important content while robots.txt disallow directives prevent crawl waste on low-value areas.
Common robots.txt mistakes
Blocking CSS and JavaScript files: preventing crawlers from accessing CSS and JavaScript that the site relies on for rendering. Googlebot renders pages similarly to browsers, if it cannot access CSS and JavaScript it cannot render pages correctly. Poorly rendered pages may be assessed as lower quality. CSS and JavaScript should generally be crawlable.
Blocking the entire site on production: deploying a staging robots.txt configuration, Disallow: /: to a production server accidentally. This is one of the most damaging technical SEO mistakes possible, preventing Googlebot from crawling any content immediately begins the deindexation process. Always verify robots.txt configuration after deployments.
Expecting robots.txt to prevent sensitive content indexation: relying on robots.txt to keep sensitive content out of search results. As described above robots.txt prevents crawling, not indexation. Pages linked from other pages can be indexed without being crawled. Sensitive content requires authentication, not robots.txt.
Inconsistent rules for important crawlers: creating overly broad rules in the wildcard block that accidentally block Googlebot from important content. Always test robots.txt rules against actual URL patterns to verify the intended crawlers are blocked or allowed as intended.
robots.txt syntax errors: incorrect syntax in the robots.txt file causes unpredictable behaviour. Missing line breaks between user-agent blocks, incorrect directive spellings, incorrect path formats, all can cause rules to be interpreted incorrectly or ignored. Validating robots.txt through Google Search Console’s robots.txt tester identifies syntax errors.
Testing and validating robots.txt
Several tools verify that robots.txt is correctly configured before errors affect crawling.
Google Search Console robots.txt tester: a tool within Google Search Console that shows the current robots.txt file as Googlebot sees it and allows testing specific URLs to verify whether they are blocked or allowed. This is the definitive tool for verifying robots.txt rules from Googlebot’s perspective.
robots.txt validators: various online tools parse robots.txt files and check for syntax errors, confirming that the file is correctly formatted and that rules are interpretable by crawlers.
Manual URL testing: checking specific URLs against robots.txt rules manually, tracing which user-agent blocks and disallow/allow directives apply to each URL, to verify that the intended URLs are blocked or allowed.
Server log analysis: reviewing server logs for Googlebot requests, confirming that Googlebot is crawling the intended pages and not making requests to disallowed URLs. Googlebot requests to disallowed URLs in server logs indicate the crawler is not respecting robots.txt, a signal worth investigating.