SEO & Redirects
What is a noindex tag?
A noindex tag is a directive placed on a webpage that instructs search engine crawlers not to include that page in their search index, preventing it from appearing in organic search results. When Googlebot or another search engine crawler crawls a page and finds a noindex directive it processes the instruction and removes the page from the index, or does not add it if the page was not previously indexed. The page continues to exist and be accessible to visitors but it is invisible to organic search.
The noindex directive takes two primary forms, an HTML meta tag in the page’s <head> section and an HTTP response header. Both communicate the same instruction to crawlers, do not index this page.
HTML meta tag form:
HTTP response header form:
The meta tag form is the most commonly implemented, it requires only HTML access and is supported by all major content management systems. The HTTP response header form is more flexible, it works for non-HTML content types like PDFs, images, and JSON responses that cannot contain HTML head elements, but requires server configuration access.
The noindex directive is one of the most powerful and direct tools in technical SEO. Used correctly it precisely controls which pages appear in search results, keeping the index clean and focused on high-quality content. Used incorrectly, particularly when accidentally deployed to pages that should be indexed, it is one of the most damaging technical configurations possible, silently removing valuable pages from search results.
How noindex works
The noindex directive operates through the crawling pipeline, it is only processed when a crawler actually fetches the page and reads its instructions. This crawl dependency has important implications for how quickly noindex takes effect and what happens when crawling is blocked.
Crawl dependency: a noindex directive on a page has no effect until a crawler crawls the page and reads the directive. A page added to the index last month that has a noindex tag added today continues to appear in search results until Googlebot next crawls it. For most pages on well-maintained sites with reasonable crawl frequency this means noindex takes effect within days. For pages that are crawled infrequently the delay may be weeks.
Crawling must be permitted: for a noindex directive to be processed the crawler must be able to fetch the page. A page that is blocked in robots.txt cannot be crawled, Googlebot cannot read the noindex directive on a page it cannot fetch. The counterintuitive consequence is that blocking a page in robots.txt and adding a noindex tag is contradictory, the robots.txt block prevents Googlebot from discovering the noindex instruction.
For pages that should be definitively excluded from the index the correct configuration is allowing crawling in robots.txt, so Googlebot can fetch and process the noindex directive, while including the noindex tag on the page.
Noindex vs robots.txt: this distinction between noindex and robots.txt is fundamental and frequently misunderstood. Robots.txt controls crawling, whether pages are fetched. Noindex controls indexing, whether fetched pages are stored in the index. They operate at different stages of the search engine pipeline and serve different purposes.
A page that needs to be excluded from search results but must be allowed to be crawled, for example to enable noindex discovery, uses noindex without robots.txt blocking. A page that should not be crawled at all, such as internal admin pages, uses robots.txt blocking. For complete exclusion from both crawling and indexing both may be combined, but the robots.txt block must understand it prevents noindex discovery.
Robots meta tag full syntax: the robots meta tag supports multiple directives combined in one tag:
noindex: do not index this page. nofollow: do not follow links on this page. noarchive: do not cache this page. nosnippet: do not show a snippet in search results. noimageindex: do not index images on this page.
Directives can be combined, content="noindex, nofollow" instructs crawlers not to index the page and not to follow its links. Each directive operates independently, noindex alone allows links to be followed while excluding the page from the index.
Bot-specific noindex: the name="robots" attribute applies to all crawlers. Bot-specific variants target specific crawlers:
This applies only to Googlebot, Bingbot and other crawlers are not affected. Bot-specific noindex is used when different search engines should have different indexing instructions for the same page, rarely needed for most sites.
When to use noindex
Noindex is appropriate for specific page types and scenarios, used appropriately it improves site quality and search performance by keeping low-value or sensitive content out of the search index.
Staging and development environments: staging sites, development environments, and preview deployments contain incomplete, incorrect, or duplicate versions of production content. These should never appear in search results. A global noindex, applied through a CMS setting or server configuration, prevents staging content from indexing. Every staging deployment should verify noindex is correctly configured before content is published.
Duplicate content management: pages that are intentional duplicates of canonical content, print versions, alternative format pages, session ID variants, can be noindexed to prevent them from competing with the canonical version in search results. Canonical tags are generally preferred over noindex for duplicate content management, they consolidate SEO equity while allowing the page to be accessible and crawled. Noindex is a stronger signal that is appropriate when canonical tags are not being respected or when the duplicate page provides no SEO value.
Internal search results: site search result pages, example.com/search?q=term: create nearly infinite URL variants with thin, duplicate content. These pages rarely provide value in organic search results, users who want to search a site prefer the site’s own search interface over finding search result pages through Google. Noindexing internal search results prevents this crawl waste and duplicate content.
Faceted navigation pages: e-commerce filter combinations, example.com/products?colour=red&size=large: create large numbers of URL variants with near-duplicate content. Noindexing filter combinations that do not have sufficient unique content worth indexing reduces duplicate content and focuses crawl budget on canonical category and product pages.
Admin and utility pages: account pages, checkout flows, cart pages, login pages, and other utility interfaces should not appear in search results. Users should not find these pages through search, they serve authenticated users in the middle of site interactions. Noindex combined with appropriate authentication prevents these pages from appearing in search results.
Thin content pages: pages with minimal unique content, auto-generated pages, placeholder pages, or pages that are entirely templated with minimal unique value, may benefit from noindex until enough substantive content has been added to make the page worth indexing. Noindexing thin pages prevents them from dragging down site-wide quality assessments.
Privacy-sensitive pages: pages containing personal information, internal documentation, or other content that should not be publicly discoverable in search results should be noindexed. Noindex provides a layer of protection against accidental search indexation, though it should be combined with authentication for truly sensitive content.
Paginated content beyond a threshold: for very long paginated series, page/2, page/3 through page/200: noindexing pages beyond a reasonable threshold while allowing important early pages to be indexed can reduce index bloat. Whether this is appropriate depends on whether the paginated pages contain unique, valuable content that users might find through search.
When not to use noindex
As important as knowing when to use noindex is understanding when not to, misapplied noindex is one of the most common causes of unintentional de-indexing.
Pages that should rank in search results: the most obvious case. Any page that should appear in organic search results must not have a noindex tag. Content pages, product pages, service pages, landing pages, blog posts, any page intended to receive organic search traffic must not have noindex.
Canonical pages in canonical tag relationships: if a page is designated as the canonical version through canonical tags it should not have a noindex directive. A canonical URL that is noindexed creates a contradiction, the canonical tag says the page is authoritative while the noindex says it should not be indexed. Remove noindex from all pages that are canonical destinations in canonical tag relationships.
Redirect destinations: pages that are the destinations of 301 redirects should not have noindex tags. A redirect transfers SEO equity from the source URL to the destination, if the destination is noindexed it cannot appear in search results and the equity transfer serves no purpose. Audit redirect destinations for noindex tags before and after configuring redirects.
Pages linked from other indexed pages: noindexing a page that has many internal links pointing to it disrupts internal link juice flow. PageRank flowing through internal links to a noindexed page is not completely lost, crawlers can still follow links on the page, but the page’s inability to rank reduces the value of the internal links pointing to it. Consider whether noindex or a 301 redirect to a relevant page better serves the SEO purpose.
Noindex and redirects
The interaction between noindex directives and redirects requires careful coordination, configurations that seem reasonable in isolation can create conflicting signals when combined.
Noindex on redirect source pages: pages that redirect all visitors should not typically have noindex tags as well. The redirect is the primary signal that the page is non-canonical, adding noindex creates redundant and potentially confusing signals. If a page redirects visitors to a destination that is the indexed canonical version the redirect alone communicates the canonicalization intent.
Redirect destination noindex, critical failure mode: the most damaging noindex-redirect conflict. A 301 redirect transfers SEO equity from the source to the destination. If the destination has a noindex tag it cannot be indexed, the equity transfer is wasted. The source URL is de-indexed because of the redirect. The destination is not indexed because of noindex. The result is that neither URL appears in search results and all accumulated equity is lost.
This configuration is a critical technical SEO failure that must be prevented through pre-redirect destination auditing and ongoing monitoring. Every redirect destination must be confirmed as indexable before the redirect is deployed.
Temporary noindex during migrations: during domain migrations some organisations temporarily noindex the new domain while redirects are configured and tested, preventing premature indexation of incomplete configurations. This is a legitimate use case but requires careful timing, the noindex must be removed before the migration goes live and before redirect signals are expected to propagate. A forgotten temporary noindex that persists after migration launch prevents the new domain from being indexed despite correct redirect configurations.
Removing noindex and recovering indexation
When noindex is intentionally removed, allowing a previously excluded page to be indexed, recovery requires the crawler to revisit the page and process the updated configuration.
Crawl and processing delay: removing a noindex directive does not immediately re-index the page. Googlebot must crawl the page again to discover that the noindex has been removed and then process the updated page for indexation. For pages that are crawled frequently the delay may be hours to days. For infrequently crawled pages the delay may be weeks.
Google Search Console URL Inspection: the URL Inspection tool’s Request Indexing button submits a specific URL for prioritised crawl consideration, accelerating recrawl after noindex removal. For individual important pages this is the fastest path to re-indexation after removing noindex.
Sitemap resubmission: for large numbers of pages where noindex has been removed resubmitting the XML sitemap signals multiple URLs for crawl consideration simultaneously, accelerating the recrawling process at scale.
Monitoring recovery: after removing noindex track index recovery through Google Search Console Coverage reports, monitoring the count of indexed pages for the affected section. Organic traffic recovery tracks behind index recovery, pages may be re-indexed before their rankings fully recover, particularly after extended noindex periods.
Common noindex mistakes
Global noindex in CMS settings: accidentally enabling a CMS setting that applies noindex globally, often a single checkbox in WordPress or another CMS that says “discourage search engines.” This single setting noindexes the entire site, all pages become invisible in search results. Every deployment to production should verify this setting is disabled.
Noindex in staging that carries to production: staging configurations that include noindex being deployed to production through CMS export, configuration file deployment, or server configuration propagation. Staging noindex configuration must be explicitly removed before production deployment.
Noindex on template-level includes redirect destination pages: a noindex tag added to a template that affects more page types than intended. A developer adds noindex to a template thinking it only affects a specific page type but the template is shared across many page types, including important content pages.
Forgetting to remove temporary noindex: a noindex added during a site migration or major update intended as temporary but never removed after the transition period ends. Pages remain excluded from the index indefinitely after the reason for their exclusion no longer applies.
Using noindex instead of canonical tags for duplicate content: applying noindex to URL parameter variants rather than canonical tags. Noindex prevents the variant from being indexed but does not consolidate its equity to the canonical URL, the variant’s backlinks and any accumulated signals are lost rather than transferred. Canonical tags consolidate equity while excluding the variant from independent indexation.