A user agent is a string transmitted in the User-Agent request header that identifies the software client making an HTTP request, communicating the type of client, its name, version, and often information about the operating system and platform on which it is running. Every HTTP client, web browsers, search engine crawlers, API clients, mobile apps, and automated scripts, includes a user agent string that identifies what it is to the servers it communicates with.

The term user agent comes from the original HTTP specification, a user agent is software acting on behalf of a user to make HTTP requests. A web browser is a user agent, it acts on the user’s behalf to retrieve and display web content. A search engine crawler is a user agent, it acts on the search engine’s behalf to discover and index web content. The user agent string is how the client introduces itself to the server, providing identifying information the server can use to tailor its response.

User agent strings have become remarkably complex over the history of the web, a modern Chrome browser on Windows might have a user agent string like Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36: a string that references Mozilla, WebKit, Gecko, Chrome, and Safari despite being none of those things in the strict sense. This complexity is a historical artefact, browsers have progressively added compatibility tokens to their user agent strings to avoid being excluded from content served only to recognised browser types.

For redirect management user agents are significant because they enable identification of crawlers, particularly Googlebot: and because device type detection through user agent parsing enables mobile-specific redirect rules. Understanding user agents helps implement redirect logic that serves different redirect destinations to different client types, while avoiding cloaking violations that result from serving substantially different content to crawlers versus users.

User agent string structure

User agent strings follow a loose format, originally defined in the HTTP specification but extended in practice through browser competition and compatibility requirements.

Original format: the HTTP specification defined a simple user agent format: ProductName/Version (Comment). A minimal compliant user agent string might be MyApp/1.0 or Googlebot/2.1. The product name identifies the software, the version identifies its version, and the optional comment provides additional information in parentheses.

Browser user agent complexity: browser user agents evolved far beyond this simple format through a series of historical compatibility decisions. The progression is instructive:

When Mosaic, the first graphical browser, was released with user agent NCSA_Mosaic/1.0 web servers could identify it. When Netscape Navigator launched it introduced Mozilla/1.0: Mozilla being Netscape’s internal codename. Webmasters began serving advanced features only to Mozilla user agents.

Internet Explorer launched with Mozilla/1.22 (compatible; MSIE 2.0; Windows 95): including Mozilla compatibility token so it would receive the same advanced content as Netscape. The pattern of including other browsers’ tokens for compatibility continued, WebKit added Gecko tokens, Chrome added Safari tokens. Each browser accumulated more compatibility tokens.

Modern Chrome user agent anatomy: dissecting a modern Chrome browser user agent:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36

Mozilla/5.0: legacy compatibility token. All modern browsers include this. (Windows NT 10.0; Win64; x64): operating system information. Windows 10, 64-bit. AppleWebKit/537.36: rendering engine. Chrome uses Blink, a fork of WebKit. (KHTML, like Gecko): compatibility token indicating KHTML/Gecko compatibility. Chrome/120.0.0.0: the actual browser identification, Chrome version 120. Safari/537.36: compatibility token for Safari compatibility.

Crawler user agents: search engine crawlers use more straightforward user agent strings that clearly identify themselves:

Googlebot, Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Bingbot, Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Yandex, Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) DuckDuckBot, DuckDuckBot/1.0; (+http://duckduckgo.com/duckduckbot.html)

Crawler user agents typically include a URL to documentation about the crawler, allowing server operators to understand and verify the crawler’s identity.

User agent detection

Servers parse user agent strings to identify client types, enabling device detection, browser detection, and crawler identification.

Browser detection: identifying the browser type and version from the user agent string. User agent parsing libraries, ua-parser, device-detector, extract structured information from user agent strings. Detection enables browser-specific content serving, delivering alternative content for browsers with limited capabilities.

Browser detection from user agent strings has largely fallen out of favour in modern web development, feature detection through JavaScript APIs is more reliable than user agent-based inference. A browser that claims to support a feature in its user agent string may have a buggy implementation, testing the feature directly is more accurate than inferring support from the user agent.

Device detection: identifying device type, desktop, mobile, tablet, from user agent characteristics. Mobile browsers include device-specific identifiers, iPhone user agents include iPhone, Android browser user agents include Android. Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Mobile/15E148 Safari/604.1 is clearly an iPhone browser user agent.

Device detection enables mobile redirect rules, redirecting mobile users to mobile-optimised content or mobile app deep links. Device-based redirects are a legitimate redirect pattern, serving device-appropriate content to different client types. The distinction from cloaking is that device-based redirects serve equivalent content in an appropriate format rather than serving fundamentally different content to crawlers versus users.

Crawler identification: identifying search engine crawlers from their user agent strings. Googlebot’s user agent contains Googlebot: a reliable identifier for Google’s main search crawler. Crawler identification enables crawler-specific behaviour, serving XML sitemaps, enforcing crawl rate limits, logging crawler activity separately from user traffic.

Crawler verification: user agent strings can be spoofed, any HTTP client can set any User-Agent value. A malicious scraper or DDoS attacker might set their user agent to Googlebot to bypass rate limiting or access crawler-specific content. Verifying that a request actually originates from Google requires reverse DNS lookup: resolving the connecting IP address to a hostname and verifying it is in Google’s published IP range, googlebot.com. Google publishes its crawler IP ranges for verification purposes.

User agents and redirects

User agent information influences redirect behaviour in several important ways, enabling device-specific and crawler-specific redirect logic.

Mobile device redirects: redirecting mobile users to mobile-optimised content. A desktop website at example.com might redirect mobile users to m.example.com or example.com/mobile based on mobile device detection in the user agent. The redirect fires when the user agent indicates a mobile device, desktop users receive no redirect.

Modern responsive web design has reduced the need for mobile-specific redirect domains, responsive sites serve appropriate layouts to all devices from the same URL. But mobile redirects remain common for legacy sites, app deep link redirects, and cases where mobile users should be directed to a mobile app rather than the mobile web.

User agent based redirect rules in Nginx:

# Redirect mobile users to mobile site
if ($http_user_agent ~* "(Mobile|Android|iPhone|iPad)") {
    return 302 https

# Redirect mobile users to mobile site
if ($http_user_agent ~* "(Mobile|Android|iPhone|iPad)") {
    return 302 https

# Redirect mobile users to mobile site
if ($http_user_agent ~* "(Mobile|Android|iPhone|iPad)") {
    return 302 https

User agent based redirect rules in Apache:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} "(Mobile|Android|iPhone|iPad)" [NC]
RewriteRule ^(.*)$ https://m.example.com/$1 [R=302,L]

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} "(Mobile|Android|iPhone|iPad)" [NC]
RewriteRule ^(.*)$ https://m.example.com/$1 [R=302,L]

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} "(Mobile|Android|iPhone|iPad)" [NC]
RewriteRule ^(.*)$ https://m.example.com/$1 [R=302,L]

Cloaking and user agent redirects: the critical distinction between legitimate user agent redirects and cloaking is content equivalence. Redirecting mobile users to a mobile-optimised version of the same content is legitimate, the content is equivalent, only the presentation differs. Redirecting Googlebot to different content than human users receive, serving keyword-optimised content to crawlers while showing different content to users, is cloaking and violates search engine guidelines.

Legitimate user agent redirect patterns treat Googlebot as a desktop user, serving Googlebot the same redirect responses that desktop browser users receive. Redirect rules that specifically detect Googlebot and serve different redirect destinations than regular users are cloaking regardless of the stated justification.

Googlebot user agent in redirect testing: Googlebot’s user agent is visible in server logs, enabling analysis of how Google’s crawler is experiencing redirect configurations. Checking server logs for Googlebot requests confirms whether Googlebot is crawling redirect source domains and following redirects to destinations, useful for verifying that redirect signals are being received and processed by Google.

User agent changes and deprecation

User agent strings have been a source of ongoing tension in web development, their complexity, unreliability, and privacy implications have motivated significant changes in recent years.

User agent reduction, Chrome: Google has been progressively reducing the information content of Chrome’s user agent string through User-Agent Reduction, a process that replaces specific version numbers and platform details with fixed values. The goal is to reduce user fingerprinting, the ability to identify specific users by their unique user agent string combined with other browser characteristics. Reduced user agents make browser-version-specific targeting less precise but preserve basic browser identification.

User-Agent Client Hints, UA-CH: a replacement mechanism for user agent-based client information. Rather than a single complex user agent string containing all device and browser information Client Hints provide structured data through separate request headers, Sec-CH-UA, Sec-CH-UA-Mobile, Sec-CH-UA-Platform. Servers must explicitly request the hints they need, browsers include only the requested hints rather than broadcasting all information in every request. Client Hints improve privacy, only requested information is shared, and provide more structured data, separate headers for mobile status, platform, and browser version are easier to parse than a complex user agent string.

Mobile detection with Client Hints: Sec-CH-UA-Mobile: ?1 indicates a mobile device, ?1 for true, ?0 for false. Servers that need mobile device information for redirect decisions should request and use Client Hints rather than parsing user agent strings, Client Hints are more reliable and future-proof.

Common user agents in redirect management

Understanding specific user agents encountered in redirect infrastructure helps interpret server logs and configure appropriate redirect rules.

Googlebot: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html). Google’s primary search crawler. Verifiable through reverse DNS. Crawls redirect source domains to discover and process redirect signals. Redirect responses seen by Googlebot initiate index updates.

Googlebot Smartphone: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html). Google’s mobile crawler, crawls pages as a mobile device to evaluate mobile experience. Mobile redirect rules should treat this user agent identically to Googlebot, not redirecting it to different content than mobile users receive.

Bingbot: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm). Microsoft Bing’s search crawler. Processes redirect signals similarly to Googlebot.

curl: curl/8.4.0. Command-line HTTP client widely used for testing and API access. Redirect testing with curl using curl -I -L https://example.com/redirect-source follows redirects and displays response headers for each hop, essential for verifying redirect configurations.

Postman: PostmanRuntime/7.35.0. API testing tool user agent. Common in server logs from developers testing APIs and redirect configurations.