Infrastructure & Networking
What is a request header?
A request header is a key-value pair included in an HTTP request that provides metadata about the request, communicating information about the client making the request, the content of the request body, the capabilities of the client, and the context in which the request is being made. Request headers accompany every HTTP request, sent by browsers, API clients, crawlers, and other HTTP clients before the request body, providing the server with the information it needs to process the request appropriately and return a suitable response.
Every HTTP request contains multiple headers, a typical browser request to load a web page includes 10-20 headers communicating the browser type, accepted content formats, accepted languages, connection preferences, caching preferences, and authentication credentials. Each header is a name-value pair, Host: example.com, User-Agent: Mozilla/5.0..., Accept: text/html,application/xhtml+xml: transmitted as text in HTTP/1.1 or as binary-encoded pairs in HTTP/2 and HTTP/3.
Request headers are one of the primary mechanisms through which web infrastructure makes routing, caching, security, and redirect decisions. A reverse proxy inspects the Host header to determine which virtual host should handle the request. A CDN inspects the Accept-Encoding header to determine whether to serve compressed content. A redirect rule checks the Host header to match domain-specific redirect configurations. A web application firewall inspects headers for attack patterns.
For redirect management request headers provide essential context for redirect rule evaluation, domain-based redirect rules match against the Host header, geographic redirect rules use IP address data derived from client connection information, and device-based redirects check the User-Agent header to distinguish mobile from desktop clients.
HTTP request structure
Understanding where headers fit in the HTTP request structure clarifies how they are transmitted and processed.
Request line: the first line of an HTTP/1.1 request, specifying the method, path, and HTTP version:
Request headers: one header per line following the request line, each header on its own line in Name: Value format:
Empty line: a blank line separating the headers from the request body.
Request body: optional content, present for POST, PUT, and PATCH requests. GET requests typically have no body.
In HTTP/2 this text-based format is replaced by binary-encoded frames, headers are compressed using HPACK compression and transmitted as binary data rather than text. The logical structure, method, headers, body, remains the same but the wire format is binary.
Essential request headers
The web’s core request headers communicate the most important context about every HTTP request.
Host: the most critical request header. Specifies the domain name and optional port of the server being requested, Host: example.com or Host: example.com:8443. The Host header enables virtual hosting, multiple websites served from a single IP address. The server reads the Host header and routes the request to the appropriate virtual host configuration.
For redirect management the Host header is the primary header used in domain-based redirect rules. A redirect rule that should fire for old-domain.com but not for new-domain.com matches against the Host header, if Host == old-domain.com then redirect. Nginx server blocks and Apache virtual hosts both use the Host header for request routing.
User-Agent: a string identifying the client software making the request. Browsers include their browser name, version, rendering engine, and operating system. Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36: Chrome on Windows 10. Search engine crawlers include their crawler identifier, Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html).
User-Agent headers enable device detection, distinguishing mobile browsers from desktop browsers for device-specific redirect rules. They enable crawler identification, serving specific content or redirect responses to known crawlers. User-Agent headers can be spoofed, any HTTP client can set any User-Agent string, making them unreliable for security decisions but useful for optimisation and personalisation.
Accept: specifies the content types the client is willing to receive. Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 indicates the client prefers HTML but accepts XML and any other format. Servers use Accept headers to implement content negotiation, returning the most appropriate content format for each client.
Accept-Language: specifies the client’s preferred languages in priority order. Accept-Language: en-GB,en;q=0.9,fr;q=0.8 indicates a preference for British English, then English generally, then French. Servers use Accept-Language for language-specific content serving, returning the appropriate language version of a page. Redirect management platforms may use Accept-Language for language-based redirect rules, redirecting users to language-specific content based on their browser language preference.
Accept-Encoding: specifies compression formats the client supports. Accept-Encoding: gzip, deflate, br indicates support for gzip, deflate, and Brotli compression. Servers use Accept-Encoding to determine whether to compress responses. CDNs serve pre-compressed assets to clients that advertise compression support, reducing bandwidth consumption significantly.
Connection: controls connection persistence. Connection: keep-alive requests that the server maintain the TCP connection for multiple requests. Connection: close requests that the server close the connection after the current response. Persistent connections, keep-alive, reduce connection establishment overhead for multiple requests to the same server. HTTP/2 makes this header largely redundant, HTTP/2 connections are persistent by default.
Cache-Control: in request context specifies the client’s caching preferences for this specific request. Cache-Control: no-cache requests that the server validate the response even if a fresh cached version exists. Cache-Control: max-age=0 indicates the client does not want a cached response older than zero seconds. A browser performing a hard refresh, Ctrl+F5, sends Cache-Control: no-cache to bypass all caches and request fresh content from the origin.
Authorization: provides authentication credentials for accessing protected resources. Authorization: Bearer eyJhbGciOiJIUzI1NiJ9... provides a JWT bearer token. Authorization: Basic dXNlcjpwYXNzd29yZA== provides Base64-encoded username and password credentials. Authorization headers are inspected by authentication middleware to verify request credentials before processing the request.
Cookie: transmits stored cookies from the browser’s cookie store for the request’s domain. Cookie: session=abc123; preferences=dark-mode; _ga=GA1.2.123456789 sends three cookies. Cookies are used for session management, identifying authenticated users, personalisation preferences, and analytics tracking. Redirect rules may read cookie values to make personalisation-based redirect decisions.
Referer: specifies the URL of the page that initiated the current request. Referer: https://example.com/blog indicates the user navigated from the blog page. Note the intentional misspelling, Referer rather than Referrer: is a historical typo in the HTTP specification that has been maintained for compatibility. The Referer header is used for analytics, understanding where traffic originates, and for security, validating that form submissions originate from expected pages.
Security and redirect-related headers
Several request headers have specific security and redirect management implications.
X-Forwarded-For: a de facto standard header added by proxies and load balancers to forward the original client IP address. When a request passes through a proxy the proxy’s IP address becomes the apparent client IP, the original client IP is lost without explicit forwarding. X-Forwarded-For: 203.0.113.42, 10.0.0.1 indicates the original client IP 203.0.113.42: with 10.0.0.1 being an intermediate proxy. Servers use X-Forwarded-For to access the real client IP for logging, rate limiting, and geographic routing.
For geo-based redirect rules the real client IP, from X-Forwarded-For, rather than the proxy IP is needed for accurate geolocation. Redirect management platforms running behind load balancers must read X-Forwarded-For to access the original client IP for geographic redirect decisions.
X-Forwarded-Proto: indicates the protocol, HTTP or HTTPS, used by the original client. When a load balancer performs SSL termination and forwards requests to backend servers over HTTP the backend receives HTTP requests regardless of whether the original client used HTTPS. X-Forwarded-Proto: https tells the backend that the original client connection used HTTPS, important for forced HTTPS redirect logic. Without X-Forwarded-Proto a backend behind an SSL-terminating load balancer would incorrectly redirect all traffic to HTTPS, even requests already on HTTPS.
X-Forwarded-Host: the original host header value before proxying. Some proxy configurations change the Host header, forwarding requests to backend servers with a backend-specific host value. X-Forwarded-Host preserves the original Host header for use by backend applications that need to generate correct absolute URLs or apply host-based redirect rules.
If-None-Match: conditional request header for cache validation. If-None-Match: "abc123def456" asks the server to return the resource only if its ETag has changed since the value provided. If the resource has not changed the server returns 304 Not Modified: the client uses its cached version. ETag-based validation reduces bandwidth by avoiding retransmission of unchanged resources.
If-Modified-Since: date-based conditional request header. If-Modified-Since: Wed, 15 Jan 2025 10:00:00 GMT requests the resource only if it has been modified after the specified date. The server returns 304 Not Modified if the resource has not changed, otherwise returning the updated resource with a 200 OK response.
Request headers in redirect rules
Request headers are evaluated by redirect management systems to determine which redirect rule, if any, applies to each incoming request.
Host header matching: the foundation of domain-based redirect management. Every redirect rule is scoped to a specific domain, the redirect platform matches the request’s Host header against configured redirect source domains. A request with Host: old-domain.com matches redirect rules for old-domain.com. A request with Host: different-domain.com does not match those rules.
Multi-tenant redirect management platforms serve many customer domains, the Host header is the primary routing mechanism that determines which customer’s redirect rules to apply to each incoming request.
Path-based matching: combined with Host header matching URL path matching determines which specific redirect rule applies within a domain’s rules. The request line provides the path, /old-page: and the redirect platform evaluates path-based rules in order of specificity, exact matches before prefix matches before wildcard matches.
User-Agent based redirects: some redirect configurations serve different redirect destinations based on client type, detected through the User-Agent header. Mobile users may be redirected to mobile-optimised versions of content. Search engine crawlers may receive specific redirect behaviour. User-Agent based redirects require careful implementation, avoiding cloaking: serving different content to crawlers than to users, which violates search engine guidelines.
Geographic redirects from IP headers: geographic redirect rules derive user location from the client IP address, either the direct connection IP or the X-Forwarded-For header for proxied requests. CDN and edge network platforms provide geographic data directly, Cloudflare Workers expose request.cf.country without requiring IP geolocation lookups. Redirect rules using geographic information route users to region-specific content based on their location.
Referer-based redirect rules: redirect rules that consider the Referer header enable context-sensitive redirects, sending users to different destinations based on where they came from. A user arriving from a social media campaign might be redirected to a campaign-specific landing page. A user arriving from a search result might be redirected to content matching their search context. Referer-based redirect rules require careful implementation, Referer headers are not always present and can be spoofed.
Custom request headers
Beyond standard HTTP headers applications frequently use custom headers for application-specific communication.
X- prefix convention, deprecated: custom headers historically used an X- prefix to distinguish them from standard headers, X-Request-ID, X-Correlation-ID, X-Custom-Header. The X- prefix convention was formally deprecated in RFC 6648, the concern being that X- prefixed headers sometimes became widely adopted standards without the prefix being dropped, creating inconsistency. Modern custom headers are defined without the X- prefix though many existing X- headers remain in widespread use.
Request tracing headers: custom headers used for distributed tracing, correlating requests across multiple services. X-Request-ID or X-Correlation-ID carries a unique identifier that is forwarded through all services handling a request, enabling trace reconstruction across service boundaries. Redirect management platforms may generate and forward request tracing headers, enabling correlation of redirect events with downstream destination server logs.
Authentication token headers: beyond the standard Authorization header applications may use custom headers for API authentication. X-API-Key: abc123def456 is a common pattern for API key authentication. Custom authentication headers are processed by authentication middleware, redirect management platforms may use authentication headers to gate access to redirect management APIs.