X-Robots-Tag Header
What the X-Robots-Tag Header Is
The X-Robots-Tag is an HTTP header that a server includes in its response to tell search engines how to index the file being requested. It carries the same kinds of instructions as the meta robots tag, such as noindex and nofollow, but it is sent in the response headers rather than inside the file itself.
A typical header looks like this: X-Robots-Tag: noindex. Because the header is attached to the response, it works for any file type a server can deliver, not only HTML pages.
When Headers Are More Useful Than Tags
HTML meta tags can only be placed inside HTML documents. Many files on a website have no such place for instructions. A PDF report, a downloadable spreadsheet or an image file can all appear in search results, and an X-Robots-Tag header is the usual way to control them.
Headers are also practical when many URLs need the same rule. A server can apply one setting to an entire folder of documents instead of editing every file. This suits a directory of internal guides that should stay out of search, or files that existed only for a campaign that has ended.
Managing Indexing Headers Carefully
Because headers are invisible on the page, they are easy to forget. A rule added years ago can keep a useful section out of search without anyone noticing. Teams should record where such headers are set and check them whenever a site changes hosts, server software or content rules. Leftover headers are one reason copies of sites end up in results, as described in why staging sites get indexed.
The header is still subject to the same limits as other indexing controls. The file must be crawlable for the header to be seen, and search engines may take time to drop files that were already indexed. A server line such as Header set X-Robots-Tag "noindex" shows how the rule is expressed in one common web server.
A Price List PDF That Should Not Rank
A manufacturer publishes a price list as a PDF each season. Last year's catalog stays on the server, and it begins to appear in search results, where it competes with current product pages and shows outdated prices.
The team cannot add a meta tag to a PDF, so they send an X-Robots-Tag header with noindex for the archive folder. The current catalog remains indexable. This is a clear case where a header is the only practical option.
Frequently asked questions
Can X-Robots-Tag Replace Robots.txt?
Not for the same purpose. The X-Robots-Tag controls indexing, but it only works if the crawler can fetch the file to read the header. Robots.txt controls crawling and works differently. Using both on the same URL usually causes conflicts. The difference is explained in robots.txt vs noindex.
Does X-Robots-Tag Work on Images and PDFs?
Yes. The header can be sent with image files, PDFs, videos and other non-HTML formats, and search engines read it whenever they fetch the file. Because it is set at the server level, one rule can cover many files of the same type.