Meta Robots Tag Explained
How the Meta Robots Tag Works
The meta robots tag is an HTML element placed in the head section of a page that tells search engines how to treat that page. A typical example is <meta name="robots" content="noindex">.
Unlike robots.txt, which applies to whole sets of URLs before they are requested, the meta robots tag applies to the page itself and is read only after the page has been fetched. Because the crawler has to fetch the page to see the tag, the page must not be blocked from crawling for the tag to have any effect.
Common Values: Noindex, Nofollow and Snippet Controls
- noindex asks search engines not to include the page in their index.
- nofollow asks them not to follow the links on the page. Search engines treat this as a hint.
- noarchive asks them not to offer a stored copy of the page.
- nosnippet asks them not to show a text excerpt or video preview in results.
- max-snippet limits how long a text excerpt may be, measured in characters.
Several values can be combined in one tag, separated by commas, and the tag can name a specific crawler instead of all of them. Not every value is honored by every search engine, so the tag works best for simple, clear goals.
When a Page-Level Directive Is the Right Choice
The tag is most useful for individual pages that should stay out of results but remain reachable by visitors, such as thank-you pages, internal search result pages or drafts that were published by mistake. Because it is set per page, it gives precise control that a folder rule in robots.txt cannot.
The meta robots tag works only in HTML pages. Other file types need a different method, which is covered in the X-Robots-Tag header.
Frequently asked questions
Is Noindex the Same as Disallow?
No. Disallow in robots.txt stops crawling, while noindex stops indexing. A page with noindex must be crawled for the instruction to be read, so the two should not be combined on the same URL. The difference is explained in robots.txt vs noindex.
Does a Nofollow Meta Tag Apply to Every Link?
Yes, it applies to every link on the page and treats them all alike. Search engines read it as a hint rather than a strict rule. For control over individual links, the rel attributes described in nofollow link attributes are more precise.