Meta Robots Tag: The Complete Guide

Search engines don’t automatically know how you want each page on your site treated. The meta robots tag fixes that gap — it’s a small piece of HTML that tells crawlers whether to index a page, follow its links, cache it, or show it in search results at all. Get it wrong, and you can accidentally hide revenue-driving pages or let junk pages flood the index. This guide breaks down how the tag works, every directive worth knowing, and the mistakes that trip up even experienced SEOs.

What Is the Meta Robots Tag?
How It Differs From Robots.txt
Core Directives
Advanced Directives
X-Robots-Tag: The HTTP Header Alternative
Common Mistakes
Conclusion
FAQs

Key Takeaways

The meta robots tag lives in a page’s <head> and controls indexing, link-following, and snippet/cache behavior on a per-page basis.
Robots.txt controls whether a page gets crawled; the meta robots tag controls what happens once it has been crawled — the two serve different jobs and shouldn’t be confused.
Never disallow a page in robots.txt while also trying to noindex it — crawlers can’t see a directive on a page they’re blocked from fetching, so the page can stay indexed indefinitely.
Directives are comma-separated and not case-sensitive, so NOINDEX, NOFOLLOW and noindex,nofollow behave identically.
For non-HTML files like PDFs and images, which have no <head>, the X-Robots-Tag HTTP header is the only way to send the same instructions.
Forgetting to remove a site-wide noindex after a staging-to-production migration is one of the most common causes of a site vanishing from search results.

What Is the Meta Robots Tag?

The meta robots tag is an HTML element placed in a page’s <head> section that instructs search engine crawlers on how to handle that specific URL. Without one, crawlers default to indexing the page and following its links — which can be a problem for pages like internal admin panels, thank-you pages, or duplicate content that were never meant to surface in search results.

How It Differs From Robots.txt

It’s easy to conflate the two, but they operate at different stages. Robots.txt controls crawl access at the site level — it tells bots which paths they’re allowed to request. The meta robots tag, by contrast, only matters once a page has actually been fetched, since it governs indexing and link behavior for that specific document. A disallowed URL can still appear in search results with no snippet if other sites link to it, precisely because Google never got the chance to see a noindex tag on the page itself.

Core Directives

index / noindex — Whether the page should appear in search results. Index is the default, so it’s rarely written explicitly.
follow / nofollow — Whether crawlers should follow (and pass authority through) the links on the page. Note that a link’s fate can still depend on other pages linking to the same URL without nofollow.
nosnippet — Prevents a text (or video) snippet from showing in search results for that page.
noarchive — Blocks a cached copy of the page. Its practical effect is limited today since Google no longer surfaces cached page links in results.

Advanced Directives

Beyond the basics, several directives give finer control:

noimageindex — Prevents images on the page from being indexed.
notranslate — Stops Google from offering a translated version of the page in results.
unavailable_after — Sets a date after which the page should stop appearing in search results.
max-snippet, max-image-preview, max-video-preview — Limit the length or size of previews shown in search results.
indexifembedded — Allows indexing of content when it’s embedded in another page via iframe, even if that content carries a noindex on its own.

X-Robots-Tag: The HTTP Header Alternative

Files without an HTML <head> — PDFs, images, videos — can’t carry a meta tag, so the X-Robots-Tag HTTP header delivers the same directives at the server level instead. It supports every directive available to the meta robots tag and can even target specific crawlers, such as sending different instructions to Googlebot versus Bingbot. It’s also useful for applying rules programmatically across large batches of URLs.

Common Mistakes

Combining disallow with noindex. If robots.txt blocks a URL, crawlers never see the noindex tag, so the page can remain indexed.
Pairing noindex with a canonical tag. This sends mixed signals and can, in rare cases, cause the noindex to carry over to the canonical target.
Applying noindex to paginated pages. Google generally understands pagination without needing this intervention.
Leaving a staging noindex in place after launch. A site-wide noindex set during development is easy to forget, and it can pull an entire site out of search results after go-live.

Conclusion

The meta robots tag is a small piece of markup with outsized consequences. Used correctly, it keeps your index clean, protects link equity, and controls exactly how your pages are presented in search results. The key is understanding that it works downstream of crawling, not instead of it — so pair it thoughtfully with robots.txt rather than treating the two as interchangeable.

Frequently Asked Questions

Does the meta robots tag block crawling? No. It only affects what happens after a page is crawled — indexing, link-following, and presentation. To block crawling itself, use robots.txt.

Where does the meta robots tag need to be placed? Conventionally in the <head>, though Google has confirmed it will also respect the tag if it appears in the <body>.

Can I target specific search engines? Yes. Using googlebot or bingbot as the tag’s name attribute instead of robots sends directives to that crawler only.

What happens if I use noindex and disallow together? The noindex is likely to be ignored, since a blocked page is never fetched and the crawler never sees the tag — the page can stay indexed.

Is the meta robots tag case-sensitive? No. Directives work regardless of capitalization, and spacing after commas is optional.