Robots.txt: The Complete Guide

What It Actually Is

A robots.txt file is a plain text file sitting at your site’s root (yoursite.com/robots.txt) that tells search engine crawlers which URLs they’re allowed to request. Its main job is to prevent your site from being overloaded with crawler requests — it was never designed to keep pages out of Google’s index. That distinction trips up a lot of SEOs, so it’s worth internalizing early: robots.txt controls crawling, not indexing. 

The Disallow Trap

Here’s the part that surprises most beginners: even if you block a URL in robots.txt, Google can still find and index that URL if it’s linked from somewhere else on the web. When that happens, the URL can show up in search results with just the address and maybe some anchor text — no real snippet, since Google never crawled the content. If your actual goal is “don’t let this page appear in search,” robots.txt is the wrong tool. The correct approach is a noindex tag or header, or password-protecting the page entirely.

Core Syntax

The file works with a handful of directives:

• User-agent — which bot the rule applies to (* for all, Googlebot for Google specifically)
• Disallow — the path you don’t want crawled
• Allow — an exception carved out of a broader Disallow
• Sitemap — points crawlers to your XML sitemap location

A basic block looks like:

• User-agent: *
• Disallow: /admin/
• Disallow: /cart/
• Sitemap: https://yoursite.com/sitemap.xml

You can stack multiple User-agent blocks to give different bots different instructions — say, allowing Googlebot broader access while restricting a scraper bot entirely.

What’s Actually Worth Blocking

Common real-world uses: admin panels, internal search results pages, filter/parameter URLs that create duplicate content, staging environments, and shopping carts. The goal is reducing wasted crawl budget on pages with no search value — not hiding sensitive content, since robots.txt files are public by nature and offer zero security.

The Rules Can Fight Each Other

Combining crawling rules (robots.txt) with indexing rules (noindex tags) carelessly can cause them to counteract each other — for instance, if Googlebot is blocked from crawling a page, it also can’t see a noindex tag sitting on that page, so the tag never gets honored. If you truly want a page gone from search, don’t disallow it in robots.txt; let it be crawled so the noindex tag can do its job.

Testing Before You Ship

Never edit robots.txt blind. Google Search Console’s robots.txt tester (or the URL Inspection tool for individual pages) lets you confirm a rule blocks what you intend — and nothing else. A single misplaced / can accidentally deindex an entire site.

The One-Line Takeaway

Robots.txt manages crawler traffic and server load. It is not a privacy tool, not a security tool, and not a guaranteed way to keep something out of Google. Treat it as traffic control, and use noindex or password protection for anything that actually needs to disappear from search.