Robots.txt is one of the smallest files on your entire site — usually just a few lines of plain text — and also one of the easiest to get catastrophically wrong. One misplaced line can quietly deindex an entire section of a site, and because there's no error message, it can go unnoticed for weeks.
What Robots.txt Actually Controls
Robots.txt sits at the root of your domain (yoursite.com/robots.txt) and tells well-behaved crawlers which parts of the site they're allowed to request. It's a set of instructions, not a lock — it doesn't hide content from people who have the URL, and it doesn't remove already-indexed pages.
What it's genuinely useful for:
- Keeping crawlers out of admin areas, internal search results, or staging environments
- Preventing crawl budget from being wasted on low-value, duplicate, or infinite URL parameters
- Pointing crawlers to your XML sitemap
What it can't do:
- Remove a page that's already indexed (that requires a noindex tag or removal request)
- Stop a page from being accessed directly by URL
- Guarantee compliance from every crawler — malicious bots ignore it entirely
A Simple Framework
- Start permissive: allow everything by default unless there's a specific reason to block it
- Disallow only what genuinely shouldn't be crawled — admin panels, internal search, filtered/faceted URLs that create duplicate content
- Add a
Sitemap:line pointing to your XML sitemap - Test the file before publishing, not after
> Tip: Never block a page with robots.txt if the actual goal is to remove it from search results. Blocking a crawler from seeing a page doesn't remove it if it's already indexed — it can even prevent a noindex tag from ever being read, keeping the page stuck in the index indefinitely.
Common Mistakes
- Accidentally leaving
Disallow: /in place after launch, blocking the entire site - Blocking CSS and JavaScript files, which can prevent search engines from properly rendering and evaluating the page
- Confusing robots.txt with noindex tags — they solve different problems
- Never testing changes before pushing them live
A robots.txt generator is a fast way to build a correctly formatted file without hand-writing syntax that's easy to get subtly wrong.
It's a small file, but it sits in front of every crawler that ever visits the site — worth the five minutes it takes to get right, and worth double-checking after every major site change.