πŸ“š General & Other

The One-Line Mistake That Removes a Site From Search

in a live robots.txt blocks the entire site. It is a single line, it is trivially easy to deploy by accident, and it is the most damaging SEO error that exists.

Disallow: / in a live robots.txt blocks the entire site. It is a single line, it is trivially easy to deploy by accident, and it is the most damaging SEO error that exists.

How it happens

A staging environment carries a blanket disallow to keep it out of the index. Someone deploys the staging configuration to production. Nothing looks wrong β€” the site loads perfectly for visitors β€” and rankings begin disappearing within days.

Test before and after every deploy

Check a handful of important URLs against the live file after any release that touches server configuration. It takes a minute and catches the one error that can undo a year of work.

Monitor rather than remember

Set up an alert on the contents of the file. Relying on someone thinking to check is how this survives long enough to cost traffic β€” the damage compounds daily and recovery takes far longer than the outage did.

Try it: Robots.txt Tester on SeoWolf's Notepad.