On a large site, robots.txt is a single file that any team can change and that can remove the entire domain from search. Governance matters more than knowing the syntax.
The staging file reaching production
The most damaging error in SEO is a blanket disallow deployed with a configuration promotion. It requires no malice and no incompetence — just an environment file copied along with everything else.
Treat it as protected configuration
Version control it, require review on changes, and add a deployment check that fails if production contains a blanket disallow. Relying on someone remembering to look is how this survives for weeks.
Monitor the live file, not the repository
What matters is what is served, which can differ from what is committed — a CDN rule, a plugin or a platform default can all override it. Alert on the content of the live file changing.
Blocking is the wrong tool for most requests
Teams ask for robots.txt blocks when they want pages out of the index. That is noindex, not disallow. Blocking prevents the crawler from reading the directive, so blocked pages can persist in results indefinitely — and the person who requested it will report it as not working.