Robots meta directives give you page-level control over how search engines treat individual pages — distinct from robots.txt, which operates at a broader, site-structural level. Misunderstanding the difference is a common source of costly mistakes.
noindex: Keeping a Page Out of Search Results
This directive tells search engines not to include a specific page in search results, while still allowing them to crawl it. Useful for thank-you pages, internal search results pages, or thin content you want to keep accessible but not indexed.
nofollow: Controlling Link Equity Flow
Applied to a specific link, nofollow indicates you don't want to vouch for or pass ranking signal to the linked page — commonly used for paid links, user-generated content links, or untrusted external links.
noarchive: Preventing Cached Snapshots
This directive prevents a search engine from storing and displaying a cached version of the page, relevant for content that changes frequently or shouldn't be accessible through an outdated snapshot.
The Critical Difference From Robots.txt
Robots.txt tells crawlers not to access a page at all, while meta noindex allows crawling but prevents indexing — using robots.txt to block a page you actually want deindexed can backfire, since the crawler may never see the noindex directive if it's blocked from accessing the page in the first place.
Common Costly Mistakes
- Accidentally leaving a sitewide noindex directive active after moving from a staging environment
- Blocking a page via robots.txt while also trying to noindex it, preventing the noindex from ever being seen
- Using nofollow indiscriminately across all internal links, unnecessarily restricting internal link equity flow
Auditing Your Directive Implementation
Given how consequential a misapplied directive can be, periodically verifying that your important pages don't have unintended noindex or blocking directives is a worthwhile, simple technical health check.
Catch directive errors before they cost you rankings. SeoWolf's SEO Audit flags unintended noindex tags and directive conflicts across your site.