Crawlability vs. Indexing: Two Different Problems That Look the Same

"Why isn't my page ranking?" often gets answered as a content problem, when the real issue is much more basic: the page was never crawled, or it was crawled but never indexed. Content quality can't help a page that search engines don't know exists yet.

Two Separate Gates

Crawlability is whether a search engine's bots can access a page at all โ€” whether they can find it and successfully request it.

Indexing is whether, having crawled it, the search engine chose to add that page to its index and make it eligible to appear in results.

A page can be perfectly crawlable and still not get indexed, if it's judged too thin, too duplicate, or too low-value. A page can also be well-written and valuable and still never get indexed, simply because it was never crawled in the first place.

Common Reasons a Page Isn't Crawled

  • No internal links point to it (an orphan page)
  • It's blocked in robots.txt, intentionally or by mistake
  • It sits too many clicks deep in the site structure
  • It's a genuinely new page and hasn't been discovered yet

Common Reasons a Crawled Page Isn't Indexed

  • A noindex tag is present, intentionally or left over from staging
  • The content is judged too thin or too similar to existing indexed pages
  • A canonical tag points to a different URL
  • The page returns inconsistent signals (indexed one moment, blocked the next)

A Simple Framework

  1. Confirm the page is actually being crawled โ€” check server logs or crawl stats for bot activity
  2. If it's not being crawled, check for orphan status, robots.txt blocks, or excessive depth
  3. If it is being crawled but not indexed, check for noindex tags, canonical conflicts, or thin content
  4. Request indexing directly through Search Console once the underlying issue is fixed

> Tip: Always check the actual live source before assuming โ€” a noindex tag left over from a staging environment is one of the most common and most overlooked causes of an otherwise-great page simply never showing up.

Common Mistakes

  • Assuming a ranking problem is always a content quality problem
  • Never checking the raw source code, only the rendered page in a browser
  • Fixing the issue but never requesting re-indexing afterward
  • Blocking a page in robots.txt while also trying to remove it with noindex โ€” the crawler needs to reach the page to read that tag

Pulling a page's source code directly and checking it against a Google index checker is the fastest way to tell which of the two problems you're actually dealing with.


Before questioning whether a page is good enough, it's worth confirming search engines have actually had the chance to judge it at all.