crawlability

Crawling is the first stage of how Search works. Google “downloads text, images, and videos from pages it found on the internet with automated programs called crawlers” (Googlebot). It finds URLs three ways: revisiting pages it already knows, extracting links from known pages, and reading sitemaps you submit. So crawlability leans heavily on internal linking. Google can only follow a link if it is an <a> element with an href attribute. A <span> with a click handler or a javascript: pseudo-link usually will not be discovered.

What blocks a crawler

A page is effectively invisible when it is disallowed in robots.txt, sits behind a login, returns server errors, or depends on JavaScript that never renders the content or its links. Larger sites also run into crawl limits: Google wants to crawl a site “without overwhelming your servers,” so orphaned pages, long redirect chains, and slow responses spend that budget on the wrong URLs.

Crawlability is not the same as indexability. A page can be crawled and still kept out of the index if Google judges it thin or duplicate. But crawling comes first. Nothing gets indexed, ranked, or quoted in an AI Overview or a ChatGPT answer if the crawler (Googlebot, or an AI search crawler) cannot reach it. That makes crawlability the base layer every other ranking factor sits on top of.