crawling
Also called: crawl, web crawling
Crawling is how search engines discover web pages. Automated programs called crawlers, such as Googlebot, fetch URLs, download their content, and follow the links on each page to reach more pages. It is the first step in search, and a page must be crawled before it can be indexed or ranked.
Search engines find pages in three ways: revisiting URLs they already know, extracting links from a known page to a new one (a category page linking to a fresh post, for example), and reading URLs you submit in a sitemap. A crawler such as Googlebot then fetches each URL, downloads the text, images, and video, and renders the page by running its JavaScript in a recent version of Chrome.
What limits a crawl
Not every discovered URL gets fetched. Googlebot crawls most sites no more than once every few seconds on average, reads roughly the first 2MB of an HTML file, and skips anything blocked in robots.txt or behind a login. One trap worth naming: robots.txt stops crawling but does not guarantee a page stays out of results, while noindex only works if the bot is allowed to crawl the page and read the tag. Block the crawl and the noindex is never seen.
Because Google now indexes mobile-first, most crawl requests come from the smartphone crawler, so the mobile version of your page is the one that counts.
The same mechanics decide whether AI answer engines can reach your content. They send their own crawlers to gather source material, and a page that cannot be fetched cannot be cited. Crawlability is the shared gate for both classic search and AI search.
How it affects your traffic
Crawling is the gate before everything else: a URL that is never fetched cannot be indexed, ranked, or cited by an AI answer engine, so any crawl problem caps your traffic before content or links even matter. Common leaks include crawl budget spent on faceted or parameter URLs, blocked CSS and JavaScript that breaks rendering, slow servers that make Googlebot back off, and orphan pages with no internal links to discover them. Our Technical SEO service audits log files, robots rules, and internal linking so both search and AI crawlers reach the pages you actually want ranked.
Get Technical SEO that moves the needle
We turn terms like this into ranked pages and qualified pipeline. Start with a free Initial SEO Strategy.