Googlebot
Also called: google crawler, google bot, googlebot smartphone
Googlebot is the crawler Google Search uses to discover and fetch pages from the web. It comes in two variants, Googlebot Smartphone and Googlebot Desktop, and since mobile-first indexing most sites are crawled primarily by the smartphone crawler.
How Googlebot works
Googlebot discovers URLs three ways: revisiting pages it already knows, following links it extracts from those pages, and reading sitemaps you submit. Crawling is not guaranteed. In Google’s words, “Googlebot doesn’t crawl all the pages it discovered,” because some are blocked in robots.txt or sit behind a login.
When it fetches a page, Googlebot downloads the first 2MB of an HTML file (up to 64MB for PDFs) and requests linked CSS and JavaScript separately. It then renders the page and runs JavaScript “using a recent version of Chrome,” so client-side content can be read, but only after that render step completes. Google keeps the load light: it “shouldn’t access your site more than once every few seconds on average.”
Two things trip people up. First, the user agent string is easy to spoof, so verify real Googlebot with a reverse DNS lookup or by matching the source IP against Google’s published ranges, never by trusting the header alone. Second, both variants obey the same robots.txt product token, so you cannot allow the desktop crawler while blocking the mobile one.
The order matters: a page has to be crawled before it can be indexed, and indexed before it can rank. If Googlebot cannot reach or render a URL, nothing downstream happens.
How it affects your traffic
Every organic visit, and every AI Overview citation, starts with a page Googlebot could reach, render, and read. When crawling breaks (a stray robots.txt block, JavaScript that never renders, a slow server that throttles the crawl, orphan pages with no internal links), those URLs never enter the index and earn zero traffic no matter how good the content is. Clearing those blockers is the core of a Technical SEO engagement: log analysis to confirm what Googlebot actually fetches, crawl-budget cleanup on large sites, and render checks so mobile-first crawling sees your full page.
Get Technical SEO that moves the needle
We turn terms like this into ranked pages and qualified pipeline. Start with a free Initial SEO Strategy.