web crawler

A crawler starts from a set of known URLs (previously visited pages, links found on other pages, or the XML sitemap you submit), fetches each page, and extracts its links to queue up new URLs to visit. Google’s crawlers then render the page with a recent version of Chrome and run its JavaScript before the content moves to indexing. Google runs its crawl across many machines in distributed data centers, so your server logs often show hits from several IP addresses.

What you can control

You decide what a crawler is allowed to reach. A robots.txt file at your domain root tells compliant crawlers which paths to skip, and Google’s common crawlers (like Googlebot) always respect those rules. Crawlers identify themselves three ways: the user-agent string, the source IP, and a reverse-DNS hostname you can verify. Google fetches over HTTP/1.1 and HTTP/2 and accepts gzip, deflate, and Brotli compression. For an HTML page, Googlebot reads only the first 2MB and ignores anything past that cutoff (Google lowered this from 15MB in March 2026; a 15MB default still applies to other Google crawlers that do not set their own limit).

Not every discovered URL actually gets crawled. Server errors (HTTP 500s), network timeouts, and robots.txt blocks all keep pages out, and Google deliberately throttles its crawl rate to avoid overloading your server. The practical takeaway: a page that is never crawled can never be indexed, which makes crawlability the first gate every other ranking factor sits behind.