crawl rate
Also called: crawl rate limit, crawl speed, Googlebot crawl rate
Crawl rate is how fast a search bot like Googlebot requests pages from your site, expressed as the number of parallel connections it opens and the delay between fetches. Google sets it automatically from your server's health, crawling as much as it can without overloading you.
How Google decides crawl rate
Google measures a site’s crawl capacity limit as “the maximum number of simultaneous parallel connections that Google can use to crawl a site, as well as the time delay between fetches.” The rate tracks your server. Fast, healthy responses let Googlebot open more connections. Slow responses, or a run of 500, 503, or 429 status codes, make it back off. Crawl rate is one half of crawl budget; the other half is crawl demand, which is how much Google wants to crawl you based on your site’s size, freshness, and quality.
You cannot dial crawl rate up by hand. Google retired the Search Console crawl rate limiter tool on January 8, 2024, so there is no slider anymore. The levers now are indirect: serve pages faster and free server capacity to earn more crawling, or return 500/503/429 responses for a day or two in an emergency to force Googlebot to slow down. Google is explicit that you can report over-crawling but cannot request an increase.
Watch the Crawl stats report in Search Console (under Settings) for requests per day, average response time, and host status. A rising response-time line usually comes right before Google throttles you.
Example: an ecommerce site whose faceted navigation spawns thousands of near-duplicate ?color=&size= URLs burns its crawl rate on junk instead of product pages. Fixing the crawl trap, not asking for “more crawl,” is the win.
How it affects your traffic
When Googlebot spends its crawl rate on redirect chains, faceted-navigation duplicates, and slow responses, your genuinely important pages get crawled less often, so new content and edits take longer to appear in search results and in AI answers that pull from the fresh index. On a small brochure site this rarely bites, but past a few thousand URLs, wasted crawl rate turns into stale rankings and slow indexation of anything new. A technical SEO pass tightens server response times, prunes crawl traps, and cleans up status-code signals so more of your budget lands on pages that actually earn traffic. That work sits at the center of our Technical SEO service.
Get Technical SEO that moves the needle
We turn terms like this into ranked pages and qualified pipeline. Start with a free Initial SEO Strategy.