crawl budget
Also called: crawl quota
Crawl budget is the number of URLs Googlebot will crawl on your site in a given period. It is set by how much crawling your server can handle (the crawl capacity limit) and how much Google wants to crawl your pages (crawl demand).
Google splits crawl budget into two levers. The crawl capacity limit is how many simultaneous connections Googlebot will open, plus the delay between them. It rises when your server responds fast and cleanly, and drops when pages slow down or throw 5xx errors. Crawl demand is how badly Google wants your URLs, driven by inventory size, popularity, and how stale the content looks.
Who actually needs to care
Google says most sites do not. The official guidance targets sites with roughly 1M+ pages that change weekly, 10K+ pages that change daily, or a large share of URLs stuck as Discovered - currently not indexed in Search Console. Below that, Google usually crawls everything fine on its own.
Waste is the real issue. Faceted navigation, session parameters, duplicate URLs, redirect chains, and soft 404s all burn crawls on pages that never earn traffic, so the pages you care about get seen less often. Google’s own fixes: consolidate duplicates, block low-value paths in robots.txt, return 404/410 for dead URLs, and keep sitemaps clean. One trap to avoid is using noindex to save budget. It does not work, because Google still has to crawl the page to read the tag.
How it affects your traffic
Crawl budget is a technical problem, not a content one. If Googlebot spends its visits on parameter junk and redirect chains, your new product pages, posts, and landing pages can sit undiscovered for weeks, which is pure lost organic traffic. A technical SEO engagement fixes this at the source: log-file analysis to see what Googlebot actually hits, robots and canonical cleanup, sitemap hygiene, and faster server response so more of every crawl lands on pages that convert.
Get Technical SEO that moves the needle
We turn terms like this into ranked pages and qualified pipeline. Start with a free Initial SEO Strategy.