crawl-delay
Also called: crawl delay, crawl-delay directive
Crawl-delay is a non-standard robots.txt directive that asks a crawler to wait a set number of seconds between requests to your server, throttling how fast it fetches pages. Google and Yandex ignore it; Bing still honors it.
How each engine reads it
You place the directive inside a user-agent group in robots.txt:
User-agent: bingbot
Crawl-delay: 10Support is inconsistent, which is the main thing to know. Google never adopted it: its parser only recognizes user-agent, allow, disallow, and sitemap, and silently ignores everything else. Yandex stopped honoring it on February 22, 2018 and moved the control into Yandex Webmaster. Bing is the notable holdout that still reads it, treating the number as a relative throttle rather than an exact second count. Bing’s own guidance describes 1 as ‘slow’, 5 as ‘very slow’, and 10 as ‘extremely slow’, and warns against values above 10 because they starve the crawler.
A few things crawl-delay does not do. It never blocks or deindexes a page; it only paces fetching. It also cannot slow Googlebot, so reaching for it as a server-load fix for Google is pointless. To ask Google to ease off, you manage crawl rate on Google’s side (Google crawls algorithmically and backs off when your server returns 500, 503, or 429 responses).
How it affects your traffic
A high crawl-delay quietly caps how many pages Bing will fetch per day, so on a large site new and updated URLs get discovered and indexed slower, which drags on Bing and Yahoo visibility. And because Google ignores the directive outright, leaning on it to calm Googlebot does nothing for server load. If crawler traffic is genuinely straining your origin, the real fix is server-side (caching, rate limits, correct 503 handling) paired with a clean robots.txt, which is core Technical SEO work.
Get Technical SEO that moves the needle
We turn terms like this into ranked pages and qualified pipeline. Start with a free Initial SEO Strategy.