ClaudeBot
Also called: Anthropic crawler, ClaudeBot user agent
ClaudeBot is Anthropic's web crawler that collects public web content to help train its Claude AI models. It identifies itself with the user-agent token ClaudeBot, obeys robots.txt, and is one of three Anthropic crawlers alongside Claude-User and Claude-SearchBot.
Anthropic’s three-crawler split
ClaudeBot is the training crawler. Two sibling agents do different jobs: Claude-User fetches a page in real time when someone asks Claude to read a specific URL, and Claude-SearchBot indexes pages so Claude can cite them in search-style answers. Each uses its own user-agent token, so a robots.txt rule blocks only the agent you name.
ClaudeBot’s full user-agent looks like this:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)To keep it off your site entirely:
User-agent: ClaudeBot
Disallow: /Anthropic states that ClaudeBot honors robots.txt and the non-standard Crawl-delay directive, and publishes its source IPs at claude.com/crawling/bots.json. IP-level blocking is unreliable, though, because the bots run on public cloud ranges.
The strategic catch: blocking ClaudeBot removes your future pages from model training, but it does not stop Claude from reading or citing you at query time, which is governed separately by Claude-User and Claude-SearchBot. Deciding which agents to allow is a deliberate visibility choice, not a single on/off switch.
How it affects your traffic
If your goal is to show up when buyers ask Claude a question, the agents that matter most are Claude-User and Claude-SearchBot, not the training crawler ClaudeBot. Many sites reflexively block every AI crawler and quietly cut themselves out of Claude's answers. Getting the robots.txt policy right, and structuring pages so Claude can extract and cite them, is core work in AI SEO. We audit which Anthropic agents you allow and align that with your AI-search visibility goals.
Get AI SEO that moves the needle
We turn terms like this into ranked pages and qualified pipeline. Start with a free Initial SEO Strategy.