AI crawler
Also called: AI bot, LLM crawler, AI web crawler
An AI crawler is an automated bot that fetches web pages to train large language models or to ground the answers AI assistants give. Examples include OpenAI's GPTBot, Anthropic's ClaudeBot, and Google's Google-Extended token, most of which site owners manage through robots.txt.
Training crawlers vs retrieval fetches
Not every AI bot does the same job. Some crawl broadly to build training datasets; others fetch a single page in real time because a user asked a question. OpenAI runs several, and three matter here: GPTBot, which “crawl[s] content that may be used in training our generative AI foundation models”; OAI-SearchBot, “used to surface websites in search results in ChatGPT’s search features”; and ChatGPT-User for live user-triggered visits. Anthropic mirrors the split with ClaudeBot (training), Claude-SearchBot (search quality), and Claude-User (live requests).
Google is the outlier. Google-Extended is not a fetching bot at all. It is a control token you set in robots.txt to say whether content Googlebot already crawled may train Gemini or ground its answers. Google states that using it “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal.”
You steer most of these by user-agent. To block training but keep AI search visibility:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /That is the tradeoff worth thinking through. Blocking training crawlers protects your content from model training, but blocking retrieval crawlers removes you from the AI answer surfaces where buyers now look.
How it affects your traffic
AI assistants now sit between your site and a growing share of buyers, and whether you show up in their answers depends on which of these bots you let in. Block the wrong user-agent and you can quietly vanish from ChatGPT, Gemini, or Claude results while your Google rankings still look healthy. Our Technical SEO work audits your robots.txt and server logs so training, search, and user-triggered bots are each handled on purpose, not by a copy-pasted block rule that costs you visibility where it earns traffic.
Get Technical SEO that moves the needle
We turn terms like this into ranked pages and qualified pipeline. Start with a free Initial SEO Strategy.