CCBot
Also called: Common Crawl bot, CCBot/2.0
CCBot is the web crawler run by Common Crawl, a nonprofit that publishes a free, petabyte-scale public archive of web pages. It identifies itself with the user-agent CCBot/2.0, and its open dataset has been used to train large language models such as GPT-3, LLaMA, and BLOOM.
Common Crawl runs CCBot on a roughly monthly cadence and stores raw pages, metadata, and extracted text in public files on Amazon S3. Because that archive is free and openly available, it became a default training source for many foundation models, so one crawl feeds dozens of downstream AI systems at once. Blocking CCBot therefore reaches further than blocking any single company’s bot.
Controlling CCBot
CCBot honors robots.txt. To keep it out entirely:
User-agent: CCBot
Disallow: /To slow it instead of blocking, Common Crawl obeys Crawl-delay (for example Crawl-delay: 2 limits it to one request every two seconds). It also respects the nofollow attribute on links. The real CCBot runs from dedicated IP ranges with reverse DNS to crawl.commoncrawl.org, and those ranges are published as JSON at index.commoncrawl.org/ccbot.json, so you can verify a request rather than trust the header (spoofed CCBot user-agents are common).
The real decision is not whether CCBot is polite, but whether you want your content in the open training pool. Pages in Common Crawl can be summarized, memorized, and cited by models that never send you a click. That is the core GEO tradeoff: reach inside AI answers versus control over how your words get reused. One nuance worth knowing: a robots rule that targets CCBot applies only to CCBot, so it does not change how Googlebot crawls or ranks you.
How it affects your traffic
Whether CCBot can read your site decides if your pages enter the open training pool behind assistants like ChatGPT and Claude, which shapes whether those tools name and recommend you. Many CDNs and firewalls now block CCBot by default, so sites opt out of AI visibility without realizing it. Blocking it does not change your Google rankings, only your presence in Common-Crawl-derived datasets. Our AI SEO work audits which AI crawlers you allow, fixes accidental blocks, and structures pages so models can quote you accurately.
Get AI SEO that moves the needle
We turn terms like this into ranked pages and qualified pipeline. Start with a free Initial SEO Strategy.