Google-Extended
Also called: Google-Extended crawler, Google-Extended user agent, Google-Extended token
Google-Extended is a robots.txt control token that lets publishers decide whether Google may use their crawled content to train and ground Gemini models and Vertex AI. It is not a separate crawler, and it does not affect Google Search indexing or rankings.
Google introduced this token on September 28, 2023, first framed around Bard and Vertex AI, now covering Gemini. What makes it unusual is that it is not really a crawler. Google-Extended has no separate HTTP user-agent string of its own. Googlebot still does the fetching. The token only tells Google whether content it already crawled may feed downstream AI uses.
Per Google’s crawler docs, it governs three things: Gemini Apps (training and grounding), the Vertex AI API for Gemini (training), and Grounding with Google Search on Vertex AI. “Grounding” is the part SEOs should care about, since that is how live pages get pulled into and cited by AI answers.
What it does not touch
Google states plainly that Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal.” It also does not control AI Overviews or AI Mode, which run on the regular Search index through Googlebot. So blocking Google-Extended will not pull you out of AI Overviews, and it will not hurt classic rankings.
To opt out, add this to robots.txt:
User-agent: Google-Extended
Disallow: /You can scope it to specific sections by changing the path after Disallow. Leaving it allowed, which is the default, keeps you eligible for Gemini grounding and training.
How it affects your traffic
For a site chasing visibility in AI answers, Google-Extended is a lever, not a threat. Blocking it removes you from Gemini grounding and training, so if getting cited inside Gemini and Vertex-powered answers matters to you, the token should usually stay allowed. The common mistake we fix is a blanket "block every AI bot" robots.txt that quietly cuts a brand out of the exact surface it wants to win. Our AI SEO work audits these crawler tokens and shapes content so Gemini can actually retrieve and cite you.
Get AI SEO that moves the needle
We turn terms like this into ranked pages and qualified pipeline. Start with a free Initial SEO Strategy.