Google-Extended: what it is and how to block it
What is Google-Extended?
Google-Extended is not a separate crawler. It is a robots.txt token that controls whether content Google crawls can be used to train Gemini models and to ground Gemini answers.
- Operator
- Used for
- Model training
- robots.txt token
Google-Extended- robots.txt
- Control token
- IP ranges
- Not published
Checked against Google's documentation
What is the Google-Extended user agent string?
Google-Extended has no user agent string. Google crawls with its usual user agents and reads this token from robots.txt.
How do I block Google-Extended in robots.txt?
Add these two lines to the robots.txt file at the root of your domain. It only exists as a robots.txt token. Google says its common crawlers always obey robots.txt rules.
User-agent: Google-Extended
Disallow: /To allow it everywhere instead, use:
User-agent: Google-Extended
Allow: /What happens if you block Google-Extended?
Google says it does not affect a site's inclusion in Google Search and is not a ranking signal.
Should you block Google-Extended?
Block it if you do not want your content used for Gemini training and grounding. Your place in Google Search stays the same.
Other AI crawlers
- GPTBotOpenAI, model training
- OAI-SearchBotOpenAI, AI search
- ChatGPT-UserOpenAI, user-triggered
- ClaudeBotAnthropic, model training
- PerplexityBotPerplexity, AI search
- Perplexity-UserPerplexity, user-triggered
See the full AI crawler list or read the official Google docs
A blog AI search can read
Every Ranken blog ships a robots.txt that welcomes AI search crawlers, plus server-rendered pages, schema and llms.txt.