OAI-SearchBot: what it is and how to block it
What is OAI-SearchBot?
OAI-SearchBot is OpenAI's crawler for ChatGPT search. It finds and refreshes the pages ChatGPT can show and link to in its answers, and it is separate from GPTBot, which collects training data.
- Operator
- OpenAI
- Used for
- AI search
- robots.txt token
OAI-SearchBot- robots.txt
- Opt out in robots.txt
- IP ranges
- Published list
Checked against OpenAI's documentation
What is the OAI-SearchBot user agent string?
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbotOpenAI publishes this as an example. The browser and bot versions can change.
How do I block OAI-SearchBot in robots.txt?
Add these two lines to the robots.txt file at the root of your domain. OpenAI documents the OAI-SearchBot token in robots.txt to control whether a site appears in ChatGPT search.
User-agent: OAI-SearchBot
Disallow: /To allow it everywhere instead, use:
User-agent: OAI-SearchBot
Allow: /What happens if you block OAI-SearchBot?
OpenAI says sites that opt out are not shown in ChatGPT search answers, though they can still appear as navigational links.
Should you block OAI-SearchBot?
Allow it if you want ChatGPT to cite and link your pages. Blocking OAI-SearchBot is the main way a site drops out of ChatGPT search answers.
Other OpenAI crawlers and AI agents
- GPTBotOpenAI, model training
- ChatGPT-UserOpenAI, user-triggered
- ClaudeBotAnthropic, model training
- PerplexityBotPerplexity, AI search
- Perplexity-UserPerplexity, user-triggered
- Google-ExtendedGoogle, model training
See the full AI crawler list or read the official OpenAI docs
A blog AI search can read
Every Ranken blog ships a robots.txt that welcomes AI search crawlers, plus server-rendered pages, schema and llms.txt.