Meta-ExternalAgent: what it is and how to block it
What is Meta-ExternalAgent?
Meta-ExternalAgent is Meta's crawler for uses such as training its AI models or improving products by indexing content directly.
- Operator
- Meta
- Used for
- Model training
- robots.txt token
meta-externalagent- robots.txt
- Opt out in robots.txt
- IP ranges
- Not published
Checked against Meta's documentation
What is the Meta-ExternalAgent user agent string?
meta-externalagent/1.1Meta documents the token and version as they appear in server logs, sometimes followed by a link to its crawler docs.
How do I block Meta-ExternalAgent in robots.txt?
Add these two lines to the robots.txt file at the root of your domain. Meta gives robots.txt rules for blocking meta-externalagent in its crawler documentation.
User-agent: meta-externalagent
Disallow: /To allow it everywhere instead, use:
User-agent: meta-externalagent
Allow: /What happens if you block Meta-ExternalAgent?
Meta does not say that blocking it changes how links to your site appear in its apps. Link previews come from a different agent.
Should you block Meta-ExternalAgent?
Block it if you do not want Meta to use your content for AI training. Link previews on Facebook and Instagram use a different agent.
Other AI crawlers
- GPTBotOpenAI, model training
- OAI-SearchBotOpenAI, AI search
- ChatGPT-UserOpenAI, user-triggered
- ClaudeBotAnthropic, model training
- PerplexityBotPerplexity, AI search
- Perplexity-UserPerplexity, user-triggered
See the full AI crawler list or read the official Meta docs
A blog AI search can read
Every Ranken blog ships a robots.txt that welcomes AI search crawlers, plus server-rendered pages, schema and llms.txt.