meta-externalagent user agent string
Meta's crawler for uses such as training AI models and indexing content to improve its products. Meta says its crawlers may cache robots.txt for up to 24 hours. Below is the exact string it sends, what each part means, and how to allow or block it.
meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)
At a glance
| Type | AI crawler |
| Operator | Meta |
| Purpose | AI training and product indexing |
| Version | 1.1 |
| robots.txt token | meta-externalagent |
Allow or block it in robots.txt
Block meta-externalagent from the whole site
User-agent: meta-externalagent
Disallow: /Allow it again
User-agent: meta-externalagent
Allow: /String breakdown
| Fragment | Meaning |
|---|---|
meta-externalagent/1.1 | meta-externalagent 1.1 — an automated crawler/bot, not a real browser. |
+https://developers.facebook.com/docs/sharing/webmasters/crawler | Info page the bot operator publishes for webmasters. |
Use it in code
curl
curl -A "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)" https://example.comPython (requests)
import requests
requests.get("https://example.com", headers={"User-Agent": "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"})Playwright (Node.js)
const context = await browser.newContext({ userAgent: "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)" });Source: Meta Web Crawlers · checked September 2026