GPTBot user agent string
OpenAI's crawler for content that may be used to train its generative AI models. Blocking it opts a site out of training and is set independently of OAI-SearchBot. Below is the exact string it sends, what each part means, and how to allow or block it.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
At a glance
| Type | AI crawler |
| Operator | OpenAI |
| Purpose | Collects content for AI model training |
| Version | 1.4 |
| robots.txt token | GPTBot |
Allow or block it in robots.txt
Block GPTBot from the whole site
User-agent: GPTBot
Disallow: /Allow it again
User-agent: GPTBot
Allow: /String breakdown
| Fragment | Meaning |
|---|---|
Mozilla/5.0 | Historical compatibility token. Nearly every browser claims "Mozilla/5.0" so that old servers do not serve it broken pages. |
AppleWebKit/537.36 | WebKit rendering engine, build 537.36. Chromium browsers keep this token for compatibility. |
KHTML, like Gecko | Compatibility note: the engine descends from KHTML and behaves like Gecko. Kept for legacy browser sniffers. |
compatible | Compatibility token — historically sent by IE, now mostly by bots. |
GPTBot/1.4 | GPTBot 1.4 — an automated crawler/bot, not a real browser. |
+https://openai.com/gptbot | Info page the bot operator publishes for webmasters. |
Use it in code
curl
curl -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" https://example.comPython (requests)
import requests
requests.get("https://example.com", headers={"User-Agent": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot"})Playwright (Node.js)
const context = await browser.newContext({ userAgent: "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" });Source: Overview of OpenAI Crawlers · checked September 2026