CCBot user agent string
The crawler of the non-profit Common Crawl, whose free web archive is widely used for research and analysis. It checks robots.txt first and obeys Crawl-delay. Below is the exact string it sends, what each part means, and how to allow or block it.
CCBot/2.0 (https://commoncrawl.org/faq/)
At a glance
| Type | AI crawler |
| Operator | Common Crawl |
| Purpose | Builds Common Crawl's open web dataset |
| Version | 2.0 |
| robots.txt token | CCBot |
Allow or block it in robots.txt
Block CCBot from the whole site
User-agent: CCBot
Disallow: /Allow it again
User-agent: CCBot
Allow: /String breakdown
| Fragment | Meaning |
|---|---|
CCBot/2.0 | CCBot 2.0 — an automated crawler/bot, not a real browser. |
https://commoncrawl.org/faq/ | Info page the bot operator publishes for webmasters. |
Use it in code
curl
curl -A "CCBot/2.0 (https://commoncrawl.org/faq/)" https://example.comPython (requests)
import requests
requests.get("https://example.com", headers={"User-Agent": "CCBot/2.0 (https://commoncrawl.org/faq/)"})Playwright (Node.js)
const context = await browser.newContext({ userAgent: "CCBot/2.0 (https://commoncrawl.org/faq/)" });Source: CCBot · checked September 2026