How to block Baiduspider: robots.txt and HTTP requests

How to block Baiduspider with robots.txt

Add the Baiduspider rule above to your site's root robots.txt, for example https://example.com/robots.txt. Merge it with existing rules rather than replacing rules for other crawlers. Disallow: / asks Baiduspider to stop crawling the whole site.

To block only a directory, use a path such as /private/ instead. These rules apply to Baiduspider, rather than every search engine. Blocking crawling can prevent Baidu from refreshing your content; it does not guarantee that an already known URL disappears from search results.

User-agent: Baiduspider
Disallow: /private/

How to refuse Baiduspider requests with Cloudflare

For a hostname whose traffic is proxied through Cloudflare, go to Security rules, choose Create rule > Custom rules and enter the expression below. Replace example.com with the hostname you want to protect; add a separate host condition if you also need www. Choose Block to refuse matching HTTP requests, rather than only asking a crawler not to fetch them.

Check the hostname and matching scope before deploying the rule. User agents can be spoofed: this matches a reported name, not verified Baidu identity. Use authentication to protect private content.

(http.host eq "example.com" and lower(http.user_agent) contains "baiduspider")

Test the HTTP block and compare a normal request

After replacing example.com with your hostname, run both commands below and check Cloudflare Security Events for the matching rule. With the Block action, the matching request should be refused; the default block status is 403. The comparison request should retain the behavior intended for ordinary visitors.

A 403 alone is not proof that this rule matched: other security rules can also deny a request. Changing the test UA exercises the matching condition; it does not impersonate or verify a real Baidu crawler.

curl -I -A "Baiduspider" https://example.com/
curl -I -A "UA-rule-check/1.0" https://example.com/

Check robots.txt and verify the request source

Open your public robots.txt and confirm that the expected text is served successfully. Then review server access logs for subsequent requests. Robots rules are instructions for cooperative crawlers, not an access-control mechanism; use authentication for private content.

A user agent containing Baiduspider can be spoofed. For a request you need to verify, check its reverse DNS hostname and resolve that hostname back to the original IP. A matching user agent alone does not prove the request came from Baidu.