robots.txt is blocked or challenged for an AI crawler request

Why it matters

robots.txt is where you tell crawlers what they may fetch. Anthropic says its bots honour robots.txt directives and do not try to get past CAPTCHAs, and OpenAI says that disallowing GPTBot signals that a site's content should not be used for training. If the file itself is refused or challenged, those rules cannot be read.

What to do

1. Compare the recorded browser and GPTBot profiles and their request headers. Use the cited status or challenge fields to identify the observed access restriction.
2. In your CDN or firewall, find the rule that matches AI crawler user-agents and exclude /robots.txt from it.
3. Keep your choice about the crawler in robots.txt itself, with a User-agent group for its token.

What we read

We request the registered HTTPS /robots.txt twice with the browser control for the audit's smartphone or desktop collection profile and twice with the pinned GPTBot 1.4 user-agent. Accept and Accept-Encoding stay the same; no language header or Cookie is sent on any physical request or redirect. Both browser controls must return a complete, parseable 200 file without a challenge.

Both GPTBot-shaped requests must return the same 403, 406, 429 or 5xx status, or a complete 200 response with exactly one cf-mitigated: challenge header. A parseable 200 file is readable regardless of its directives. Matching 404 or 410 responses mean an accessibly missing file.

Example & evidence

Illustrative example — one audit, not a live result.

An audit of bakery.example requests the file with two user-agents, twice each:

GET https://bakery.example/robots.txt
browser_control -> 200, 200
ai_crawler:GPTBot -> 403, 403

The browser reads the file and the crawler request is refused both times, so the site is reported. A possible correction: the firewall rule skips /robots.txt; both requests then return the file, and nothing is reported for this crawler.

Confirm & check again

Check for yourself: request /robots.txt with a command-line client twice, once with a browser user-agent and once with the string the crawler's operator publishes, and compare the status codes.

Check again: Full audit — Run a new audit. A complete clean recheck closes the same site finding only when every previously blocked or challenged token is now readable or accessibly missing. A withdrawn or untested token remains unconfirmed. Unprofiled reviewed crawlers are named limits and do not prevent a clean result for the pinned GPTBot profile.

Limits & sources

Only the pinned GPTBot 1.4 user-agent is requested. Other reviewed crawler tokens are named as outside the tested universe; a clean result covers GPTBot only. Every request comes from one audit location, so a firewall may answer a real crawler differently.

The homepage still requires audit robots permission. Profile requests to the exact /robots.txt path bypass only that path's permission check; redirects to another path obey ordinary audit robots policy. HTTPS, public-address safety, pacing and request budgets still apply.

Readable-file and challenge conclusions require complete bodies and response-header evidence. Two matching supported block statuses can still fire when the crawler body or response-header inventory is incomplete. A 401, differing answers, more than four redirects, timeout, permission refusal or request-budget limit remains unconfirmed; CAPTCHA text or a changed body alone does not prove the supported challenge.

Confidence: High because each answer is recorded together with the request headers that were actually sent, and the result is limited to what a copied user-agent string can show.

Sources

• Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler? — says Anthropic's bots honour robots.txt directives and do not attempt to bypass CAPTCHAs on the sites they crawl. Checked 14 October 2026.
• OpenAI, Overview of OpenAI crawlers — publishes the user-agent string that GPTBot sends, with version 1.4 in the example and a note that the version number may change, and says GPTBot crawls content that may be used to train OpenAI models. Checked 14 October 2026.
• Cloudflare, Detect a Challenge Page response — says every Challenge Page response carries the cf-mitigated header, that challenge is its only valid value, and that the header is a reliable way to tell whether a response was challenged. Checked 14 October 2026.

Editorial review: 3 November 2026