An AI crawler request receives a challenge page with status 200

Why it matters

A challenge response can return status 200, so status alone cannot show that the normal page was received. Cloudflare documents cf-mitigated: challenge as the challenge-response marker. This check compares the captured emulated and browser responses; it does not test whether a real crawler can solve the challenge.

What to do

1. Look at the finding: it names the crawler and lists the pages, with the header received on each crawler request.
2. In Cloudflare, review the security rules and bot settings that apply a challenge to this user-agent.
3. If you want the crawler to read your pages, exclude it from the challenge. If you do not, blocking it and saying so in robots.txt is clearer than a challenge.

When to leave it as it is: if you challenge this crawler on purpose, you can leave it. The finding shows what the crawler request received; it does not say the rule is wrong.

What we read

For the homepage and a sample of crawled pages, we send two requests with a desktop browser user-agent and two with the crawler's published user-agent string, changing no other header. A page is compared only when both browser requests return a complete 200 page with the same content and no challenge header.

A page is reported when both crawler requests return a complete 200 HTML page carrying exactly one cf-mitigated response header whose value is challenge. A page passes when the crawler requests carry no such header and show the same main content as the browser requests.

Example & evidence

Illustrative example — one audit, not a live result.

An audit of bakery.example requests one page with the crawler's user-agent, twice:

GET https://bakery.example/recipes/sourdough/
User-Agent: (the GPTBot string that OpenAI publishes, version 1.4)

HTTP/1.1 200 OK
content-type: text/html
cf-mitigated: challenge

Both crawler requests return this answer, and both browser requests return the recipe page without the header, so the page is listed in the finding for this crawler.

Confirm & check again

Check for yourself: request one of the listed pages with the string the crawler's operator publishes and look for a cf-mitigated response header.

Check again: Full audit — run a new audit of the site. The finding closes when every page it listed is requested again and the crawler requests return the page without a challenge header and with the same main content as the browser requests. A page that is not requested again, or whose answer is unclear, keeps it open.

Limits & sources

Only the Cloudflare challenge header is recognised. A challenge from another provider, a page that mentions a CAPTCHA, a verification widget, or a body that simply differs from the browser's is left unconfirmed, not reported and not passed.

A repeated cf-mitigated header, any other value, a header that could not be read, an incomplete body and a status other than 200 are also left unconfirmed. If the browser request is challenged too, nothing is reported.

The request copies the user-agent string that the crawler's operator publishes and is sent from the audit's own location. A site that decides by network address can answer the real crawler differently, in either direction.

The check uses the GPTBot string published by OpenAI, version 1.4 at the time of review. A crawler whose operator publishes only a robots.txt token, without a full user-agent string, is not requested and is left unconfirmed.

Confidence: Medium because the result rests on one response header that Cloudflare documents for challenge pages, seen on two requests; it does not read or judge the page body.

Sources

• Cloudflare, robots.txt setting — says robots.txt compliance is voluntary, that the file expresses preferences but does not prevent crawlers from accessing content at a technical level, and that access restrictions are enforced by other controls. Checked 14 October 2026.
• Cloudflare, Detect a Challenge Page response — says every Challenge Page response carries the cf-mitigated header, that challenge is its only valid value, and that the header is a reliable way to tell whether a response was challenged. Checked 14 October 2026.
• Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler? — says Anthropic's bots do not attempt to bypass CAPTCHAs on the sites they crawl. Checked 14 October 2026.
• OpenAI, Overview of OpenAI crawlers — publishes the user-agent string that GPTBot sends, with version 1.4 in the example and a note that the version number may change, and says GPTBot crawls content that may be used to train OpenAI models. Checked 14 October 2026.

Editorial review: 15 October 2026