An AI crawler request is refused on pages a browser can read

Why it matters

robots.txt states a preference; a CDN or firewall enforces access. Cloudflare has announced that it is changing its default to block AI crawlers, so a site can refuse a crawler without its owner having written a rule for it.

What to do

1. Look at the finding: it names the crawler and lists the pages, each with the two browser answers and the two crawler answers.
2. Find the setting that refuses the crawler: an AI crawler or bot management option in your CDN, a firewall rule on the user-agent, or a security plugin.
3. If you want the crawler to read your pages, allow it there. If you do not, say so in robots.txt as well, so that both layers give the same answer.

When to leave it as it is: if you block this crawler on purpose, you can leave it. The finding only shows that the refusal happens at the server and is not stated in robots.txt.

What we read

For the homepage and a sample of crawled pages, we send two requests with a desktop browser user-agent and two with the crawler's published user-agent string, changing no other header. A page is compared only when both browser requests return a complete 200 page with the same content and no challenge header.

A page is reported when both crawler requests return the same 403, 406 or 429 status, or the same 5xx status. Findings are grouped by crawler: one finding lists every page refused for it.

Example & evidence

Illustrative example — one audit, not a live result.

An audit of bakery.example requests one page with two user-agents, twice each:

GET https://bakery.example/recipes/sourdough/
User-Agent: (desktop browser)
200 text/html     200 text/html

GET https://bakery.example/recipes/sourdough/
User-Agent: (the GPTBot string that OpenAI publishes, version 1.4)
403               403

robots.txt has no rule against GPTBot, and both crawler requests are refused, so the page is listed in the finding for this crawler. If both crawler requests had returned 200, the page would not be listed.

Confirm & check again

Check for yourself: request one of the listed pages with a command-line client, once with a browser user-agent and once with the string the crawler's operator publishes, and compare the status codes.

Check again: Full audit — run a new audit of the site. The finding closes when every page it listed is requested again with both user-agents and the crawler receives 200 both times. A page that is not requested again, or whose answers are unclear, keeps the finding open; a different page that answers normally does not close it.

Limits & sources

The request copies the user-agent string that the crawler's operator publishes and is sent from the audit's own location. A site that decides by network address can answer the real crawler differently, in either direction.

The check uses the GPTBot string published by OpenAI, version 1.4 at the time of review. A crawler whose operator publishes only a robots.txt token, without a full user-agent string, is not requested and is left unconfirmed.

A page is skipped, not passed, when the audit's own reading of your robots.txt disallows the request for the crawler or for the audit. A 401, 404 or 410, different answers on the two requests and a timeout are left unconfirmed, as is any page where the browser request did not return a stable 200.

Confidence: High because each status is recorded with the request headers that were actually sent, and both the refusal and the browser answer were seen twice.

Sources

• Cloudflare, Content Independence Day: no AI crawl without compensation! — announces that Cloudflare is changing its default to block AI crawlers unless they pay creators for their content. Checked 14 October 2026.
• OpenAI, Overview of OpenAI crawlers — publishes the user-agent string that GPTBot sends, with version 1.4 in the example and a note that the version number may change, and says GPTBot crawls content that may be used to train OpenAI models. Checked 14 October 2026.

Editorial review: 15 October 2026