Should I block AI bots from my website? I looked at what was knocking on mine, and most of it was not AI.

You saw a number. On a dashboard, in a report your host sends, in a log somebody forwarded — a line that counts requests from AI bots, and beside it an arrow pointing up. Or somebody saw it for you and offered to make it go away for a monthly fee. That is where this question comes from. Nobody types it from principle.

The pages that answer it argue from principle anyway. I read eleven before writing this. Three of them tell you to read your logs. Not one of the eleven reports a single thing its author saw on a server he controls. So this article is only that: what the screen said on one of my sites, what was actually there when I looked, and what changed when I acted. One site, three twenty-four-hour readings across six days. It proves nothing about your site. It shows what the question looks like once somebody measures it.

What the screen said

On 10 September 2026 I opened Cloudflare’s AI Crawl Control for kovalweb.com. Cloudflare sits between the internet and that site, so every request reaches it first, and AI Crawl Control is the screen in it that counts AI bots. (Not sure you have Cloudflare? Type your domain followed by /cdn-cgi/trace into the address bar: plain text lines mean yes, a not-found means no — the previous article explains.) Overview, last 24 hours:

Requests from AI crawlers        1.1k   (up 84.1%)
Allowed                          438
Unsuccessful                     661    (up 197.75%)
Crawls that received HTTP 502    533

Busiest, by name    ChatGPT-User 108 · Claude-User 92 · Perplexity-User 54
                    Applebot 48 · Googlebot 37

Read at face value, that says AI bots are hammering the site and two thirds of them are failing. It is exactly the screen that sends an owner to type this question. Two words on it decide everything, so here they are in plain terms. Unsuccessful is not blocked — Cloudflare’s own page for the screen says “Unsuccessful requests may come from any rule or response error, not just the block action in AI Crawl Control.” And 502 is not a refusal at all. Every request to a site gets a three-digit answer: 200 is here it is, 404 is no such page, 403 is I heard you, and no. 502 belongs to the 500s, the family that means the server itself failed. Those 533 requests were not turned away. They died.

What was actually wrong

So before deciding anything about AI, I measured the site from outside. Five fetches of the home page in a row, timed, with nothing else running: 0.10 s, 0.11 s, 5.11 s, 5.12 s, 0.10 s. A fifty-fold swing. The notes sent back with every page said why: cf-cache-status: DYNAMIC on every page I tried. Cloudflare can keep a copy of a page in front of the site, so the server behind it does not have to build the same page again; DYNAMIC means it kept none. Every request — bot, scanner, customer — reached the site’s software and had it build the page from nothing. That is what the server could not keep up with.

My other site, kovalseo.com, is plain files with nothing to build; in the same window it took 84 AI bot requests, 66 of them successful, and answered in 0.07 s.

The site was not under attack by AI. It had no cache.

Here is the first thing this article can give you. If the pitch you were given is that bots are driving up your costs, the whole pitch rests on a question it may never have asked: is the hosting bill actually higher than before, and by how much? Ask that first. Then, if what alarmed you is load, the next question is not a block either. Ask whoever runs the site: are the public pages served from a saved copy, or built fresh for every request? With no Cloudflare, the question goes to your host or your platform instead. If the answer is built fresh, that is the thing to fix first, and the message at the end asks for it in words you can send.

The fix blocked nobody

One rule in Cloudflare: keep a copy of every public page for two hours, leaving out the admin area, anyone logged in, and the site’s own working addresses. Then clear the old copies. The exclusions are the part to get right: a cached page must never be one person’s own. The rule above was written for a blog. If your site has a basket or customer logins, that list is what to ask about before anyone switches caching on. Same method, same day:

	Before	After
Home page, five fetches	Before0.10 · 0.11 · 5.11 · 5.12 · 0.10 s	After0.051 · 0.069 · 0.056 · 0.057 · 0.068 s
Copies kept in front	Beforenone, on every page tried	AfterHIT on /, /uk/, /what-is-seo/, /blog/
AI bots	Beforeevery request built the page	AfterChatGPT-User, Claude-User and PerplexityBot served from the copy

The five-second spikes were gone. The spread collapsed from five seconds to eighteen milliseconds. Not one bot was refused to get there. The next day’s screen had no 500s at all, and six days later, with the request count up sharply, it still had none.

Then what most of them really were

Next day, the Metrics tab of the same tool, after the fix: 966 requests. Allowed 136. Unsuccessful 830. By answer code: 830 in the 400s (refused, or not found), 120 in the 200s (here it is), 16 in the 300s (moved), and nothing in the 500s. So the failures were gone and the unsuccessful number had not fallen with them — it went up. That is when I opened the table the Overview does not have, Most crawled paths. There were 367 of them, and they read like this:

/rclone.conf
/.env.backup
/.env.production
/@fs/home/ubuntu/.aws/credentials
/.env
/backend/.env
/serverless.yaml
/secrets.env
/.aws/credentials
/terraform.tfstate
/.git-credentials
/.ssh/id_dsa
/.docker/config.json

None of those is a page. They are where a carelessly set-up site might leave its passwords and keys. No AI bot asks for .env.backup. These were programs scanning for credentials, and the dashboard filed them under AI because of the name they gave. I opened every listed address myself: 404 or 403 on all of them, nothing exposed — but each 404 had cost about 32 kB, a fully built page, because until the cache rule a not-found was built like everything else.

Why they counted as AI bots is on Cloudflare’s page for the free plan — Cloudflare’s free plan, which is the one I am on:

Cloudflare’s documentation on AI Crawl Control, free plan

On the free plan, AI Crawl Control identifies AI crawlers based on their user agent strings. This enables AI Crawl Control to detect well-known, self-identifying AI crawlers.

A user agent string is the name a program gives for itself when it asks for a page. It is text, typed by whoever wrote the program. Anything that sends the name GPTBot is counted as GPTBot. OpenAI and Anthropic publish the addresses their bots come from, so a check is possible; on the free plan it is not done, in either direction.

So on that site, in that window, 830 of 966 — 86% — of what the dashboard labelled AI bot traffic was scanners wearing the names. Mine, that day; not yours. And not all of it was scanning: 120 requests succeeded, and that list was mostly the names Googlebot, Applebot, Claude-SearchBot and BingBot fetching real pages.

Five days later, a second window. About 1.35k requests the screen could classify, 1.19k of them in the 400s, against 156 distinct addresses — /public../.env, /host.key, /awsconfiguration.json, /credentials.js. Same shape.

One screen, the same name on both lists

Filter the Metrics tab by answer class and the second window’s traffic splits into two populations.

Answered in the 400s, by name: DuckAssistBot 153 · ChatGPT-User 135 · CCBot 109 · OAI-SearchBot 102 · Googlebot 84. What they asked for: the credential files above.

Answered in the 200s, by name: Googlebot 36 · Claude-SearchBot 12 · BingBot 9 · PetalBot 7 · Bytespider 6. What they asked for: /sitemap_index.xml, the home page, the stylesheet, articles.

Googlebot sits near the top of both — hunting password files on one list, reading real pages on the other. The name carries no information about who is asking. And Googlebot’s place on the first list says the impostors are not limited to AI bot names. Google’s page on its own bots puts it in one line: “The HTTP user agent string can be spoofed.”

Why the block-by-name switch is the wrong tool

Cloudflare offers, beside each bot’s name, a switch that blocks it. I nearly used it. The same screen shows why not. In one day Googlebot read Allowed 40 / Unsuccessful 84, OAI-SearchBot 9 / 102, Claude-SearchBot 16 / 34, Applebot 4 / 45. The unsuccessful half asked for files that do not exist. The allowed half asked for real pages. Which of them was the vendor’s own crawler, the screen cannot say — on the free plan it never checks. Block the name and you refuse every request carrying it, real or not, while the impostor, already using a dozen names, takes a thirteenth. What a block on a search bot’s name costs is written by its owner; Anthropic’s page on its bots says disabling Claude-SearchBot “may reduce your site’s visibility and accuracy in user search results”.

Block by what is asked for, never by who asks

What I did instead: a rule in Cloudflare that refuses any request for an address that looks like a secret file, whoever asks. Three classes. Addresses that begin with a dot, like /.env and /.git/config — with /.well-known excluded, because certificate renewal lives there. Addresses containing ../, which is a program trying to climb out of the site’s folder. And addresses containing the file types secrets are kept in: .key .pem .p12 .pfx .sql .bak .ini .yaml .yml .cfg .conf .tfstate — containing, not ending in, because on the free plan the rule editor has no ends-with test.

Before switching it on I tested it against 173 real addresses from both sites’ sitemaps and pages: 0 matched. Against 15 article addresses I might write one day: 0. Against 31 addresses the scanners had actually asked for: 28. Five patterns had to change on the way, four narrowed and one dropped, because titles I could plausibly write one day — /blog/how-to-reset-passwd/, /blog/seo-secrets/ — would have matched. Then live, from outside: home page, blog, sitemap, robots.txt and feed all answered 200; /.well-known/acme-challenge/test answered 404, not 403; /.env, /host.key and /wp-config.php answered 403 — the last of those had been getting through.

Its limits, plainly. Three addresses I left unmatched — /config.json and two more ordinary-looking filenames — because they are too generic to refuse without risking something real. And the AI dashboard will not get quieter, for a reason below.

If the site is not yours to change, send this to whoever runs it: three questions, two fixes that need no bot’s name, and one for AI training if you decide you want it.

If you still want to keep AI training out

A separate question, and the measurements above do not answer it; they only say it is not the one your dashboard was asking. Two things are documented.

The blunt switch has a listed price, and the price is a name already on my own answered list. Cloudflare’s panel offers a Training row; its own blog puts the consequence in one clause — “multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training”. Requests under Googlebot’s name fetched real pages of mine 36 times in the second window — the sitemap, the home page, articles. That switch refuses those.

The precise instrument is a name in your robots.txt — a short public text file at a fixed address on your site, asking named programs to keep out of named places; the previous article is about who else writes in it. Google’s page on its bots gives the training-only name and its guarantee:

Google’s documentation, on the Google-Extended name

Google-Extended does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.

OpenAI’s GPTBot and Anthropic’s ClaudeBot are likewise training names separate from their search bots; the message’s last line asks for exactly those three. A line in robots.txt is a request, not a lock — the standard that defines the file says “These rules are not a form of access authorization.” — and Anthropic’s wording for what a block does is “future materials”. Nothing I read says a block takes anything back that was already collected. And it stops nobody who was never the bot: when I took the first reading, robots.txt on this site already carried Disallow: / for nine names, GPTBot among them, put there by a Cloudflare setting switched on the day before. GPTBot’s name still appeared 87 times in the 400s of that reading.

How to know it worked

A screenshot of a setting proves only that the setting is there. Three things proved more, and each is a measurement you can ask for.

The time from outside, before and after: five fetches of the home page in a row, timed, nothing else running, and the gap between the fastest and the slowest. The Metrics tab the next day: on Cloudflare’s free plan it keeps only 24 hours, so look the day after, and look at the answer-code chart rather than the total. Mine lost its 500s and kept them lost. And the cost of a refused request: in the first window each not-found had cost about 32 kB, a whole built page; in the second, the address asked for most often cost about 2.4 kB a request. A refused request now costs about a thirteenth of what it did.

What that screen cannot tell you is whether a request was refused by your rule or simply found nothing; both are 400s there, and separating them takes Cloudflare’s security events list, which I did not capture. And the total will not fall, because the scanners have not stopped — the second window carried more requests than the first, not fewer. They only stopped costing anything.

The question was never whether to let them in

The screen said AI. What arrived was a server building every page from scratch, and a stranger trying doors under a dozen borrowed names, Google’s among them. A block on any of those names would have refused every request carrying it, the ones reading real pages included, and missed the stranger, and would have left the server exactly as slow. What worked was a saved copy of every page and a refusal of every address that looks like a secret — neither of which asked anyone’s name.

So, should you block? Not by name, and not to save money before anyone has shown you the bill. If the alarm was cost or load, the answers are above, and neither refuses a bot. If what you want is your pages kept out of AI training, the answer is three names in robots.txt — one of which Google promises costs nothing in its search — and not the switch that takes Google’s bot with it. Whatever you decide, decide it from a list of what was asked for. Until you have that list, the number was never measuring what you thought.

Written by Oleh Koval at Specia1ne, who reads these reports for a living.