Crawl-delay is declared for a bot without documented support

Why it matters

A robots declaration cannot be assumed to enforce request pacing when its applying bot has no documented support for that field. Explicit lack of support and absent documentation are shown separately.

What to do

Review the bot, support basis and exact captured User-agent and Crawl-delay lines, including inherited wildcard groups.
Remove Crawl-delay from groups that apply to the reported bots. Keep declarations intended for bots with documented support, such as bingbot.
Changing the value to zero, blank or malformed text leaves the declaration present and does not fix this notice.
If enforceable request limits are needed, assess server or edge controls separately, then run another full audit of the same site.

When to leave it as it is: Keep a shared declaration if its benefit for a supported bot is intentional, while recognizing the documented limitation for the other reported bots. Moving it into User-agent: * does not remove that limitation.

What we read

The claim-bound root robots.txt response, original UTF-8 line text and byte positions, and the existing shared REP group resolver.

The admitted bot registry: non-Google bots with crawler or user-fetch roles, each bot’s Crawl-delay capability, publisher citations and freshness.

Named applying groups and User-agent: * inherited by bots without a matching named group. Any captured Crawl-delay field counts, independently of numeric validity.

Example & evidence

Illustrative removal for Applebot, whose reviewed documentation explicitly says Crawl-delay is unsupported.

User-agent: Applebot
Crawl-delay: 5
Allow: /

User-agent: Applebot
Allow: /

Confirm & check again

Check for yourself: Check the captured applying group and publisher citation for each bot. A named group with no Crawl-delay prevents that bot from inheriting a wildcard declaration.

Check again: Full audit — A subsequent full audit uses the shared evaluator and lifecycle to compare the same-site declaration. Sufficient fresh capture and reference coverage are required; could not check never proves correction.

Limits & sources

Google-family directives belong to robots_txt_unsupported_directives. A shared wildcard or Google/named-bot row can be reported by both rules for their respective bots.

Genuinely unregistered bots and non-crawling usage controls are outside this check and do not block the site.

Superficial variants of a registered token, ambiguous groups or unknown capability remain not checked; they cannot manufacture a correction. Definite other bot declarations can remain visible alongside incomplete coverage.

Unavailable, redirected, incomplete, non-UTF-8 or unbound robots evidence, and stale or missing reference evidence cannot prove removal.

The current reference was reviewed on 9 September 2026 and is fresh through 9 October 2026. This page does not claim a new source review.

This notice does not prove that a bot fetched any page or that request pacing changed. Zero, blank and malformed values do not imply valid delays.

Confidence: High High confidence pairs an exact captured applying directive with an admitted documented capability. No traffic, load or throttling effect is inferred.

Sources

Anthropic, Anthropic crawlers — Documents robots policy and Crawl-delay support for the reviewed Claude bots. Checked 12 July 2026.
Amazon, Amazonbot — Publisher documentation for the registered Amazon bots; missing documented Crawl-delay support is not evidence of measured throttling. Checked 9 September 2026.
Apple, About Applebot — Documents Applebot robots behaviour and the explicit lack of Crawl-delay support. Checked 9 September 2026.
OpenAI, OpenAI bots — Describes registered OpenAI bot roles and robots policy; the admitted registry records Crawl-delay as not documented. Checked 9 September 2026.
Perplexity, Perplexity crawlers — Describes reviewed Perplexity bots; no documented support is distinct from an explicit unsupported statement. Checked 9 September 2026.

Editorial review: 2 November 2026