A wildcard robots rule applies to AI crawlers without named groups
In the reportAI crawlers inherit wildcard robots restrictions without a named group
ai_bot_blocked_only_by_wildcard · Review item
Review whether the wildcard restriction matches your intended treatment of the listed AI crawlers.
Severity: Low. Review item: this can be intentional; it is not a ranking or indexing diagnosis.
Why it matters
A restriction written for conventional crawlers may also apply to training, grounding and AI search crawlers. The finding lists the known affected crawlers and the inherited captured patterns.
What to do
- Read the affected crawler, purpose, wildcard group and exact captured Disallow lines.
- If the inherited restriction is unintended, add explicit named groups for the crawlers whose treatment should differ, or change the relevant wildcard restriction.
- Run another full audit; a correction is confirmed only with sufficient same-site coverage.
When to leave it as it is: Keep or dismiss an intentional wildcard restriction; this finding does not decide your AI content policy.
What we read
The captured site robots.txt and its original line and byte positions, resolved by the shared robots policy owner.
The admitted reviewed bot subset supplies crawler purpose and documented robots applicability. User-triggered fetchers and usage-control tokens are outside this crawler check.
The current supported route can prove wildcard inheritance when the document has only wildcard groups. Unproven unnamed fallback in documents with named groups remains not checked.
Example & evidence
Illustrative removal of an unintended wildcard restriction; this is not a live audit.
User-agent: *
Disallow: /privateUser-agent: *
Disallow:Confirm & check again
Check for yourself: Confirm that the reported lines are served by your robots.txt and that the intended crawler policy is correct.
Check again: Full audit — Run another full site audit. The same-site finding resolves only after the shared evaluator actually checks enough fresh captured evidence; unknown coverage does not demonstrate a fix.
Limits & sources
This is a reviewed subset, not a complete universe of AI bots. Unverified records and unresolved fallback remain explicit unknowns even when known affected bots produce a finding.
An unavailable or partial robots body, an unsupported charset, stale reference data, unproven 404 redirect coverage or unresolved pattern coverage cannot confirm a correction.
This check does not prove a live crawler visited or blocked a page and does not change robots permissions.
Confidence: High The finding records a proven captured wildcard policy for the listed reviewed crawlers. It does not infer owner intent or publisher caching behavior.
Sources
- Anthropic, Anthropic crawlers and robots controls — Documents training, search and user-request bots and standard robots directives. Checked 3 October 2026.
Editorial review: 2 November 2026