AI search crawlers are disallowed across your site
In the reportAI search crawlers are disallowed across the site
ai_search_crawler_blocked_in_robots · Review item
A sitewide robots restriction applies to a reviewed search-purpose AI crawler. This is a review item: it may be deliberate. The audit does not prove that an answer engine fetched, indexed or stopped citing your pages.
Severity: Medium. Review item because the access restriction may be intentional.
Why it matters
Search-purpose crawlers and training crawlers have different documented purposes. Review their permissions separately instead of assuming that a training opt-out must also exclude search crawling.
What to do
- Use the affected-bot list and original line evidence to identify the applying groups.
- If you want those search crawlers to access all pages, remove the sitewide restriction or add full named Allow groups for them. Keep any separate training controls you intend.
- Run a complete audit again; adding a single allowed health or admin path does not establish access to your public content.
When to leave it as it is: Leave the restriction if it matches the access or usage policy you want.
What we read
One complete original robots.txt from the registered origin, the shared REP resolver and the admitted bot-purpose reference.
Only documented robots-respecting search crawlers in the reviewed subset are assessed, currently OAI-SearchBot, Claude-SearchBot, PerplexityBot and Amzn-SearchBot. Usage-control tokens are not crawlers.
An effective all-path Disallow without an opening Allow can fire. If a crawler’s resolved group has no all-path Disallow and no unresolved declarations, path-limited restrictions pass this sitewide check. An all-path Disallow with only specific sections opened by Allow remains unknown. Ambiguous agent names, invalid paths and unresolved fallback also remain unknown. Known blocks stay visible beside unknown bots; incomplete coverage cannot prove a fix.
Example & evidence
Illustrative permission correction; not evidence of requests or citations.
User-agent: OAI-SearchBot
Disallow: /User-agent: OAI-SearchBot
Allow: /Confirm & check again
Check for yourself: Open the original robots.txt and compare the named agent and wildcard groups, including every Allow exception.
Check again: Full audit — Run a new complete audit of the same site. A complete re-check closes the earlier finding when no reviewed search crawler is restricted sitewide, even if valid path-limited Disallow rules remain. An all-path Disallow opened only for specific sections, unresolved declarations, an unavailable robots.txt or incomplete reference coverage cannot prove a fix.
Limits & sources
The reference is a reviewed subset, not an inventory of every bot. Named competing groups can leave an unnamed crawler unresolved.
The finding describes permissions; it does not predict answer citations, discover bot requests or infer your intent.
Valid path-limited Disallow rules produce no sitewide finding. An all-path Disallow with only specific sections opened by Allow stays unknown and cannot close an earlier sitewide finding.
Confidence: High The captured declarations and reviewed publisher metadata establish the permission scope, not actual fetching or use.
Sources
- Amazon, Amazon crawler documentation — Documents the publisher search crawler and its purpose or robots controls; it does not establish actual visits or a universal citation guarantee. Checked 9 September 2026.
- OpenAI, Overview of OpenAI crawlers — Documents the publisher search crawler and its purpose or robots controls; it does not establish actual visits or a universal citation guarantee. Checked 9 September 2026.
- Perplexity, Perplexity crawlers — Documents the publisher search crawler and its purpose or robots controls; it does not establish actual visits or a universal citation guarantee. Checked 9 September 2026.
- Anthropic, Anthropic crawler controls — Documents the publisher search crawler and its purpose or robots controls; it does not establish actual visits or a universal citation guarantee. Checked 9 September 2026.
Editorial review: 2 November 2026