A named user-fetch robots restriction is documented as ineffective
In the reportrobots.txt attempts to block user-fetch AI agents documented to ignore it
user_fetch_bot_robots_rule_ineffective · Issue
Use the captured bot-specific policy and publisher source to decide whether this robots declaration meets your exclusion goal.
Severity: Notice. Advice: review whether the declaration serves your exclusion goal. The documented limitation concerns the listed user-fetch bot, not all AI crawlers or your search indexing.
Why it matters
A robots declaration expresses a preference. For a user-request bot documented to ignore it, the declaration alone does not enforce the intended restriction.
What to do
- Read the named bot, documented applicability and exact original User-agent and Disallow lines.
- Remove or adjust an ineffective robots declaration. If access exclusion is required, evaluate access controls separately.
- Run another full audit of the same site. The checked declaration can resolve only with sufficient fresh evidence.
When to leave it as it is: Keep a robots declaration if it is an intentional expression of policy; understand that this check does not prove it enforces access exclusion for the reported bot.
What we read
The current audit’s claim-bound root robots.txt capture, original UTF-8 lines and byte positions, and shared REP group selection and Allow precedence.
The admitted bot reference supplies the reviewed bot role, robots applicability, reference date and publisher citation.
The currently supported positive subset is Perplexity-User, documented as generally ignoring robots.txt. Claude-User respects robots.txt and does not fire. Restrictive ChatGPT-User or Amzn-User rules have conditional applicability and remain not checked.
Example & evidence
Illustrative removal of an ineffective named declaration; this is not evidence of edge blocking.
User-agent: Perplexity-User
Disallow: /privateUser-agent: Perplexity-User
Disallow:Confirm & check again
Check for yourself: Confirm that the captured lines are served by your site and that the cited documented behaviour applies to the reported user-request bot.
Check again: Full audit — Run another full site audit. The shared evaluator and lifecycle can confirm checked removal or neutralization of the same-site robots declaration; insufficient coverage never proves a fix.
Limits & sources
This is a reviewed documented subset. Conditional or undocumented behaviour next to a restrictive relevant declaration remains unknown and cannot prove a fix.
Wildcard-only or prefix-like declarations do not name a reviewed user-fetch product token for this rule. Equal-pattern Allow wins; unresolved pattern materiality remains unknown.
Unavailable, redirected, incomplete, non-UTF-8 or unbound source evidence, or stale/missing reference data cannot confirm a correction.
This finding does not prove that a bot fetched a page, that a firewall blocked it, or that removal of robots instructions enforces edge exclusion.
Confidence: High The finding pairs a captured named restrictive declaration with the listed bot’s admitted publisher-documented applicability. It does not guess intent or treat conditional applicability as definite ignoring.
Sources
- OpenAI, OpenAI bots — Distinguishes user-request fetching and crawler robots applicability. Conditional applicability does not establish a deterministic ineffective control. Checked 9 September 2026.
- Perplexity, Perplexity crawlers — Documents Perplexity-User as a user-request fetcher that generally ignores robots.txt; it does not prove an actual fetch or access blocking. Checked 10 October 2026.
Editorial review: 2 November 2026