AI search20 rules onhow your site handles AI crawlers, controls and content.
AI crawlers in robots.txt, what they are served, content signals, llms.txt, answers, citations and statistics in your content.
AI crawlers
- A named user-fetch robots restriction is documented as ineffectiveA captured named Disallow targets a reviewed user-request bot whose publisher documents that it ignores or generally ignores robots.txt.In the report: robots.txt attempts to block user-fetch AI agents documented to ignore it
- Crawl-delay is declared for a bot without documented supportAn original robots.txt Crawl-delay declaration applies to a registered non-Google bot whose reviewed capability is unsupported or not documented.In the report: robots.txt declares Crawl-delay without documented support
- robots.txt is blocked or challenged for an AI crawler requestTwo cookie-free requests for robots.txt with the pinned GPTBot user-agent were blocked or challenged while matching browser controls could read it.In the report: robots.txt is blocked or challenged for an AI crawler request
- The captured origin root is blocked for Bingbot in robots.txtThe retained robots.txt denies Bingbot access to / while the directly captured root meets this check’s page qualifications.In the report: The captured origin root is blocked for Bingbot in robots.txt
- A wildcard robots rule applies to AI crawlers without named groupsA captured wildcard Disallow also restricts reviewed AI crawlers without an explicit named decision.In the report: AI crawlers inherit wildcard robots restrictions without a named group
- AI search crawlers are disallowed across your siteThe complete robots.txt disallows all paths for one or more reviewed AI search crawlers.In the report: AI search crawlers are disallowed across the site
- AI training decision is undecidedYour recorded training decision is undecided, and the reviewed crawlers have unrestricted robots permissions.In the report: AI training decision is undecided with unrestricted robots permissions
- Google-Extended is disallowed: review its usage scopeA named Google-Extended group disallows all paths; review what this separate usage control covers.In the report: Google-Extended is disallowed; review its Gemini usage scope
Bot access
- An AI crawler request is refused on pages a browser can readPages that answer 200 to a browser request returned a block status to a request with an AI crawler's published user-agent string.In the report: An AI crawler request receives a block status on pages a browser can read
- An AI crawler request receives a challenge page with status 200Pages that a browser request reads normally answered an AI crawler request with a Cloudflare challenge page under status 200.In the report: An AI crawler request receives a challenge page instead of the content
- Raw HTML may omit the main contentThe page's original HTML looks like a shell that may need JavaScript to show its main content.In the report: Raw HTML may omit the main content
Signals and controls
- Content-Signal declaration does not match the documented syntaxAn original Content-Signal entry uses a documented key with a definitely unsupported or missing value.In the report: Content-Signal declaration does not match the documented syntax
- Content-Signal and usage-control declarations express opposed preferences at /A no usage signal and an explicit winning Allow for an admitted usage-control token appear in the same unambiguous group.In the report: Content-Signal and usage-control declarations express opposed preferences at /
- Pages carry noarchive or nocacheCaptured head declarations on indexable pages include noarchive or nocache.In the report: Pages carry NOARCHIVE/NOCACHE — Bing keeps their rankings but Copilot answers can't use their content; confirm that's intended
llms.txt
- Published llms.txt has a format errorOriginal bytes in the optional origin-root guidance file show a definite error against a fixed version of the llms.txt proposal.In the report: Published llms.txt has a definite proposal-format error
- A link listed in llms.txt was unavailable to the auditA resource linked from the site's root /llms.txt returned HTTP 404 or 410 to the audit's request, or the DNS lookup made for that request ended with NXDOMAIN.In the report: Links in llms.txt were unavailable to the audit
- No llms.txt or llms-full.txt at the site rootThe site's root /llms.txt and /llms-full.txt both answered HTTP 404 or 410 directly to the audit's requests.In the report: Neither root llms.txt guidance file was found
Citable content
- Main content has no recognized external source linkA qualifying page's main content has no visible link to a website on a different registrable domain.In the report: Main content has no recognized external source link
- Main content has no recognized statistical statementA qualifying page's main content has no statement matching the check's recognized statistic patterns.In the report: Main content has no recognized statistical statement
- No direct-answer pattern follows a question headingA heading asks what or who something is, but the paragraph after it does not open by naming that topic and answering.In the report: No direct-answer pattern follows a question heading