robots.txt contains unsupported directives
In the reportRobots.txt contains directives Google ignores (noindex / crawl-delay / nofollow)
robots_txt_unsupported_directives · Review item
This appears when robots.txt has lines Google does not support, such as Noindex, Crawl-delay or Nofollow, in a Google-attributed or shared group. Google supports only user-agent, allow, disallow and sitemap, and ignores other lines. Put what you meant into a form Google reads, and remove the lines.
Severity: Medium. This becomes a notice when every line reported is Crawl-delay, which other crawlers may still read; a Noindex line there means pages you think are excluded may not be.
Why it matters
Google's robots.txt specification lists user-agent, allow, disallow and sitemap as the fields it supports, says other fields such as crawl-delay are not supported, and that Google ignores invalid lines. So a Noindex line there does not keep a page out of Google, and a Crawl-delay line does not slow Googlebot.
What to do
- Look at the finding: it shows each line, its number and byte, and the group it sits in.
- For
Noindex, use anoindexrobots meta tag or anX-Robots-Tagheader on the pages themselves, and keep those pages crawlable so Google can see it. - For
Crawl-delay, remove it from Google-attributed or shared groups. If another crawler needs it, put it under a separateUser-agentgroup for that crawler;Crawl-delayalone does not separate consecutive groups. - Correct misspelled fields such as
Dissallow: they count as unsupported too.
When to leave it as it is: if the only lines reported are Crawl-delay and you keep them for other crawlers on purpose, you can leave them; Google ignores them.
What we read
We read the same copy of robots.txt as the size check: the audit’s captured response, indexed only when it answers 200, with every field: value line indexed. Any field other than user-agent, allow, disallow and sitemap, in any letter case, counts as unsupported, so a misspelled field does too.
A line is reported when it sits in the User-agent: * group, in a group naming a Google crawler that follows robots.txt, such as Googlebot or AdsBot-Google, or before any User-agent line. This is group attribution, not proof that a particular crawler will select that group. Lines in groups only for named AI crawlers or other bots, such as GPTBot or bingbot, are not reported. Crawlers are recognised from a list of 32 bot names kept with the product.
The check cannot conclude when robots.txt does not answer 200 with its content — a 404 included, so a site without the file gets no clean result — when the file could not be indexed in full, or when the bot list is missing or out of date. An unsupported line in a group for a crawler not on the list, such as Yandex, also rules out a clean result, although lines in the groups Google reads are still reported.
Example & evidence
Illustrative example — one robots.txt, not a live audit.
https://bakery.example/robots.txt contains:
User-agent: *
Disallow: /checkout/
Noindex: /drafts/
Crawl-delay: 10Noindex and Crawl-delay are not fields Google supports, and they sit in the * group Google reads, so the site is reported with both lines.
User-agent: *
Disallow: /checkout/A possible correction: remove both lines, and add a noindex robots meta tag to the pages under /drafts/.
Confirm & check again
Check for yourself: open your robots.txt and look for lines that start with anything other than User-agent, Allow, Disallow, Sitemap or a # comment.
Check again: Full audit — run a new full audit. The finding can be confirmed fixed when robots.txt answers 200, its lines are indexed in full, the bot reference list is current, and no unsupported line remains in a Google-attributed or shared group or a group with unknown attribution. Deleting robots.txt does not confirm the fix; leaving only Crawl-delay lines keeps the notice open. Incomplete audit coverage cannot confirm a fix.
Limits & sources
The check reads the text of one copy of the file; it does not test how Google or any other search engine applies it. Crawlers are recognised from a list of 32 names, and lines past Google's 500 KiB limit are still reported.
Confidence: High because the lines are read directly from the file.
Sources
- Google, How Google interprets the robots.txt specification — lists user-agent, allow, disallow and sitemap as the fields Google supports, says other fields such as crawl-delay are not supported, and that Google ignores invalid lines. Checked 28 September 2026.
Editorial review: 29 September 2026