A listed address is blocked by robots.txt

Why it matters

Google asks for the URLs you want to see in its results to be listed in your sitemap. When the robots.txt of the listed address's host disallows it for Googlebot, Google cannot read the page's content, although it may still show the address in results without a snippet.

What to do

1. Look at the finding: it shows the sitemap entry and the robots.txt line that blocks it, with its line number.
2. If the page should be in search, remove or narrow that line, or add an Allow rule for the page.
3. If it should not be crawled, remove it from the sitemap, usually a setting in your SEO plugin.

What we read

We match page-like addresses retained by the sitemap collector on the same HTTP origin as the captured robots.txt policy, except the homepage, against the rules for Googlebot, or for * when there is no Googlebot group; the longest matching rule wins, and an allow rule wins a tie. File-looking paths such as .pdf or .jpg are excluded before this check. No listed address is fetched for this check, and there is no page limit within the checked origin.

The check needs robots.txt to answer 200; a 4xx other than 429 means there are no rules, and a 429, a network failure or a server error makes the check inconclusive. The captured policy is applied by path only to listed addresses on that same scheme, host and port; a www. or protocol twin is a separate origin.

Example & evidence

Illustrative example — one sitemap entry and one robots.txt, not a live audit.

https://bakery.example/robots.txt answers 200 with:

User-agent: *
Disallow: /catering/

Sitemap: https://bakery.example/sitemap.xml

and the sitemap lists:

<url><loc>https://bakery.example/catering/wedding-cakes</loc></url>

The path /catering/wedding-cakes matches line 2, so the listed page is reported with the sitemap entry and that line.

A possible correction: add Allow: /catering/wedding-cakes to the same group, a longer rule that wins, or remove the entry from the sitemap.

Confirm & check again

Check for yourself: take the address from your sitemap and compare its path with the rules in your robots.txt; Search Console's URL Inspection reports whether robots.txt blocks a page.

Check again: Full audit — run a new audit of the site. While the address stays listed, allowing it can be confirmed once it answers directly with 200 and complete HTML, every sitemap is read in full, and no listed address remains outside the checked robots.txt origin. The finding can also close when the address answers directly with a 200 file that is not HTML, or starts with a permanent redirect within your site, and the check no longer reports it, provided the sitemap coverage is complete. Removing the address from the sitemap while the block stays is not accepted as proof.

Limits & sources

The check matches paths only and does not fetch the page, so it says nothing about the page's status or content. Only Googlebot's rules are applied, and a robots.txt that Google holds from an earlier fetch is not known.

Confidence: High because the sitemap entry and the blocking rule are both read directly.

Sources

• Google, Build and submit a sitemap — asks you to include in your sitemap the URLs you want to see in Google's search results. Checked 25 September 2026.
• Google, How Google interprets the robots.txt specification — explains how Google chooses the group and rule that apply, and that it cannot read disallowed pages. Checked 25 September 2026.

Editorial review: 29 September 2026

Related

• A listed page asks to stay out of search — the listed page was fetched and itself asks to stay out of search.
• A listed page returns 404 or 410 — the listed page was fetched and answered 404 or 410.
• A sitemap cannot be fetched or read — the sitemap file itself failed.