robots.txt blocks your homepage
In the reportHomepage blocked by robots.txt
homepage_blocked_by_robots · Issue
This appears when your robots.txt, read with the rules that apply to Googlebot, forbids crawling your homepage. Googlebot does not fetch the page, other crawlers may follow different rules, and our audit does not fetch it either. Remove or narrow the rule, unless the site is meant to be closed to crawling.
Severity: Critical. Fix it first: while the homepage is blocked, Googlebot cannot read your main page.
Why it matters
robots.txt tells crawlers which addresses they may fetch. Google's specification applies the group of rules that matches its crawler most specifically, and notes that it cannot read the content of a disallowed page, although it may still show the address in results without a snippet.
What to do
- Open your site's robots.txt and find the line the finding names; the finding gives the rule and its line number.
- Remove that rule, or narrow it to the paths you really want kept out, such as
/checkout/. If your website editor or SEO plugin writes robots.txt for you, change it there. - Run a new audit once the new file is published.
When to leave it as it is: if the site is not meant to be public yet, such as a staging copy, the block is doing its job; lift it at launch.
What we read
We fetch /robots.txt from your site's address, following up to six redirects and reading its first 500 KiB. We apply the group written for Googlebot, or the * group when there is none; groups for Googlebot-News or Googlebot-Image do not count. As in Google's specification, the longest matching rule wins, and an allow rule wins a tie.
The homepage is your site's address, or the page it redirects to. It is reported when robots.txt answered 200 and its rules forbid that address, whether the file blocks only the homepage or the whole site. If robots.txt answers a 4xx error other than 429, there are no rules to apply and the homepage counts as allowed; if it cannot be read at all, the check cannot conclude.
Example & evidence
Illustrative example — one robots.txt, not a live audit.
Address: https://bakery.example/robots.txt, answered 200:
User-agent: *
Disallow: /
Allow: /menu/
Sitemap: https://bakery.example/sitemap.xmlThere is no group for Googlebot, so the * group applies. The homepage's path / matches Disallow: /, and Allow: /menu/ does not match it, so the homepage is blocked. The finding names line 2, Disallow: /, and our crawler does not fetch the homepage.
Because this group has an allow line, the check for a whole-site block does not report it, although only /menu/ stays open.
A possible correction:
User-agent: *
Disallow: /checkout/
Sitemap: https://bakery.example/sitemap.xmlNo rule matches / any more, so the homepage is allowed.
Confirm & check again
Check for yourself: open your robots.txt in a browser, find the group for Googlebot or, if there is none, for *, and look for a Disallow rule that matches /. Google Search Console's robots.txt report shows the version Google fetched.
Check again: Full audit — run a new audit of the site; it fetches robots.txt again and crawls the homepage. The finding is confirmed fixed only when that audit is complete: not cut short by its page limit, with every address it found fetched and answered 200, and none refused by robots.txt.
Limits & sources
Only the rules for Googlebot are applied; rules for Bingbot, AI crawlers or Googlebot-Image are not judged. We fetch robots.txt with our own user agent, so a server that gives Googlebot a different file is not detected, and a block alone does not show whether the homepage is in Google's index.
Confidence: High because the file and the matching line are read directly; if robots.txt cannot be read, the check makes no claim.
Sources
- Google, How Google interprets the robots.txt specification — explains how Google chooses a group and a rule, and that a disallowed address can still appear in results without a snippet. Checked 25 September 2026.
Editorial review: 29 September 2026