robots.txt blocks the whole site

Why it matters

robots.txt tells crawlers which addresses they may fetch, and a blanket disallow keeps each crawler it applies to out of every page. Google cannot read the content of disallowed pages, although it may still list their addresses without a snippet.

What to do

Open your site's robots.txt and find the line the finding names, usually Disallow: / under User-agent: *.
Delete it, or replace it with rules for the paths you really want kept out. A blanket block is often left over from a staging copy or a pre-launch setting.
Run a new audit once the new file is published.

When to leave it as it is: if the site is deliberately closed, for example a staging copy or a site not yet launched, leave the block and lift it at launch.

What we read

We fetch /robots.txt from your site's address, read its first 500 KiB and apply the group for Googlebot, or the * group when there is none, as Google's specification describes.

The site is reported when robots.txt answered 200, its rules forbid the root /, the deciding rule, as parsed, is Disallow: / or Disallow: /*, whatever its letter case, spacing or trailing comment, and the group has no allow line at all; even an allow for one folder stops this report. Blocks written another way, such as Disallow: *, are not reported here, and the homepage check still reports them when they block the homepage.

Example & evidence

Illustrative example — one robots.txt, not a live audit.

Address: https://bakery.example/robots.txt, answered 200:

# copied from staging
User-agent: *
Disallow: /

There is no group for Googlebot, so the * group applies. Line 3, Disallow: /, blocks every path and no line allows any back, so the finding names that line. The homepage check reports the same line, and our crawler fetches no pages.

A possible correction:

User-agent: *
Disallow:

An empty Disallow: is no rule at all, so nothing is blocked.

Confirm & check again

Check for yourself: open your robots.txt in a browser and read the group for Googlebot, or for * if there is none: a Disallow: / with no allow line closes the site.

Check again: Full audit — run a new audit of the site; it fetches robots.txt again. Once the block is lifted, the audit can crawl your pages and no longer reports it. A robots.txt that now answers 404 or another 4xx other than 429 counts as having no crawl rules; a network failure, 429 or server error does not confirm that the block was lifted.

Limits & sources

Only Googlebot's rules are applied, and we fetch robots.txt with our own user agent, so a file served differently to Googlebot is not seen. The check does not tell whether the block is intended, or whether your pages are in Google's index.

Confidence: High because the file and the deciding line are read directly.

Sources

Google, How Google interprets the robots.txt specification — explains how Google chooses the group and the rule that apply, and that it cannot read disallowed pages. Checked 25 September 2026.

Editorial review: 29 September 2026

robots.txt blocks your homepage — only the homepage needs to be blocked; it also reports blocks this rule leaves out.
robots.txt fails with a server error — robots.txt cannot be read at all.
robots.txt blocks a stylesheet or script the page loads — robots.txt blocks only some stylesheet or script paths, not the site.