robots.txt was available at one point of the audit and unavailable at another

Why it matters

Google says that when a robots.txt file exists but cannot be fetched because of a server error, it stops crawling the site for the first 12 hours while it keeps trying to fetch the file, and then uses the last good version for the next 30 days. A file that fails some of the time can therefore pause crawling of the whole site.

What to do

1. Look at the finding: it shows the three readings with their time, status and the number of attempts each one took.
2. Check your server and CDN logs for /robots.txt at those times. Look for rate limiting, a firewall rule, or an application that builds the file on each request and fails under load.
3. Serve robots.txt as a static file where you can, and exclude it from rate limits.

What we read

We request /robots.txt at three points of the same audit: at the start, after the pages have been crawled, and near the end. No waiting time is added between them. Each reading is a separate request with its own recorded start and end time.

A reading counts as available when it ends in a complete robots.txt file, or in a 404 or 410. It counts as unavailable when it ends in a 5xx or 429 status and the automatic repeat of that same request failed the same way. The finding needs all three readings, in order, with at least one available and one unavailable.

Example & evidence

Illustrative example — one audit, not a live result.

An audit of bakery.example reads the file three times within eleven minutes:

10:02:14  GET https://bakery.example/robots.txt   200 text/plain
10:07:41  GET https://bakery.example/robots.txt   503
10:07:43  GET https://bakery.example/robots.txt   503   (automatic repeat of the same request)
10:13:05  GET https://bakery.example/robots.txt   200 text/plain

The second reading failed twice, so it counts as unavailable, while the first and third were served. The site is reported. If all three readings had answered 200, nothing would be reported.

Confirm & check again

Check for yourself: request /robots.txt several times over a few minutes and look for a 5xx or 429 status in your server or CDN logs for that address.

Check again: Full audit — run a new audit of the site. The finding closes only when the new audit takes all three readings and each of them is available in the same way. An audit that stops early, a reading with an unclear answer, or three failed readings leave it unconfirmed.

Limits & sources

Three readings from one location within one audit are a small sample. They show that the file failed during this audit; they do not measure uptime or show how often it fails.

A request that failed once and succeeded on its automatic repeat counts as available. The repeat is part of the same reading, not a fourth one.

A file that fails on every reading is the subject of the 5xx and 429 robots.txt checks, not this one. A mix of 200 and 404 answers, an HTML page, an incomplete file, a timeout or a redirect that could not be followed is left unconfirmed.

Confidence: Medium because each reading is recorded with its status and time, while three readings in one audit remain a limited sample of how the file behaves.

Sources

• Google, How Google interprets the robots.txt specification — says that when a robots.txt file exists but cannot be fetched because of a server error, Google stops crawling the site for the first 12 hours while it keeps trying, and then uses the last good version for the next 30 days; and that 4xx answers other than 429 are treated as if no robots.txt file existed. Checked 14 October 2026.

Editorial review: 15 October 2026