robots.txt fails with a server error
In the reportrobots.txt 5xx
robots_txt_5xx · Issue
This appears when our request for robots.txt ended in a server error (5xx) even after one retry. While Google cannot read the file it may pause crawling, use a saved copy, or later act as if there were no file. Make robots.txt reliably reachable.
Severity: High. Fix it soon: while robots.txt fails, Google may slow or pause crawling of the whole site.
Why it matters
Google's robots.txt specification describes what happens when the file answers with a server error: for the first 12 hours Google stops crawling the site while it keeps trying, then for up to 30 days it uses the last good copy, and after that it decides by whether the site is generally available. A file that cannot be fetched because of DNS or network problems is treated the same way.
What to do
- Open
/robots.txton your site in a browser; if it loads, ask your host whether a firewall, CDN or rate rule answers some requests with errors. - Serve robots.txt as a plain file that does not depend on your application being up, if your host allows it.
- Run a new audit to see whether it now loads.
What we read
We request /robots.txt from your site's address with our own user agent, following up to six redirects. If the first answer is a server error, a 429 or a network failure, we ask once more, straight away or after the delay the server asks for in Retry-After, up to 60 seconds.
The site is reported when that second answer is any 5xx status. If the second answer is a 429, the too-many-requests check reports it instead; if it is a network failure, the check cannot conclude.
Example & evidence
Illustrative example — robots.txt read twice in one audit, not a live audit.
GET https://bakery.example/robots.txt
HTTP/1.1 503 Service Unavailable
GET https://bakery.example/robots.txt
HTTP/1.1 503 Service UnavailableThe first answer was a server error, so we asked again at once, and the second answer was a server error too. The finding records the final status, 503, with both attempts and when each was made.
If the second answer had been 200 OK with the file, there would be no finding: a single failure followed by a success is not reported.
Confirm & check again
Check for yourself: open your robots.txt a few times in a browser, and ask your host for error logs from the times the finding records.
Check again: Full audit — run a new audit of the site; it requests robots.txt again, with one retry. The finding is confirmed fixed only when that audit is complete: not cut short by its page limit, with every address it found fetched and answered 200, and none refused by robots.txt.
Limits & sources
Two requests a moment apart show the file failing during the audit, not for how long, and not that Google saw the same. They carry our own user agent, so a CDN or firewall may answer Googlebot differently, and a network failure is reported as not checked, although Google treats it like a server error.
Confidence: Medium because two failed requests within one audit are a short sample; the finding shows the time and status of each.
Sources
- Google, How Google interprets the robots.txt specification — describes how Google handles a robots.txt that returns a server error or cannot be fetched. Checked 25 September 2026.
Editorial review: 29 September 2026