robots.txt turns crawlers away as too many requests
In the reportrobots.txt 429
robots_txt_429 · Issue
This appears when our request for robots.txt ended in HTTP 429, the too-many-requests response, even after one retry. Google counts a 429 as a server error, so while it cannot read robots.txt it may pause or slow crawling. Exempt robots.txt from rate limits so crawlers can read it whenever they ask.
Severity: High. Fix it soon: while robots.txt answers 429, Google may slow or pause crawling of the site.
Why it matters
Google treats a 429 as a sign that the server is overloaded and counts it as a server error. For robots.txt, its specification describes pausing crawling at first and then relying on a saved copy while the file cannot be read.
What to do
- Ask your host or CDN which rate limit or bot protection answers 429, and exempt
/robots.txtfrom it. - Check that the exemption covers the crawlers you care about, not only browsers.
- Run a new audit to see whether the file now loads.
What we read
We request /robots.txt from your site's address with our own user agent, following up to six redirects. If the first answer is a server error, a 429 or a network failure, we ask once more, straight away or after the delay the server asks for in Retry-After, up to 60 seconds.
The site is reported when that second answer is exactly 429. A server error on the second try is reported by the 5xx check instead, and a success means no finding.
Example & evidence
Illustrative example — robots.txt read twice in one audit, not a live audit.
GET https://bakery.example/robots.txt
HTTP/1.1 429 Too Many Requests
Retry-After: 10
GET https://bakery.example/robots.txt
HTTP/1.1 429 Too Many RequestsThe first answer asked us to wait ten seconds, so we did and asked again, and the second answer was 429 too. The finding records the final status, 429, with both attempts.
Once the rate limit leaves /robots.txt alone and the file answers 200 OK, there is no finding.
Confirm & check again
Check for yourself: open your robots.txt in a browser; if it loads, look at your CDN or firewall rules for limits that count repeated automated requests.
Check again: Full audit — run a new audit of the site; it requests robots.txt again, with one retry. The finding is confirmed fixed only when that audit is complete: not cut short by its page limit, with every address it found fetched and answered 200, and none refused by robots.txt.
Limits & sources
Two requests show the limit during the audit, not how long it lasts. They carry our own user agent, so a limit aimed at our requests may not apply to Googlebot, and a limit that applies to Googlebot may not show here.
Confidence: Medium because two requests within one audit are a short sample, and rate limits often depend on who is asking.
Sources
- Google, How Google interprets the robots.txt specification — excludes 429 from the 4xx errors Google treats as a missing robots.txt, and describes how it handles server errors. Checked 25 September 2026.
- Google, How HTTP status codes affect Google's crawlers — says Google treats 429 as a sign that the server is overloaded and as a server error. Checked 25 September 2026.
Editorial review: 29 September 2026