robots.txt approaches or exceeds the 500 KiB limit
In the reportRobots.txt approaches or exceeds Google's 500 KiB parse limit
robots_txt_over_size_limit · Issue
This appears when your robots.txt is 400 KiB or larger, or when an Allow or Disallow line in it starts after the first 500 KiB. Google documents that it ignores content past 500 KiB, so such a line would not apply for Google. Consolidate rules so the file stays well under the limit.
Severity: High. Most serious when a rule line starts past the limit, because under Google's documented limit that line would not apply; a notice when the file is only close to it.
Why it matters
Google's robots.txt specification sets a file size limit of 500 kibibytes and says content after that size is ignored. A rule that starts past it would not apply for Google, whatever it was meant to allow or block.
What to do
- Look at the finding: it shows the file's size and each
AlloworDisallowline that starts past 500 KiB, with its line number and byte. - Consolidate rules: one pattern such as
Disallow: /search/replaces many lines for single addresses, and material you exclude can live in one directory. - Move the rules that matter most to the top of the file, and remove rules for sections that no longer exist.
When to leave it as it is: if the file is only close to the limit and every rule line starts well inside it, you can leave it for now; the review is a warning, not proof that a line is ignored.
What we read
We use the audit’s captured /robots.txt response for your site. For a 200 response, the audit retains up to the first 1 MiB and indexes complete UTF-8 field: value lines with their starting byte positions, excluding comments. This rule does not make another request.
The site is reported as an issue when an Allow or Disallow line starts at byte 512,000 or later, in any group; other fields such as Sitemap do not count. When there is no such line and the whole file was read, it is kept for review once the file reaches 409,600 bytes. Both limits are fixed, not settings.
The check cannot conclude when robots.txt answers 429, a server error or a network failure, or when the file could not be indexed in full — it is over 1 MiB, a line is longer than 16 KiB, it has more than 10,000 field: value lines, or it is not valid UTF-8 — unless a rule line past 500 KiB was already found. A 4xx answer other than 429, including 403 or 404, counts as no file, and the check passes.
Example & evidence
Illustrative example — one robots.txt, not a live audit.
https://bakery.example/robots.txt is 530,112 bytes, and its line 9,406 starts at byte 512,340:
Disallow: /checkout/The line starts past 512,000 bytes, beyond Google's documented limit, so the site is reported with the line, its number and its byte.
A possible correction: replace thousands of lines for single addresses with patterns such as Disallow: /search/, and move Disallow: /checkout/ near the top.
Confirm & check again
Check for yourself: save your robots.txt from the browser and check its size in bytes: Google reads the first 512,000.
Check again: Full audit — run a new full audit. A complete indexed 200 response below 400 KiB, with no Allow or Disallow line beyond the limit, lets this check pass. A conclusive 4xx no-file response other than 429 also lets it pass. A review-range file keeps the finding open; incomplete audit coverage cannot confirm a fix.
Limits & sources
One captured response from our crawler: a CDN or server may give Googlebot a different file. The check measures where lines start in the copy we received and applies Google's documented limit; it does not see Google ignoring a line, and nothing past the first 1 MiB is read.
Confidence: High because the byte positions are measured in the file as received.
Sources
- Google, How Google interprets the robots.txt specification — sets a robots.txt file size limit of 500 kibibytes, says content after it is ignored, and suggests consolidating rules to reduce the size. Checked 28 September 2026.
Editorial review: 29 September 2026