Crawlability15 rules onwhat can get in the way of crawling your pages.
robots.txt, server errors, redirects and the files a page needs to load.
robots.txt
- robots.txt approaches or exceeds the 500 KiB limitThe captured robots.txt approaches 500 KiB, or a captured Allow or Disallow directive begins beyond that limit.In the report: Robots.txt approaches or exceeds Google's 500 KiB parse limit
- robots.txt blocks the whole siteThe robots.txt group for Googlebot disallows the whole site with a Disallow: / rule and has no allow line at all.In the report: robots.txt disallow all
- robots.txt blocks your homepageThe robots.txt rules for Googlebot disallow the homepage.In the report: Homepage blocked by robots.txt
- robots.txt fails with a server errorAfter one retry, the request for robots.txt still ended in a server error (5xx).In the report: robots.txt 5xx
- robots.txt is reached through more than five redirectsThe request for your robots.txt file passed through more than five redirect responses.In the report: robots.txt is reached through more than five redirects
- robots.txt turns crawlers away as too many requestsAfter one retry, the request for robots.txt still ended in HTTP 429, the too-many-requests response.In the report: robots.txt 429
- robots.txt was available at one point of the audit and unavailable at anotherWithin one audit, robots.txt answered normally on one request and failed with a server error or 429 on another.In the report: robots.txt availability flapped during the audit — intermittent failures risk a site-wide crawl stop
- robots.txt contains unsupported directivesUnsupported directives appear in the
*group, before any User-agent line, or in a group naming a Google crawler; an unsupported directive whose crawler group cannot be identified prevents a clean result.In the report: Robots.txt contains directives Google ignores (noindex / crawl-delay / nofollow)
Responses
- A page fails with a server errorA crawled page kept returning a server error (5xx) across repeated requests.In the report: Page server error
- A referenced page answers with an errorA page referenced by an internal link, a structured-data breadcrumb or a sitemap entry returns 404 or 410; 401 and 403 need review.In the report: Referenced page HTTP error
- Your homepage fails with a server errorThe homepage kept returning a server error (5xx) across repeated requests.In the report: Homepage server error
Redirects
- A redirect loops or exceeds the redirect limitFollowing a page's redirects does not reach a final page, because they loop or run past the redirect limit.In the report: Redirect loop or limit exceeded
- An address takes a long chain of redirectsAn address passes through three or more redirects before its final answer, which may itself be an error.In the report: Long redirect chain
CSS and JavaScript
- A stylesheet or script the page declares returns an errorA stylesheet or script named in a page's HTML returned 404, 410 or a lasting server error.In the report: CSS or JavaScript resource returned an error
- robots.txt blocks a stylesheet or script the page loadsA stylesheet or script named in a page's HTML is disallowed for Googlebot by robots.txt.In the report: Rendering resource blocked by robots.txt