"Blocked by robots.txt": Google was asked to stay out — and it did. The question is who asked.

The word blocked does its worst work on the person who blocked nothing — and that person is this row’s usual audience. The row sits in Search Console’s Page indexing report, in the table headed Why pages aren’t indexed: Blocked by robots.txt, with a count beside it. Mine says nine.

If you blocked nothing on purpose, hold on to that, because it is the common case. The sentence this whole article stands on: this row usually means a block worked. A file on your site asked search engines not to visit certain addresses. Google honoured the request — then counted the addresses the rule covers and showed you a sample. Click the row and read that sample: feeds, search pages, addresses with tracking tails — the usual tenants, none of them broken. In Google’s help forum — in a thread about this row’s sibling, which you will meet below — a Product Expert (a badge Google gives its forum’s volunteer regulars) answered the worry that covers both rows:

A Product Expert in Google’s help forum, as reported

No, not at all, it’s no problem to have URLs indexed which are intentionally blocked by robots.txt. The search console just shows these “errors” to make sure such pages aren’t blocked by mistake.

(“Indexed”, if the word is new, is the trade term for listed in Google’s search results.) Not an alarm, a confirmation: these addresses are blocked — was that the plan? For most rows the true answer is yes, and the work ends there. On my own site the somebody who asked was not me: I never wrote my robots.txt — an SEO specialist I hired did, years ago. How that file came to hold the one line behind all nine of my blocked addresses turned into this article’s best story. First, the file.

The small file behind all of it

An owner in the same thread asked the three questions most guides skip straight past:

An owner in the same thread, as reported

What is robots.txt? Why is it there? Where is it?

What it is: one plain text file at the front of your site, holding a short list of addresses — or patterns of addresses — that search engines are asked not to fetch. Not a program, not a lock: a list of requests, in public, readable by anyone. Where it is: always the same place — your domain, then /robots.txt. Mine is at kovalweb.com/robots.txt; type yours into a browser and the file itself appears. Why it is there: because something put it there. Site platforms make a basic one on their own; plugins — add-on programs that extend what your site can do — can write into it; and, the part this article turns on, people paste “improved” versions found in guides.

A request not to visit is not an instruction to remove

What the file can and cannot do, Google states on its own page introducing robots.txt — with the middle clause set in bold by Google itself:

Google’s introduction to robots.txt — “crawlers” are the programs that read pages for a search engine

A robots.txt file tells search engine crawlers which URLs the crawler can access on your site. This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google. To keep a web page out of Google, block indexing with noindex or password-protect the page.

Not a mechanism for keeping a web page out of Google — bolded by the people who run the index. Asking Google not to visit an address and instructing Google not to list it are two different sentences, and robots.txt only knows how to say the first one. Two consequences follow.

First: a blocked page can still be listed. Google’s help page for the report concedes the point inside this reason’s own definition:

Google’s help page, inside this reason’s definition

If Google can find other information about this page without loading it, there is a very small chance that the page might still be indexed.

When that small chance lands, the address moves to this row’s sibling — a reason headed Indexed, though blocked by robots.txt, filed under Warning, which the same help page defines: Google keeps its word and never reads the page, but can still list the address, built from what the pages linking to it say — the way a shop can appear in a directory whose writer never walked in.

Second: blocking a page hides your own instructions from Google. There is a line a page can carry that really does say don’t list this one — the noindex, and it has its own article in this series, including the part where Google documents that a robots.txt block stops a noindex from ever being seen. So to get an already-listed address dropped, the order runs backwards from instinct: unblock it first, so the noindex can be read on the next visit. The file cannot remove a page, and it can smother the one instruction that would. What it does do, reliably, is keep Google away from addresses somebody chose. Whether the choice still speaks for you is the row’s real question — first your addresses, then, because mine needed it, the somebody.

Should you unblock?

The honest answer: it depends, on less than you would think. Blocking things on purpose is a legitimate use of the file — Google’s own blog, in 2025:

Google’s Search Central blog, 2025

You can use a robots.txt file for solving technical issues (such as unnecessary paginated pages), or for editorial or personal reasons (such as just not wanting certain things crawled).

So the question is never “how do I clear this row”. It is: do these rules still say what I want? Judge your row’s addresses against what typically sits here:

• Either is fine Feed addresses — copies of your posts written for programs rather than people; RSS is the name you may have met. The published record points both ways at once — it follows below.
• Leave it Your site’s search and filter addresses — the near-infinite variations a search box or a filter can produce; the file’s classic job.
• Leave it Addresses with tracking tails — the decorated duplicates that newsletters and ads create, one page under many spellings.
• Leave it Technical plumbing — addresses only programs ever visit. No customer will miss them.
• Fix the rule A page customers should find — the one genuine fault this row can hold. The page is fine; a rule was written wider than its author noticed, and the rule, not the page, is what needs editing. The last section is yours.

On feeds, the published record is short. Google has never recommended blocking them. Its only statement — a 2009 post about how feeds help Google notice new pages, old enough that Google’s own banner now warns it may be outdated — leans the other way. And when an owner put the question straight, a Product Expert, in a 2023 thread, declined to pretend a rule exists:

A Product Expert in a 2023 thread, as reported

The is no absolute right answer. Yes by blocking in robots.txt can save the crawling. Although won’t crawl, they could still index. But that doesn’t really matter, the URLs are surely not private. … So generally may as well block.

No absolute right answer — either choice is defensible. But notice what both defences assume: somebody choosing. That is the one thing my line never had.

The line arrived with a template

On my site, the entire row — nine addresses — came down to one line of that file:

Disallow: */feed

Came down: that line is no longer in my file. I removed it on 24 August 2026, the day I finished the tracing this section walks through. (If you open kovalweb.com/robots.txt right now and still find it, you have just met this article’s other discovery — a cache can keep serving a file its owner has already replaced — and it gets its own section below.)

WordPress makes feeds on its own: for the site, for each author, for each post’s comments, each at an address ending in /feed/. Nobody builds them, and almost nobody looks at them. So who told Google to stay away from them? Start with who did not: me. I never wrote this file. It was written by an SEO specialist I hired — about six years ago, though that is my estimate, not a record — who was, back then, a beginner at this work. And nobody, beginner or veteran, sits down and invents Disallow: */feed out of nothing. Where did it come from?

Not from an SEO plugin, it turns out. Rank Math documents the default rules it writes — three lines and a sitemap, nothing about feeds — and one owner pasted her Yoast-generated file in full on wordpress.org, fenced by the plugin’s own START and END markers: no feed rule there either. The line comes from somewhere older: a family of “correct robots.txt for WordPress” templates — ready-made files published on blogs and in guides, meant to be copied into your site whole — that circulated for years and are still easy to find. One long-running page of “better robots.txt rules” for WordPress includes Disallow: /feed/, reasoning that “You also want to make sure that feeds and trackbacks aren’t included in the search results.” Another — a “correct robots file for wordpress” — carries my line literally: Disallow: */feed, star and all.

How do I know the specialist worked from that tradition, rather than composing every line by hand? The rest of the file testifies. One long-running guide in that tradition — a page on configuring robots.txt for WordPress — prints, under the heading “Optimal code for robots.txt for WordPress”, a recommended file with the guide’s own comment beside each rule. Seven of its lines:

Disallow: /cgi-bin              # Standard folder on hosting.
Disallow: /?                    # All query parameters on the main page.
Disallow: *?s=                  # Search.
Disallow: *&s=                  # Search.
Disallow: /search               # Search.
Disallow: *utm*=                # Links with utm tags
Disallow: *openstat=            # Links with openstat tags

My file carried all seven — six to the letter, the utm one a star short (*utm=). And the tell was *openstat=: openstat is a tracking parameter used by a Russian analytics service. Nobody composes that line by hand for a Ukrainian WordPress site; it arrives when a ready-made file does. Which particular page my copy descended from, I could not establish — an overlap like this identifies a shared tradition, not a single source.

And here is the turn I did not see coming. The line that filled my report is not in that recommended code at all. On the whole page, */feed appears under one heading only — “Controversial Recommendations”, in a subsection headed “Do Not Disallow Feeds: */feed” — where the same guide argues against the line, because other services still need open feeds. So the provenance reads plainly: my file matched a published guide’s recommended code on its most distinctive lines — and the one line that had put nine addresses into my report was the line that same guide prints under do not do this.

And the beginner detail lands in their favour. A specialist starting out did what a careful beginner should: worked from what the field itself published under headings like optimal and correct. So the answer to “who blocked my pages?” — for mine, and quite possibly for yours if someone else set your site up and your row is full of /feed/ addresses — is: a person doing the job properly, by the advice of its day, and a line that outlived the day. Nothing in that calls for embarrassment — theirs, mine, or yours.

So on 24 August 2026 I replaced my robots.txt. The feed line is gone, and so are the tracking-tail rules — the utm and openstat lines above. The blocks on search addresses and tag archives stayed, on purpose: that part of the inheritance I would have written myself. Do not read my removal as the feeds verdict — the record above points both ways. What I removed was not a block. It was a rule nobody had chosen. What I kept, I chose. Either one is a decision, and a decision is the only thing this row was ever waiting for.

Nine addresses, three answers, one rule

Now the row itself. Mine reads Blocked by robots.txt, 9 pages, and every address my report shows me ends in /feed/. Shows me, not lists — Google’s help page for the report says of its examples table:

Google’s help page, on the examples table

The list does not necessarily show all URLs with that issue, and is limited to 1,000 rows.

Three of my sample, exactly as the report prints them:

https://kovalweb.com/author/okoval/feed/
https://kovalweb.com/uk/what-is-html-microdata/feed/
https://kovalweb.com/uk/author/myhalchakmyhajlyna/feed/

An author’s feed, a post’s feed, another author’s feed — WordPress plumbing, addressed to programs. One short line held all nine because of the star: a wildcard, which Google’s technical description of the file defines as standing for “0 or more instances of any valid character”. */feed means anything, then /feed — so it caught every feed the site had.

I opened those three addresses myself, on 24 August 2026, and no two of them are in the same state. The first has stopped being a page in its own right: ask for it and the site answers moved, forwarding the visitor to the front page — a redirect my SEO plugin makes and signs, in a technical line reading x-redirect-by: Yoast SEO. The second is a working feed, serving its machine-readable copy the way it always has. The third is gone: that author was deleted, and the address now answers not found.

One forwards, one serves, one is dead — and Google knows none of it, because the rule forbade the visit that would have told it: a program that honours Disallow: */feed never fetches any of the three. The report files all nine under the one word blocked, and that is accurate — it is a fact about the rule, not about the pages. Which is this article’s sentence, demonstrated on its own addresses: a request not to visit is not an instruction to remove — and it is not a report on the page behind the address, either.

One more strip sits above my examples, because I put it there: Validation failed — Started: 7/30/26, Failed: 8/8/26, the details counting PENDING 6 · FAILED 3. This row is the very case Google’s own help page for the report names:

Google’s help page, on when not to bother validating

It might not always make sense to fix and validate a specific issue on your website: for example, URLs blocked by robots.txt are probably intentionally blocked. Use your judgment when deciding whether to address a given issue.

“Probably intentionally blocked” — rows here are usually somebody’s decision — and VALIDATE FIX asks Google to confirm the opposite: that the blocking has stopped. I pressed it anyway, at the end of July, before the tracing; with the line still in the file, the verdict was never in doubt — why one leftover instance fails a whole batch is its own article. A Product Expert had spelled it out, for an owner validating a robots.txt block back in 2020:

A Product Expert in Google’s help forum, as reported

Well validation will fail, because they STILL blocked in robots.txt … The only way to pass the validation would be to remove the robots.txt block.

Each of my Failed verdicts is the same sentence read again: the rule is still there. If you have pressed the button on a block you have since decided to keep, read the red strip the same way: Failed is the interface’s only vocabulary for still true, deliberately.

The removed line that kept answering

Removing the line, it turned out, was not even the end of the day.

Later that same day I fetched kovalweb.com/robots.txt, the way any visiting program would, to look at the new version — and the old file came back. Feed line present, tracking tails present. Then the twist: the same address with a meaningless tail on it — /robots.txt?anything=1 — returned the new file. One address, two files. The edit was real on my server, and my server was not the one answering.

What answered was a cache. My site sits behind Cloudflare — one of the services that keep copies of a site’s pages and hand them out on the site’s behalf, so the site itself is asked less often. Its copy of my robots.txt came back wearing these technical lines:

cf-cache-status: HIT
age: 1621861
cache-control: max-age=31536000
last-modified: Wed, 03 Feb 2021 10:26:29 GMT

HIT is the service saying this answer came from my saved copy, not from the site. age, counted in seconds, is how old that copy was: nearly three weeks. And max-age=31536000 is the permission the copy was saved under — one year, in seconds; a year-long licence to hand out an old robots.txt without ever asking my server whether the file changed. Google asks for /robots.txt plain, no tail — so Google got the saved copy too, feed line and all. My edit existed on my server and nowhere that mattered, and until a cache like that one is cleared, that stays true.

So if you ever edit your robots.txt and the world carries on as if you hadn’t, run the two fetches that caught this. Open yourdomain.com/robots.txt in a browser. Then open the same address with ?anything=1 on the end: caches file their copies by the exact address, so a tail the cache has never seen forces a fresh answer straight from your site. If the two differ, a cache is still serving the old file — and the cure is purging that cache, not waiting on Google. (While you are in there: a robots.txt allowed to go a year without a freshness check is worth a shorter setting.)

The stale copy had one more thing to say before it went. Its last-modified line — the date the file was last saved — read 3 February 2021: five and a half years, and not one write to that file. A saved-date records the last write, never the first, so it cannot confirm my “about six years ago” — but it agrees with it as far as a date can.

A row that stays should be somebody’s decision

For a deliberate block — a template’s line you have decided to keep, or a rule of your own — the settled state looks like nothing happening. No button to press, no validation to run, and a row that may sit in the report for as long as the rule sits in the file. The row counts addresses the rule covers, the rule still covers them, and the two are in perfect agreement. Walking away from an agreement is allowed.

For the genuine mistake — a page you want found, caught by a wide rule — the fix is never the page; it is the line. Open your /robots.txt and find the rule that matches the trapped address, remembering that a star matches anything; make it narrower, or remove it. If someone else manages the site, that one line is the entire bug report to send them. The proof afterwards: paste the address into URL Inspection at the top of Search Console and run the live test — the 403 article walks through where the buttons sit — and its answer is about the page right now, not the page as Google remembers it. Once the live test agrees the block is gone, yours is the one case where a fix actually happened, so VALIDATE FIX is now the right button — press it, and expect this row to empty rather than to stay.

And my own nine? As I write this, three days before it publishes, nothing has moved — and nothing should have: the cache above had been cleared only hours earlier, so Google had not yet read a single changed character of the new file. What I expect, as expectation and nothing more: this row should drain over weeks rather than days, and two of the nine should surface in other rows rather than vanish — the feed that answers with a redirect, the deleted author’s feed that answers not found — because unblocking an address does not restore it; it only lets Google see, at last, what the hand-check above already saw. When the report actually moves, this page will say so, with the date.

Either way, the sentence to leave with is the one the file itself taught: a request not to visit is not an instruction to remove. Behind my file there was a person all along; behind yours there is something just as ordinary — a platform’s default, a plugin’s setting, a person with a template. Neither the file nor the report can decide whether the request still speaks for you. Nine feeds, one line — written by someone else, inherited by me, and decided, at last, by me.

Written by Oleh Koval at Specia1ne, who reads these reports for a living.