Google is indexing URLs with parameters that are disallowed in robots.txt despite canonical URLs without parameters listed in the sitemap

Question

All of my webpages are showing ?mode=grid & ?mode=list in the google coverage. But submitted sitemap shows normal URLs. For example:

example.com/page/?mode=grid
example.com/page/?mode=list
example.com/page/ ---> [url in sitemap]

And the robots.txt has a command Disallow: /*? which has led to blocking of all webpages from index. I don't want to remove the disallow command. How can I get the webpages indexed, removing command will show ?mode=grid & ?mode=list in google searches. Also this is a WordPress website.

Disallowing in robots.txt doesn't block indexing, it only blocks crawling. Google usually won't index URLs it can't crawl, but sometimes it does — Stephen Ostermiller, Jun 20 '20 at 14:43
Also, if Google can't crawl, it can't see your canonical tag. So it does not know that it is the wrong URL to index. — Tony McCreath, Jul 09 '20 at 09:45

EVE Milano · Answer 1 · 2022-08-17T11:05:04.083

The Disallow directive doesn't ask search engines not to INDEX a resource, just not to crawl it. If google finds a link to that resource, it indexes it, even without being able to scan it (enter the link graph). Also, the rel canonical tag cannot have any effect, since disallowing prevents google from reading the page (and thus finding the canonical rel). In order not to index a page you should use NOINDEX, or a rel canonical, but NEVER a Disallow.

score 1 · Answer 2 · answered Aug 05 '22 at 13:16

I had the same issue and it turned out that if a page uses the parameters then the page with the param may get indexed. Once indexed, Google kept preferring the parametrized page over the non-parameter version.

I took multiple measures and I am getting rid of them. Use the measure that are appropriate for you -

Use the Google Removal Tool to remove the param version of the url
Stop internal (and external) linking to param versions
Do a 301 redirect from the param version to the non-param version. (Or return a 404, it depends on your situation).
Ensure the canonical in the param version points to the non param version.

score 0 · Answer 3 · edited Jul 08 '20 at 13:11

0

Thinking outside of the box you may be able to apply regex to your robots.txt file to disallow any variable parameters such as https://example.php/?p=x. You may be able to filter the question mark indicating a variable in your robots.txt with regex in the file itself. I actually just Googled this and although the robots.txt file does not allow regex it looks like Google crawlers allow pattern matching. This may be the answer you are looking for. See https://searchengineland.com/a-deeper-look-at-robotstxt-17573

edited Jul 08 '20 at 13:11

Stephen Ostermiller

98,758
18
137
361

answered Jun 20 '20 at 15:55

Michael Lazin

46
1

Why do you suggest regex before saying that robots.txt doesn't support them? – Stephen Ostermiller Jul 08 '20 at 13:13

Google is indexing URLs with parameters that are disallowed in robots.txt despite canonical URLs without parameters listed in the sitemap

3 Answers3