1

I am working on an e-commerce website (similar to e-bay) where users can buy or sell products.

The urls for buying a product are like:

example.com/browse/cars
example.com/browse/toys
etc

And the urls for selling products are like:

example.com/post-ad/cars
example.com/post-ad/toys
etc

When selling a product, user just need to fill a form and click on Save. Also user needs to be logged-in, in order to sell a product, therefore I have decided to prevent this page from being crawled and added the following to my robots.txt

Allow: /

Disallow: //post-ad
Disallow: /account/login?ReturnUrl=

Now I am getting the following warning in Google Search Console, for of of the post-ad pages:

Indexed, though blocked by robots.txt

How can I resolve this issue?

Stephen Ostermiller
  • 98,758
  • 18
  • 137
  • 361
Hooman Bahreini
  • 538
  • 4
  • 21

1 Answers1

2

Google can still index based on links to these pages.

Below is a passage from Google support: (https://support.google.com/webmasters/answer/7440203?hl=en)

The page was indexed, despite being blocked by your website's robots.txt file. Google always respects robots.txt, but this doesn't necessarily prevent indexing if someone else links to your page. Google won't request and crawl the page, but we can still index it, using the information from the page that links to your blocked page. Because of the robots.txt rule, any snippet shown in Google Search results for the page will probably be very limited.

The support page also states that if you really do not want the page to be indexed, you should have to implement "noindex" either as meta tag or as http header.

Kannan
  • 2,343
  • 11
  • 24