4

Google is indexing some pages which require login on our site, but just the title of each page.

I am quite baffled as to how this can happen, Has anyone experienced something similar?

One thing that comes to my mind is Google Analytics js sending the title of the page to Google, but that doesn't make sense.

We are also using some other 3rd party like optimizely, which might be causing this.

Any ideas what might be causing Google to index the titles of these pages?

ustun
  • 173
  • 1
  • 5
  • 1
    The answer is probably a safe "no", but is there any way you can paste a link? The fact that wget returns the redirected page makes this a tough one. Was there possibly one point in time when Google was able to index these pages normally? – Charlie Schliesser Mar 14 '13 at 20:03

2 Answers2

8

If the pages in question are in robots.txt, Google still may index the pages but use the anchor text of any inbound links as the page title. My guess is that you have links to these pages that have anchor text that is identical or similar to the page titles. Clearly if Google can't crawl the page they can't get the page contents, so they aren't going to index that.

If you don't want these pages to appear in search results you actually have to let Googlebot crawl them and return the appropriate meta robots tag

<meta name="robots" content="noindex, nofollow">

Googlebot would also be unlikely to index these URLs if they redirect to your login page and Googlebot can see that by crawling them. Especially if your login page has the robots noindex meta tag.

Stephen Ostermiller
  • 98,758
  • 18
  • 137
  • 361
  • Thank you, this seems like a plausible answer. I need to do some more investigation to find out if that is the case, I'll try altering the anchor texts on the login page to see if Google picks them up. – ustun Mar 15 '13 at 09:22
  • 1
    Usually these kinds of pages don't show up in the normal search results (but you might find them in site:-queries), so in most cases, I'd just leave them robotted out. Using the noindex (as mentioned above, you need to allow crawling then) is also fine. – John Mueller Mar 19 '13 at 20:18
5

This is purely down to the way your site is operating, the titles are being processed without requiring authentication. As long as the rest of the page is not rendering then its not a serious problem and since Google isn't indexing the contents of the page then its a good sign that this isn't the case.

The mistake here is that you have not blocked the administrator/private areas of the site.

You should use both of these methods to stop Google even attempting to index these pages,

On all administrator/private pages use:

<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

Additional a extra layer of protection against indexing is using the robots.txt, you can block Google and others indexing these pages by using:

User-agent: *
Disallow: /administrator/
Simon Hayter
  • 32,999
  • 7
  • 59
  • 119
  • This is what I thought initially, and still think that it could be the answer, however we check the logged-in status of users, and redirect them to login page without returning anything. So, for example if I wget the file Google indexed, I get redirected with no trace of the login-only page. – ustun Mar 14 '13 at 19:53