3

In my robots.txt file (http://www.tutorvista.com/robots.txt), I'm using Noindex: /content/... to disallow indexing:

enter image description here

This should mean that http://www.tutorvista.com/content/ and anything below this URL shouldn't be indexed. But in the image of my search results below, you can see that pages under this URL are being indexed:

enter image description here

Additionally, I'm using Disallow: /biology/ which means that http://www.tutorvista.com/biology/ and anything below this shouldn't be crawled. But in the image of my search results, you can see that pages under this URL are being crawled and indexed.

enter image description here

So can anyone tell me what's wrong with my robots.txt directives?

dan
  • 15,123
  • 11
  • 44
  • 52
Goutham Jabez GJ
  • 63
  • 1
  • 7
  • Google's webmaster tools (select a site and go to Crawl | Robots tester) allows you to test your robots.txt against various paths on your site. Doing that with your robots.txt shows that (for example) Allow: /biology/ is the justification for /biology/abdominal-cavity-organs being allowed rather than the root, and because these rules come before the exact disallow rule. – Zhaph - Ben Duguid Sep 12 '17 at 07:14
  • And with reference to your search result shot, you have the following allow rule: Allow: /biology/animations/ above the disallow block, so that will take precedence. – Zhaph - Ben Duguid Sep 12 '17 at 07:17
  • @Zhaph-BenDuguid Thank you for your reply. Yes you are right! Now I understand. – Goutham Jabez GJ Sep 13 '17 at 10:53

2 Answers2

7

"noindex" directives should not be used in your robots.txt file, instead a noindex meta tag should be added to any pages that you don't want indexed in Google.

A NOINDEX tag looks like the below and it should be placed in the section of any page you do not want indexed:

<meta name="robots" content="noindex">

More information can be found here.

In the second example while you do have "Disallow: /biology/" in your robots.txt file, a few lines above this you also have "Allow: /biology/animations/" hence why this page in indexed in your example.

Hope this helps!

Matthew Elshaw
  • 81
  • 3
  • 2
    Actually Noindex is supposed to work in robots.txt: How does “Noindex:” in robots.txt work? but Google has said they may remove support for it, which may be what happened here. If you are going to use it, use it in conjunction with the noindex meta tag. – Stephen Ostermiller Sep 01 '17 at 10:21
  • @matthew Thanks for the reply. In the Second example image, check the fourth result it has /biology. but i have actually disallowed using Disallow: /biology/ .

    how did it show up?

     – Goutham Jabez GJ Sep 05 '17 at 09:36
  • 1
    Yeah, always have a backup plan, like @StephenOstermiller suggested :). Unofficial extensions may or may not stick around, and behavior will be different across search engines anyway. – John Mueller Sep 06 '17 at 09:16
6

Note that Noindex is not part of the original robots.txt specification. Google supported it as experimental feature (see: How does “Noindex:” in robots.txt work?), but it’s not clear if that is still the case (as they didn’t document it to begin with). But let’s assume it is.

Your robots.txt has two problems.

Empty lines

A record must not contain empty lines. Empty lines are used to separate records.

A conforming bot (which doesn’t identify as Googlebot-Image/Adsbot-Google/Mediapartners-Google) uses this record:

User-agent: *
Allow: /

So none of the following Disallow/Allow/Noindex lines apply.

Of course a bot may try to "fix" this and interpret the following lines to be part of this record (i.e., ignoring the blank lines), but the robots.txt spec doesn’t define this, so I wouldn’t count on it.

... in Noindex values

If Noindex works like Disallow (which we don’t know for sure, as Noindex is not specified/documented, but I guess it wouldn’t make sense to specify it differently), the ... you appended to the values mean that ... must appear in the URLs you want to noindex.

The line

Noindex: /content/biology/...

would apply to a URL like /content/biology/.../foobar, but not to a URL like /content/biology/foobar nor /content/biology/.

So if you want every URL whose paths starts with /content/biology/ to be noindexed, you would have to specify:

Noindex: /content/biology/
unor
  • 21,739
  • 3
  • 46
  • 117