2

Does google REALLY respect noindex, nofollow and blocking content to being indexed and seen via .htacess or robots.txt file?

I need to be sure of this 100%, I need to use duplicate content on some high ranking sites, but just for text display, I won't want Google to even know the content even exists.

Is this possible?

unor
  • 21,739
  • 3
  • 46
  • 117
Bruno Vincent
  • 259
  • 1
  • 9

2 Answers2

2

There is no foolproof method that will prevent Google from knowing that your pages exist. Google knows that pages exist as soon as it finds a link to them. It is not possible to prevent linking to your private pages from every public site. Any of your users can create links.

You also say that you want to prevent problems with duplicate content. In that case, either noindex or robots.txt disallow should work fine. Either would prevent Google from indexing the duplicate content. However even if you did allow Google to index duplicate content, Google rarely penalizes for it. When Google finds duplicate content, it usually just chooses one of the two copies to put in its search index. If you are syndicating content from somewhere else, they may require that you hide your copy from Google so there is no possibility that it is chosen for index instead of theirs. See What is duplicate content and how can I avoid being penalized for it on my site?

nofollow

<a href="http://example.com/some-page.html" rel="nofollow">

Google just announced that nofollow is now a hint rather than something they will 100% respect. Don't rely on nofollow to prevent Google from seeing links.

noindex

<meta name="robots" content="noindex">
<meta name="googlebot" content="noindex">
X-Robots-Tag: noindex

If Google finds a meta tag with noindex or a HTTP header with noindex, it will not put that page into its search index. That page will not appear on Google. Note that noindex won't prevent Google from knowing the page is there. In fact, Googlebot will crawl the page and see all the contents. However, it will not put the page into the search index.

robots.txt

Disallow: /some-page.html

If you disallow a page in robots.txt Google won't crawl the page. Even when Google finds links to the page and knows it exists, Googlebot won't download the page or see the contents. Google will usually not choose to index the URL, however that isn't 100%. Google may include the URL in the search index along with words from the anchor text of links to it if it feels that it may be an important page.

password protection

AuthType Basic
AuthName "Password Required"
AuthUserFile /directory/.htpasswd
Require valid-user

If you truly have sensitive content on your site, it is best to password protect it. Again, password protection won't prevent Google from knowing the page exists, but Google won't be able to crawl the content, nor will it include the pages in the search index.

Stephen Ostermiller
  • 98,758
  • 18
  • 137
  • 361
1

It's a little confusing but since noindex is a meta tag on the page, the page has to be allowed to be crawled (i.e. not in robots.txt) for it to take effect.

Putting a page as disallow in robots.txt will still allow Google to index it if it finds it via a link from another page. It just won't be loaded (per robots.txt) so the content including meta description will not show up in searches, however it may show when Google believes it is relevant such as when the URL contains keywords.

So my understanding to achieve zero indexing what you want is to let Google crawl it but add the noindex meta tag.

You can see some discussion around this interaction at https://news.ycombinator.com/item?id=20326445

Nick P
  • 143
  • 4