1

I have a site with many pages (30MM+ indexed) and have been getting a extremely high number of URLs on your site warning message for a long time. I'm thinking of reducing the numbers of URLs Google sees especially between similar pages, specifically on pages that have a different language template but have the same content.

Current for every page I have this in my <head>:

  <link rel="alternate" hreflang="en x-default" href="http://mydomain.com/dir/page" />
  <link rel="alternate" hreflang="ja" href="http://ja.mydomain.com/dir/page" />
  <link rel="alternate" hreflang="ko" href="http://ko.mydomain.com/dir/page" />
  <link rel="alternate" hreflang="th" href="http://th.mydomain.com/dir/page" />
  <link rel="alternate" hreflang="es" href="http://es.mydomain.com/dir/page" />
  <link rel="alternate" hreflang="pt" href="http://pt.mydomain.com/dir/page" />

Will adding a rel=nofollow or a robots.txt disallow help Google focus on the more important pages he has to crawl instead of crawling a different template of the same page?

Stephen Ostermiller
  • 98,758
  • 18
  • 137
  • 361
Noam
  • 2,008
  • 1
  • 21
  • 28
  • 1
    It's only a warning and nothing that you need to take action on. How many hreflang templates do you have in the <head> of each page? – zigojacko Jun 18 '14 at 08:37
  • Exactly like in my code snippet - 6. I know it doesn't necessarily mean I should take action, but it feels he doesn't really need to invest his time crawling these template pages, doesn't it? – Noam Jun 18 '14 at 09:49
  • They aren't going to be showing that warning because of 6 hreflang links in the <head> - that's nothing. How many other URL's are on the rest of your pages, what type of website is it? – zigojacko Jun 18 '14 at 09:53
  • @zigojacko for every page I have 6 language versions, so it's 6x not just 6. For example: http://twtrland.com/profile/zigojacko and http://ko.twtrland.com/profile/zigojacko ... (you can also get the feeling of what type of website it is) – Noam Jun 18 '14 at 10:00
  • Yes I get that but 6 links on 1 page multiplied by 30 million is not an issue in terms of links per page. It is only because you have such a large site that you see that warning. I wouldn't even expect Google WMT to be referring to the URL's in the ` of your pages with this warning either tbh. (Signing up to your website though as it looks useful) :) – zigojacko Jun 18 '14 at 10:05
  • WMT doesn't show these as example pages in the warning, but I'm trying to remove as much as I can. Thanks and happy to have you on-board, would love to hear your thoughts – Noam Jun 18 '14 at 10:14

2 Answers2

3

Adding nofollow generally does not prevent Googlebot from discovering content. See: Does a "nofollow" attribute on a link prevent URL discovery by search engines?

If the nofollow link is the only link to that page, then Google won't index it. In your case, I highly doubt that your alternate language links are the only links into your other sites. Once Google finds a dofollow link to any page on the alternate language subdomain, they would be able to discover the rest with dofollow links there.

The "extremely high number of URLs on your site" warning from webmaster tools is nothing to worry about if your site is supposed to have that many pages. Google is more worried about cases such as:

  • A site with session ids in the URL that make the site appear to have a different URL every time Google crawls it
  • Sites that can auto-generate a page for any URL

Every site with millions of pages gets this message in Webmaster Tools. There is no problem with a site having that many pages if it has the content to back it up. I've worked with several such sites myself. None of them have had trouble with rankings because of it.

Community
  • 1
Stephen Ostermiller
  • 98,758
  • 18
  • 137
  • 361
0

Search engines other than Google, of which there are several, sometimes do pay attention to the nofollow attribute. I worked on a site with only nofollow links from its parent company and no (0) pages indexed on Bing, Yahoo, or DuckDuckGo. Within 8 weeks of having the nofollow attribute removed, we started getting indexed in those engines.

Vic Shoup
  • 1