3

When I create an xml sitemap using an online sitemap creator, it adds the following

<url>
   <loc>http://www.mysite.co.uk/</loc>
   <lastmod>2017-09-26T18:03:59+00:00</lastmod>
   <changefreq>monthly</changefreq>
   <priority>1.0000</priority>
</url>
<url>
   <loc>http://www.mysite.co.uk/index.html</loc>
   <lastmod>2017-09-26T18:03:59+00:00</lastmod>
   <changefreq>monthly</changefreq>
   <priority>0.8000</priority>
</url>

This seems to add the same page twice, as the index.html is the same page as the domain name.

I don't want Google to class this as duplicate content

Do I need to add a canonical expression to my index.html page - like this?

<link rel="canonical" href="http://www.mysite.co.uk/ >

If I do that, do I remove this line from the sitemap?

 <url>
       <loc>http://www.mysite.co.uk/index.html</loc>
       <lastmod>2017-09-26T18:03:59+00:00</lastmod>
       <changefreq>monthly</changefreq>
       <priority>0.8000</priority>
    </url>

The links to my homepage on all my other pages look like this

<a href="/index.html">Home</a>

Do I need to change these links so they read

<a href="http://www.mysite.co.uk">Home</a>

Any help would be appreciated

Thanks in advance

Raul Reyes
  • 2,476
  • 12
  • 22
Joel
  • 33
  • 1
  • 4
  • Does your all pages have directory structure I mean www.example.com/contact/ and www.example.com/contact/index.html display same thing, or it is only for home page? – Goyllo Sep 27 '17 at 08:47
  • Sitemap has no bearing on what Google classifies what duplicate content is, in fact Google can crawl most sites without a sitemap, nowadays many sites operate without one. Sitemaps are only useful if you pages that are not linked such as landing pages, or have crawling issues. Canonical links should always be used if your site or page can be accessed more than one way, i.e if https://www.example.com is your desired page then http://197.241.215.194/ http://example.com, http://www.example.com would all need a canonical pointing to the page you prefer, or alternatively you can redirect. – Simon Hayter Sep 27 '17 at 13:00
  • @Joel Hi, do you think either of us have helped you finding an answer? if so, please mark the one you think answers your question. :) – Raul Reyes Sep 29 '17 at 14:08
  • Hi guys. Thank you all for your help - I didn't realise this was such a contentious issue! – Joel Sep 29 '17 at 16:40
  • What I did was this

    Replaced all link to home page with href="/"

    Removed

    http://www.mysite.co.uk/index.html 2017-09-26T18:03:59+00:00 monthly 0.8000

    I didn't do a 301 redirect as I was worried about the repercussions if I did it wrong

    Thanks again to all who responded for for your help and time

     – Joel Sep 29 '17 at 16:49

2 Answers2

0

You need to fix this issue by “homogenizing” the way the internal links are set across the whole website. When you create internal links you have basically two options 1) use relative or 2) absolute paths.

For example, the same internal link could be set either way like this:

href=“/index.html“ (1)

href=“http://www.example.com/“ (2)

You need to decide if you are going to use relative or absolute paths. Pick one way and stick with it. Make sure you are consistent across the website.

Your sitemap however, should alway contain the website URLs using absolute paths. In your case, I would definitely create one and only one reference to the homepage, even if you have created links using relative paths to the file (/index.html), you need to use absolute paths here.

To avoid having this issue make sure you do the following:

  • decide between relative vs absolute paths.

  • if you choose relative make sure all your internal links that point to the homepage via navigation menu, logos or any other internal links are using href=“/“ instead of href=“/index.html”.

  • fix this canonical issue by creating 301 redirects from example.com/index.html to example.com. One way to do this is via the .htaccess file if you are using Apache as the web server.

Note: the online sitemap creator that you used is an automated tool or web crawler that will retrieve the urls found according to the way the web server is configured, if the crawler or bot is smart he will classify /index.html the same as example.com. My recommendation here is always double check the output online tools and in this case, feel free to edit the sitemap according to what I said above.

Raul Reyes
  • 2,476
  • 12
  • 22
  • Even if you use relative paths, you can omit the "index.html" in links: href="/" or href="./" or href="../" for example – Stephen Ostermiller Sep 28 '17 at 15:33
  • @StephenOstermiller true, but I needed to create an answer using the same question examples, making sure I am understood and create some sort of context. – Raul Reyes Sep 28 '17 at 21:21
-1

XML Sitemaps are still very important to Google.

I manage a site with 100m+ pages and can tell you from experience that they make a strong, positive, impact, on increasing our crawl rate (as seen in Google Search Console).

Answer: Keep your URLs clean throughout your codebase.

For your internal links, canonicals, and XML Sitemaps:

  • Do not include .html
  • When linking to your homepage use <a href="/" title="home">
  • BE CONSISTENT with your trailing slashes. (example.com/about is different than example.com/about/) Choose one pattern and stick to it.

Try going to your home page via all combinations to see if the page loads.

If you go to example.com, example.com/ and example.com/index.html they should all properly 301 redirect to your preferred canonical version of the URL.

Andy Muns
  • 110
  • 6
  • XML sitemaps are not at all important to Google. For almost every site, Google can crawl and index every page just fine without sitemaps. Sitemaps may help minimally on very large websites that don't link to all their content. Even in those cases, pages would rank much better if linked from the site rather than only included in the sitemap. A sitemap is most useful for getting extra stats about your site though Google Search Console. See: The Sitemap Paradox – Stephen Ostermiller Sep 27 '17 at 19:26
  • Your advice about not including .html is poor. In many cases omitting the .html from the URL will cause it not to work. In those cases, including the .html on the URL is necessary when linking or creating a sitemap. I think you may mean that choosing URLs without .html is preferable to creating URLs that have it. That is probably a bit better for usability, but shouldn't change SEO or rankings. – Stephen Ostermiller Sep 27 '17 at 19:29
  • I can tell you, with significant data (i.e. 300k pageviews/day), that sitemaps are important to Google, even when linking internally to pages. I agree that Google doesn't require an XML sitemap, and you can still do well in SERPs without one, but if the goal of your site is to have more organic traffic, then use a properly formatted sitemap. – Andy Muns Sep 27 '17 at 19:30
  • 301 redirects for the home page index.html can be a good idea, but your advice about the home page with and without a slash is wrong. The URL with and without the slash are exactly the same. It is impossible (per the HTTP spec) to make a request for the home page without the slash. Therefore, it is not possible to redirect to add it. Browsers typically do so for you. You also don't mention the alternative of canonical tags which was asked about in the question and can be just as good for SEO as redirects. – Stephen Ostermiller Sep 27 '17 at 19:30
  • After all this bad advice, you don't actually answer the question. You don't say if example.com/index.html should be omitted from the sitemap. You don't say whether or not a canonical tag is needed. – Stephen Ostermiller Sep 27 '17 at 19:32
  • Yes, I did answer that question. Are you referring to the "bad advice" that I use in practice and that put my sites in the top 0.0001% of sites with the most organic traffic in the world? – Andy Muns Sep 27 '17 at 19:39