2

Sitemaps are necessary part of our web work, and we need them to get more "Google love". But for the past one month or so, I have not been able to submit sitemaps on Google search console.

I used https://technicalseo.com/tools/fetch-render/ to check, and I passed there successfully. I used sitemap validator and that too showed no errors. It seems that something might be wrong with the search console.

Yesterday I started another site and used the same Wordpress plugin to generate a sitemap and then used the same google account to log-in to search console, and tried adding a property, this time it worked.

If I submit any new sitemap, it says couldn't fetch and if I try to resubmit existing one, then it is successful. But last read date is 6 months back, so I assume it doesn't work.

What are the possible solutions? The live testing says, that it's unable to fetch, because it is noindex. I feel that there is nothing wrong with keeping sitemaps noindex, because sitemaps should not appear in search. In-fact my second domain also got sitemaps submitted with noindex

I am using WordPress php 7.3, and HTTP/2

My sitemap Url is https://milyin.com/sitemap.xml

My HTTP headers are: 

HTTP/2 200 
server: nginx
date: Thu, 22 Aug 2019 12:23:21 GMT
content-type: text/xml; charset=UTF-8
vary: Accept-Encoding
x-robots-tag: noindex, follow
link: <https://milyin.com/wp-json/>; rel="https://api.w.org/"
last-modified: Wed, 21 Aug 2019 14:03:48 GMT
etag: W/"59184deaadc3de11f553f5a8fbaac7f0"
x-httpd: 1
x-robots-tag: index, follow
host-header: 192fc2e7e50945beb8231a492d6a8024
x-proxy-cache: MISS
content-encoding: gzip
Anuvesh
  • 1,222
  • 1
  • 7
  • 20
Amit Bajar
  • 23
  • 4
  • Not necessarily related to your problem, but you have 2 conflicting x-robots-tag headers in your response. (Not necessarily an issue, but you also have 2 overlapping Vary response headers when looking at the response.) You mention .htaccess, however, your response headers are coming from an Nginx server? You also seem to be employing service workers (to help generate the sitemap)? Maybe related, maybe not, but when I test your sitemap using a local CURL gui I don't get any response for some reason?! – DocRoot Aug 28 '19 at 15:21

1 Answers1

2

Don't use noindex, it's just not necessary. It seems you have a XSL/XSLT for the sitemap, which doesn't help and is the most likely cause for Google adding the sitemaps in its' index by mistake.

EDIT: Unless you specifically restrict indexing of your sitemaps, Google will archive them in its' index. However, if you have followed the rules and guidelines, it will be hidden from most SERPs unless specifically searched for. Also, this from Google

Using a sitemap doesn't guarantee that all the items in your sitemap will be crawled and indexed, as Google processes rely on complex algorithms to schedule crawling. However, in most cases, your site will benefit from having a sitemap, and you'll never be penalized for having one.

Remove the timestamps from your robots.txt. See below for example. More about robots Sitemap: https://milyin.com/sitemap.xml Sitemap: https://milyin.com/sitemap-home.xml Sitemap: https://milyin.com/sitemap-posttype-post.2019.xml

... and so on.

Afterwards, submit it as your sitemap-index in GSC.

Trickytree22
  • 141
  • 5
  • 1
    Google will index XML sitemaps, as it will any XML file, regardless of whether XSLT are applied. https://webmasters.stackexchange.com/questions/62978/prevent-xml-sitemaps-from-showing-up-in-google-search-results – MrWhite Aug 22 '19 at 14:43
  • Search for: filetype:xml sitemap. – MrWhite Aug 22 '19 at 15:26
  • You mean if I specifically look for xml sitemaps, Google returns those in it's result. Hmmm, working as intended. But you have a point, I'll edit the wording of my post. – Trickytree22 Aug 22 '19 at 15:34
  • Does that mean, that when when i try to index sitemap as a normal url it should get indexed? I have another site, and on it my sitemap got submitted even when it was not getting indexed. Is NoIndex a hurdle in submition of sitemap? – Amit Bajar Aug 23 '19 at 02:20
  • Sitemaps exist for search engines. Google have spent countless of hours so that you can submit one as conveniently as possible. They have standards in place for a reason, and not once have they claimed that it should be noindexed. Don't spend time second guessing them, use your valuable time on making great content instead. – Trickytree22 Aug 23 '19 at 07:57
  • Dude, the problem remains, that i use wordpress, and my plugin automatically ads noindex, and i edited .htaccess file to index all urls on my site, but that too doesn't help – Amit Bajar Aug 23 '19 at 08:39
  • Your robots.txt still contains timestamps. .htaccess is incompatible with Nginx. Try using a converter. (https://winginx.com/en/htaccess) – Trickytree22 Aug 23 '19 at 09:10
  • @AdityaAgarwal Which part of this answer resolved your issue? I'm having a similar problem, but having checked your site/sitemap you don't appear to have employed any of the recommendations mentioned here? – DocRoot Aug 28 '19 at 15:27
  • 1
    I just tried to index my sitemaps in Search Console, it threw an error, after which it showed me a Live Test, where scrolling down a bit, said page fetch successful. Which meant that it was just a buf from thwir end, and my sitemap works. After a week, my urls from that sitemap automatically indexed... – Amit Bajar Aug 29 '19 at 12:37