-1

Posting this here because this post doesn't seem to show up on the Google Webmaster Community forum (must have some sort of keyword that causes it to be shadow banned). I want to stress that I would put a lot of money on this being a genuine bug with Google - I'm not just blaming Google for this on a whim - this has been an ongoing issue for months that I've invested tens of hours into trying to fix.

Simply put, there is a bug with Google which is causing it to not crawl our sitemaps. This is specific to one of our sites (we run 10+). The website URL is https://minecraftitemids.com.

Symptoms:

  • Google doesn't pick up on site changes, new pages, etc for MONTHS (e.g. we removed a page in February, but it was still in the index last week). I don't think this is a bug with search console - it happened in the old search console too - due to these symtpoms

  • Google Webmasters reports that the sitemap was last crawled LAST YEAR:

    enter image description here

The website receives thousands of visitors from Google every day - I really don't think this could be a priority issue.

There is no issue at all with the sitemap. You can find it at https://minecraftitemids.com/sitemap.xml. Run it through any tester, etc - it passes.

I have made two previous posts about this on the Webmaster forums. I even submitted a question to the Google Webmasters Hangout on Youtube (but didn't hear back after they told me to leave a comment).

Where can I report this bug? What else can I do? I've tried everything to fix this and/or get in touch with someone at Google.

Stephen Ostermiller
  • 98,758
  • 18
  • 137
  • 361
Daniel Hunt
  • 1
  • Removing a URL from a sitemap will never remove it from search results. That isn't what sitemaps are for. I'm not sure what you are hoping for with sitemaps, but they usually have little to no bearing on how your site is crawled and indexed. If, like most sites, your site has links to all its pages from other pages, you don't need a sitemap at all. Google doesn't usually choose to index a page just because it is in a sitemap. Including a page in the sitemap doesn't help that page rank better. See https://webmasters.stackexchange.com/questions/4803/the-sitemap-paradox – Stephen Ostermiller Jun 14 '19 at 20:49
  • @StephenOstermiller Appreciate the response, thank you. I understand that removing a page from a sitemap wouldn't remove it from search - I probably explained my point there poorly. The point was that Google isn't picking up on changes made to the site. The sitemap issue is a very good way of showing that there's something wrong here. Why is Google not crawling a sitemap after 6 months? This isn't normal behavior, and it would be stupid to say that it is. – Daniel Hunt Jun 14 '19 at 21:36
  • You shouldn't be relying on sitemaps to notify Google of new pages. Simply link to new pages from other popular pages of your site and Google will crawl them much sooner. – Stephen Ostermiller Jun 14 '19 at 22:55
  • @StephenOstermiller I think you are misunderstanding my point. The point isn't that sitemaps have any relevance - I'm not arguing that they do or don't. There is a bug with Google that is causing it to not crawl our sitemaps. I mentioned that it doesn't detected changes on our site for months because I feel this is also an issue that is related to the sitemap issue (two separate issues, maybe both caused by the same bug). – Daniel Hunt Jun 15 '19 at 10:44
  • I'm saying that it shouldn't matter whether or not they crawl your sitemap regularly. Sitemaps just don't do that much. Just forget about it and focus on more important things. – Stephen Ostermiller Jun 15 '19 at 12:33
  • @StephenOstermiller I should be more clear - Google isn't ranking new pages, it isn't detecting changes on the site for MONTHS. Essentially, this site will only rank for content that it had up before this issue started - I've made new content in January that is for <10 volume keywords that isn't ranking. The sitemap issue is a tangible issue that I think is related - that is why I am making a point about the sitemap - because if I say "my site is having issues ranking" this will be treated like there is an issue on my end. – Daniel Hunt Jun 15 '19 at 12:49
  • To develop further, this sitemap behavior is not normal - IMO it proves that there is definitely something wrong here. It doesn't matter if Google actually crawls my sitemap - maybe that's correct - but it should matter that Google's behavior for crawling my site (and my sitemap) is suggesting that there is an issue or bug. – Daniel Hunt Jun 15 '19 at 12:51

1 Answers1

0

Imho it is pretty sure a priority issue. The relation of content amount to url is such small, so Google doesn't spent crawling capabilities to your urls. You have a notable thin content problem on your urls (some of i looked into).

I would deindex and remove from sitemap all urls without substantial traffic. After this urls, which remain in the sitemap will be crawled more often.

Evgeniy
  • 10,062
  • 1
  • 17
  • 48