3

I have content that was deleted several years ago and from time to time Googlebot still attempts to access those pages, filling up my logs with lots of 404, making the 'real' problems harder to find and to read.

I have found Google is still crawling and indexing my old, dummy, test pages which now are 404 not found already, but that question is more about removing pages from the index. My pages are no longer indexed, but I'd like Google to stop attempting to open them.

I also believe this is not related to How to effectively close a page?. The page was closed years ago (maybe we did it wrong back then), but I would like to prevent Google from still crawling those old URL's (they have already been removed from the index years ago).

Would a 410 work? Indicating that the content will not come back? Or is this something we have no control over?

Community
  • 1
Neograph734
  • 419
  • 1
  • 3
  • 10

2 Answers2

3

Yes you are absolutely correct. You need to throw 410 to indicate that you have permanently disabled the page from your site.

404 page does not indicate that to crawler. Assume that there is some temporary issue on your site due to which 404 is shown, you don't want crawler to remove ur pages from index in that case.

Also make sure you remove these pages from sitemap, also from any other linked pages + disallow them from robots.txt.

You can find a good fix by seeing from where your pages which are throwing 404 are linked to. You can identify those resources and remove the page links from there in addition to throwing 410.

abhinsit
  • 1,714
  • 11
  • 17
  • Even with a 410 status, Googlebot will come back and check sometimes. I've seen Googlebot check 15 year old removed URLs. – Stephen Ostermiller Apr 08 '16 at 13:39
  • Thanks @StephenOstermiller for adding to abhinsit's answer and confirming what I was afraid of. Just to be sure, if I add the path to robots.txt Google will never pick up on the 410. So I suppose I shouldn't do that? – Neograph734 Apr 08 '16 at 13:44
  • Yes I think you are correct .. If removed from robots it will not know the page's status has changed .. You can wait till all are crawled as 410, then add that to robots. – abhinsit Apr 08 '16 at 13:46
  • It is a forum we've removed long ago, and Google is still attempting to fetch all the threads we had there. I am pretty sure this is based on a list Google keeps somewhere. – Neograph734 Apr 08 '16 at 13:49
  • 1
    If you can put them in robots.txt Googlebot won't crawl them. Presumably they have been 404 long enough that Google isn't indexing any of them. – Stephen Ostermiller Apr 08 '16 at 14:01
  • @StephenOstermiller Good idea. Although, if the reason Google keeps crawling those URLs is because some other sites still link to them, disallowing them in robots.txt may cause Google to reindex them based on the links alone. – Ilmari Karonen Apr 11 '16 at 11:10
  • I'm not 100% sure but I don't think Google would index something that was 404 last time they crawled it. Most are likely crawled because the once existed, not because they are linked. – Stephen Ostermiller Apr 11 '16 at 11:42
1

Better yet: 301 to the correct page.

Edit: since this forum is gone, 301 to a page that explains that the forum has been permanently shut down. That's because external links still point to interior pages of that dead forum. That's not your fault, but on the other hand you sure enjoyed collecting all that free link juice. Surely this has happened to you: you click a link that seems perfect, only to land at some jerk's homepage and go "WTH, this stinks." Back up, look at the URLs and find out the jerk destroyed the content you needed. And in that moment, who are you angry at? Bingo.

As far as Google mistaking that for doorways, no worries - people shut down forums all the time, and that's certainly better than leaving them up to be stuffed full of spam by robots. I don't know if your platform allows this, but the HTTP protocol allows you to serve a 404 that is both a redirect and an actual web page: include a Location: in the HTTP header, and an HTML header/body with the usual redirects.

The crawling continues because external sites still link to the old page location; or Google is aware of that page being bookmarked by users. That is absolutely free link juice that you have earned! If it brings enough traffic to be worth the bother, and the content is any good, restore the forum in archive mode.

Even if you 410, that doesn't mean Google will disappear from the logs forever; as long as those external links and bookmarks live, the Free Traffic Fairy (er, Google) will check up periodically to see if the URLs have Lazarus'd.

As far as the access log pollution, grep -v ’/forum-URL-pattern-here/.*404' Though you may want to know about organic traffic to those pages; it is worth money.

Harper - Reinstate Monica
  • 836
  • 5
  • 6
  • These pages have no replacement pages. We shut down the forum and that was is. Redirecting to the homepage feels a bit like tricking Googlebot, which I suppose it won't like. – Neograph734 Apr 09 '16 at 09:43
  • That's easy, redirect to a page that says "we shut down the forum." Google issue solved. These are coming from links external to your website - links made by other webmasters or posters on other websites. I don't know if you've ever clicked a link like that, but it's very annoying and confusing when the link takes you somewhere unexpected without any explanation to the surfer of why the link didn't work. Your web site is not there to make your logs easier to read. It is there to bring customers to your company. – Harper - Reinstate Monica Apr 09 '16 at 17:29
  • What advantage (besides the dubious one of sponging up residual link juice) do you think serving a 301 redirect to a "forum closed" page would have over simply serving a 404 / 410 error page with the same content? – Ilmari Karonen Apr 11 '16 at 11:03
  • That's fine too, if you're comfortable that enough browsers support displaying content on a 404/410. That wasn't always so, some browsers would either not render or pre-empt the content with their own content. Either way, the goal is customer engagement, which is the ultimate goal of link juice and traffic. It won't help link juice enough to bother... The only thing that will is leaving the forum up, perhaps in an archive mode. – Harper - Reinstate Monica Apr 11 '16 at 21:38