0

Our website http://www.example.com has a sub-domain http://jobs.example.com that uses a tool which shares our job listings. We would like for job listings to become indexed in Google search results. The biggest hurdle is that we are unable to upload a sitemap to the sub-domain (long story,) so I am trying some work-arounds to get Google to notice these pages.

I have checked against all of the items in this post: Why isn't my website in Google search results?. As I mentioned, submitting the sitemap isn't possible. We do have incoming links, and the fetch test for individual links presented no problems, though none were ultimately indexed.

Now, I'm trying to see if the robots.txt file is the problem. If I can rule that out, then I think I can say that there is something in the code that is causing an issue. The provider of the tool says that pages are generated dynamically, but I don't see why that would be a problem for indexation.

Here is the information from the robot.txt file:

User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /feed/
Disallow: /trackback/
#Disallow: wp-login.php
Disallow: wp-signup.php
Disallow: /xmlrpc.php

My suspicion is that this is some issue with the tool on the sub-domain that I am not seeing, but I'd love to find out that there is something else going on that we can fix.

Stephen Ostermiller
  • 98,758
  • 18
  • 137
  • 361
KDP
  • 267
  • 1
  • 6
  • 1
    I am a bit confused so be patient with me. Your robots.txt file on your parent domain example.com cannot effect the sub-domain at all. You would need a robots.txt file on your sub-domain as well. Is the robots.txt file in your question referring to a robots.txt for your sub-domain or parent domain?? – closetnoc Sep 02 '15 at 14:15
  • Have you tried using the Google 'fetch URL' tool that is mentioned in the question you linked to? That would give you a definitive answer. – Tim Fountain Sep 02 '15 at 16:53
  • And how long as the site been online? – Simon Hayter Sep 02 '15 at 19:08
  • @closetnoc, this is the robots.txt file from the sub-domain. Tim, yes, I have tried the Google 'fetch URL' tool with several URLs on the sub-domain. Status is "Complete" and I am notified that the URL has been submitted to index. Simon, I believe the site has been online for four or more months. – KDP Sep 03 '15 at 18:46

1 Answers1

1

That looks like a fairly standard (WordPress) robots.txt file, so it shouldn't really be blocking anything of significance, regardless of where it is actually located (although that would be useful to know).

the fetch test for individual links presented no problems

Assuming this is the fetch tool within Google Search Console (formerly Google Webmaster Tools), then you've already answered your question. The tool would be unable to fetch your page if it (or a linked resource) was blocked by robots.txt.

MrWhite
  • 42,784
  • 4
  • 49
  • 90
  • Oh... if I had just looked a little harder- good catch!! – closetnoc Sep 02 '15 at 19:40
  • @w3d, yes, this was the fetch tool within the Google Search Console. I was notified that the fetch was "complete" and "URL submitted to index." It is very odd that the job postings are not appearing in the index while other pages on the sub-domain do, but it sounds like I can rule out the robots.txt, right? – KDP Sep 03 '15 at 18:48
  • The Google "fetch tool" would have given you an immediate error/warning if any part of the page was blocked with robots.txt. Do these pages not appear in the index at all (ie. do a site: search for the specific URL) or can you just not find them when doing an ordinary Google search? – MrWhite Sep 03 '15 at 19:19