3

My company is renting a CentOS server, with one main domain and ~30 subdomains (the number varies, but it's usually around 30) hosted on it. The subdomains themselves are used for various purposes (mostly development), and are usually deleted once they outlive their use.

The setup is as follows

public_html / 
    maindomain_com
    subdomain1_maindomain_com
    subdomain2_maindomain_com
    .
    .
    .
    subdomain30_maindomain_com
    subdomain31_maindomain_com
    .
    .

Recently it has come to our attention that some of the search engines have indexed some of our defunct and some of our active subdomains.

What would be the best way of preventing search engines and crawlers from crawling and indexing the main site and its subdomains?

I am aware that it's possible to create a robots.txt file with noindex, nofollow, etc directives, but I'd like to avoid it, if possible (going through ~30 subdomains, setting up a new robots.txt when a new subdomain is created, etc).

I'm also aware that it's possible to use:

<IfModule mod_headers.c>
  Header set X-Robots-Tag "noindex, noarchive, nosnippet, nofollow, noodp, noydir"
</IfModule>

in .htaccess. What I'm unsure of is whether I could place this directive inside of main domain's .htaccess, in hopes of preventing crawling for both the main domain and its subdomains.

Is there a better way of doing this? I've come across a StackOverflow answer that Alias can be used in httpd.conf for robots.txt. Can this be used for .htaccess as well?

FiddlingAway
  • 193
  • 7
  • Crawling and indexing are two different things. It is usually impossible to totally prevent both. You can either prevent all crawling and accept that a URL or two occasionally gets indexed, or you can allow crawling and prevent indexing. – Stephen Ostermiller Feb 12 '21 at 08:34
  • Robots.txt doesn't support nofollow. Google used to support noindex in robots.txt but no longer does. The only relevant robots.txt directive that is available is Disallow:. – Stephen Ostermiller Feb 12 '21 at 08:38
  • Why do you want a .htaccess solution rather than something for httpd.conf? – Stephen Ostermiller Feb 12 '21 at 11:28
  • @StephenOstermiller Thank you for the first two remarks, I'll keep them in mind. As for the .htaccess solution, it seemed to be the easiest to implement in case a crawlable domain is added to the server. But in this case, it's only one main domain, and xy subdomains. – FiddlingAway Feb 12 '21 at 11:54
  • Do all of these subdomains have the same document root are are you serving them out of different directories? – Stephen Ostermiller Feb 12 '21 at 12:09
  • @StephenOstermiller Every subdomain has its own directory. I've added the structure of public_html to my question. – FiddlingAway Feb 12 '21 at 12:19
  • "place this directive inside of main domain's .htaccess" - Although from your file structure it looks like subdirectory maindomain_com that presumably serves the main domain's content is at the same level as the subdomain directories? – MrWhite Feb 12 '21 at 13:25
  • @MrWhite maindomain_com serves the main domain's content, and all of the subdomains are on the same level, and have contents of their own. – FiddlingAway Feb 12 '21 at 14:42

0 Answers0