I have several websites on one hosting account. The main website is in the root. Other websites are in sub-folders off the root directory.
In the robots.txt file for the main website, do I need to Disallow other website directories?
Asked
Active
Viewed 67 times
2
1 Answers
1
In the robots.txt file for the main website, do I need to Disallow other website directories?
Only if you are wanting to block these directories on the main site, which shouldn't be necessary, since mainsite.com/subsite/ should ideally be blocked at the server level (ie. in .htaccess).
Additional... if these "websites" each have their own domain (they are "add-on" domains) then you will need a separate robots.txt for each site in its own root directory, which appears to be a sub-folder of the main site in this case. Providing you do actually want to block crawlers on these sub sites.
MrWhite
- 42,784
- 4
- 49
- 90