2

I have a project and I would like to disallow everything starting with root.

From what I understand I think I can do so by doing this

Disallow: /
Disallow: /*

However I would like to allow 4 subdirectories and everything under those subdirectories.

This is how I think it should be done

Allow: /directory_one/
Allow: /directory_one/*
Allow: /directory_two/
Allow: /directory_two/*
Allow: /directory_six/
Allow: /directory_six/*
Allow: /about/
Allow: /about/*

So how would I go about disallowing everything starting from root but allowing only those 4 directories and everything under them?

Also if I want to allow specific directory and everything under it, do I have to declare it twice?

Will webcrawler be able to navigate to those subdirectories if root is disallowed?

Quillion
  • 185
  • 3

2 Answers2

2

What you have would seem to be "about" correct, assuming you have the appropriate User-agent directive that precedes this?

Disallow: /
Disallow: /*

However, you don't need to repeat the same directive, one with a trailing * and one without. robots.txt is prefix matching. The trailing * is superfluous and these match the same URLs.

User-agent: *
Allow: /directory_one/
Allow: /directory_two/
Allow: /directory_six/
Allow: /about/
Disallow: /

Whilst Google (and the big search engines) match using the longest-path-matched method, I believe you should put the Allow directives first for those search engines that use the first-path-matched method.

Will webcrawler be able to navigate to those subdirectories if root is disallowed?

Yes, that is the reason for the overriding Allow directives. Strictly speaking, the Allow directive is a more recent addition to the "standard", but all main search engines support it.

DocRoot
  • 4,297
  • 12
  • 20
  • I've not heard of any crawlers that use "first matching rule". I can't find many references to such a rule online. One of the few is our question here: What's the proper way to handle Allow and Disallow in robots.txt? which doesn't link to any sources or documentation for it. – Stephen Ostermiller Jul 21 '20 at 15:49
  • I thought it was how some "older" bots behaved, and I believe is how Baidu still does: "Please note that the order of Disallow and Allow records is important as robot takes the records in sequence from the beginning of the file. Once matched, robot will perform the job accordingly.". – DocRoot Jul 21 '20 at 17:24
  • Wikipedia - "Robots exclusion standard" (also linked to in that question) under the "Allow directive" does state, "by standard implementation the first matching robots.txt pattern always wins, Google's implementation differs" and suggests using Allow before Disallow for better compatibility. – DocRoot Jul 21 '20 at 17:25
  • Since the original "spec" only used the Disallow directive, "standard implementation" would only require a simpler first-match method. For a developer implementing an Allow directive there wouldn't necessarily be an incentive to also change the implementation, since a first-match method does "work" and is easier to implement. – DocRoot Jul 21 '20 at 17:27
2

I would not recommend trying to set up your site to disallow everything except certain directories.

  • The home page of your site will be blocked from crawling. Because most sites get many links to their home page, your site will be throwing away SEO value from a lot of incoming links. You will need to have direct external links to your deep crawlable pages to make your SEO work.
  • Futhermore, the home page is usually where bots find navigation to the other pages on your site. Without allowing crawling of the home page, bots may not be able to crawl all the pages of your site. You'd need to ensure that there are links between your deep crawlable pages.
  • Even though your home page wouldn't be crawled, it would likely still be indexed. When somebody searches for your brand name, the Google search results would likely have link to your home page with the unfriendly message "A description for this result is not available because of the site's robots.txt -- learn more"
  • Most crawlers don't know how to deal with Allow directives. Most crawlers will be disallowed from crawling your entire site. Only a few crawlers will know they are allowed to crawl those subdirectories. Luckily the major search engine bots process Allow directives.

I would recommend moving all disallowed content into its own subdirectory and disallowing it. For example, put all your blocked content into /private/ and use the following in robots.txt:

User-Agent: *
Disallow: /private

If you are comfortable blocking most bots other than search engine crawlers and with having worse SEO because your home page is not crawled, your idea of using Allow for specific directories could work. There is no need to use the wildcard * at the end of any directive. All robots.txt rules without a wildcard are "starts with" rules. There is an implied * at the end of each and every one. All you would need would be:

User-Agent: *
Disallow: /
Allow: /directory_one
Allow: /directory_two
Allow: /directory_six
Allow: /about

The order of the rules shouldn't matter. When multiple rules match, the longest rule should apply. So:

  • / (the home page) matches Disallow: / and crawling is not allowed
  • /foo matches Disallow: / and crawling is not allowed
  • /about/bob matches both Disallow: / and Allow: /about. The longer rule will apply and crawling would be allowed.

You can test this with Google's robots.txt testing tool that is part of search console.

Stephen Ostermiller
  • 98,758
  • 18
  • 137
  • 361
  • That is an amazing answer!!! Thank you :) and thanks for the info about SEO!!! I am more than content with blocking my main homepage from being indexed – Quillion Jul 21 '20 at 20:17