Is it possible set the crawl delay for all bots to 10 seconds except Googlebot and Bing/Yahoo which can proceed at any pace?
I like being indexed by all these bots but with Yandex and Baidu it is just a bit overwhelming for my app which is very very database heavy.
Eventually I will cache stuff but right now i'm in beta. I do enjoy the traffic. Just need to slow these bots down a bit.
crawl-delayin robots.txt is non-standard and most bots don't support it. Yandex and Bing are about the only two that do support it. Googlebot does not. – Stephen Ostermiller Mar 14 '17 at 18:44