Will this robots.txt file work?

Question

User-agent: *
Allow: /sitemapindex.xml
Allow: /post_google_news.xml
Allow: /news/
Allow: /category/news/
Disallow: /

I want to exclude everything from being indexed by Google except for example.com/sitemapindex.xml, example.com/post_google_news.xml, example.com/news/anything and example.com/category/news/anything/anything etc.

If you want to test a specific robots.txt file, you should probably just use Google's tool for doing so: https://support.google.com/webmasters/answer/6062598?hl=en — Stephen Ostermiller, Aug 31 '19 at 00:04
You should also know that robots.txt can't prevent indexing, it only prevents crawling. Google sometimes chooses to index pages even if Googlebot can't crawl them. See Why do Google search results include pages disallowed in robots.txt? — Stephen Ostermiller, Aug 31 '19 at 00:07

score 1 · Accepted Answer · answered Aug 26 '19 at 04:46

Yes, this robots.txt file will work. Only comment here is, your /news/ and /category/news/ seem like they might be two different file paths to the same content? If so, I'm assuming you've already selected your canonical URL's and noindexed anything that shouldn't show up in search results using meta tags; in that case, feel free to include only the file path you want to appear in the SERP's, and take out the other one.

Will this robots.txt file work?

1 Answers1