robots.txt - just a guess about wild-card

Question

so if I disallow tempPage, does it mean tempPage_1, temp_Page_2, tempPage_x are also disallowed? I tried to google this up, but I don't know...

score 3 · Accepted Answer · answered Nov 19 '10 at 20:06

3

1) If tempPage_1, temp_Page_2, tempPage_x are subdirectories of tempPage then, yes, they are blocked.

2) If they are parallel directories then, no, they are not blocked.

3) If they are separate files then, no, they are not blocked.

Assuming #1 is not true, to block them all you would use:

disallow tempPage*

answered Nov 19 '10 at 20:06

John Conde

86,255
27
146
241

Pretty sure this is incorrect... – DisgruntledGoat Nov 20 '10 at 00:33
what is incorrect? – TPR Nov 20 '10 at 00:36
1

Google, Yahoo, and Microsoft honor wildcards in the disallow statement – John Conde Nov 20 '10 at 00:47

score 0 · Answer 2 · answered Nov 20 '10 at 00:32

Yes those pages would be blocked, assuming they are at the same directory level. There is no * wildcard in robots.txt, except a single * for the User agent.

In other words this...

User-agent: *
Disallow: /tempPage

...would block yoursite.com/tempPage, yoursite.com/tempPage_1, yoursite.com/temp_Page_2, yoursite.com/tempPage_x.

robots.txt - just a guess about wild-card

2 Answers2