How to retry for 404 link not found in scrapy?

Asked Jan 07 '12 at 09:13

Active Jan 07 '12 at 12:03

Viewed 3,713 times

10

Some sites give 404 error temporarily. But I paste in on browser it works. How to tell scrapy to retry the 404 status code links for 5 times.

asked Jan 07 '12 at 09:13

nizam.sp

3,892
5
37
60

Maybe you are making too many requests using scrapy and sites just returninig 404? – adt Jan 07 '12 at 09:15
Even if I try 1 requests per second, it fails sometimes. Is there a way to say in scrapy to retry 404 status? – nizam.sp Jan 07 '12 at 09:17

1 Answers1

18

There are two Scrapy settings relevant to what you need:

RETRY_HTTP_CODES: you should override the default value in your project to include 404
RETRY_TIMES: just set it to 5

answered Jan 07 '12 at 12:03

Pablo Hoffman

1,508
13
19