Scrapy throws error ReactorNotRestartable when runnning on AWS Lambda

Question

I have deployed a scrapy project which crawls whenever an lambda api requests comes.

It runs perfectly for the first api call but later on it fails and throws ReactorNotRestartable error.

As far as I can understand the AWS Lambda ecosystem is not killing the process, hence reactor is still present in the memory.

The lambda log error is as follows:

Traceback (most recent call last):
File "/var/task/aws-lambda.py", line 42, in run_company_details_scrapy
process.start()
File "./lib/scrapy/crawler.py", line 280, in start
reactor.run(installSignalHandlers=False)  # blocking call
File "./lib/twisted/internet/base.py", line 1242, in run
self.startRunning(installSignalHandlers=installSignalHandlers)
File "./lib/twisted/internet/base.py", line 1222, in startRunning
ReactorBase.startRunning(self)
File "./lib/twisted/internet/base.py", line 730, in startRunning
raise error.ReactorNotRestartable()
ReactorNotRestartable

The lambda handler function is:

def run_company_details_scrapy(event, context):
   process = CrawlerProcess()
   process.crawl(CompanyDetailsSpidySpider)
   process.start()

I had a workaround by not stopping the reactor by inserting a flag in the start function

process.start(stop_after_crawl=False)

But the problem with this was that I had to wait until the lambda call timed out.

Tried other solutions, but none of them seems to work.Can anyone guide me how to solve this problem.

Huh. Lambda re-uses your Python process to handle multiple events? And your handler has to complete synchronously? — Jean-Paul Calderone, Feb 22 '17 at 16:37
@firefox Since you marked the question as solved. Can you describe how you used crochet to solve your issue ? — Hugo, Apr 14 '17 at 07:08
@firefox Im having a hard time trying to run scrapy in an aws lambda.. how did you create your zip file? I'm having an `ImportError: cannot import name 'etree'` — Emannuel Carvalho, May 31 '17 at 21:35

score 4 · Answer 1 · answered May 17 '18 at 16:48

Had the same problem recently, and Crochet didn't want to work for various reasons.

Eventually we went for the dirty solution: just call sys.exit(0) (or sys.exit(1) if an error was caught, not that anything looks at the return code AFAICT) at the end of the lambda handler function. This worked perfectly.

Obviously no good if you're intending to return a response from your Lambda, but if you're using Scrapy, data's probably being persisted already via your Pipelines, with a scheduler as the trigger for your Lambda, so no response needed.

Note: you will get a notice from AWS in CloudWatch:

RequestId: xxxx Process exited before completing request

Kishan Patel · Answer 2 · 2021-08-06T10:14:20.930

I faced error ReactorNotRestartable on AWS lambda and after I came to this solution

By default, the asynchronous nature of scrapy is not going to work well with Cloud Functions, as we'd need a way to block on the crawl to prevent the function from returning early and the instance being killed before the process terminates.

Instead, we can use scrapydo to run your existing spider in a blocking fashion:



import scrapy
import scrapy.crawler as crawler
rom scrapy.spiders import CrawlSpider
import scrapydo

scrapydo.setup()

# your spider
class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = ['http://quotes.toscrape.com/tag/humor/']

    def parse(self, response):
        for quote in response.css('div.quote'):
            print(quote.css('span.text::text').extract_first())

scrapydo.run_spider(QuotesSpider)

score 1 · Accepted Answer · answered Feb 22 '17 at 16:42

1

You could try using https://pypi.python.org/pypi/crochet to coordinate use of a reactor running in a non-main thread from the Lambda-controlled main thread.

Crochet will do the threaded reactor initialization for you and provides tools to make it easy to call code in the reactor thread from the main (and get the results).

This might be more in line with the expectations Lambda has of your code.

answered Feb 22 '17 at 16:42

Jean-Paul Calderone

46,779
5
89
118

2

Does anyone have an example on how this is done with Scrapy? – Jamie Jan 31 '20 at 11:43
@Jamie This answer worked perfectly for me: https://stackoverflow.com/a/57347964/308204 – LaundroMat Feb 22 '20 at 12:21

score 1 · Answer 4 · edited May 23 '17 at 12:02

1

This problem isn't unique to AWS Lambda - see running a spider in a Celery task.

You might try ScrapyScript (disclosure: I wrote it). It spawns a subprocess to support the Twisted reactor, blocks until all of the supplied spiders have finished, and then exits. It was written with Celery in mind, but the use case is similar.

In your case, this should work:

from scrapyscript import Job, Processor
def run_company_details_scrapy(event, context):
    job = Job(CompanyDetailsSpidySpider())
    Processor().run(job)`

edited May 23 '17 at 12:02

Community

1
1

answered Feb 23 '17 at 02:15

jschnurr

1,111
6
8

Careless combination of Twisted and multi-processing results in bizarre hard-to-debug errors. For example - http://stackoverflow.com/questions/42347121/why-i-am-getting-sigchldwaker-object-has-no-attribute-dowrite-in-scrapy - does ScrapyScript address such things? – Jean-Paul Calderone Feb 23 '17 at 13:07
@jschnurr I have a different error using scrapy on lambda "module initialization error: 'twisted.internet.reactor'" I tried scrapyscript without success. Do you have any thoughts on the subject? – Hugo Apr 14 '17 at 08:22

Scrapy throws error ReactorNotRestartable when runnning on AWS Lambda

4 Answers4

Linked

Related