1

I have been using the method described on stackoverflow (https://stackoverflow.com/a/43661172/5037146) , to make scrapy run from script using Crawler Runner to allow to restart the process.

However, I don't get any console logs when running the process through CrawlerRunner, whereas when I using CrawlerProcess, it outputs the status and progress.

Code is available online: https://colab.research.google.com/drive/14hKTjvWWrP--h_yRqUrtxy6aa4jG18nJ

Aerodynamic
  • 496
  • 3
  • 12

1 Answers1

1

With CrawlerRunner you need to manually setup logging, which you can do with configure_logging(). See https://docs.scrapy.org/en/latest/topics/practices.html#run-scrapy-from-a-script

Gallaecio
  • 3,329
  • 2
  • 23
  • 62
  • Thanks man, it works ! I also found out that using CrawlerProcess instead or CrawlerRunner automatically yields logs. – Aerodynamic Sep 04 '19 at 08:27