1

I need to scrape a website with Scrapy that uses AJAX Pagination. How can I be sure that the page is fully loaded ?

I used: 

Download_Delay = 2

I need something like document.ready in js.

RattleyCooper
  • 4,737
  • 5
  • 23
  • 40
parik
  • 2,160
  • 12
  • 37
  • 64
  • scrapy doesn't have that, try with selenium or [Splash](https://github.com/scrapinghub/splash). – eLRuLL May 04 '16 at 14:49
  • you could still check which specific request was done to get the information you need and replicate that with scrapy. To debug requests I would recommend [Firebug](http://getfirebug.com/) for firefox. – eLRuLL May 04 '16 at 14:51
  • On the Splash note I recommend this library [ScrapyJS](https://github.com/scrapy-plugins/scrapy-splash) – Rafael Almeida May 04 '16 at 16:04
  • @RafaelAlmeida on Scrapy we can use ScrapyJs too, but i send my requests by FormRequest and i don't use scrapyJS for clicking – parik May 04 '16 at 16:11
  • 1
    FormRequest doesn't use any javascript, it only makes the post or get request on a form with the data provided. ScrapyJS loads the javascript before returning the page source which is precisely what you want – Rafael Almeida May 04 '16 at 16:14
  • 1
    @parik the great mighty Alecxe explains best. Check out his answers http://stackoverflow.com/questions/17975471/selenium-with-scrapy-for-dynamic-page/17979285#17979285 http://stackoverflow.com/questions/30345623/scraping-dynamic-content-using-python-scrapy – Rafael Almeida May 04 '16 at 16:18

0 Answers0