0

I'm trying to scrape JSON response from an API call on a website using scrapy. And to get that data, I'm using 'FormRequest' inside a for loop. The only thing that changes in the for loop is the 'URL'. Other things like headers and formdata remain the same for every call. The request runs successfully when I run it for a single URL. Although, when I try to run it from within the loop, it doesn't give any response. It does not create a response even for the first request. I've tried solutions from these answers on stackoverflow:

  1. Loop on scrapy FormRequest but only one item created
  2. Scrapy Crawl URLs in Order

The function looks like this:

def parse_after_login(self,response):
        print("Logged in")
        for i in self.ids:
            time.sleep(10)
            yield scrapy.FormRequest(
                self.patient_documents_url % (i),
                self.parse_patient_documents_POST,
                dont_filter = True,
                formdata={'file_action': 'load_docs'},
                headers={
                    'content-type': 'application/x-www-form-urlencoded; charset=UTF-8',
                    'newrelic': '',
                    'x-csrftoken': '',
                    'x-newrelic-id': '',
                    'x-requested-with': ''
                },
            )
def parse_patient_documents_POST(self, response):
        #print("\n\nData: ", response.text)
        print("\n\nData: ", response.text, "\n\n")

I'm afraid I cannot post the whole code. ANy help is appreciated!

Vishal A.
  • 1,234
  • 7
  • 17

0 Answers0