How to send post data in start_urls of the scrapy spider

Question

I want to crawl a website which supports only post data. I want to send the query params in post data in all the requests. How to achieve this?

score 15 · Accepted Answer · edited Oct 12 '14 at 14:04

15

POST requests can be made using scrapy's Request or FormRequest classes.

Also, consider using start_requests() method instead of start_urls property.

Example:

from scrapy.http import FormRequest

class myspiderSpider(Spider):
    name = "myspider"
    allowed_domains = ["www.example.com"]

    def start_requests(self):
        return [ FormRequest("http://www.example.com/login",
                     formdata={'someparam': 'foo', 'otherparam': 'bar'},
                     callback=self.parse) ]

Hope that helps.

edited Oct 12 '14 at 14:04

KrisWebDev

9,052
4
37
57

answered Jul 12 '13 at 22:46

alecxe

441,113
110
1,021
1,148

1

But which parameter of `Request` or `FormRequest` should I past the post data to? `body` or `formdata`? – Friedmannn Mar 20 '14 at 03:42
You should use the `formdata` dict for the post data. – lgaggini May 07 '14 at 14:50
How do you do it with a list of urls? – CodeGuru Sep 17 '17 at 13:34

How to send post data in start_urls of the scrapy spider

1 Answers1

Linked