发起post请求

scrapy发起post请求的两种方式

一定要对start_requests方法进行重写

　1.Request方法中给method属性复制成post

　2.FormRequest()进行post请求的发送

 

方法1(不建议使用):

　

import scrapy


class PostexampleSpider(scrapy.Spider):
    name = 'postexample'
    # allowed_domains = ['www.baidu.com']
    start_urls = ['https://www.baidu.com/']

    def start_requests(self):
     
        for url in self.start_urls:
            yield scrapy.Request(url=url, callback=self.parse,  method="post")

    def parse(self, response):
        pass

 

方法2（推荐用scrapy.FormRequest）：

import scrapy


class PostexampleSpider(scrapy.Spider):
    name = 'postexample'
    # allowed_domains = ['www.baidu.com']
    start_urls = ['https://fanyi.baidu.com/sug']

    def start_requests(self):
        print("start_requests()")
        # post请求参数
        data = {
            "kw": "dog"
        }
        for url in self.start_urls:
            # formdata 请求参数对应的字典参数
            yield scrapy.FormRequest(url=url, formdata=data, callback=self.parse)

    def parse(self, response):
        print(response.text)