scrapy框架使用imgpipeline

1.可以在原来的items的基础上添加一个新的item类型

class imgItem(scrapy.Item):
    # define the fields for your item here like:
    title = scrapy.Field()
    link = scrapy.Field()

2.去pipeLine文件内新建一个专属img的管道

class imgsPipeLine(ImagesPipeline):

    # 就是可以根据图片地址进行图片数据的请求
    def get_media_requests(self, item, info):
        if isinstance(item, imgItem):  # 判断item是否为imgItem类型
            yield scrapy.Request(url=item['link'])

    # 指定图片存储的路径(这里只返回了图片的名字，图片的存储路径需要去配置)
    def file_path(self, request, response=None, info=None, *, item=None):
        imgName = item['title'] + '.jpg'
        return imgName

    def item_completed(self, results, item, info):
        return item  # 返回给下一个即将被执行的管道类

3.给其他的管道类添加上各自处理的item类型判断

4.在spider中写上各自item的内容，然后yield

item_1 = imgItem()
            if movie_name != None:
                item_1['title'] = movie_name
            else:
                item_1['title'] = ''
            if li.xpath('.//div[@class="pic"]/a/img/@src').extract_first() != None:
                item_1['link'] = li.xpath('.//div[@class="pic"]/a/img/@src').extract_first()
            else:
                item_1['link'] = ''
            yield item_1

5.还需要开启setting中的pipeline和img_store

IMAGES_STORE = './imgs'

ITEM_PIPELINES = {
    'douban250.pipelines.Douban250Pipeline': 300,
    'douban250.pipelines.SQLPipeline': 301,
    'douban250.pipelines.imgsPipeLine': 299,
}

6.执行即可