scrapy-例：douyu图片下载案例

一. 新建项目(scrapy startproject)

scrapy startproject douyuSpider

 

二、明确目标(douyuSpider/items.py)

import scrapy

class DouyuspiderItem(scrapy.Item):
    # define the fields for your item here like:
    room_name = scrapy.Field()
    imagelink=scrapy.Field()
    imagepath=scrapy.Field()

 

三、制作爬虫 （spiders/douyu.py）

　1、scrapy genspider douyu "capi.douyucdn.cn"

scrapy shell调试

 

   2、打开 tencentSpider/spider目录里的 douyu.py，代码如下：

# -*- coding: utf-8 -*-
import scrapy
import json
from ..items import DouyuspiderItem

class DouyuSpider(scrapy.Spider):
    name = 'douyu'
    allowed_domains = ['capi.douyucdn.cn']

    url="http://capi.douyucdn.cn/api/v1/getVerticalRoom?limit=20&offset="
    offset=0

    start_urls = [url+str(offset)]

    def parse(self, response):
        # 把json格式的数据转换为python格式，data段是列表
        data=json.loads(response.text)['data']
        for each in data:
            item=DouyuspiderItem()
            item['room_name']=each['room_name']
            item['imagelink']=each['vertical_src']

            yield item

        if self.offset<100:
            self.offset+=20

        yield scrapy.Request(self.url+str(self.offset),callback=self.parse)

 

四、存储内容 （pipelines.py)

修改settings.py以下几个地方：

ROBOTSTXT_OBEY = False

DEFAULT_REQUEST_HEADERS = {
    "User-Agent" : "DYZB/1 CFNetwork/808.2.16 Darwin/16.3.0"
}

ITEM_PIPELINES = {
   'douyuSpider.pipelines.ImagesPipeline': 300,
}

IMAGES_STORE=r"E:\python_practice_ku\pachong\douyuSpider\douyuSpider\images"

编写pipelines.py文件（图片下载可参考：https://blog.csdn.net/kuangshp128/article/details/80321099）

from scrapy.utils.project import get_project_settings
from scrapy.pipelines.images import ImagesPipeline
import scrapy
import os

#下载图片可以使用该方法，函数名及参数都是固定写法
class ImagesPipeline(ImagesPipeline):
    # 获取settings文件里设置的变量值
    IMAGES_STORE=get_project_settings().get('IMAGES_STORE')

    def get_media_requests(self,item,info):
        image_url=item['imagelink']   #获取图片链接
        yield scrapy.Request(image_url)   #再次请求图片链接

    def item_completed(self,result,item,info):
        #result返回的是一个列表，例：[(True, {'url': 'https://p0.ssl.qhimgs1.com/t01a098025e4214bacc.jpg', 'path': 't01a098025e4214bacc.jpg', 'checksum': '7adb29c836cde7a422c740aac3f86234'})]
        image_path=[x["path"] for ok,x in result if ok]
        #对图片重命名
        os.rename(self.IMAGES_STORE+'/'+image_path[0],self.IMAGES_STORE+'/'+item['room_name']+'.jpg')
        item['imagepath']=self.IMAGES_STORE+'/'+item['room_name']
        return item

命令执行：