Scrapy框架爬取图片

一、任务

爬取该网站（https://desk.zol.com.cn/bizhi/9506_115438_2.html）壁纸，并保存

二、项目代码

1.创建项目

 scrapy startproject zol 

2.修改配置信息：

　　USER_AGENT

　　ROBOTSTXT_OBEY 改成 False

 

　　开启ITEM_PIPLINES

　　设置图片保存位置：

IMAGES_STORE = "d:/pics"

 

3.爬虫文件 -- zol.py

 

scrapy genspider zol zol.com.cn

 

import scrapy


class ZolSpider(scrapy.Spider):
    name = 'zol'
    allowed_domains = ['zol.com.cn']
    start_urls = ['http://desk.zol.com.cn/bizhi/9506_115438_2.html']

    def parse(self, response):
        image_url = response.xpath('//img[@id="bigImg"]/@src').extract_first()
        image_name = response.xpath('string(//h3)').extract_first()
        yield {
            # 'image_urls': [image_url],
            'image_url': image_url,
            'image_name': image_name
        }
        next_url = response.xpath('//a[@id="pageNext"]/@href').extract_first()
        yield scrapy.Request(response.urljoin(next_url), callback=self.parse)

 

4.pipelines.py

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html


# useful for handling different item types with a single interface
import scrapy
from itemadapter import ItemAdapter
from scrapy.pipelines.images import ImagesPipeline
from scrapy import Request

class ImagePipeline(ImagesPipeline):
    def get_media_requests(self, item, info):
        yield Request(item['image_url'],meta={'name': item['image_name']})

    def file_path(self, request, response=None, info=None):
        name = request.meta['name'].strip().replace('\r\n\t\t', '')
        name = name.replace('/', '_')
        return name + '.jpg'

三、结果