Scrapy 基本用法之安装、目录结构、分布式、代理、增量式

一、安装

Linux：

pip install scrapy

Windows:

第一步：pip install wheel

第二步：先进入网址：https://www.lfd.uci.edu/~gohlke/pythonlibs/#Twisted 下载对于Twisted 使用pip安装

第三步：pip install pywin32

第四步：pip install scrapy

二、目录结构

Scrapy主要包括了以下组件：

引擎(Scrapy)
用来处理整个系统的数据流处理, 触发事务(框架核心)
调度器(Scheduler)
用来接受引擎发过来的请求, 压入队列中, 并在引擎再次请求的时候返回. 可以想像成一个URL（抓取网页的网址或者说是链接）的优先队列, 由它来决定下一个要抓取的网址是什么, 同时去除重复的网址
下载器(Downloader)
用于下载网页内容, 并将网页内容返回给蜘蛛(Scrapy下载器是建立在twisted这个高效的异步模型上的)
爬虫(Spiders)
爬虫是主要干活的, 用于从特定的网页中提取自己需要的信息, 即所谓的实体(Item)。用户也可以从中提取出链接,让Scrapy继续抓取下一个页面
项目管道(Pipeline)
负责处理爬虫从网页中抽取的实体，主要的功能是持久化实体、验证实体的有效性、清除不需要的信息。当页面被爬虫解析后，将被发送到项目管道，并经过几个特定的次序处理数据。
下载器中间件(Downloader Middlewares)
位于Scrapy引擎和下载器之间的框架，主要是处理Scrapy引擎与下载器之间的请求及响应。
爬虫中间件(Spider Middlewares)
介于Scrapy引擎和爬虫之间的框架，主要工作是处理蜘蛛的响应输入和请求输出。
调度中间件(Scheduler Middewares)
介于Scrapy引擎和调度之间的中间件，从Scrapy引擎发送到调度的请求和响应。

Scrapy运行流程大概如下：

引擎从调度器中取出一个链接(URL)用于接下来的抓取
引擎把URL封装成一个请求(Request)传给下载器
下载器把资源下载下来，并封装成应答包(Response)
爬虫解析Response
解析出实体（Item）,则交给实体管道进行进一步的处理
解析出的是链接（URL）,则把URL交给调度器等待抓取

三、基本语法

1. scrapy startproject 项目名称

- 在当前目录中创建中创建一个项目文件（类似于Django）

2. 先 cd 到项目名称下再执行 scrapy genspider [-t template] <name> <domain>

- 创建爬虫应用

如：

scrapy genspider -t basic baidu baidu.com

scrapy genspider -t xmlfeed baidu baidu.com.cn

　创建crawlspider应用：scrapy genspider -t crawl baidu baidu.com.cn

PS:

查看所有命令：scrapy genspider -l

查看模板命令：scrapy genspider -d 模板名称

3. scrapy list

- 展示爬虫应用列表

4. scrapy crawl 爬虫应用名称 --nolog

- 运行单独爬虫应用

windows编码问题：

sys.stdout = io.TextIOWrapper(sys.stdout.buffer,encoding='db18030')

content = str(response.body,encoding='utf-8')

生成exe 这个与爬虫无关

pyinstaller -F -w --icon=start.ico .\test.py

三、Scrapy 发起 Post：使用scrapy.FormRequest方法发送

import scrapy


class D1Spider(scrapy.Spider):
    name = 'post1'
    allowed_domains = ['baidu.com']
    start_urls = ['https://fanyi.baidu.com/sug']  # 默认所有请求都是GET

    def start_requests(self):
        data = {
            'kw':'dog'
        }
        for url in self.start_urls:
            # yield scrapy.Request(url=url,callback=self.parse)   # GET请求
            yield scrapy.FormRequest(url=url,formdata=data,callback=self.parse)  # POST请求

    def parse(self, response):
        print(response.text)

四、Scrapy 使用下载中间件：DownloaderMiddleware

settings.py

DOWNLOADER_MIDDLEWARES = {
   'middlePro.middlewares.MiddleproDownloaderMiddleware': 543,
}

middlewares.py

class MiddleproDownloaderMiddleware(object):
    # Not all methods need to be defined. If a method is not defined,
    # scrapy acts as if the downloader middleware does not modify the
    # passed objects.

    user_agent_list = [
        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 "
        "(KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1",
        "Mozilla/5.0 (X11; CrOS i686 2268.111.0) AppleWebKit/536.11 "
        "(KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11",
        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.6 "
        "(KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6",
        "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.6 "
        "(KHTML, like Gecko) Chrome/20.0.1090.0 Safari/536.6",
        "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.1 "
        "(KHTML, like Gecko) Chrome/19.77.34.5 Safari/537.1",
        "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.5 "
        "(KHTML, like Gecko) Chrome/19.0.1084.9 Safari/536.5",
        "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 "
        "(KHTML, like Gecko) Chrome/19.0.1084.36 Safari/536.5",
        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 "
        "(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
        "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.3 "
        "(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_0) AppleWebKit/536.3 "
        "(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
        "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 "
        "(KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",
        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 "
        "(KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",
        "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 "
        "(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 "
        "(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
        "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.3 "
        "(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
        "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 "
        "(KHTML, like Gecko) Chrome/19.0.1061.0 Safari/536.3",
        "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.24 "
        "(KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24",
        "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 "
        "(KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24"
    ]

    # 可被选用的代理IP
    PROXY_http = [
        '153.180.102.104:80',
        '195.208.131.189:56055',
    ]
    PROXY_https = [
        '121.204.150.193:8118',
        '139.199.90.111:1080',
        '183.65.42.187:3128',
        '218.60.8.83:3129'
    ]

    # 拦截所有未发生异常的请求
    def process_request(self, request, spider):
        # 使用UA池进行代理伪装
        request.headers['User-Agent'] = random.choice(self.user_agent_list)
        print(request.headers['User-Agent'])

        # 使用代理池进行代理伪装
        # 对拦截到请求的url进行判断（协议头到底是http还是https）
        # request.url返回值：http://www.xxx.com
        h = request.url.split(':')[0]  # 请求的协议头
        if h == 'https':
            ip = random.choice(self.PROXY_https)
            request.meta['proxy'] = 'https://' + ip
        else:
            ip = random.choice(self.PROXY_http)
            request.meta['proxy'] = 'http://' + ip

        print(request.meta['proxy'])
        return None

    # 拦截所有响应
    def process_response(self, request, response, spider):
        # Called with the response returned from the downloader.

        # Must either;
        # - return a Response object
        # - return a Request object
        # - or raise IgnoreRequest
        return response

    # 拦截所有发生异常的请求
    def process_exception(self, request, exception, spider):
        # 使用代理池进行代理伪装
        # 对拦截到请求的url进行判断（协议头到底是http还是https）
        # request.url返回值：http://www.xxx.com
        print(request)
        h = request.url.split(':')[0]  # 请求的协议头
        if h == 'https':
            ip = random.choice(self.PROXY_https)
            request.meta['proxy'] = 'https://' + ip
        else:
            ip = random.choice(self.PROXY_http)
            request.meta['proxy'] = 'http://' + ip

五、Scrapy 使用meta参数在请求中传值

spider.py

# -*- coding: utf-8 -*-
import scrapy
from moviePro.items import MovieproItem

class MovieSpider(scrapy.Spider):
    name = 'movie'
    # allowed_domains = ['www.xxx.com']
    start_urls = ['https://www.4567tv.tv/frim/index1.html']

    # 解析详情页数据
    def parse_detail(self, response):
        item = response.meta['item']
        actor = response.xpath('/html/body/div[1]/div/div/div/div[2]/p[3]/a/text()').extract_first()
        item['actor'] =actor
        yield item

    def parse(self, response):
        li_list = response.xpath('//li[@class="col-md-6 col-sm-4 col-xs-3"]')
        for li in li_list:
            item = MovieproItem()
            name = li.xpath('./div/a/@title').extract_first()
            detail_url ='https://www.4567tv.tv/' + li.xpath('./div/a/@href').extract_first()

            item['name'] = name
            # 通过meta参数可以将参数传递
            yield scrapy.Request(url=detail_url,callback=self.parse_detail,meta={'item':item})

items.py

import scrapy


class MovieproItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    name = scrapy.Field()
    actor = scrapy.Field()

pipelines.py

class MovieproPipeline(object):
    def process_item(self, item, spider):
        print(item)
        return item

settings.py

BOT_NAME = 'moviePro'

SPIDER_MODULES = ['moviePro.spiders']
NEWSPIDER_MODULE = 'moviePro.spiders'


# Crawl responsibly by identifying yourself (and your website) on the user-agent
#请求头设置
USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36'

# Obey robots.txt rules
ROBOTSTXT_OBEY = False

ITEM_PIPELINES = {
   'moviePro.pipelines.MovieproPipeline': 300,
}


# 只输出错误
LOG_LEVEL = 'ERROR'
# 将所有日志保存再log.txt
LOG_FILE = './log.txt'

六、Scrapy 中使用 selenium

spider.py

# -*- coding: utf-8 -*-
import scrapy
from selenium import webdriver

class WangyiSpider(scrapy.Spider):
    name = 'wangyi'
    # allowed_domains = ['www.xxx.com']
    start_urls = ['https://war.163.com/']

    def __init__(self):
        self.bro = webdriver.Chrome(executable_path='chromedriver.exe')

    def parse(self, response):

        div_list = response.xpath('//div[@class="data_row news_article clearfix "]')
        for div in div_list:
            title = div.xpath('.//div[@class="news_title"]/h3/a/text()').extract_first()
            print(title)

    def closed(self, spider):
        print("关闭浏览器对象")
        self.bro.quit()

middlewares.py

# -*- coding: utf-8 -*-

# Define here the models for your spider middleware
#
# See documentation in:
# https://doc.scrapy.org/en/latest/topics/spider-middleware.html

from scrapy import signals
from scrapy.http import HtmlResponse
from time import sleep

class WangyiproDownloaderMiddleware(object):
    # Not all methods need to be defined. If a method is not defined,
    # scrapy acts as if the downloader middleware does not modify the
    # passed objects.

    def process_request(self, request, spider):
        # Called for each request that goes through the downloader
        # middleware.

        # Must either:
        # - return None: continue processing this request
        # - or return a Response object
        # - or return a Request object
        # - or raise IgnoreRequest: process_exception() methods of
        #   installed downloader middleware will be called
        return None

    def process_response(self, request, response, spider):
        print("即将返回一个新的响应对象！！！")
        bro = spider.bro
        bro.get(url=request.url)
        sleep(3)
        # 包含了动态加载的新闻数据
        page_text = bro.page_source
        sleep(2)
        return HtmlResponse(url=spider.bro.current_url,body=page_text,encoding='utf-8',request=request)

    def process_exception(self, request, exception, spider):
        # Called when a download handler or a process_request()
        # (from other downloader middleware) raises an exception.

        # Must either:
        # - return None: continue processing this exception
        # - return a Response object: stops process_exception() chain
        # - return a Request object: stops process_exception() chain
        pass

    def spider_opened(self, spider):
        spider.logger.info('Spider opened: %s' % spider.name)

settings.py

BOT_NAME = 'wangyiPro'

SPIDER_MODULES = ['wangyiPro.spiders']
NEWSPIDER_MODULE = 'wangyiPro.spiders'


# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'wangyiPro (+http://www.yourdomain.com)'
USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36'

# Obey robots.txt rules
ROBOTSTXT_OBEY = False

DOWNLOADER_MIDDLEWARES = {
   'wangyiPro.middlewares.WangyiproDownloaderMiddleware': 543,
}

七、Crawlspider 的基本使用:自带url去重和获取正则匹配的所有url递归

# -*- coding: utf-8 -*-
import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule


class BaiduSpider(CrawlSpider):
    name = 'couti'
    start_urls = ['https://dig.chouti.com/']

    rules = (
        Rule(LinkExtractor(allow=r'all/hot/recent/\d+'), callback='parse_item', follow=True),
    )

    def parse_item(self, response):
        i = {}
        div_list = response.xpath('//div[@class="news-content"]')
        for div in div_list:
            title = div.xpath('./div[@class="part1"]/a/text()').extract()
            print(title)
        return i

八、提高Scrapy的运行效率

增加并发：
默认scrapy开启并发线程32个，可以适当增加，settings配置文件中：CONCURRENT_REQUESTS = 32，并发设置成为了32

降低日志级别：
在运行scrapy时，会有大量的日志信息输出，为了减少CPU的使用率。可以将设置log输出信息为INFO或者ERROR。配置文件中设置：LOG_LEVEL = 'ERROR'

禁止cookie：
如果不是真的需要cookie，可以禁止cookie来减少CPU的使用率，提升爬取效率。在配置文件中设置：COOKIES_ENABLED = False

禁止重试：
对失败的HTTP进行重新请求（重试）会减慢爬取效率，因此禁止重试。在配置文件中编写：RETRY_ENABLED = False

减少下载超时：
如果对于一个非常慢的链接进行下载，可以减少超时时间的设置，让其超时链接丢失，从而提高效率。配置文件设置：DOWNLOAD_TIMEOUT = 10 设置超时时间为10s

九、分布式Scrapy

安装 pip install scrapy-redis

引入from scrapy_redis.spiders import RedisCrawlSpider包

继承 RedisCrawlSpider

设置redis key：redis_key = "couti:start_urls"

配置settings文件，需要重新定义scrapy的url调度器和数据管道，使用redis队列实现

# -*- coding: utf-8 -*-
import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
from scrapy_redis.spiders import RedisCrawlSpider
from coutiPro.items import CoutiproItem

class CoutiSpider(RedisCrawlSpider):
    name = 'couti'
    # allowed_domains = ['www.xxx.com']
    # start_urls = ['http://www.xxx.com/']

    # 启动爬虫的命令
    redis_key = "couti:start_urls"

    rules = (
        Rule(LinkExtractor(allow=r'all/hot/recent/\d+'), callback='parse_item', follow=True),
    )

    def parse_item(self, response):
        div_list = response.xpath('//div[@class="news-content"]')
        for div in div_list:
            title = div.xpath('./div[@class="part1"]/a/text()').extract_first()
            print(title)
            item = CoutiproItem()
            item['title'] = title

            yield item

spider.py

BOT_NAME = 'coutiPro'

SPIDER_MODULES = ['coutiPro.spiders']
NEWSPIDER_MODULE = 'coutiPro.spiders'


# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'coutiPro (+http://www.yourdomain.com)'
USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36'

# Obey robots.txt rules
ROBOTSTXT_OBEY = False

# 使用scrapy-redis里的去重组件，不使用scrapy默认的去重方式
DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter"
# 使用scrapy-redis里的调度器组件，不使用默认的调度器
SCHEDULER = "scrapy_redis.scheduler.Scheduler"
# 允许暂停，redis请求记录不丢失
SCHEDULER_PERSIST = True
# 默认的scrapy-redis请求队列形式（按优先级）
SCHEDULER_QUEUE_CLASS = "scrapy_redis.queue.SpiderPriorityQueue"
# 队列形式，请求先进先出
#SCHEDULER_QUEUE_CLASS = "scrapy_redis.queue.SpiderQueue"
# 栈形式，请求先进后出
#SCHEDULER_QUEUE_CLASS = "scrapy_redis.queue.SpiderStack"

# 只是将数据放到redis数据库，不需要写pipelines文件
ITEM_PIPELINES = {
    'scrapy_redis.pipelines.RedisPipeline': 400,
}

LOG_LEVEL = 'ERROR'

# Introduce an artifical delay to make use of parallelism. to speed up the
# crawl.
DOWNLOAD_DELAY = 1
# 指定数据库的主机IP
REDIS_HOST = "127.0.0.1"
# 指定数据库的端口号
REDIS_PORT = 6379
REDIS_ENCODING = 'utf-8'
# REDIS_PARAMS = {'password':'123456'}

settings.py

redis列表操作补充

windows启动redis
redis-cli.exe

查看所有key
keys * 

加入列表
lpush [key] [value]

清空所有
flushall

获取列表所有元素
lrange [key] 0 -1

十、增量式Scrapy

如何进行增量式爬取工作：
-- 在发送请求之前判断这个URL之前是不是爬取过
-- 在解析内容之后判断该内容之前是否爬取过
-- 在写入存储介质时判断内容是不是在该介质中

示例一：url检验的方式

spiders.py

# -*- coding: utf-8 -*-
import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
from redis import Redis
from IncrementalspiderPro.items import IncrementalspiderproItem

class ExampleSpider(CrawlSpider):
    name = 'movie'
    start_urls = ['https://www.4567tv.tv/index.php/vod/show/id/7.html']

    rules = (
        Rule(LinkExtractor(allow=r'/index.php/vod/show/id/7/page/\d+\.html'), callback='parse_item', follow=True),
    )

    def parse_item(self, response):
        conn = Redis(host='127.0.0.1', port=6379)
        detail_url_list = response.xpath('//li[@class="col-md-6 col-sm-4 col-xs-3"]/div/a/@href').extract()
        for url in detail_url_list:
            url = 'https://www.4567tv.tv'+url
            ex = conn.sadd('urls', url)
            # 等于1 的时候 说明数据还没有存储到redis中  等于0 的时候 说明redis中已经存在该数据
            if ex == 1:
                yield scrapy.Request(url=url, callback=self.parse_detail)
            else:
                print("网站中无数据更新，没有可爬取得数据！！！")

    def parse_detail(self, response):
        item = IncrementalspiderproItem()
        item['name'] = response.xpath('/html/body/div[1]/div/div/div/div[2]/h1/text()').extract_first()
        item['actor'] = response.xpath('/html/body/div[1]/div/div/div/div[2]/p[3]/a/text()').extract_first()

        if item['name']:
            item['name'] = item['name']
        else:
            item['name'] = '无'

        if item['actor']:
            item['actor'] = item['actor']
        else:
            item['actor'] = '无'

        yield item

items.py

# -*- coding: utf-8 -*-

# Define here the models for your scraped items
#
# See documentation in:
# https://doc.scrapy.org/en/latest/topics/items.html

import scrapy


class IncrementalspiderproItem(scrapy.Item):
    # define the fields for your item here like:
    name = scrapy.Field()
    actor = scrapy.Field()

pipelines.py

# -*- coding: utf-8 -*-

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://doc.scrapy.org/en/latest/topics/item-pipeline.html

import redis,json
class IncrementalspiderproPipeline(object):
    def __init__(self):
        self.conn = None

    def open_spider(self, spider):
        pool = redis.ConnectionPool(host='127.0.0.1', port=6379)
        self.conn = redis.Redis(connection_pool=pool)

    def process_item(self, item, spider):
        print('有新的数据正在入库')
        self.conn.lpush('data', str(item))
        return item

settings.py

BOT_NAME = 'IncrementalspiderPro'

SPIDER_MODULES = ['IncrementalspiderPro.spiders']
NEWSPIDER_MODULE = 'IncrementalspiderPro.spiders'


# Crawl responsibly by identifying yourself (and your website) on the user-agent
USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36'

# Obey robots.txt rules
ROBOTSTXT_OBEY = False
ITEM_PIPELINES = {
   'IncrementalspiderPro.pipelines.IncrementalspiderproPipeline': 300,
}

如何查看redis里的data数据：创建s1.py

import redis,json

pool = redis.ConnectionPool(host='127.0.0.1', port=6379)

r = redis.Redis(connection_pool=pool)

data = r.lrange('data',0,-1)
for i in data:
    # 对redis里面的数据进行字节转字符串，再将单引号替换成双引号
    item = i.decode('utf-8').replace("'", "\"")
    try:
        print(json.loads(item))
    except Exception as e:
        print(e)
        print(item)

示例二：数据指纹检验的方式

spiders.py

# -*- coding: utf-8 -*-
import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
from IncrementalspiderPro2.items import Incrementalspiderpro2Item
from redis import Redis
import hashlib

class QiubaiSpider(CrawlSpider):
    name = 'qiubai'
    start_urls = ['https://www.qiushibaike.com/text/']

    rules = (
        Rule(LinkExtractor(allow=r'/text/page/\d+/'), callback='parse_item', follow=True),
    )

    def parse_item(self, response):
        div_list = response.xpath('//div[@class="article block untagged mb15 typs_hot"]')
        conn = Redis(host='127.0.0.1', port=6379)
        for div in div_list:
            item = Incrementalspiderpro2Item()
            item['content'] = div.xpath('.//div[@class="content"]/span//text()').extract()
            item['content'] = ''.join(item['content'])

            item['author'] = div.xpath('./div/a[2]/h2/text() | ./div[1]/span[2]/h2/text()').extract_first()

            sourse = item['content'] + item['author']
            # 自己定制一种形式得数据指纹
            hashvalue = hashlib.sha256(sourse.encode()).hexdigest()

            ex = conn.sadd('qiubai_hash', hashvalue)
            if ex == 1:
                yield item
            else:
                print('没有可更新的数据可爬取')

items.py

import scrapy


class Incrementalspiderpro2Item(scrapy.Item):
    # define the fields for your item here like:
    author = scrapy.Field()
    content = scrapy.Field()

pipelines.py

import redis
class Incrementalspiderpro2Pipeline(object):
    def __init__(self):
        self.conn = None

    def open_spider(self, spider):
        pool = redis.ConnectionPool(host='127.0.0.1', port=6379)
        self.conn = redis.Redis(connection_pool=pool)

    def process_item(self, item, spider):
        print(item)
        print('爬取到一条数据,正在入库......')
        self.conn.lpush('data', str(item))
        return item

settings.py

BOT_NAME = 'IncrementalspiderPro2'

SPIDER_MODULES = ['IncrementalspiderPro2.spiders']
NEWSPIDER_MODULE = 'IncrementalspiderPro2.spiders'


# Crawl responsibly by identifying yourself (and your website) on the user-agent
USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36'

# Obey robots.txt rules
ROBOTSTXT_OBEY = False

ITEM_PIPELINES = {
   'IncrementalspiderPro2.pipelines.Incrementalspiderpro2Pipeline': 300,
}

查询redis中的数据：新建文件s1.py

import redis

pool = redis.ConnectionPool(host='127.0.0.1', port=6379)

r = redis.Redis(connection_pool=pool)

data = r.lrange('data',0,-1)
for i in data:
    # 对redis里面的数据进行字节转字符串，再将单引号替换成双引号
    item = i.decode('utf-8')
    print(item)

posted @ 2019-03-29 18:00 我在地球凑人数的日子阅读(561) 评论(0) 收藏举报

刷新页面返回顶部

鲁迅说

时间就像海绵里的水，只要愿挤，总还是有的！

Scrapy 基本用法之安装、目录结构、分布式、代理、增量式

一、安装

二、目录结构

三、基本语法

三、Scrapy 发起 Post：使用scrapy.FormRequest方法发送

四、Scrapy 使用下载中间件：DownloaderMiddleware

settings.py

middlewares.py

五、Scrapy 使用meta参数在请求中传值

spider.py

items.py

pipelines.py

settings.py

六、Scrapy 中使用 selenium

spider.py

settings.py

七、Crawlspider 的基本使用:自带url去重和获取正则匹配的所有url递归

八、提高Scrapy的运行效率

九、分布式Scrapy

redis列表操作补充

十、增量式Scrapy

示例一：url检验的方式

示例二：数据指纹检验的方式