Scrapy框架

• Scrapy简介
• 环境安装
• 基础命令
• settings配置文件

 

Scrapy简介

Scrapy是一个为了爬取网站数据，提取结构性数据而编写的应用框架，
非常出名，非常强悍。所谓的框架就是一个已经被集成了各种功能（高性能异步下载，队列，分布式，解析，持久化等）的具有很强通用性的项目模板。
对于框架的学习，重点是要学习其框架的特性、各个功能的用法即可。

 

环境安装

- Linux：

　　- 直接 pip install scrapy 即可

　

- Windows：

- 先下载 wheel
    pip3 install wheel

- 再下载twisted  进入官网找到 对应的python解释器版本的文件
    下载地址：http://www.lfd.uci.edu/~gohlke/pythonlibs/#twisted

- 找到 twisted 文件的下载目录执行命令：
    pip3 install Twisted‑17.1.0‑cp35‑cp35m‑win_amd64.whl(下载文件名的全称)

- 然后命令：
    pip install pywin32
    pip install scrapy

 

基础命令

- 创建项目：

　　scrapy startproject 项目名称

 

- 项目目录结构：

项目名称/
   scrapy.cfg：
   project_name/
       __init__.py
       items.py
       pipelines.py
       settings.py
       spiders/
           __init__.py


scrapy.cfg   项目的主配置信息。（真正爬虫相关的配置信息在settings.py文件中）
items.py     设置数据存储模板，用于结构化数据，如：Django的Model
pipelines    数据持久化处理
settings.py  配置文件，如：递归的层数、并发数，延迟下载等
spiders      爬虫目录，如：创建文件，编写爬虫解析规则

 

- 在项目中创建爬虫程序：

　　- 进入项目目录： cd 项目名称

　　- 创建命令：

　　　　scrapy genspider 应用名称 爬取网页的起始url （例如：scrapy genspider qiubai www.qiushibaike.com）

 

　　- 命令执行结束后 会在 spiders 文件中生成一个 上面  应用名称.py 的文件； 源码如下：

# -*- coding: utf-8 -*-
import scrapy

class QiubaiSpider(scrapy.Spider):
    name = 'qiubai' #应用名称

    #允许爬取的域名（如果遇到非该域名的url则爬取不到数据）
    allowed_domains = ['https://www.qiushibaike.com/']
    
    #起始爬取的url
    start_urls = ['https://www.qiushibaike.com/']

     #访问起始URL并获取结果后的回调函数，该函数的response参数就是向起始的url发送请求后，获取的响应对象.该函数返回值必须为可迭代对象或者NUll 
     def parse(self, response):
        print(response.text) #获取字符串类型的响应内容
        print(response.body)#获取字节类型的相应内容

 

- 执行爬虫应用：

　　scrapy crawl  应用名称

　　ps：scrapy crawl 爬虫名称 --nolog：该种执行形式不会显示执行的日志信息

 

settings配置文件

- 源码文件：

# -*- coding: utf-8 -*-

# Scrapy settings for scrapyproDemo project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
#     https://doc.scrapy.org/en/latest/topics/settings.html
#     https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
#     https://doc.scrapy.org/en/latest/topics/spider-middleware.html

BOT_NAME = 'scrapyproDemo'

SPIDER_MODULES = ['scrapyproDemo.spiders']
NEWSPIDER_MODULE = 'scrapyproDemo.spiders'


# Crawl responsibly by identifying yourself (and your website) on the user-agent
# 通过在用户代理上标识您自己(和您的网站)，负责任地爬行
#USER_AGENT = 'scrapyproDemo (+http://www.yourdomain.com)'

# Obey robots.txt rules
# 是否遵循 robots协议，默认为True 但是一般都要改成False
ROBOTSTXT_OBEY = True

# Configure maximum concurrent requests performed by Scrapy (default: 16)
# 配置由Scrapy执行的最大并发请求(默认:16)
#CONCURRENT_REQUESTS = 32

# Configure a delay for requests for the same website (default: 0)
# See https://doc.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
#DOWNLOAD_DELAY = 3
# The download delay setting will honor only one of:
#CONCURRENT_REQUESTS_PER_DOMAIN = 16
#CONCURRENT_REQUESTS_PER_IP = 16

# Disable cookies (enabled by default)
#COOKIES_ENABLED = False

# Disable Telnet Console (enabled by default)
#TELNETCONSOLE_ENABLED = False

# Override the default request headers:
#DEFAULT_REQUEST_HEADERS = {
#   'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
#   'Accept-Language': 'en',
#}

# Enable or disable spider middlewares
# See https://doc.scrapy.org/en/latest/topics/spider-middleware.html
#SPIDER_MIDDLEWARES = {
#    'scrapyproDemo.middlewares.ScrapyprodemoSpiderMiddleware': 543,
#}

# Enable or disable downloader middlewares
# See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
#DOWNLOADER_MIDDLEWARES = {
#    'scrapyproDemo.middlewares.ScrapyprodemoDownloaderMiddleware': 543,
#}

# Enable or disable extensions
# See https://doc.scrapy.org/en/latest/topics/extensions.html
#EXTENSIONS = {
#    'scrapy.extensions.telnet.TelnetConsole': None,
#}

# Configure item pipelines
# See https://doc.scrapy.org/en/latest/topics/item-pipeline.html
#ITEM_PIPELINES = {
#    'scrapyproDemo.pipelines.ScrapyprodemoPipeline': 300,
#}

# Enable and configure the AutoThrottle extension (disabled by default)
# See https://doc.scrapy.org/en/latest/topics/autothrottle.html
#AUTOTHROTTLE_ENABLED = True
# The initial download delay
#AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
#AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False

# Enable and configure HTTP caching (disabled by default)
# See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
#HTTPCACHE_ENABLED = True
#HTTPCACHE_EXPIRATION_SECS = 0
#HTTPCACHE_DIR = 'httpcache'
#HTTPCACHE_IGNORE_HTTP_CODES = []
#HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'

 

- 伪装请求载体：USER_AGENT

- 忽略robots协议：ROBOTSTXT_OBEY = False