scrapy CrawlSpiders-爬取url实例2

一. 新建项目(scrapy startproject)

scrapy startproject dongguanSpider

 

二、明确目标(dongguanSpider/items.py)

import scrapy

class DongguanspiderItem(scrapy.Item):
    # define the fields for your item here like:
    url = scrapy.Field()
    title = scrapy.Field()
    content = scrapy.Field()
    number = scrapy.Field()

 

三、制作爬虫 （spiders/donguan.py或spiders/xixi.py）

scrapy genspider -t crawl donguan "wz.sun0769.com"

1、spiders/donguan.py使用CrawlSpider方法提取url

# -*- coding: utf-8 -*-
import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
from ..items import DongguanspiderItem

class DongguanSpider(CrawlSpider):
    name = 'dongguan'
    allowed_domains = ['wz.sun0769.com']
    start_urls = ['http://wz.sun0769.com/political/index/politicsNewest?id=1&page=']

    rules = (
        #follow是一个布尔(boolean)值，指定了根据该规则从response提取的链接是否需要跟进。 如果callback 为None，follow 默认设置为True，否则默认为False
        Rule(LinkExtractor(allow=r'id=1&page=\d+'),follow=True),  #表示链接需要跟进，假如这里follow=False,那只会爬取第一页的数据，之后的页面没有根据就不会爬取
        #Rule(LinkExtractor(allow=r'id=1&page=\d+'),process_links = "deal_links"),   # 如果url被web服务器篡改，需要调用process_links来处理提取出来的url
        Rule(LinkExtractor(allow='index\?id=\d+'), callback='parse_item'),   #这个问号坑爹啊，正则表达式问号要转义！！！！！！！对正则还是不熟没能及时发现问题
    )


    # # 如果url被web服务器篡改，需要调用process_links来处理提取出来的url
    # def deal_links(self,links):  # links 是当前response里提取出来的链接列表
    #     for each in links:
    #         each.url=each.url.replace("?","&").replace("Type&","Type?")
    #         return links
    #

    def parse_item(self, response):
        print(response.url)
        item = DongguanspiderItem()
        item['title']=response.xpath("//div[@class='mr-three']/p/text()").extract()[0]
        item['content']=response.xpath("//div[@class='details-box']/p/text()").extract()[0].replace("\n","").strip()   #strip()去掉空格
        item['url']=response.url
        item['number']=response.xpath("//div[@class='mr-three']/div/span[4]/text()").extract()[0].split("：")[-1]

        yield item

2、或者spiders/xixi.py使用Spider方法爬虫（有时候CrawlSpider提取的url可能被篡改，发送请求发不出去，不想深挖解决的话。可以使用Spider）

# -*- coding: utf-8 -*-
import scrapy
from ..items import DongguanspiderItem

class XixiSpider(scrapy.Spider):
    name = 'xixi'
    allowed_domains = ['wz.sun0769.com']

    url="http://wz.sun0769.com/political/index/politicsNewest?id=1&page="
    offset=0
    start_urls = [url+str(offset)]

    def parse(self, response):
        # 每一页里的所有帖子的链接集合
        links=response.xpath("//li[@class='clear']/span[3]/a/@href").extract()
        # 迭代取出集合里的链接
        for link in links:
            # 提取列表里每个帖子的链接，发送请求放到请求队列里,并调用self.parse_item来处理
            yield scrapy.Request(r'http://wz.sun0769.com/'+link,callback=self.parse_item)

        # 页面终止条件成立前，会一直自增offset的值，并发送新的页面请求，调用parse方法处理
        if self.offset<=20:
            self.offset+=1
            # 发送请求放到请求队列里，调用self.parse处理response
            yield scrapy.Request(self.url+str(self.offset),callback=self.parse)

    # 处理每个帖子的response内容
    def parse_item(self,response):
        print(response.url)
        item=DongguanspiderItem()
        item['title'] = response.xpath("//div[@class='mr-three']/p/text()").extract()[0]
        item['content'] = response.xpath("//div[@class='details-box']/p/text()").extract()[0].replace("\n","").strip()  # strip()去掉空格
        item['url'] = response.url
        item['number'] = response.xpath("//div[@class='mr-three']/div/span[4]/text()").extract()[0].split("：")[-1]

        # 交给管道
        yield item

 

四、存储内容 （pipelines.py)

修改settings.py以下几个地方(可将日志打印在文件里)：

 1 # 保存日志信息的文件名
 2 LOG_FILE='dongguanlog.log'
 3 # 保存日志等级，低于|等于此等级的信息都被保存
 4 LOG_LEVEL='DEBUG'
 5 
 6 DEFAULT_REQUEST_HEADERS = {
 7   "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.87 Safari/537.36",
 8   'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
 9   'Accept-Language': 'en',
10 }
11 
12 ITEM_PIPELINES = {
13    'wdzurlSpider.pipelines.DongguanspiderPipeline': 300,
14 }

编写pipelines.py文件

import json

class DongguanspiderPipeline(object):
    def process_item(self, item, spider):
        jsontext=json.dumps(dict(item),ensure_ascii=False)+',\n'
        with open('dongguan.json','a') as f:
            f.write(jsontext)
        return item

命令执行：

 或：