scrapy简单使用

创建scrapy项目
scrapy startproject first_scrapy

cmd命令切换目录到first_scrapy
cd first_scrapy

创建爬虫文件
scrapy genspider first_crawl www.web.com

cmd命令切换目录到first_crawl文件所在目录
cd spiders

运行first_crawl
scrapy crawl first_crawl

first_crawl.py文件里的代码

import scrapy


class FirstSpider(scrapy.Spider):
    """
     爬取糗事百科的title和author
    """
    name = 'first'
    # allowed_domains = ['www.xxx.com']
    start_urls = ['https://www.qiushibaike.com/']

    def parse(self, response):
        li_list = response.xpath('//*[@id="content"]/div/div[2]/div/ul/li')
        for li in li_list:
        	title = li.xpath('./div/a/text()')[0].extract()
        	author = li.xpath('./div/div/a/span/text()')[0].extract()
        	print(title)
        	print()
        	print(author)