spiders 爬虫

Spider are classes which define how a certain site (or a group of sites) will be scraped, including how to perform the crawl and how to extract strutured data from their pages.

Spider are the place where you define the custom behaviour for crawling and parsing pages for a particular site.

1. a spider's scraping cycle

  1. You start by generating the initial Requests to crawl the first URLs, and specify a callback function to be called with the response downloaded from those requests. the first requests to perform are obtained bu calling the start_requests() method which (by default) generates Request fro the URLs specified in the start_urls and the parse method as callback function for the Requests.

  2. In the callback function, you parse the response (web page) and return either dicts with extracted data, Item objects, Request objects, or an iterable of thest objects. Those Requests will also contain a callback (maybe the same) and will then be downloaded by Scrapy and then their response handled by the specified callback.

  3. In callback function, you parse the page contents, typically using Selectors (but you can also use BeautifluSoup, lxml or whatever mechanism you perfer) and generate items with the parsed data.

  4. Finally , the items returned from the spider will be typically persisted to a database (in some Item Pipeline) or written to a file using Feed exports.

2. scrapy.Spider

class scrapy.spiders.Spider : This is the simplest spider, and the one from which every other spider must inhert (including spiders that come bundled with Scrapy, as well as spiders that you write yourself).

It doesn't provied any special functionality. It just provides a default start_requests() implementation which sends requests from the start_urls spider attribute and calls the spider's method parse() for each of the resulting responses.

name

A string which defines the name for this spider. it must be unique in a single project.

In Python 2 this must be ASCII only.

allowed_domains

An optional list of strings containing domains that this spider is allowed to crawl.

Requests for URLs not belonging to the domain names specified in this list (or theri subdomains) won't be followed if OffsiteMiddleware is enabled.

start_urls

A list of URLs where the spider will begin to crawl from .

custom_settings

A dictionary of settings that will be overridden from the project wide configuration when running this spider. It must be defined as a class attribute since the settings are updated before instantiation.

available built-in settings list

crawler

This attribute is set by the from_crawler() class method after initialization the class, and links to the Crawler object to which this spider instance is bound.

Crawlers encapsulate a lot of components in the project for theri single entry access (such as extensions, middlewares, signals managers, etc)

settings

Configuration fot running this spider.

This is a settings instacne.

logger

Python logger created with the Spider's name. you can use it to send log messages.

from_crawler(crawler, *args, **kwargs)

This is the class method used by Scrapy to create your spiders.

You probably won't need to override this directly because the default implementation acts as a proxy to the __init__() method, calling it with the given arguments args and named arguments kwargs.

Nonetheless(尽管如此), this method sets the crawler and settings attributes in the new instance so they can be accessed later inside the spider's code.

  • crawler : crawler instance, crawler to which the spider will be bound.
  • args : arguments passed to the __init__() method.
  • kwargs : keyword arguments passed to the __init__() method.

start_requests()

This method must return an iterable wiht the first Requests to crawl for this spider. It is called by Scrapy when the spider is opened for scraping .

Scrapy calls it only once, so it is safe to implement start_requests() as a generator.

You can override this method to change the Requests used to start scraping a domain.

Example: you need to start by logging in using a POST request.

class MySpider(scrapy.Spider):
	name = "myspider"

	def start_requests(selt):
		return [scrapy.FormRequest('http://www.exmaple.com/login', formdata={'user': 'tom', 'pass': 'secret'}, callback=self.logged_in)]

	def logged_in(self, response):
		# here you would extract links to follow and return Requests for each of them , with another callback.
		pass

parse(response)

This is the default callback used by Scrapy to process downloaded responses, when theri requests don't specify a callback.

The parse method is in charge of processing the response and returning scraped data and/or more URLs to follow. Other Requests callbacks have the same requirements as the Spider class.

return :

  • an iterable of Request
  • dicts
  • Item objects

response : the response to parse.

log(message [, level, component])

Wrapper that sends a log message through the Spider's logger, kept for backwards(向后的) compatibility(兼容性).

closed(reason)

Called when the spider closes. This method provides a shortcut to signals.connect() for the spider_closed signal.

example

use Items to make data more structure.

import scrapy
from myproject.items import MyItem

class MySpider(scrapy.Spider):
	name = 'example.com'
	allowed_domains=['example.com']

	def start_requests(self):
		yield scrapy.Request('http://www.example.com/1.html', self.parse)
		yield scrapy.Request('http://www.example.com/2.html', self.parse)
		yield scrapy.Request('http://www.example.com/3.html', self.parse)

	def parse(self, response):
		for h3 in response.xpath('//h3').extract():
			yield MyItem(title=h3)

		for url in response.xpath("//a/@href").extract():
			yield scrapy.Request(url, callback=self.parse)

3. spider arguments

Spiders can receive arguments that modify their behaviour. Some common uses for spider arguments are to define the start URLs or to restrict the crawl to certain sections of the site, but they can be used to configure any functionality of the spider.

Spider arguments are passed through the crawl command using the -a option.

$ scrapy crawl myspider -a category=electronics

# set http auth credentials use by HttpAuthMiddleware or the user agent used by UserAgentMiddleware
$ scrapy crawl myspider -a http_user=myuser -a http_pass=mypasswd -a user_agent=myagent

Spiders can acess arguments in their __init__ methods, the default __init__ method will take any spider arguments and copy them to the spider as attribute.

import scrapy
class MySpider(scrapy.Spider):
	name = 'myspider'

	def __init__(self, category=None, *args, **kwargs):
		super(MySpider,self).__init__(*args, **kwargs))
		self.start_urls = ["http://example.com/categories/%s" % category]
		# ...

# the follow example working the same way as above.

import scrapy
class MySpider(scrapy.Spider):
	name = 'myspider'
	def start_requests(self):
		yield scrapy.Request("http://example.com/categories/%s" % self.category)

Spider arguments are only string. The spider will not do any parsing on its own.

If you were to set the start_urls attribute from the command line, you wonlud have to parse it on your own into a list using something like atr.literal_eval or json.loads and set it as an attribute.

Spiders arguments can also be passed through the Scrapyd schedule.json.

4. Generic Spiders

Scrapy comes with some useful generic spiders that you can use to subclass your spiders form . Their aim is to provide convenient functionality for a few common scraping cases.

For the examples used in the following spiders, we'll assume you have a project with a TestItem deckared in a myproject.items module.

import scrapy

class TestItem(scrapy.Item):
	id = scrapy.Field()
	name = scrapy.Field()
	description = scrapy.Field()

4.1 CrawlSpider

This the most commonly used spider for crawling regular website, as it providfes a convenient mechanism for following links by defining a set of rules.

You can start from it and override it as needed for more custom functionality , or just implement your own spider.

4.1.1 scrapy.spiders.CrawlSpider
class scrapy.spiders.CrawlSpider
	rules =(,)

	def parse_start_url(response):
		pass

  • rules :

    A list of one or more Rule object. Each Rule defines a certain behaviour for crawling the site. If multiple rules match the same link, the first one will be used.

  • parse_start_url(response) :

    This method is called for the start_urls responses. It allows to parse the initial responses and must return either an Item object, a Request object, or an iterable containing any of them . This method is overrideable .

4.1.2 scrapy.spiders.Rule
class scrapy.spiders.Rule(link_extractor, callback=None, cb_kwargs=None, follow=None, process_links=None, process_request=None):

  • link_extractor : is a Link Extractor object which defines how links will be extracted from each crawled page.

  • callback : a callable or a string (in which case a method from the spider object with that name will be used) to be called for each link extracted with the spiecified link_extractor.This callback receives a response as its frist argument and must return a list containing Item and/or Request objects (or any subclass of them).

    When writing crawl spider rules, avoid using parse as callback, since the CarwlSpider uses the parse method itself to implement its logic. So if you override the parse method, the crawl spider will no longer work.

  • cb_kwargs : a dict containing the keywork arguments to be passed to the callback function.

  • follow : a boolean which specifies if links should be followed from each response extracted with this rule. If callback is None, follow defaults to True; otherwise it defaults to False.

  • process_links : a callable or a string (in which case a method from the spider object with that name will be used) which will be called for each list of links extracted from each response using the specified link_extractor. This is mainly used for filtering purpose.

  • process_request : a callable or a string (in which case a method from the spider object with that name will be used) which will be called with every request extraced by this rule, and must return a request or None (to filter out the request).

4.1.3 CrawlSpider Example:
import scrapy 
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor

class MySpider(CrawlSpider):
	name = "example.com"
	allowed_domains = ["example.com"]
	start_urls = ["http://www.example.com"]

	rules = (
		# Extract links matching 'category.php' (but not matching 'subsection.php')
		# and follow links from them (since no callback means follow=True by default)
		Rule(LinkExtractor(allow=('category\.php',), deny=('subsection\.php',))),

		# Extract links matrching 'item.php' and parse them with the spider's method parse_item
		Rule(LinkExtractor(allow=('item\.php',)), callback='parse_item'),
	)

	def parse_item(self, response):
		self.logger.info("Hi, this is an item page ! %s ", response.url)
		item = scrapy.Item()
		item['id'] = response.xpath('//td[@id="item_id"]/text()').re(r'ID: (\d+)')
        item['name'] = response.xpath('//td[@id="item_name"]/text()').extract()
        item['description'] = response.xpath('//td[@id="item_description"]/text()').extract()
        return item

4.2 XMLFeedSpider

XMLFeedSpider is designed for parsing XML feeds by iterating through them by a certain node name. The iterator can be chosen from : iternodes, xml, html.

It's recommended to use the iternodes iterator for performance resons, since the xml and html iterators generate the whole DOM at once in order to parse it.

However, using html as the iterator may be useful when parsing XML with bad markup.

class scrapy.spider.XMLFeedSpider

To set the iterator and the tag name, you must define the following calss attributes:

  • iterator

    A string which defines the iterator to use. it can be either:

    • iternodes : a fast iterator based on regular expressions. default
    • html : an iterator which uses Selector. This must load all DOM in memory which could be a problem for big feeds.
    • xml : an iterator which uses Selector. This must load all DOM in memory which could be a problem for big feeds.

  • itertag

    A string with the name of the node (or element) to iterate in .

      itertag = "product"

  • namespaces

    A list of (prefix, uri) tuples which define the namespaces available in that document that will be processed with this spider.

    The prefix and uri will be used to automatically register namespaces using the register_namespace() method.

    you can then specify nodes with namespaces in the itertag attribute.

      class MySpider(XMLFeedSpider):

  	namespaces = [('n', 'http://www.sitemaps.org/schemas/sitemap/0.9')]

  	itertag = 'n:url'

Methods :

  • adapt_response(response)

    A method that recveives the response as soon as it arrives from the spider middleware, before the spider starts parsing it.

    It can be used to modify the response body before parsing it. This method receives a response and also return a response (it could be the same, or anther one ).

  • parse_node(response, selector)

    This method is called for the nodes matching the provided tag name itertag. Receives the response and an Selector for each node.

    Overriding this method is mandatory(强制的). otherwise, your spider won't work.

    This method must return either a Item object , a Request object, or an iterable containing any of them .

  • process_result(response, result)

    This method is called for each result (item or request) returned by the spider, and it's intended to perform any last time processing required before returning the retult to the framework core, for example setting the item IDs.

    It receives a list of retults and the response which originated those results. It must be return a list of result (Items or Requests)

XMLFeedSpider Example

from scrapy.spider import XMLFeedSpider
from myproject.items import TestItem

class MySpider(XMLFeedSpider):
	"""
	Basically what we did up there was to create a spider that downloads a feed from the given start_urls, and then iterates through each of its item tags, prints them out, and stores some random data in an Item.
	"""
	name = 'example.com'
	allowed_domain = ['example.com']
	start_urls = ['http://www.example.com/feed.xml']
	iterator = "iternodes"
	itertag = "item"

	def parse_node(self, response, node):
		self.logger.info("Hi, This is a <%s> node! : %s", self.itertag, ''.join(node.extract()))

		item = TestItem()
		item['id'] = node.xpath('@id').extract()
        item['name'] = node.xpath('name').extract()
        item['description'] = node.xpath('description').extract()
        return item

4.3 CSVFeedSpider

This spider is very similar to the XMLFeedSpider, except that it iterates over rows, instead of nodes.

class scrapy.spider.CSVFeedSpider

  • delimiter

    A string with the separator charcter for each field in the CSV file Deafults to ',' (comma).

  • quotechar

    A string with the enclosure character for each field in the CSV file Defaults to '"'(quotation mark)

  • headers

    A list of the column names in the CSV file.

  • parse_row(response, row)

    Receives a response and a dict (representing each row) with a key for each provided (or detected) header of the CSV file.

  • adapt_response

    pre and post-processing purpose.

  • process_results

    pre and post-processing purpose.

CSVFeedSpider Example

from scrapy.spider import CSVFeedSpider
from myproject.items import TestItem

class MySpider(CSVFeedSpider):
    name = 'example.com'
    allowed_domains = ['example.com']
    start_urls = ['http://www.example.com/feed.csv']
    delimiter = ';'
    quotechar = "'"
    headers = ['id', 'name', 'description']

    def parse_row(self, response, row):

        self.logger.info('Hi, this is a row!: %r', row)

        item = TestItem()
        item['id'] = row['id']
        item['name'] = row['name']
        item['description'] = row['description']
        return item

4.4 SitemapSpider

SitemapSpider allow you ti crawl a site by discovering the URLs using Sitemaps.

It supports nested sitemaps and discovering sitemap urls from robots.txt.

class scrapy.spiders.SitemapSpider

  • sitemap_urls

    A list of urls pointing to the sitemaps whose urls you want to crawl.

    You can also point to a robots.txt and it will be parsed to extract sitemap urls from it.

  • sitemap_rules

    Example:

      sitemap_rules = [('/product/', 'parse_product')]

    A list of tuples (regex, callback)

    • regex : regex is a regular expression to match urls extracted from sitemaps. it can be either a str or a compiled regex object.

    • callback : is the callback to use for processing the urls that match the regular expression. callback can be a string or a callable. parse is the default method.

    Rules are applied in order ,and only the first one that matches will be used.

  • sitemap_follow

    A list of regexes of sitemap that should be followed. This is only for sites that use Sitemap Index Files that point to other sitemap files.

    By default, all sitemaps are followed.

  • sitemap_alternate_links

    Specifies if alternate links for one url should be followed.

    These are links for the same website in another language passed within the same url block.

      <url>
      <loc>http://example.com/</loc>
      <xhtml:link rel="alternate" hreflang="de" href="http://example.com/de"/>
  </url>

    With sitemap_alternate_links set, this would retrueve both URLs.

    with sitemap_alternate_links disabled, only http://example.com would be retrieved. Default is disabled.

Examples:

# process some urls with certain callback and other urls with a different callbakc.

	from scrapy.spiders import SitemapSpider

	class MySpider(SitemapSpider):
		sitemap_urls = ["http://www.example.com/sitemap.xml"]
		sitemap_rules = [
			('/product/', 'parse_product'),
			('/category/', 'parse_category')
		]

		def parse_product(self, response):
			pass

		def parse_category(self, response):
			pass

# Follow sitemaps defined in the robots.txt fiel and only follow sitemaps whose url contains `/sitemap_shop`

	from scrapy.spider import SitemapSpider

	class MySpider(self, response):
		sitemap_urls = ['http://www.example.com/robots.txt']
		sitemap_rules = [
			('/shop/', 'parse_shop'),
		]

		sitemap_follow = ['/sitemap_shops']

		def parse_shop(self, response):
			pass

# Combine SitemapSpider with other sources of urls.

	from scrapy.spiders import SitemapSpider

	class MySpider(SitemapSpider):
		sitemap_urls = ["http://www.example.com/robots.txt"]
		sitemap_rules = [
			('/shop/', 'parse_shop'),
		]

		other_urls = ["http://www.example.com/about"]

		def start_requests(self):
			requests = list(super(MySpider, self).start_requests())
			requests += [ scrapy.Request(x, self.parse_other) for x in self.other_urls ]

		def parse_shop(self, response):
			pass

		def parse_other(self, response):
			pass
posted @ 2017-11-03 22:54  眼镜男  阅读(273)  评论(0)    收藏  举报