crawler 0.1.0 : Python Package Index
crawler 0.1.0
python crawler.
Latest Version:
0.1.2
python crawler. ===== ## Example ===== from crawler.crawler import Crawler mycrawler = Crawler() seeds = ['http://www.example.com/'] # list of url mycrawler.add_seeds(seeds) url_patterns = ['^(.+example\.com)(.+)
] # list of regular expression for urls that crawler will work on. mycrawler.start(url_patterns) # start crawling ################# data files ################# three database (Berkeley DB) files will be generated. queue.db webpage.db duplcheck.db
