随笔分类 -  爬虫

authproxyhandler
摘要：私密代理 阅读全文
posted @ 2018-01-25 17:26 zy-- 阅读(303) 评论(2) 推荐(0)
ProxyHandler
摘要：#coding:utf-8 import urllib2 proxyswitch = True httpproxy_handler = urllib2.ProxyHandler({"http":"211.159.177.212:3128"}) nullproxy_handler = urllib2.ProxyHandler({}) if proxyswitch: opener =... 阅读全文
posted @ 2018-01-25 16:50 zy-- 阅读(646) 评论(1) 推荐(0)
HTTPHandler
摘要：#coding:utf-8 import urllib2 #构建一个HTTPandler处理对象，支持处理HTTP的请求 http_handler = urllib2.HTTPHandler() #调用build_opener()方法构建一个自定义的opener对象，参数是构建的处理器对象 opener = urllib2.build_opener(http_handler) request... 阅读全文
posted @ 2018-01-25 15:09 zy-- 阅读(351) 评论(1) 推荐(0)
cookie
摘要：#coding:utf-8 import urllib2 url = "http://www.renren.com/247277361/profile" headers = { "host":"www.renren.com", "connection":"keep-alive", #"upgrade-insecure-requests":"1", "use... 阅读全文
posted @ 2018-01-25 00:33 zy-- 阅读(210) 评论(0) 推荐(0)
Ajax练习
摘要：#coding:utf-8 import urllib import urllib2 url = "https://movie.douban.com/j/new_search_subjects?" headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, li... 阅读全文
posted @ 2018-01-24 23:54 zy-- 阅读(120) 评论(0) 推荐(0)
百度贴吧
摘要：0 0 #coding:utf-8 import urllib import urllib2 def loadPage(url,filename): ''' 作用：根据url发送请求，获取服务器响应文件 url：需要爬去的url地址 filename : 处理的文件名 ''' print "正在下载 阅读全文
posted @ 2018-01-21 19:56 zy-- 阅读(752) 评论(0) 推荐(0)
baidu.com关键字查询
摘要：# coding:utf-8 import urllib import urllib2 url = "http://wwww.baidu.com/s" headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3... 阅读全文
posted @ 2018-01-19 18:32 zy-- 阅读(332) 评论(0) 推荐(0)
fiddler手机抓包
摘要：下载地址：https://www.telerik.com/fiddler APP手机爬取(转：http://blog.csdn.net/c406495762/article/details/76850843） 阅读全文
posted @ 2018-01-15 22:20 zy-- 阅读(99) 评论(0) 推荐(0)
pipelines.py
摘要：# -*- coding: utf-8 -*- # Define your item pipelines here # # Don't forget to add your pipeline to the ITEM_PIPELINES setting # See: https://doc.scrapy.org/en/latest/topics/item-pipeline.html impo... 阅读全文
posted @ 2018-01-12 00:48 zy-- 阅读(282) 评论(1) 推荐(0)
scrapy crawl itcast -o teachers.json 爬虫案列
摘要：spider.py文件配置 1 2 # -*- coding: utf-8 -*- 3 import scrapy 4 from itTeachers.items import ItteachersItem 5 6 7 class ItcastSpider(scrapy.Spider): 8 name = 'itcast' 9 allo... 阅读全文
posted @ 2018-01-11 18:45 zy-- 阅读(743) 评论(3) 推荐(0)
pyhhon2.7+scrapy+mongodb-- 爬取豆瓣电影TOP250
摘要：24 quote = each.xpath(".//p[@class='quote']/span/text()").extract() 25 if len(quote) != 0: 26 item['quote'] = quote[0] 27 28 29 yield item 30 31 if se 阅读全文
posted @ 2018-01-09 00:53 zy-- 阅读(385) 评论(0) 推荐(0)
Redis
摘要：转：http://blog.csdn.net/xiangwanpeng/article/details/54586087 ：https://www.cnblogs.com/wangchunniu1314/p/6339416.html 阅读全文
posted @ 2017-11-24 22:27 zy-- 阅读(180) 评论(0) 推荐(0)
腾讯招聘网站爬虫
摘要：7.vi pipelines.py 8.vi settings.py 9.cd spiders 12 scrapy crawl qqcent 阅读全文
posted @ 2017-11-14 20:40 zy-- 阅读(286) 评论(0) 推荐(0)
Linux下python虚拟环境和配置
摘要：sudo apt-get install python-virtualenv virtualenv py2 cd py2 cd bin 启动虚拟环境：source activate deactivate virtualenv -p /usr/bin/pyhon3 py3 cd py3 cd bin source activate pip install virtualenvwrapper... 阅读全文
posted @ 2017-11-10 12:17 zy-- 阅读(4505) 评论(1) 推荐(0)
Windows下python虚拟环境和配置
摘要：pip install virtualenv 下载模块使用国内镜像：pip install -i https://pypi.douban.com/simple/ django 新建虚拟环境：virtualenv scrapytest cd scrapytest cd Scrapytest 运行：activate.bat 退出：deactivae.bat virtualenv -p C:\User... 阅读全文
posted @ 2017-11-10 12:03 zy-- 阅读(1834) 评论(1) 推荐(0)
urllib2
摘要：Python自带的模块：/usr/lib/python2.7/urllib2.py Python的第三方模块：/usr/local/lib/python2.7/site-packages:hh 阅读全文
posted @ 2017-10-15 22:04 zy-- 阅读(120) 评论(0) 推荐(0)