Python中处理HTTP协议的库：urllib2

使用Python访问网页主要有三种方式： urllib, urllib2, httplib

urllib比较简单，功能相对也比较弱，httplib简单强大，但不支持session

 

1. 最简单的页面访问(获取服务器端的Response包)

url = 'http://www.baidu.com'
res = urllib2.urlopen(url)
print res.read()

 2. 加上要GET或POST的数据

data = {'name': 'hank', 'passwd': 'hjz'}
res = urllib2.urlopen(url, urllib.urlencode(data))
print res.read()

 3. 加上http头

header = {'User-Agent':'Mozilla/5.0 (Windows NT 6.1; rv:14.0) Gecko/20100101 Firefox/14.0.1\r\n'}
res = urllib2.Request(url)
res.add_header('User-Agent', 'Mozilla/5.0 (Windows NT 6.1; rv:14.0) Gecko/20100101 Firefox/14.0.1\r\n')
response = urllib2.urlopen(res)
print response.read()

 

#转载其他的：

使用opener和handler
opener = urllib2.build_opener(handler)
urllib2.install_opener(opener)
4. 加上session
cj = cookielib.CookieJar()
cjhandler=urllib2.HTTPCookieProcessor(cj)
opener = urllib2.build_opener(cjhandler)
urllib2.install_opener(opener)
5. 加上Basic认证
password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
top_level_url = "http://www.163.com/"
password_mgr.add_password(None, top_level_url, username, password)
handler = urllib2.HTTPBasicAuthHandler(password_mgr)
opener = urllib2.build_opener(handler)
urllib2.install_opener(opener)
6. 使用代理
proxy_support = urllib2.ProxyHandler({"http":"http://1.2.3.4:3128/"})
opener = urllib2.build_opener(proxy_support)
urllib2.install_opener(opener)
7. 设置超时
socket.setdefaulttimeout(5)