requests库

一、http协议原理

http是无状态的应用层协议

二、urllib、urllib2、requests

在python2中，有urllib和urlib2两个包，python3全变成urllib了

urllib和urllib2是相互独立的模块,比如，urllib不能伪造user agent，但是请求参数的构建，需要使用到urllib,所有经常是两者配合使用的;

requests使用了urlib3(多次请求重复使用一个socket)

三、环境准备

http://httpbin.org/

如果将该站点搭建在本地，使用pip安装就可以了

pip install gunicorn httpbin

启动:

gunicorn httpbin:app --bind 0.0.0.0:8000

发送一个http的get请求，获取本机的IP地址

import urllib2
import urllib

URL_IP='http://192.168.74.20:8000/ip'

def use_simple_urllib2():
    response=urllib2.urlopen(URL_IP)
    print ">>>>Response Headers:"
    print response.info()
    print ''.join([line for line in response.readlines()])

if __name__ == '__main__':
    print ">>>Use Simple urllib2"
    use_simple_urllib2()

结果为:
>>>Use Simple urllib2
>>>>Response Headers:
Server: gunicorn/19.7.1
Date: Sat, 08 Apr 2017 13:06:18 GMT
Connection: close
Content-Type: application/json
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
Content-Length: 31

{
  "origin": "192.168.74.1"
}

使用urllib构建请求参数,使用urllib2发送请求

def use_params_urllib2():
    """
    构建请求参数
    :return:
    """
    params = urllib.urlencode({'param1':'hello','param2':'world'})
    print 'Request Params:'
    print params
    #发送请求
    response=urllib2.urlopen('?'.join([URL_GET,'%s'])%params)
    #处理响应
    print ">>>>Response Headers:"
    print response.info()
    print ">>>Status code"
    print response.getcode()
    print ''.join([line for line in response.readlines()])
if __name__ == '__main__':
    print ">>>Use Simple urllib2"
    use_simple_urllib2()
    print ">>>Use params urllib2"
    use_params_urllib2()

结果:
>>>Use params urllib2
Request Params:
param2=world&param1=hello
>>>>Response Headers:
Server: gunicorn/19.7.1
Date: Sat, 08 Apr 2017 13:22:05 GMT
Connection: close
Content-Type: application/json
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
Content-Length: 322

>>>Status code
200
{
  "args": {
    "param1": "hello", 
    "param2": "world"
  }, 
  "headers": {
    "Accept-Encoding": "identity", 
    "Connection": "close", 
    "Host": "192.168.74.20:8000", 
    "User-Agent": "Python-urllib/2.7"
  }, 
  "origin": "192.168.74.1", 
  "url": "http://192.168.74.20:8000/get?param2=world&param1=hello"
}

使用requests模块实现上述功能

#!/usr/bin/env python
# _*_ coding:utf-8 _*_
__author__ = 'Charles Chang'

import requests
import urllib2
import urllib

URL_IP='http://192.168.74.20:8000/ip'
URL_GET='http://192.168.74.20:8000/get'

def use_simple_requests():
    response=requests.get(URL_IP)
    print ">>>>Response Headers:"
    print response.headers
    print '>>>>Response Body:'
    print response.text

def use_params_resuqets():
    """
    构建请求参数
    :return:
    """
    params = {'param1':'hello','param2':'world'}
    print 'Request Params:'
    print params
    #发送请求
    response=requests.get(URL_IP,params=params)
    #处理响应
    print ">>>>Response Headers:"
    print response.headers
    print ">>>Status code"
    print response.status_code
    print response.reason
    print ">>>>Response Body:"
    print response.json()

if __name__ == '__main__':
    print ">>>Use Simple urllib2"
    use_simple_requests()
    print
    print ">>>Use params requests"
    use_params_resuqets()

结果：
>>>Use Simple urllib2
>>>>Response Headers:
{'Content-Length': '31', 'Server': 'gunicorn/19.7.1', 'Connection': 'close', 'Access-Control-Allow-Credentials': 'true', 'Date': 'Sat, 08 Apr 2017 14:29:38 GMT', 'Access-Control-Allow-Origin': '*', 'Content-Type': 'application/json'}
>>>>Response Body:
{
  "origin": "192.168.74.1"
}


>>>Use params requests
Request Params:
{'param2': 'world', 'param1': 'hello'}
>>>>Response Headers:
{'Content-Length': '31', 'Server': 'gunicorn/19.7.1', 'Connection': 'close', 'Access-Control-Allow-Credentials': 'true', 'Date': 'Sat, 08 Apr 2017 14:29:38 GMT', 'Access-Control-Allow-Origin': '*', 'Content-Type': 'application/json'}
>>>Status code
200
OK
>>>>Response Body:
{u'origin': u'192.168.74.1'}

四、github api，requests 请求方法

https://developer.github.com/v3/

GET:查看资源

POST:增加资源

PUT：修改资源

DELETE：删除资源

HEAD:查看响应头

OPTIONS:查看可用请求方法

PATCH:使用少量的json格式的数据去更新数据的方式;

Verb	Description
`HEAD`	Can be issued against any resource to get just the HTTP header info.
`GET`	Used for retrieving resources.
`POST`	Used for creating resources.
`PATCH`	Used for updating resources with partial JSON data. For instance, an Issue resource has`title` and `body` attributes. A PATCH request may accept one or more of the attributes to update the resource. PATCH is a relatively new and uncommon HTTP verb, so resource endpoints also accept `POST` requests.
`PUT`	Used for replacing resources or collections. For `PUT` requests with no `body` attribute, be sure to set the `Content-Length` header to zero.
`DELETE`	Used for deleting resources.

requests请求方法调用github api

import json
import requests

URL='https://api.github.com'

def build_uri(endpoint):
    return '/'.join([URL,endpoint])

def better_print(json_str):
    return json.dumps(json.loads(json_str),indent=4)

def request_method():
    response=requests.get(build_uri('users/myuser'),auth=('myuser','mypassword'))
    print response.text
    print better_print(response.text)

if __name__ == '__main__':
    request_method()

结果:
{"login":"CharlesQQ","id":18431718,"avatar_url":"https://avatars2.githubusercontent.com/u/18431718?v=3","gravatar_id":"","url":"https://api.github.com/users/CharlesQQ","html_url":"https://github.com/CharlesQQ","followers_url":"https://api.github.com/users/CharlesQQ/followers","following_url":"https://api.github.com/users/CharlesQQ/following{/other_user}","gists_url":"https://api.github.com/users/CharlesQQ/gists{/gist_id}","starred_url":"https://api.github.com/users/CharlesQQ/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/CharlesQQ/subscriptions","organizations_url":"https://api.github.com/users/CharlesQQ/orgs","repos_url":"https://api.github.com/users/CharlesQQ/repos","events_url":"https://api.github.com/users/CharlesQQ/events{/privacy}","received_events_url":"https://api.github.com/users/CharlesQQ/received_events","type":"User","site_admin":false,"name":null,"company":null,"blog":null,"location":null,"email":null,"hireable":null,"bio":null,"public_repos":7,"public_gists":0,"followers":0,"following":8,"created_at":"2016-04-13T00:58:55Z","updated_at":"2017-04-08T14:36:42Z","private_gists":0,"total_private_repos":0,"owned_private_repos":0,"disk_usage":14729,"collaborators":0,"two_factor_authentication":false,"plan":{"name":"free","space":976562499,"collaborators":0,"private_repos":0}}
{
    "disk_usage": 14729, 
    "private_gists": 0, 
    "public_repos": 7, 
    "site_admin": false, 
    "subscriptions_url": "https://api.github.com/users/CharlesQQ/subscriptions", 
    "gravatar_id": "", 
    "hireable": null, 
    "id": 18431718, 
    "followers_url": "https://api.github.com/users/CharlesQQ/followers", 
    "following_url": "https://api.github.com/users/CharlesQQ/following{/other_user}", 
    "collaborators": 0, 
    "total_private_repos": 0, 
    "blog": null, 
    "followers": 0, 
    "location": null, 
    "type": "User", 
    "email": null, 
    "bio": null, 
    "gists_url": "https://api.github.com/users/CharlesQQ/gists{/gist_id}", 
    "owned_private_repos": 0, 
    "company": null, 
    "events_url": "https://api.github.com/users/CharlesQQ/events{/privacy}", 
    "html_url": "https://github.com/CharlesQQ", 
    "updated_at": "2017-04-08T14:36:42Z", 
    "plan": {
        "collaborators": 0, 
        "name": "free", 
        "private_repos": 0, 
        "space": 976562499
    }, 
    "received_events_url": "https://api.github.com/users/CharlesQQ/received_events", 
    "starred_url": "https://api.github.com/users/CharlesQQ/starred{/owner}{/repo}", 
    "public_gists": 0, 
    "name": null, 
    "organizations_url": "https://api.github.com/users/CharlesQQ/orgs", 
    "url": "https://api.github.com/users/CharlesQQ", 
    "created_at": "2016-04-13T00:58:55Z", 
    "avatar_url": "https://avatars2.githubusercontent.com/u/18431718?v=3", 
    "repos_url": "https://api.github.com/users/CharlesQQ/repos", 
    "following": 8, 
    "login": "CharlesQQ", 
    "two_factor_authentication": false
}

五、带参数的请求

方式1:

search_product.html?cat=19283377&...

params: requests.get(url,params={'key1':''value1})

方式2：表单数据提交

Content-Type: application/x-www-form-urlencoded

requests.post(url,data={'key1':'value1','keys':'value2'})

方式3：json参数提交

Content-type:application/json

requests.post(url,json={'key1':'value1','key2':'value2'})

使用params获取user信息

def params_requests():
    response=requests.get(build_uri('users'),params={'since':11})
    print better_print(response.text)
    print response.request.headers
    print response.url
if __name__ == '__main__':
    params_requests()

结果：
    {
        "following_url": "https://api.github.com/users/tomtt/following{/other_user}", 
        "events_url": "https://api.github.com/users/tomtt/events{/privacy}", 
        "organizations_url": "https://api.github.com/users/tomtt/orgs", 
        "url": "https://api.github.com/users/tomtt", 
        "gists_url": "https://api.github.com/users/tomtt/gists{/gist_id}", 
        "html_url": "https://github.com/tomtt", 
        "subscriptions_url": "https://api.github.com/users/tomtt/subscriptions", 
        "avatar_url": "https://avatars1.githubusercontent.com/u/31?v=3", 
        "repos_url": "https://api.github.com/users/tomtt/repos", 
        "received_events_url": "https://api.github.com/users/tomtt/received_events", 
        "gravatar_id": "", 
        "starred_url": "https://api.github.com/users/tomtt/starred{/owner}{/repo}", 
        "site_admin": false, 
        "login": "tomtt", 
        "type": "User", 
        "id": 31, 
        "followers_url": "https://api.github.com/users/tomtt/followers"
    }, 
    {
        "following_url": "https://api.github.com/users/railsjitsu/following{/other_user}", 
        "events_url": "https://api.github.com/users/railsjitsu/events{/privacy}", 
        "organizations_url": "https://api.github.com/users/railsjitsu/orgs", 
        "url": "https://api.github.com/users/railsjitsu", 
        "gists_url": "https://api.github.com/users/railsjitsu/gists{/gist_id}", 
        "html_url": "https://github.com/railsjitsu", 
        "subscriptions_url": "https://api.github.com/users/railsjitsu/subscriptions", 
        "avatar_url": "https://avatars1.githubusercontent.com/u/32?v=3", 
        "repos_url": "https://api.github.com/users/railsjitsu/repos", 
        "received_events_url": "https://api.github.com/users/railsjitsu/received_events", 
        "gravatar_id": "", 
        "starred_url": "https://api.github.com/users/railsjitsu/starred{/owner}{/repo}", 
        "site_admin": false, 
        "login": "railsjitsu", 
        "type": "User", 
        "id": 32, 
        "followers_url": "https://api.github.com/users/railsjitsu/followers"
    }, 
    {
        "following_url": "https://api.github.com/users/nitay/following{/other_user}", 
        "events_url": "https://api.github.com/users/nitay/events{/privacy}", 
        "organizations_url": "https://api.github.com/users/nitay/orgs", 
        "url": "https://api.github.com/users/nitay", 
        "gists_url": "https://api.github.com/users/nitay/gists{/gist_id}", 
        "html_url": "https://github.com/nitay", 
        "subscriptions_url": "https://api.github.com/users/nitay/subscriptions", 
        "avatar_url": "https://avatars1.githubusercontent.com/u/34?v=3", 
        "repos_url": "https://api.github.com/users/nitay/repos", 
        "received_events_url": "https://api.github.com/users/nitay/received_events", 
        "gravatar_id": "", 
        "starred_url": "https://api.github.com/users/nitay/starred{/owner}{/repo}", 
        "site_admin": false, 
        "login": "nitay", 
        "type": "User", 
        "id": 34, 
        "followers_url": "https://api.github.com/users/nitay/followers"
    }, 
    {
        "following_url": "https://api.github.com/users/kevwil/following{/other_user}", 
        "events_url": "https://api.github.com/users/kevwil/events{/privacy}", 
        "organizations_url": "https://api.github.com/users/kevwil/orgs", 
        "url": "https://api.github.com/users/kevwil", 
        "gists_url": "https://api.github.com/users/kevwil/gists{/gist_id}", 
        "html_url": "https://github.com/kevwil", 
        "subscriptions_url": "https://api.github.com/users/kevwil/subscriptions", 
        "avatar_url": "https://avatars1.githubusercontent.com/u/35?v=3", 
        "repos_url": "https://api.github.com/users/kevwil/repos", 
        "received_events_url": "https://api.github.com/users/kevwil/received_events", 
        "gravatar_id": "", 
        "starred_url": "https://api.github.com/users/kevwil/starred{/owner}{/repo}", 
        "site_admin": false, 
        "login": "kevwil", 
        "type": "User", 
        "id": 35, 
        "followers_url": "https://api.github.com/users/kevwil/followers"
    }, 
    {
        "following_url": "https://api.github.com/users/KirinDave/following{/other_user}", 
        "events_url": "https://api.github.com/users/KirinDave/events{/privacy}", 
        "organizations_url": "https://api.github.com/users/KirinDave/orgs", 
        "url": "https://api.github.com/users/KirinDave", 
        "gists_url": "https://api.github.com/users/KirinDave/gists{/gist_id}", 
        "html_url": "https://github.com/KirinDave", 
        "subscriptions_url": "https://api.github.com/users/KirinDave/subscriptions", 
        "avatar_url": "https://avatars1.githubusercontent.com/u/36?v=3", 
        "repos_url": "https://api.github.com/users/KirinDave/repos", 
        "received_events_url": "https://api.github.com/users/KirinDave/received_events", 
        "gravatar_id": "", 
        "starred_url": "https://api.github.com/users/KirinDave/starred{/owner}{/repo}", 
        "site_admin": false, 
        "login": "KirinDave", 
        "type": "User", 
        "id": 36, 
        "followers_url": "https://api.github.com/users/KirinDave/followers"
    }, 
    {
        "following_url": "https://api.github.com/users/jamesgolick/following{/other_user}", 
        "events_url": "https://api.github.com/users/jamesgolick/events{/privacy}", 
        "organizations_url": "https://api.github.com/users/jamesgolick/orgs", 
        "url": "https://api.github.com/users/jamesgolick", 
        "gists_url": "https://api.github.com/users/jamesgolick/gists{/gist_id}", 
        "html_url": "https://github.com/jamesgolick", 
        "subscriptions_url": "https://api.github.com/users/jamesgolick/subscriptions", 
        "avatar_url": "https://avatars1.githubusercontent.com/u/37?v=3", 
        "repos_url": "https://api.github.com/users/jamesgolick/repos", 
        "received_events_url": "https://api.github.com/users/jamesgolick/received_events", 
        "gravatar_id": "", 
        "starred_url": "https://api.github.com/users/jamesgolick/starred{/owner}{/repo}", 
        "site_admin": false, 
        "login": "jamesgolick", 
        "type": "User", 
        "id": 37, 
        "followers_url": "https://api.github.com/users/jamesgolick/followers"
    }, 
    {
        "following_url": "https://api.github.com/users/atmos/following{/other_user}", 
        "events_url": "https://api.github.com/users/atmos/events{/privacy}", 
        "organizations_url": "https://api.github.com/users/atmos/orgs", 
        "url": "https://api.github.com/users/atmos", 
        "gists_url": "https://api.github.com/users/atmos/gists{/gist_id}", 
        "html_url": "https://github.com/atmos", 
        "subscriptions_url": "https://api.github.com/users/atmos/subscriptions", 
        "avatar_url": "https://avatars0.githubusercontent.com/u/38?v=3", 
        "repos_url": "https://api.github.com/users/atmos/repos", 
        "received_events_url": "https://api.github.com/users/atmos/received_events", 
        "gravatar_id": "", 
        "starred_url": "https://api.github.com/users/atmos/starred{/owner}{/repo}", 
        "site_admin": false, 
        "login": "atmos", 
        "type": "User", 
        "id": 38, 
        "followers_url": "https://api.github.com/users/atmos/followers"
    }, 
    {
        "following_url": "https://api.github.com/users/errfree/following{/other_user}", 
        "events_url": "https://api.github.com/users/errfree/events{/privacy}", 
        "organizations_url": "https://api.github.com/users/errfree/orgs", 
        "url": "https://api.github.com/users/errfree", 
        "gists_url": "https://api.github.com/users/errfree/gists{/gist_id}", 
        "html_url": "https://github.com/errfree", 
        "subscriptions_url": "https://api.github.com/users/errfree/subscriptions", 
        "avatar_url": "https://avatars1.githubusercontent.com/u/44?v=3", 
        "repos_url": "https://api.github.com/users/errfree/repos", 
        "received_events_url": "https://api.github.com/users/errfree/received_events", 
        "gravatar_id": "", 
        "starred_url": "https://api.github.com/users/errfree/starred{/owner}{/repo}", 
        "site_admin": false, 
        "login": "errfree", 
        "type": "Organization", 
        "id": 44, 
        "followers_url": "https://api.github.com/users/errfree/followers"
    }, 
    {
        "following_url": "https://api.github.com/users/mojodna/following{/other_user}", 
        "events_url": "https://api.github.com/users/mojodna/events{/privacy}", 
        "organizations_url": "https://api.github.com/users/mojodna/orgs", 
        "url": "https://api.github.com/users/mojodna", 
        "gists_url": "https://api.github.com/users/mojodna/gists{/gist_id}", 
        "html_url": "https://github.com/mojodna", 
        "subscriptions_url": "https://api.github.com/users/mojodna/subscriptions", 
        "avatar_url": "https://avatars1.githubusercontent.com/u/45?v=3", 
        "repos_url": "https://api.github.com/users/mojodna/repos", 
        "received_events_url": "https://api.github.com/users/mojodna/received_events", 
        "gravatar_id": "", 
        "starred_url": "https://api.github.com/users/mojodna/starred{/owner}{/repo}", 
        "site_admin": false, 
        "login": "mojodna", 
        "type": "User", 
        "id": 45, 
        "followers_url": "https://api.github.com/users/mojodna/followers"
    }, 
    {
        "following_url": "https://api.github.com/users/bmizerany/following{/other_user}", 
        "events_url": "https://api.github.com/users/bmizerany/events{/privacy}", 
        "organizations_url": "https://api.github.com/users/bmizerany/orgs", 
        "url": "https://api.github.com/users/bmizerany", 
        "gists_url": "https://api.github.com/users/bmizerany/gists{/gist_id}", 
        "html_url": "https://github.com/bmizerany", 
        "subscriptions_url": "https://api.github.com/users/bmizerany/subscriptions", 
        "avatar_url": "https://avatars1.githubusercontent.com/u/46?v=3", 
        "repos_url": "https://api.github.com/users/bmizerany/repos", 
        "received_events_url": "https://api.github.com/users/bmizerany/received_events", 
        "gravatar_id": "", 
        "starred_url": "https://api.github.com/users/bmizerany/starred{/owner}{/repo}", 
        "site_admin": false, 
        "login": "bmizerany", 
        "type": "User", 
        "id": 46, 
        "followers_url": "https://api.github.com/users/bmizerany/followers"
    }, 
    {
        "following_url": "https://api.github.com/users/jnewland/following{/other_user}", 
        "events_url": "https://api.github.com/users/jnewland/events{/privacy}", 
        "organizations_url": "https://api.github.com/users/jnewland/orgs", 
        "url": "https://api.github.com/users/jnewland", 
        "gists_url": "https://api.github.com/users/jnewland/gists{/gist_id}", 
        "html_url": "https://github.com/jnewland", 
        "subscriptions_url": "https://api.github.com/users/jnewland/subscriptions", 
        "avatar_url": "https://avatars1.githubusercontent.com/u/47?v=3", 
        "repos_url": "https://api.github.com/users/jnewland/repos", 
        "received_events_url": "https://api.github.com/users/jnewland/received_events", 
        "gravatar_id": "", 
        "starred_url": "https://api.github.com/users/jnewland/starred{/owner}{/repo}", 
        "site_admin": true, 
        "login": "jnewland", 
        "type": "User", 
        "id": 47, 
        "followers_url": "https://api.github.com/users/jnewland/followers"
    }, 
    {
        "following_url": "https://api.github.com/users/joshknowles/following{/other_user}", 
        "events_url": "https://api.github.com/users/joshknowles/events{/privacy}", 
        "organizations_url": "https://api.github.com/users/joshknowles/orgs", 
        "url": "https://api.github.com/users/joshknowles", 
        "gists_url": "https://api.github.com/users/joshknowles/gists{/gist_id}", 
        "html_url": "https://github.com/joshknowles", 
        "subscriptions_url": "https://api.github.com/users/joshknowles/subscriptions", 
        "avatar_url": "https://avatars0.githubusercontent.com/u/48?v=3", 
        "repos_url": "https://api.github.com/users/joshknowles/repos", 
        "received_events_url": "https://api.github.com/users/joshknowles/received_events", 
        "gravatar_id": "", 
        "starred_url": "https://api.github.com/users/joshknowles/starred{/owner}{/repo}", 
        "site_admin": false, 
        "login": "joshknowles", 
        "type": "User", 
        "id": 48, 
        "followers_url": "https://api.github.com/users/joshknowles/followers"
    }, 
    {
        "following_url": "https://api.github.com/users/hornbeck/following{/other_user}", 
        "events_url": "https://api.github.com/users/hornbeck/events{/privacy}", 
        "organizations_url": "https://api.github.com/users/hornbeck/orgs", 
        "url": "https://api.github.com/users/hornbeck", 
        "gists_url": "https://api.github.com/users/hornbeck/gists{/gist_id}", 
        "html_url": "https://github.com/hornbeck", 
        "subscriptions_url": "https://api.github.com/users/hornbeck/subscriptions", 
        "avatar_url": "https://avatars0.githubusercontent.com/u/49?v=3", 
        "repos_url": "https://api.github.com/users/hornbeck/repos", 
        "received_events_url": "https://api.github.com/users/hornbeck/received_events", 
        "gravatar_id": "", 
        "starred_url": "https://api.github.com/users/hornbeck/starred{/owner}{/repo}", 
        "site_admin": false, 
        "login": "hornbeck", 
        "type": "User", 
        "id": 49, 
        "followers_url": "https://api.github.com/users/hornbeck/followers"
    }, 
    {
        "following_url": "https://api.github.com/users/jwhitmire/following{/other_user}", 
        "events_url": "https://api.github.com/users/jwhitmire/events{/privacy}", 
        "organizations_url": "https://api.github.com/users/jwhitmire/orgs", 
        "url": "https://api.github.com/users/jwhitmire", 
        "gists_url": "https://api.github.com/users/jwhitmire/gists{/gist_id}", 
        "html_url": "https://github.com/jwhitmire", 
        "subscriptions_url": "https://api.github.com/users/jwhitmire/subscriptions", 
        "avatar_url": "https://avatars0.githubusercontent.com/u/50?v=3", 
        "repos_url": "https://api.github.com/users/jwhitmire/repos", 
        "received_events_url": "https://api.github.com/users/jwhitmire/received_events", 
        "gravatar_id": "", 
        "starred_url": "https://api.github.com/users/jwhitmire/starred{/owner}{/repo}", 
        "site_admin": false, 
        "login": "jwhitmire", 
        "type": "User", 
        "id": 50, 
        "followers_url": "https://api.github.com/users/jwhitmire/followers"
    }, 
    {
        "following_url": "https://api.github.com/users/elbowdonkey/following{/other_user}", 
        "events_url": "https://api.github.com/users/elbowdonkey/events{/privacy}", 
        "organizations_url": "https://api.github.com/users/elbowdonkey/orgs", 
        "url": "https://api.github.com/users/elbowdonkey", 
        "gists_url": "https://api.github.com/users/elbowdonkey/gists{/gist_id}", 
        "html_url": "https://github.com/elbowdonkey", 
        "subscriptions_url": "https://api.github.com/users/elbowdonkey/subscriptions", 
        "avatar_url": "https://avatars0.githubusercontent.com/u/51?v=3", 
        "repos_url": "https://api.github.com/users/elbowdonkey/repos", 
        "received_events_url": "https://api.github.com/users/elbowdonkey/received_events", 
        "gravatar_id": "", 
        "starred_url": "https://api.github.com/users/elbowdonkey/starred{/owner}{/repo}", 
        "site_admin": false, 
        "login": "elbowdonkey", 
        "type": "User", 
        "id": 51, 
        "followers_url": "https://api.github.com/users/elbowdonkey/followers"
    }, 
    {
        "following_url": "https://api.github.com/users/reinh/following{/other_user}", 
        "events_url": "https://api.github.com/users/reinh/events{/privacy}", 
        "organizations_url": "https://api.github.com/users/reinh/orgs", 
        "url": "https://api.github.com/users/reinh", 
        "gists_url": "https://api.github.com/users/reinh/gists{/gist_id}", 
        "html_url": "https://github.com/reinh", 
        "subscriptions_url": "https://api.github.com/users/reinh/subscriptions", 
        "avatar_url": "https://avatars0.githubusercontent.com/u/52?v=3", 
        "repos_url": "https://api.github.com/users/reinh/repos", 
        "received_events_url": "https://api.github.com/users/reinh/received_events", 
        "gravatar_id": "", 
        "starred_url": "https://api.github.com/users/reinh/starred{/owner}{/repo}", 
        "site_admin": false, 
        "login": "reinh", 
        "type": "User", 
        "id": 52, 
        "followers_url": "https://api.github.com/users/reinh/followers"
    }, 
    {
        "following_url": "https://api.github.com/users/knzconnor/following{/other_user}", 
        "events_url": "https://api.github.com/users/knzconnor/events{/privacy}", 
        "organizations_url": "https://api.github.com/users/knzconnor/orgs", 
        "url": "https://api.github.com/users/knzconnor", 
        "gists_url": "https://api.github.com/users/knzconnor/gists{/gist_id}", 
        "html_url": "https://github.com/knzconnor", 
        "subscriptions_url": "https://api.github.com/users/knzconnor/subscriptions", 
        "avatar_url": "https://avatars0.githubusercontent.com/u/53?v=3", 
        "repos_url": "https://api.github.com/users/knzconnor/repos", 
        "received_events_url": "https://api.github.com/users/knzconnor/received_events", 
        "gravatar_id": "", 
        "starred_url": "https://api.github.com/users/knzconnor/starred{/owner}{/repo}", 
        "site_admin": false, 
        "login": "knzconnor", 
        "type": "User", 
        "id": 53, 
        "followers_url": "https://api.github.com/users/knzconnor/followers"
    }
]
{'Connection': 'keep-alive', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'User-Agent': 'python-requests/2.10.0'}
https://api.github.com/users?since=11

使用patch修改数据

def json_request():
    response=requests.patch(build_uri('user'),auth=('charlesQQ','mypass'),
                            json={'name':'charlesQQ11','email':'charles_ali@qq.com'})
    print better_print(response.text)
    print response.request.headers
    print response.request.body
    print response.status_code

{
    "disk_usage": 14729, 
    "private_gists": 0, 
    "public_repos": 7, 
    "site_admin": false, 
    "subscriptions_url": "https://api.github.com/users/CharlesQQ/subscriptions", 
    "gravatar_id": "", 
    "hireable": null, 
    "id": 18431718, 
    "followers_url": "https://api.github.com/users/CharlesQQ/followers", 
    "following_url": "https://api.github.com/users/CharlesQQ/following{/other_user}", 
    "collaborators": 0, 
    "total_private_repos": 0, 
    "blog": null, 
    "followers": 0, 
    "location": null, 
    "type": "User", 
    "email": "charles_ali@qq.com", 
    "bio": null, 
    "gists_url": "https://api.github.com/users/CharlesQQ/gists{/gist_id}", 
    "owned_private_repos": 0, 
    "company": null, 
    "events_url": "https://api.github.com/users/CharlesQQ/events{/privacy}", 
    "html_url": "https://github.com/CharlesQQ", 
    "updated_at": "2017-04-08T14:36:42Z", 
    "plan": {
        "collaborators": 0, 
        "name": "free", 
        "private_repos": 0, 
        "space": 976562499
    }, 
    "received_events_url": "https://api.github.com/users/CharlesQQ/received_events", 
    "starred_url": "https://api.github.com/users/CharlesQQ/starred{/owner}{/repo}", 
    "public_gists": 0, 
    "name": "charlesQQ11", 
    "organizations_url": "https://api.github.com/users/CharlesQQ/orgs", 
    "url": "https://api.github.com/users/CharlesQQ", 
    "created_at": "2016-04-13T00:58:55Z", 
    "avatar_url": "https://avatars2.githubusercontent.com/u/18431718?v=3", 
    "repos_url": "https://api.github.com/users/CharlesQQ/repos", 
    "following": 8, 
    "login": "CharlesQQ", 
    "two_factor_authentication": false
}
{'Content-Length': '54', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'User-Agent': 'python-requests/2.10.0', 'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Authorization': 'Basic Y2hhcmxlc1FROnN0YXJ0MzMzMzM='}
{"name": "charlesQQ11", "email": "charles_ali@qq.com"}
200

使用post增加数据

def json_request():
    #response=requests.patch(build_uri('user'),auth=('charlesQQ','mypass'),
                           # json={'name':'charlesQQ11','email':'charles_ali@qq.com'})
    response=requests.post(build_uri('user/emails'),auth=('charlesQQ','start33333'),json=['qq_c1231@163.com'])
    print better_print(response.text)
    print response.request.headers
    print response.request.body
    print response.status_code

六、请求异常处理

一般在我们调用第三方服务的时候，难以避免的是第三方服务调用出现异常，这种时候，就需要使用异常处理，来保证程序可以正常运行;

requests的异常处理，需要使用其exceptions类来实现;

将超时时间设置为0.1秒，来触发超时异常

import requests
from requests import exceptions
def timeout_request():
    try:
        response=requests.get(build_uri('user/emails'),timeout=0.1)
        response.raise_for_status()
    except exceptions.Timeout as e:
        print e.message
    except exceptions.HTTPError as e:
        print e.message
    else:
        print response.text
        print response.status_code

调用结果:
HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /user/emails (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x0000000003148CF8>, 'Connection to api.github.com timed out. (connect timeout=0.1)'))

将超时时间设置为10s，触发401错误

def timeout_request():
    try:
        response=requests.get(build_uri('user/emails'),timeout=10)
        response.raise_for_status()
    except exceptions.Timeout as e:
        print e.message
    except exceptions.HTTPError as e:
        print e.message
    else:
        print response.text
        print response.status_code

调用结果:
401 Client Error: Unauthorized for url: https://api.github.com/user/emails

七、自定义request

requests通过urlib3实现可以hold住整个socket，通过session实现;

发送的request包，包含发送的body，headers，auth，proxy，timeout，verify等，回复消息response为text,json等;

有时间去研究源码吧

def hard_requests():
    from requests import Request,Session
    s=Session()
    headers={'User-Agent':'fake1.3.4'}
    req=Request('GET',build_uri('user/emails'),auth=('myuser','mypass'),headers=headers)
    prepared=req.prepare()
    print prepared.body
    print prepared.headers

    resp=s.send(prepared,timeout=5)
    print resp.status_code
    print resp.request.headers
    print resp.text

结果:
None
{'Authorization': 'Basic Y2hhcmxlc1FROnN0YXJ0MzMzMzM=', 'User-Agent': 'fake1.3.4'}
200
{'Authorization': 'Basic Y2hhcmxlc1FROnN0YXJ0MzMzMzM=', 'User-Agent': 'fake1.3.4'}
[{"email":"qq_c123@163.com","primary":false,"verified":false,"visibility":null},{"email":"charles_ali@qq.com","primary":true,"verified":true,"visibility":"public"},{"email":"qq_c1231@163.com","primary":false,"verified":false,"visibility":null}]

八、响应基本API

Response对象的API：

status_code:如下

400：Bad request,请求服务器端无法解析； 401:unauthorized,请求没有认证; 403：forbidden； 404:Not found；

500：内部错误;

reason:

headers:

url:

history:

elapsed:

request:

encoding:

raw:读取原始的对象

content:string类型:

text:unicode;

json:

下面分别举例介绍

status_code和reason

import requests
response = requests.get('https://api.github.com')
>>> response.status_code
200
>>> response.reason
'OK'

响应headers

>>> response.headers
{'X-XSS-Protection': '1; mode=block', 'Content-Security-Policy': "default-src 'none'", 'Access-Control-Expose-Headers': 'ETag, Link, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval', 'Transfer-Encoding': 'chunked', 'Access-Control-Allow-Origin': '*', 'X-Frame-Options': 'deny', 'Status': '200 OK', 'X-Served-By': 'eef8b8685a106934dcbb4b7c59fba0bf', 'X-GitHub-Request-Id': '059D:17F65:2FAC88D:3DE5D57:58EA0875', 'ETag': 'W/"7dc470913f1fe9bb6c7355b50a0737bc"', 'Date': 'Sun, 09 Apr 2017 10:09:57 GMT', 'X-RateLimit-Remaining': '38', 'Strict-Transport-Security': 'max-age=31536000; includeSubdomains; preload', 'Server': 'GitHub.com', 'X-GitHub-Media-Type': 'github.v3; format=json', 'X-Content-Type-Options': 'nosniff', 'Content-Encoding': 'gzip', 'Vary': 'Accept, Accept-Encoding', 'X-RateLimit-Limit': '60', 'Cache-Control': 'public, max-age=60, s-maxage=60', 'Content-Type': 'application/json; charset=utf-8', 'X-RateLimit-Reset': '1491735173'}

响应url

>>> response.url
u'https://api.github.com/'

响应history

>>> response.history    #如果没有重定向，默认为空
[]

>>> response = requests.get('http://api.github.com')    #一般访问http,会跳转到https
>>> response.history
[<Response [301]>]

响应时间

>>> response.elapsed
datetime.timedelta(0, 3, 158723)

可以调用的方法

>>> dir(response)
['__attrs__', '__bool__', '__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__getstate__', '__hash__', '__init__', '__iter__', '__module__', '__new__', '__nonzero__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setstate__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_content', '_content_consumed', 'apparent_encoding', 'close', 'connection', 'content', 'cookies', 'elapsed', 'encoding', 'headers', 'history', 'is_permanent_redirect', 'is_redirect', 'iter_content', 'iter_lines', 'json', 'links', 'ok', 'raise_for_status', 'raw', 'reason', 'request', 'status_code', 'text', 'url']

请求对象和请求头

>>> response.request
<PreparedRequest [GET]>
>>> response.request.headers
{'Connection': 'keep-alive', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'User-Agent': 'python-requests/2.6.0 CPython/2.7.5 Linux/3.10.0-327.el7.x86_64'}

请求的编码

>>> response.encoding
'utf-8'

响应的内容，content为str类型

>>> response.raw.read(10)
''
>>> 
>>> 
>>> response.content
'{"current_user_url":"https://api.github.com/user","current_user_authorizations_html_url":"https://github.com/settings/connections/applications{/client_id}","authorizations_url":"https://api.github.com/authorizations","code_search_url":"https://api.github.com/search/code?q={query}{&page,per_page,sort,order}","commit_search_url":"https://api.github.com/search/commits?q={query}{&page,per_page,sort,order}","emails_url":"https://api.github.com/user/emails","emojis_url":"https://api.github.com/emojis","events_url":"https://api.github.com/events","feeds_url":"https://api.github.com/feeds","followers_url":"https://api.github.com/user/followers","following_url":"https://api.github.com/user/following{/target}","gists_url":"https://api.github.com/gists{/gist_id}","hub_url":"https://api.github.com/hub","issue_search_url":"https://api.github.com/search/issues?q={query}{&page,per_page,sort,order}","issues_url":"https://api.github.com/issues","keys_url":"https://api.github.com/user/keys","notifications_url":"https://api.github.com/notifications","organization_repositories_url":"https://api.github.com/orgs/{org}/repos{?type,page,per_page,sort}","organization_url":"https://api.github.com/orgs/{org}","public_gists_url":"https://api.github.com/gists/public","rate_limit_url":"https://api.github.com/rate_limit","repository_url":"https://api.github.com/repos/{owner}/{repo}","repository_search_url":"https://api.github.com/search/repositories?q={query}{&page,per_page,sort,order}","current_user_repositories_url":"https://api.github.com/user/repos{?type,page,per_page,sort}","starred_url":"https://api.github.com/user/starred{/owner}{/repo}","starred_gists_url":"https://api.github.com/gists/starred","team_url":"https://api.github.com/teams","user_url":"https://api.github.com/users/{user}","user_organizations_url":"https://api.github.com/user/orgs","user_repositories_url":"https://api.github.com/users/{user}/repos{?type,page,per_page,sort}","user_search_url":"https://api.github.com/search/users?q={query}{&page,per_page,sort,order}"}'

响应内容,text为unicode类型

>>> response.text
u'{"current_user_url":"https://api.github.com/user","current_user_authorizations_html_url":"https://github.com/settings/connections/applications{/client_id}","authorizations_url":"https://api.github.com/authorizations","code_search_url":"https://api.github.com/search/code?q={query}{&page,per_page,sort,order}","commit_search_url":"https://api.github.com/search/commits?q={query}{&page,per_page,sort,order}","emails_url":"https://api.github.com/user/emails","emojis_url":"https://api.github.com/emojis","events_url":"https://api.github.com/events","feeds_url":"https://api.github.com/feeds","followers_url":"https://api.github.com/user/followers","following_url":"https://api.github.com/user/following{/target}","gists_url":"https://api.github.com/gists{/gist_id}","hub_url":"https://api.github.com/hub","issue_search_url":"https://api.github.com/search/issues?q={query}{&page,per_page,sort,order}","issues_url":"https://api.github.com/issues","keys_url":"https://api.github.com/user/keys","notifications_url":"https://api.github.com/notifications","organization_repositories_url":"https://api.github.com/orgs/{org}/repos{?type,page,per_page,sort}","organization_url":"https://api.github.com/orgs/{org}","public_gists_url":"https://api.github.com/gists/public","rate_limit_url":"https://api.github.com/rate_limit","repository_url":"https://api.github.com/repos/{owner}/{repo}","repository_search_url":"https://api.github.com/search/repositories?q={query}{&page,per_page,sort,order}","current_user_repositories_url":"https://api.github.com/user/repos{?type,page,per_page,sort}","starred_url":"https://api.github.com/user/starred{/owner}{/repo}","starred_gists_url":"https://api.github.com/gists/starred","team_url":"https://api.github.com/teams","user_url":"https://api.github.com/users/{user}","user_organizations_url":"https://api.github.com/user/orgs","user_repositories_url":"https://api.github.com/users/{user}/repos{?type,page,per_page,sort}","user_search_url":"https://api.github.com/search/users?q={query}{&page,per_page,sort,order}"}'

响应内容，为json对象

>>> response.json(
... )
{u'issues_url': u'https://api.github.com/issues', u'current_user_repositories_url': u'https://api.github.com/user/repos{?type,page,per_page,sort}', u'rate_limit_url': u'https://api.github.com/rate_limit', u'repository_search_url': u'https://api.github.com/search/repositories?q={query}{&page,per_page,sort,order}', u'user_organizations_url': u'https://api.github.com/user/orgs', u'commit_search_url': u'https://api.github.com/search/commits?q={query}{&page,per_page,sort,order}', u'repository_url': u'https://api.github.com/repos/{owner}/{repo}', u'emojis_url': u'https://api.github.com/emojis', u'hub_url': u'https://api.github.com/hub', u'keys_url': u'https://api.github.com/user/keys', u'following_url': u'https://api.github.com/user/following{/target}', u'emails_url': u'https://api.github.com/user/emails', u'authorizations_url': u'https://api.github.com/authorizations', u'code_search_url': u'https://api.github.com/search/code?q={query}{&page,per_page,sort,order}', u'followers_url': u'https://api.github.com/user/followers', u'public_gists_url': u'https://api.github.com/gists/public', u'organization_url': u'https://api.github.com/orgs/{org}', u'gists_url': u'https://api.github.com/gists{/gist_id}', u'feeds_url': u'https://api.github.com/feeds', u'user_search_url': u'https://api.github.com/search/users?q={query}{&page,per_page,sort,order}', u'user_url': u'https://api.github.com/users/{user}', u'events_url': u'https://api.github.com/events', u'organization_repositories_url': u'https://api.github.com/orgs/{org}/repos{?type,page,per_page,sort}', u'current_user_url': u'https://api.github.com/user', u'issue_search_url': u'https://api.github.com/search/issues?q={query}{&page,per_page,sort,order}', u'notifications_url': u'https://api.github.com/notifications', u'starred_url': u'https://api.github.com/user/starred{/owner}{/repo}', u'starred_gists_url': u'https://api.github.com/gists/starred', u'current_user_authorizations_html_url': u'https://github.com/settings/connections/applications{/client_id}', u'user_repositories_url': u'https://api.github.com/users/{user}/repos{?type,page,per_page,sort}', u'team_url': u'https://api.github.com/teams'}

>>> response.json()['team_url']   #访问json的对象
u'https://api.github.com/teams'

九、下载图片/文件

图片/文件下载过程

浏览器模拟-->构建request(调试策略)-->读取流data-->打开文件，存入数据

有一些返回的context的内容是没有办法读取的，只有通过流的方式保存到文件中

import requests
def download_image():
    """
    demo:下载图片
    :return:
    """
    url= "http://img0.tech2ipo.com/upload/img/article/2015/06/1434420238601.png"
    headers={'User-Agent':'Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36'}
    response=requests.get(url,headers=headers,stream=True)
    with open('demo.png','wb') as fd:
        for chunk in response.iter_content(128):
            fd.write(chunk)
    print response.status_code
    # print  response.reason
    # print  response.content

使用上下午文管理的方式，读物数据流，写入文件，最后关闭流

def download_image_improved():
    """
    demo:下载图片
    :return:
    """
    #伪造headers信息
    headers={'User-Agent':'Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36'}
    #限定url
    url= "http://img0.tech2ipo.com/upload/img/article/2015/06/1434420238601.png"
    response=requests.get(url,headers=headers,stream=True)
    from contextlib import closing
    with  closing(requests.get(url,headers=headers,stream=True)) as response:
        #打开文件
        with open('demo.png','wb') as fd:
            #每128写入一次
            for chunk in response.iter_content(128):
                fd.write(chunk)

十、事件钩子(Event hooks)

类似于js中的回调函数；

IO请求-->response-->回调函数

说白了，就是异步请求，request发出之后，直接返回，剩余的IO请求，由回调函数处理;

异步处理demo

import requests

def get_key_info(response,*args,**kwargs):
    """
    回调函数
    :return:
    """
    print response.headers['Content-Type']

def main():
    """
    主程序
    :return:
    """
    requests.get('https://api.github.com',hooks=dict(response=get_key_info))

main()

执行结果:
application/json; charset=utf-8

十一、HTTP认证

1、http基本认证

requests a protected resource-->requests a username:password-->sends username:password-->returns requests resource

import requests

BASE_URL='https://api.github.com'

def construct_url(end_point):
    return '/'.join([BASE_URL,end_point])

def basic_auth():
    """
    基本认证
    :return:
    """

    response=requests.get(construct_url('user'),auth=('imoocdeo','imoocdeo123'))
    print response.text
    print response.request.headers

basic_auth()

打印结果:
{"message":"Bad credentials","documentation_url":"https://developer.github.com/v3"}
{'Authorization': 'Basic aW1vb2NkZW86aW1vb2NkZW8xMjM=', 'Connection': 'keep-alive', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'User-Agent': 'python-requests/2.10.0'}
这里字符串 aW1vb2NkZW86aW1vb2NkZW8xMjM=是使用base64编码的结果

在终端打开bpython(pip install bpython安装)

>>> import base64
>>> 
>>> base64.b64decode('aW1vb2NkZW86aW1vb2NkZW8xMjM=')
'imoocdeo:imoocdeo123'    #是用户名和密码拼凑起来的,但是这样可能不安全

2、OAUTH认证

https://www.codewars.com/ 使用的登录认证方式

APP---github

github生成access_token: https://github.com/settings/tokens-->Generate new token-->选择该token拥有的权限

def basic_oauth():
    headers={'Authorization':'token 51684c8452ff83556f1a6705d2d85d7f68cd5ca7'}
    #user/emails,通过请求头把token带过去
    response=requests.get(construct_url('user/emails'),headers=headers)
    print response.request.headers
    print response.text
    print response.status_code
basic_oauth()

打印结果:
{'Authorization': 'token 51684c8452ff83556f1a6705d2d85d7f68cd5ca7', 'Connection': 'keep-alive', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'User-Agent': 'python-requests/2.10.0'}
[{"email":"qq_c123@163.com","primary":false,"verified":false,"visibility":null},{"email":"charles_ali@qq.com","primary":true,"verified":true,"visibility":"public"},{"email":"qq_c1231@163.com","primary":false,"verified":false,"visibility":null}]
200

使用python自带的语法糖类实现上述内容,__call__可以将实例化的对象当做类进行实例化调用,即将实例化的对象在加()，就可以调用__call__函数,f()();

from requests.auth import AuthBase

class GithubAuth(AuthBase):

    def __init__(self,token):
        self.token=token

    def __call__(self, r):
        #request 加headers
        r.headers['Authorization']=' '.join(['token',self.token])
        print r
        return r

def oauth_advaced():
    auth=GithubAuth('51684c8452ff83556f1a6705d2d85d7f68cd5ca7')
    response=requests.get(construct_url('user/emails'),auth=auth)
    print response.text

oauth_advaced()

调用结果:
<PreparedRequest [GET]>
[{"email":"qq_c123@163.com","primary":false,"verified":false,"visibility":null},{"email":"charles_ali@qq.com","primary":true,"verified":true,"visibility":"public"},{"email":"qq_c1231@163.com","primary":false,"verified":false,"visibility":null}]

十二、requests库的proxy代理

import requests
proxies={'http':'sock5://127.0.0.1:1080','https':'sock5://127.0.0.1:1080'}    #设置代理地址

url='https://www.facebook.com'

response=requests.get(url,proxies=proxies,timeout=10)

十三、Session和cookie

cookie:

浏览器请求(无cookie)-->服务器-->HTTP响应(响应头有set cookie,设置有效期，权限范围等等)-->解析cookie保存本地-->HTTP请求(携带cookie)--->解析cookie识别信息--HTTP响应

缺点:cookie是在浏览器端的，每次请求，都要携带cookie，消耗带宽;cookie可以伪造,不安全;

session:

浏览器HTTP请求-->服务器存储session-->HTTP响应(set cookie-session-id 叫做session id的cookie)-->解析cookie保存本地(这个cookie非常小,仅含有session id)-->解析session id(到存储中查，该session id对于的用户信息，权限信息是什么)-->HTTP响应

posted on 2017-04-09 17:55 阿里山QQ 阅读(338) 评论(0) 编辑收藏举报

会员力量，点亮园子希望

刷新页面返回顶部

阿里山QQ

导航

公告

requests库