爬虫--处理复杂请求

设置请求头:在reqoests，设置请求头的方式非常简单，只需要在调用请求函数时为headers参数传入定制好的请求头即可，一般是将请求头中的字段与值分别作为字典的键与值，以字典的形式穿个headers参数

例:

import requests
base_url="https://www.baidu.com/"
header = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36'}
response = requests.get(url=base_url,headers=header)#根基URL和请求头构造请求，发送GET请求，接收服务器返回的响应
response.encoding ='utf-8'
print(response.text)

　　

二，验证Cookie

网站辨别是不是真人

requests库h中，发送请求时一帮直接将包含Cookie信息的请求头传入请求函数的headers参数

import requests
base_url="https://www.baidu.com/"
header = {
    'Cookie':'buvid3=3286CE99-2276-1C84-C92B-4FE5A92EFE1E13855infoc; b_nut=1695113113; _uuid=4EAFDCED-D3FD-B8B3-96BA-DB79733CAAF709503infoc; buvid_fp=7b41b51ef2328cd3a92ce25a642b7fec; buvid4=E685A39D-7041-2203-3F28-E71CDF75D58D15401-023091916-Aewihy+MgAYMoTiDBmuwDxVWh4LFtNEtwavotmzDAL9mfZuCid9n2g%3D%3D; sid=pocw6zle; innersign=0; b_lsid=6F8155BD_18B0E9579F4; bsource=search_baidu; header_theme_version=CLOSE; home_feed_column=4; browser_resolution=982-739',
    'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36'}
response = requests.get(url=base_url,headers=header)
response.encoding ='utf-8'
print(response.text)