curl_cffi 异步使用完整详解

一、基础介绍

curl_cffi 基于 libcurl 与 Chrome/Firefox TLS 指纹，解决 requests/httpx 指纹被识别问题，同时提供同步 AsyncSession、原生异步 AsyncSession 两套 API，异步版本依托 Python asyncio，性能远高于同步并发。

核心优势

完美模拟浏览器 TLS/HTTP2/ja3 指纹，反爬友好
异步 IO 基于 libcurl 多句柄，并发量大、内存占用低
自动携带完整浏览器 headers、cookie 持久化
支持代理、上传下载、websocket、拦截重定向

安装

pip install curl-cffi

二、核心异步对象：AsyncSession

AsyncSession 是异步主入口，必须配合 async/await，上下文管理器 async with 自动释放资源。

关键参数（初始化）

from curl_cffi import AsyncSession

async with AsyncSession(
    # 指纹浏览器版本（必选）
    impersonate="chrome124",
    # 超时
    timeout=30,
    # 是否开启http2
    http2=True,
    # 代理
    proxies={"http": "http://127.0.0.1:7890", "https": "http://127.0.0.1:7890"},
    # 自定义请求头
    headers={"Referer": "https://xxx.com"},
    # 自动保存cookie到文件
    cookie_file="cookies.json",
    # 禁止自动重定向
    allow_redirects=True,
    # 自定义curl底层参数
    curl_options={}
) as session:
    resp = await session.get("https://httpbin.org/get")

常用 impersonate 值：chrome120/chrome124/firefox122/safari17.4

三、基础异步请求示例

1. GET 请求（最简）

import asyncio
from curl_cffi import AsyncSession

async def demo_get():
    async with AsyncSession(impersonate="chrome124") as session:
        resp = await session.get("https://httpbin.org/get")
        print(resp.status_code)
        print(resp.json())  # json解析
        print(resp.text)    # 文本
        print(resp.content) # 二进制

if __name__ == "__main__":
    asyncio.run(demo_get())

2. POST 表单 / JSON

async def demo_post():
    async with AsyncSession(impersonate="chrome124") as session:
        # json参数
        resp1 = await session.post(
            "https://httpbin.org/post",
            json={"username": "test", "pwd": "123456"}
        )
        # form表单
        resp2 = await session.post(
            "https://httpbin.org/post",
            data={"name": "curl_cffi"}
        )
        print(resp1.json())

asyncio.run(demo_post())

3. URL 参数、自定义 Headers、Cookie

async def demo_params():
    headers = {
        "User-Agent": "self-custom-ua",
        "Accept-Language": "zh-CN,zh;q=0.9"
    }
    cookies = {"token": "abc123xyz"}
    params = {"page": 1, "size": 10}

    async with AsyncSession(impersonate="chrome124") as session:
        resp = await session.get(
            "https://httpbin.org/get",
            params=params,
            headers=headers,
            cookies=cookies
        )
        print(resp.json()["args"])

四、高并发异步批量请求（核心场景）

利用 asyncio.gather 实现批量并发，不要创建多个 AsyncSession，复用单个 session 最优，libcurl 内部自动多路复用连接。

import asyncio
from curl_cffi import AsyncSession

async def fetch(session, url):
    resp = await session.get(url, timeout=10)
    return url, resp.status_code

async def batch_request():
    urls = [
        "https://httpbin.org/get",
        "https://httpbin.org/headers",
        "https://httpbin.org/ip",
    ]
    async with AsyncSession(impersonate="chrome124") as session:
        # 构建任务列表
        tasks = [fetch(session, url) for url in urls]
        # 并发执行
        results = await asyncio.gather(*tasks)
    for url, code in results:
        print(f"{url} -> {code}")

asyncio.run(batch_request())

并发限流（防止请求爆炸）

使用 asyncio.Semaphore 控制同时并发数量：

async def fetch_limit(session, url, sem):
    async with sem:  # 信号量控制并发
        resp = await session.get(url)
        return url, resp.status_code

async def limited_batch():
    sem = asyncio.Semaphore(5)  # 最多5个并发
    urls = ["https://httpbin.org/get"] * 20
    async with AsyncSession(impersonate="chrome124") as session:
        tasks = [fetch_limit(session, u, sem) for u in urls]
        res = await asyncio.gather(*tasks)
        print(len(res))

asyncio.run(limited_batch())

五、Cookie 持久化与操作

1. 自动持久化到文件

初始化 cookie_file="cookies.json"，session 关闭自动写入，下次启动自动加载。

async with AsyncSession(impersonate="chrome124", cookie_file="cookies.json") as s:
    await s.get("https://httpbin.org/cookies/set?k1=v1")

2. 手动读取 / 修改 Cookie

async def cookie_ops():
    async with AsyncSession(impersonate="chrome124") as s:
        await s.get("https://httpbin.org/cookies/set?a=1")
        # 获取所有cookie
        cookies = s.cookies
        print(dict(cookies))
        # 手动添加cookie
        s.cookies.set("token", "xxx", domain="httpbin.org")
        resp = await s.get("https://httpbin.org/cookies")
        print(resp.json())

六、代理、上传文件、二进制下载

1. 异步代理（http/socks5）

# socks5代理
proxies = {
    "http": "socks5://127.0.0.1:1080",
    "https": "socks5://127.0.0.1:1080"
}
async with AsyncSession(impersonate="chrome124", proxies=proxies) as s:
    resp = await s.get("https://httpbin.org/ip")
    print(resp.json())

2. 异步文件上传

async def upload():
    async with AsyncSession(impersonate="chrome124") as s:
        files = {"file": open("test.txt", "rb")}
        resp = await s.post("https://httpbin.org/post", files=files)
        print(resp.json())

3. 流式大文件下载（避免一次性读入内存）

resp.iter_content() 异步迭代二进制流：

async def download_file():
    async with AsyncSession(impersonate="chrome124") as s:
        resp = await s.get("https://example.com/test.zip", stream=True)
        with open("save.zip", "wb") as f:
            async for chunk in resp.iter_content(chunk_size=8192):
                f.write(chunk)

七、底层 CURL 参数自定义（curl_options）

可传入 libcurl 原生常量，精细控制网络行为，需要导入 curl_cffi.const.CurlOpt

from curl_cffi.const import CurlOpt

async def custom_curl_opt():
    opts = {
        CurlOpt.SSL_VERIFYPEER: 0,  # 关闭ssl证书校验（测试用，生产不推荐）
        CurlOpt.CONNECTTIMEOUT: 5,
        CurlOpt.MAX_RECV_SPEED_LARGE: 1024 * 1024,  # 限速1MB/s
    }
    async with AsyncSession(impersonate="chrome124", curl_options=opts) as s:
        await s.get("https://httpbin.org/get")

八、异常捕获与错误处理

异步请求常见异常：CurlError、超时、连接失败

from curl_cffi import AsyncSession, CurlError
import asyncio

async def safe_request(url):
    try:
        async with AsyncSession(impersonate="chrome124", timeout=8) as s:
            resp = await s.get(url)
            resp.raise_for_status()  # 4xx/5xx抛异常
            return resp.json()
    except CurlError as e:
        print(f"Curl底层错误: {e.code}, {e.message}")
    except asyncio.TimeoutError:
        print("请求超时")
    except Exception as e:
        print(f"其他异常: {repr(e)}")

asyncio.run(safe_request("https://invalid.example"))

九、异步关键注意事项（避坑）

AsyncSession 复用原则

不要循环创建 AsyncSession，单个 session 可复用发起成千上万个请求，内部连接池自动复用，频繁新建会大幅损耗性能。
必须使用 async with

若手动创建 session = AsyncSession()，结束必须 await session.close()，否则 libcurl 句柄泄漏，内存暴涨。
# 不推荐写法（需手动关闭） session = AsyncSession(impersonate="chrome124") resp = await session.get(url) await session.close()
impersonate 指纹全局绑定 Session

一个 AsyncSession 只能用一种浏览器指纹；需要多种指纹并发时，创建多个独立 AsyncSession。
并发数量不要无限开大

libcurl 默认句柄池有限，上万并发建议拆分任务 + 信号量限流，否则会出现连接排队、超时。
stream 流式下载必须开启 stream=True

大文件不使用 stream 会一次性将完整响应加载进内存，极易 OOM。
与 httpx.AsyncClient 对比
- curl_cffi：底层 libcurl，完美指纹，反爬强；并发配置略复杂
- httpx：纯 Python 异步，易用，但 TLS 指纹容易被识别封禁

十、完整工业级异步爬虫模板（带限流、异常、cookie）

import asyncio
from curl_cffi import AsyncSession, CurlError

SEMAPHORE_LIMIT = 8  # 最大并发数
SEMAPHORE = asyncio.Semaphore(SEMAPHORE_LIMIT)

async def task_handler(session: AsyncSession, url: str):
    async with SEMAPHORE:
        try:
            resp = await session.get(url, timeout=12)
            resp.raise_for_status()
            data = resp.json()
            print(f"成功 {url}: {data['origin']}")
            return data
        except CurlError as e:
            print(f"curl错误 {url}: {e}")
        except Exception as e:
            print(f"失败 {url}: {repr(e)}")
        return None

async def main():
    target_urls = ["https://httpbin.org/get"] * 30
    # 全局session，持久cookie
    async with AsyncSession(
        impersonate="chrome124",
        cookie_file="spider_cookies.json",
        http2=True
    ) as session:
        tasks = [task_handler(session, u) for u in target_urls]
        await asyncio.gather(*tasks)

if __name__ == "__main__":
    asyncio.run(main())

十一、拓展：异步 Websocket

curl_cffi AsyncSession 支持 websocket 异步通信：

async def ws_demo():
    async with AsyncSession(impersonate="chrome124") as s:
        ws = await s.ws_connect("wss://echo.websocket.org")
        await ws.send_str("hello curl_cffi async")
        msg = await ws.receive()
        print(msg.data)
        await ws.close()

posted @ 2026-07-12 20:51 chenlight 阅读(45) 评论(0) 收藏举报

刷新页面返回顶部

chenlight