协程

1.什么是协程

三要素：事件循环 + 回调 (驱动生成器) + epoll (IO多路复用)

协程就是，通过一个线程实现代码块之间互相切换

实现协程有这么几种方法

1.greenlet,早期模块
2.yield关键字
3.asyncio模块（python3.4引入）
4.async wait (python3.5引入)【推荐】

asyncio遇到IO阻塞自动切换

事件循环

# 伪代码
任务列表 = [ 任务1, 任务2, 任务3,... ]
while True:
    可执行的任务列表，已完成的任务列表 = 去任务列表中检查所有的任务，将'可执行'和'已完成'的任务返回
    for 就绪任务 in 已准备就绪的任务列表:
        执行已就绪的任务
    for 已完成的任务 in 已完成的任务列表:
        在任务列表中移除 已完成的任务
    如果 任务列表 中的任务都已完成，则终止循环

import asyncio
# 1.去生成或去获取一个事件循环
loop = asyncio.get_event_loop()
# 2.讲任务放到`任务列表`
loop.run_until_complete(任务)

协程函数和协程对象★

协程函数，定义形式为 async def function的函数。

coroutine func返回的是coroutine object

# 定义一个【协程函数】
async def func():
    pass
  
# 调用协程函数，返回一个【协程对象】
# 注：函数内部代码不会执行的
result = func()

协程函数+协程对象+事件循环=运行起来

程序中，如果想要执行协程函数的内部代码，需要 事件循环 和 协程对象 配合才能实现

coroutine只有变成了task才开始被执行，asyncio.gather(*[coroutine,coroutine])或asyncio.create_task(coroutine)

await coroutine不会去创建task(不会交出控制权)，只会等待这个coroutine返回

import asyncio
async def func():
    print("协程内部代码")
# 调用协程函数，返回一个协程对象。
result = func()
# 方式一
loop = asyncio.get_event_loop() # 创建一个事件循环
loop.run_until_complete(result) # 将协程当做任务提交到事件循环的任务列表中，协程执行完成之后终止。

# 方式二
# 本质上方式一是一样的，内部先 创建事件循环 然后执行 run_until_complete，一个简便的写法。
# asyncio.run 函数在 Python 3.7 中加入 asyncio 模块，
asyncio.run(result)

asyncio.run和asyncio.get_event_loop区别

asyncio.run

asyncio.run是Python 3.7新加的接口

loop = events.new_event_loop()
loop.run_until_complete(loop.shutdown_asyncgens())

import asyncio

async def a():
    print('Suspending a')
    await asyncio.sleep(0)
    print('Resuming a')

async def b():
    print('In b')

async def main():
    await asyncio.gather(a(), b())

if __name__ == '__main__':
    asyncio.run(main())

asyncio.get_event_loop

asyncio.get_event_loop

import asyncio

async def a():
    print('Suspending a')
    await asyncio.sleep(0)
    print('Resuming a')

async def b():
    print('In b')

async def main():
    await asyncio.gather(a(), b())

if __name__ == '__main__':
  loop = asyncio.get_event_loop()
  loop.run_until_complete(main())
  loop.close()

await

await+等待对象(协程对象+Future对象+Task对象 -> IO等待)

await后面是coroutine func的时候会将coroutine func变为task

await后跟协程函数

运行时长3秒

原理:一开始只有main一个task，碰到await say_after(1, 'hello')后把它变为task运行，现在有2个task了，loop发现main需要等say_after(1, 'hello')运行完后再执行，loop就等待1秒。然后碰到await say_after(2, 'world')后把它变为task运行，现在有2个task了,loop发现main需要等say_after(2, 'world')运行完后再执行，loop等待了2秒后发现只有一个main了，接着把main也运行完成了

import asyncio
import time

async def say_after(delay, msg):
  await asyncio.sleep(delay)
	print(msg)

async def main():
	print(f'start {time.strftime("%X")}')
  await say_after(1, 'hello')
  await say_after(2, 'world')
	print(f'end {time.strftime("%X")}')
	
asyncio.run(result)

await后跟协程对象

运行时长2秒

原理: 一开始只有main一个task，main里面注册了say_1和say_2两个task，那么一个三个task，main需要等待say_1和say_2执行完成，在执行await say_1里碰到asyncio.sleep(1)，说要等待1秒后执行完成。loop把控制权交给了await say_2，说要等待2秒后执行完成。say_1和say_2两个task一起执行了,等待执行完成之后，main继续执行

import asyncio
import time

async def say_after(delay, msg):
  await asyncio.sleep(delay)
	print(msg)

async def main():
  say_1 = asyncio.create_task(say_after(1, 'hello'))
  say_2 = asyncio.create_task(say_after(2, 'world'))
	print(f'start {time.strftime("%X")}')
  await say_1
  await say_2
	print(f'end {time.strftime("%X")}')
	
asyncio.run(result)

await接收返回值

import asyncio
import time

async def say_after(delay, msg):
  await asyncio.sleep(delay)
	retun f"{msg} - {delay}"

async def main():
  say_1 = asyncio.create_task(say_after(1, 'hello'))
  say_2 = asyncio.create_task(say_after(2, 'world'))
	print(f'start {time.strftime("%X")}')
  res = await say_1
  res = await say_2
  print(res_1)
  print(res_1)
	print(f'end {time.strftime("%X")}')
	
asyncio.run(result)

使用gather

gather里可以放一个coroutine或者task，coroutine的话会转换成task，会返回一个list

import asyncio
import time

async def say_after(delay, msg):
  await asyncio.sleep(delay)
	retun f"{msg} - {delay}"

async def main():
  say_1 = asyncio.create_task(say_after(1, 'hello'))
  say_2 = asyncio.create_task(say_after(2, 'world'))
	print(f'start {time.strftime("%X")}')
  res = await asyncio.gather(say_1, say_2)
  print(res)
	print(f'end {time.strftime("%X")}')
	
asyncio.run(result)

把coroutine的话会转换成task

import asyncio
import time

async def say_after(delay, msg):
  await asyncio.sleep(delay)
	retun f"{msg} - {delay}"

async def main():
	print(f'start {time.strftime("%X")}')
  res = await asyncio.gather(say_after(1, 'hello'), say_after(2, 'world'))
  print(res)
	print(f'end {time.strftime("%X")}')
	
asyncio.run(result)

实例一

import asyncio

async def func():
    print("执行协程函数内部代码")
    # 遇到IO操作挂起当前协程（任务），等IO操作完成之后再继续往下执行。
    # 当前协程挂起时，事件循环可以去执行其他协程（任务）。
    response = await asyncio.sleep(2)
    print("IO请求结束，结果为：", response)
    
result = func()

asyncio.run(result)

示例二

import asyncio

async def others():
    print("start")
    await asyncio.sleep(2)
    print('end')
    return '返回值'
  
async def func():
    print("执行协程函数内部代码")
    # 遇到IO操作挂起当前协程（任务），等IO操作完成之后再继续往下执行。当前协程挂起时，事件循环可以去执行其他协程（任务）。
    response = await others()
    print("IO请求结束，结果为：", response)
    
asyncio.run(func())

示例三

await就是等待对象获取到值后再往下走

如果下面协程需要基于上面的协程结果在执行，那么就需要加上await

import asyncio

async def others():
    print("start")
    await asyncio.sleep(2)
    print('end')
    return '返回值'
  
async def func():
    print("执行协程函数内部代码")
    # 遇到IO操作挂起当前协程（任务），等IO操作完成之后再继续往下执行。当前协程挂起时，事件循环可以去执行其他协程（任务）。
    # 下面是同步执行的，response1返回完结果才执行response2的await others()
    response1 = await others()
    print("IO请求结束，结果为：", response1)
    response2 = await others()
    print("IO请求结束，结果为：", response2)
    
asyncio.run( func() )

Task对象

Tasks用于并发调度协程，通过asyncio.create_task(协程对象)的方式创建Task对象，这样可以让协程加入事件循环中等待被调度执行。除了使用 asyncio.create_task() 函数以外，还可以用低层级的 loop.create_task() 或 asyncio.ensure_future() 函数。不建议手动实例化 Task 对象。

本质上是将协程对象封装成task对象，并将协程立即加入事件循环，同时追踪协程的状态。

注意：asyncio.create_task() 函数在 Python 3.7 中被加入。在 Python 3.7 之前，可以改用低层级的 asyncio.ensure_future() 函数。

在事件循环中添加多个任务

2.初步使用

单个

import asyncio
import time

async def get_html(url):
    ss = time.time()
    print(f'start')
    await asyncio.sleep(2)		# 不能使用time.sleep(2),time是同步的
    print(f'end {time.time()-ss}')

if __name__ == '__main__':
    s = time.time()
    print(f'root run')
    loop = asyncio.get_event_loop()
    loop.run_until_complete(get_html('www.baidu.com'))
    print(f'root end {time.time() - s}')
    loop.close()
    
# root run
# start
# end 2.0063397884368896
# root end 2.0072431564331055

使用time会怎么样？

会阻塞

import asyncio
import time

async def get_html(url):
    ss = time.time()
    print(f'start')
    time.sleep(2)
    print(f'end {time.time()-ss}')
    
if __name__ == '__main__':
    s = time.time()
    print(f'root run')
    loop = asyncio.get_event_loop()
    tasks = [get_html('www.baidu.com') for i in range(2)]
    loop.run_until_complete(asyncio.wait(tasks))
    print(f'root end {time.time() - s}')
    loop.close()
    
# root run
# start (>>>会等待2秒<<<)
# end 2.0050899982452393
# start (>>>会等待2秒<<<)
# end 2.0049610137939453
# root end 4.0112340450286865

多个

import asyncio
import time

async def get_html(url):
    ss = time.time()
    print(f'start')
    await asyncio.sleep(2)
    print(f'end {time.time()-ss}')

if __name__ == '__main__':
    s = time.time()
    print(f'root run')
    loop = asyncio.get_event_loop()
    tasks = [get_html('www.baidu.com') for i in range(1_0000)]
    loop.run_until_complete(asyncio.wait(tasks))
    print(f'root end {time.time() - s}')
    loop.close()
    
# root run
# start
# start
# start ...(10000个start)
# end 2.xxx
# end 2.xxx
# end 2.xxx ...(10000个end)
# root end 2.1802661418914795

3.获取协程返回值

单个

loop.create_task() 和 asyncio.ensure_future() 是等效的，差别不大

loop.create_task()是Python 3.7新增的高阶API

asyncio.ensure_future( )

asyncio.ensure_future 内部调用了 loop.create_task( )

if loop is None:
    loop = events.get_event_loop()
task = loop.create_task(coro_or_future)

import asyncio
import time


async def get_html(url):
    ss = time.time()
    print(f'start')
    await asyncio.sleep(2)
    print(f'end {time.time() - ss}')
    return 'lll'

if __name__ == '__main__':
    s = time.time()
    print(f'root run')
    loop = asyncio.get_event_loop()
    # 使用 asyncio.ensure_future
    get_future = asyncio.ensure_future(get_html('www.baidu.com'))
    loop.run_until_complete(get_future)
    # 获取return结果
    print(get_future.result())
    print(f'root end {time.time() - s}')
    loop.close()
    
# root run
# start
# end 2.0059399604797363
# lll
# root end 2.007657289505005

loop.create_task( )

import asyncio
import time

async def get_html(url):
    ss = time.time()
    print(f'start')
    await asyncio.sleep(2)
    print(f'end {time.time() - ss}')
    return 'lll'

if __name__ == '__main__':
    s = time.time()
    print(f'root run')
    loop = asyncio.get_event_loop()
    # 使用 loop.create_task
    tesk = loop.create_task(get_html('www.baidu.com'))
    loop.run_until_complete(tesk)
    # 获取return结果
    print(tesk.result())
    print(f'root end {time.time() - s}')
    loop.close()
    
# root run
# start
# end 2.0059399604797363
# lll
# root end 2.007657289505005

4.回调函数

callback无参

import asyncio
import time


async def get_html(url):
    ss = time.time()
    print(f'start')
    await asyncio.sleep(2)
    print(f'end {time.time() - ss}')
    return 'lll'

def callback(future):
    # 获取return结果
    print('>>>', future.result())

if __name__ == '__main__':
    s = time.time()
    print(f'root run')
    loop = asyncio.get_event_loop()
    task = loop.create_task(get_html('www.baidu.com'))
    # 回调函数
    task.add_done_callback(callback)
    loop.run_until_complete(task)
    print(f'root end {time.time() - s}')
    loop.close()
    
# root run
# start
# end 2.0032451152801514
# >>> lll
# root end 2.0037331581115723

callback需要参数怎么办

需要使用partial来包装callback函数

import asyncio
import time
# 包装函数
from functools import partial

async def get_html(url):
    ss = time.time()
    print(f'start')
    await asyncio.sleep(2)
    print(f'end {time.time() - ss}')
    return 'lll'
  
# 注意参数必须要在future前面★★★★
# 参数不限制，最后一个必须是future
def callback(uuu, future):	
    # 获取return结果
    print(uuu, '>>>', future.result())

if __name__ == '__main__':
    s = time.time()
    print(f'root run')
    loop = asyncio.get_event_loop()
    task = loop.create_task(get_html('www.baidu.com'))
    # 回调函数
    task.add_done_callback(partial(callback, 'abc'))
    loop.run_until_complete(task)
    print(f'root end {time.time() - s}')
    loop.close()

# root run
# start
# end 2.0063300132751465
# abc >>> lll
# root end 2.007383108139038

5.asyncio.wait和asyncio.gather区别

gather更加高层，wait执行顺序是随机的，gather执行顺序是有序的

wait:
	在内部wait()使用一个set集合保存它创建的Task实例：
  1.因为set集合是无序的所以这也就是我们的任务不是顺序执行的原因。
  2.wait的返回值是一个元组，包括两个集合，分别表示已完成和未完成的任务。
  
gather:
  1.gather任务无法取消。
  2.返回值是一个结果列表。
  3.可以按照传入参数的 顺序，顺序输出。

wait

async.wait会返回两个值:done和pending，done为已完成的协程Task，pending为超时未完成的协程Task，需通过future.result调用Task的resul

import asyncio
import time

async def get_html(url):
    ss = time.time()
    print(f'start')
    time.sleep(2)
    print(f'end {time.time()-ss}')
    
if __name__ == '__main__':
    s = time.time()
    print(f'root run')
    loop = asyncio.get_event_loop()
    tasks = [get_html('www.baidu.com') for i in range(2)]
    loop.run_until_complete(asyncio.wait(tasks))
    print(f'root end {time.time() - s}')
    loop.close()

gather

async.gather返回的是已完成Task的result

import asyncio
import time

async def get_html(url):
    ss = time.time()
    print(f'start')
    await asyncio.sleep(2)
    print(f'end {time.time()-ss}')
    
if __name__ == '__main__':
    s = time.time()
    print(f'root run')
    loop = asyncio.get_event_loop()
    tasks = [get_html('www.baidu.com') for i in range(2)]
    loop.run_until_complete(asyncio.gather(*tasks))
    print(f'root end {time.time() - s}')
    loop.close()

gather返回值

res返回的是一个列表

import asyncio
import time

async def get_html(url, i):
    ss = time.time()
    print(f'start')
    await asyncio.sleep(2)
    print(f'end {time.time() - ss}')
    return url + f"/{i}"

async def main():
    tasks = [get_html('www.baidu.com', i) for i in range(2)]
    get_html_res = await asyncio.gather(*tasks)
    return get_html_res

if __name__ == '__main__':
    s = time.time()
    print(f'root run')
    loop = asyncio.get_event_loop()
    res = loop.run_until_complete(main())
    print(res)
    print(f'root end {time.time() - s}')
    loop.close()

root run
start
start
end 2.005063056945801
end 2.0051157474517822
['www.baidu.com/0', 'www.baidu.com/1']
root end 2.005621910095215

gather分组案列一

import asyncio
import time

async def get_html(url):
    ss = time.time()
    print(f'start')
    await asyncio.sleep(2)
    print(f'end {time.time()-ss}')
    
if __name__ == '__main__':
    s = time.time()
    print(f'root run')
    loop = asyncio.get_event_loop()
    task1 = [get_html('www.baidu.com') for i in range(2)]
    task2 = [get_html('www.qq.com') for i in range(2)]
    loop.run_until_complete(asyncio.gather(*task1,*task2))
    print(f'root end {time.time() - s}')
    loop.close()

root run
start
start
start
start
end 2.0015370845794678
end 2.0015711784362793
end 2.001573085784912
end 2.0015761852264404
root end 2.002058982849121

gather分组案列二

import asyncio
import time

async def get_html(url):
    ss = time.time()
    print(f'start')
    await asyncio.sleep(2)
    print(f'end {time.time()-ss}')
    
if __name__ == '__main__':
    s = time.time()
    print(f'root run')
    loop = asyncio.get_event_loop()
    task1 = [get_html('www.baidu.com') for i in range(2)]
    task2 = [get_html('www.qq.com') for i in range(2)]
    task1 = asyncio.gather(*task1)
    task2 = asyncio.gather(*task2)
    loop.run_until_complete(asyncio.gather(task1, task2))
    print(f'root end {time.time() - s}')
    loop.close()

root run
start
start
start
start
end 2.00620698928833
end 2.006431818008423
end 2.0064568519592285
end 2.0065090656280518
root end 2.007246255874634

gather取消任务时

取消掉，取消时，必须设置 return_exception为True，不然会抛异常

import asyncio
import time
async def get_html(url):
    print("start get url")
    await asyncio.sleep(2)
    print("end get url")

if __name__ == "__main__":
    start_time = time.time()
    loop = asyncio.get_event_loop()
    
    group1 = [get_html("http://projectsedu.com") for i in range(2)] # 分组
    group2 = [get_html("http://www.imooc.com") for i in range(2)]
    group1 = asyncio.gather(*group1)
    group2 = asyncio.gather(*group2)
    group2.cancel()
    loop.run_until_complete(asyncio.gather(group1, group2, return_exceptions=True))
    print(time.time() - start_time)

6.run_until_complete和run_forever

run_forever

使用loop.run_forever()启动无限循环时，task实例会自动加入事件循环。如果注释掉loop.stop()方法，则loop.run_forever()之后的代码永远不会被执行，因为loop.run_forever()是个无限循环。

未注释loop.stop()

import asyncio
import time
from datetime import datetime


async def work(loop, t):
    print(datetime.strftime(datetime.now(), '%Y-%m-%d %H:%M:%S'), '[work] start')
    await asyncio.sleep(t)  # 模拟IO操作
    print(datetime.strftime(datetime.now(), '%Y-%m-%d %H:%M:%S'), '[work] finished')
    loop.stop()  # 停止事件循环，stop后仍可重新运行


if __name__ == '__main__':
    loop = asyncio.get_event_loop()  # 创建任务，该任务会自动加入事件循环
    task = asyncio.ensure_future(work(loop, 1))
    print(datetime.strftime(datetime.now(), '%Y-%m-%d %H:%M:%S'), '[main]', task._state)
    loop.run_forever()  # 无限运行事件循环，直至loop.stop停止
    print(datetime.strftime(datetime.now(), '%Y-%m-%d %H:%M:%S'), '[main]', task._state)
    loop.close()  # 关闭事件循环，只有loop处于停止状态才会执行

2023-01-03 16:26:31 [main] PENDING
2023-01-03 16:26:31 [work] start
2023-01-03 16:26:32 [work] finished
2023-01-03 16:26:32 [main] FINISHED

注释loop.stop()

一直卡着，执行不下去

import asyncio
import time
from datetime import datetime


async def work(loop, t):
    print(datetime.strftime(datetime.now(), '%Y-%m-%d %H:%M:%S'), '[work] start')
    await asyncio.sleep(t)  # 模拟IO操作
    print(datetime.strftime(datetime.now(), '%Y-%m-%d %H:%M:%S'), '[work] finished')
    # loop.stop()  # 停止事件循环，stop后仍可重新运行


if __name__ == '__main__':
    loop = asyncio.get_event_loop()  # 创建任务，该任务会自动加入事件循环
    task = asyncio.ensure_future(work(loop, 1))
    print(datetime.strftime(datetime.now(), '%Y-%m-%d %H:%M:%S'), '[main]', task._state)
    loop.run_forever()  # 无限运行事件循环，直至loop.stop停止
    print(datetime.strftime(datetime.now(), '%Y-%m-%d %H:%M:%S'), '[main]', task._state)
    loop.close()  # 关闭事件循环，只有loop处于停止状态才会执行

2023-01-03 16:27:46 [main] PENDING
2023-01-03 16:27:46 [work] start
2023-01-03 16:27:47 [work] finished

run_until_complete

当我们传入指定的事件循环以后这个方法会在执行完事件循环以后停止事件循环

import asyncio

async main():
  ...

if __name__ == '__main__':
  loop = asyncio.get_event_loop()
  # 会在指定的future运行完给停止掉
  loop.run_until_complete(main())

取消future（task）

未报错

import asyncio
import time

async def get_html(ttt):
    ss = time.time()
    print(f'start')
    await asyncio.sleep(ttt)
    print(f'end {time.time() - ss}')

if __name__ == '__main__':
    tasks = [get_html(i) for i in range(1, 11)]
    loop = asyncio.get_event_loop()
    try:
        loop.run_until_complete(asyncio.wait(tasks))
    except KeyboardInterrupt as e:
        all_task = asyncio.Task.all_tasks()
        for task in all_task:
            print("cancel task")
            print(task.cancel())
        loop.stop()
        loop.run_forever() # 不写这个，contort+C终止会抛出异常
    finally:
        loop.close()

(venv) ➜  python3 tt.py
start
start
^Ctt.py:16: DeprecationWarning: Task.all_tasks() is deprecated, use asyncio.all_tasks() instead
  all_task = asyncio.Task.all_tasks()
cancel task
True
cancel task
True
cancel task
True

报错

import asyncio
import time

async def get_html(ttt):
    ss = time.time()
    print(f'start')
    await asyncio.sleep(ttt)
    print(f'end {time.time() - ss}')

if __name__ == '__main__':
    tasks = [get_html(i) for i in range(1, 11)]
    loop = asyncio.get_event_loop()
    try:
        loop.run_until_complete(asyncio.wait(tasks))
    except KeyboardInterrupt as e:
        all_task = asyncio.Task.all_tasks()
        for task in all_task:
            print("cancel task")
            print(task.cancel())
        loop.stop()
        loop.run_forever() # 不写这个，contort+C终止会抛出异常
    finally:
        loop.close()

(venv) ➜  python3 tt.py
start
start
^Ctt.py:16: DeprecationWarning: Task.all_tasks() is deprecated, use asyncio.all_tasks() instead
  all_task = asyncio.Task.all_tasks()
cancel task
True
cancel task
True
cancel task
True
Task was destroyed but it is pending!
task: <Task pending name='Task-3' coro=<get_html() done, defined at tt.py:4> wait_for=<Future cancelled> cb=[_wait.<locals>._on_completion() at /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/asyncio/tasks.py:507]>
Task was destroyed but it is pending!
task: <Task pending name='Task-2' coro=<get_html() done, defined at tt.py:4> wait_for=<Future cancelled> cb=[_wait.<locals>._on_completion() at /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/asyncio/tasks.py:507]>

7.协程当中嵌套协程

import asyncio

async def compute(x, y):
    await asyncio.sleep(1)
    return x + y

async def print_sum(x, y):
    result = await compute(x, y)
    print('{} + {} = {}'.format(x, y, result))

if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(print_sum(1, 2))
    loop.stop()