线程池
在一个池子里,放固定数量的线程,这些线程等待任务,一旦有任务来,就有线程自发的去执行任务。
from concurrent.futures import ThreadPoolExecutor,ProcessPoolExecutor
concurrent.futures 这个模块是异步调用的机制
concurrent.futures 提交任务都是用submit
for + submit 多个任务的提交
shutdown 是等效于Pool中的close+join,是指不允许再继续向池中增加任务,
然后让父进程(线程)等待池中所有进程执行完所有任务。
线程池,多进程,Pool进程效率对比
线程池: from concurrent.futures import ThreadPoolExecutor,ProcessPoolExecutor import time def func(num): sum = 0 for i in range(num): sum += i ** 2 print(sum) if __name__ == '__main__': tp = ThreadPoolExecutor(5) start = time.time() for i in range(1000): tp.submit(func,i) tp.shutdown() print(time.time()-start) ProcessPoolExecutor进程池 from concurrent.futures import ThreadPoolExecutor,ProcessPoolExecutor import time def func(num): sum = 0 for i in range(num): sum += i ** 2 print(sum) if __name__ == '__main__': tp = ProcessPoolExecutor(5) start = time.time() for i in range(1000): tp.submit(func,i) tp.shutdown() print(time.time()-start) from multiprocessing import Pool import time def func(num): sum = 0 for i in range(num): sum += i **2 print(sum) if __name__ == '__main__': p = Pool(5) start = time.time() for i in range(1000): p.apply_async(func,args=(i,)) p.close() p.join() print(time.time()-start)
结果:针对计算密集的程序来说
不管是Pool的进程池还是ProcessPoolExecutor()的进程池,执行效率相当
ThreadPoolExecutor 的效率要差很多
所以 当计算密集时,使用多进程。
多任务的提交
from concurrent.futures import ThreadPoolExecutor import time def func(num): sum = 0 for i in range(num): sum += i ** 2 print(sum) t = ThreadPoolExecutor(20) start =time.time() t.map(func,range(1000)) #map提交多个任务到线程池中,等效于for i in range(1000): tp.submit(func,i) t.shutdown() print(time.time() - start)
线程池的返回值
from concurrent.futures import ThreadPoolExecutor def func(num): sum = 0 for i in range(num): sum += i ** 2 return sum t = ThreadPoolExecutor(20) 下列代码是用for + submit提交多个任务的方式,对应拿结果的方法是result lst = [] for i in range(1000): ret = t.submit(func,i) lst.append(ret) t.shutdown() [print(i.result()) for i in lst] 在Pool进程池中拿结果,是用get方法。 在ThreadPoolExecutor里边拿结果是用result方法 下列代码是用map的方式提交多个任务, 对应 拿结果的方法是__next__() 返回的是一个生成器对象 res = t.map(func,range(1000)) t.shutdown() print(res.__next__()) print(res.__next__()) print(res.__next__()) print(res.__next__()) print(res.__next__()) print(res.__next__())
回调函数
from concurrent.futures import ThreadPoolExecutor import time def func(num): sum = 0 for i in range(num): sum += i ** 2 return sum def call_back_fun(res): print(res) print(res.result()) t = ThreadPoolExecutor(20) for i in range(1000): t.submit(func,i).add_done_callback(call_back_fun) t.shutdown() 线程池的回调函数不是父线程调用的 from concurrent.futures import ProcessPoolExecutor import os def func(num): sum = 0 for i in range(num): sum += i ** 2 return sum def call_back_fun(res): print(res.result(),os.getpid()) if __name__ == '__main__': print(os.getpid()) t = ProcessPoolExecutor(20) for i in range(1000): t.submit(func,i).add_done_callback(call_back_fun) t.shutdown() 不管是ProcessPoolExecutor的进程池 还是Pool的进程池,回调函数都是父进程调用的。