Python-多线程+多进程包（concurrent包，处理并发）

1、concurrent包

　　此包3.2版本之后引入，只提供了一个模块futures

　　异步并行任务编程模块，提供了一个高级的异步可执行的便利接口。

　　提供了两个池执行器

　　　　ThreadPoolExecutor 异步调用的线程池的 Executor

　　　　ProcessPoolExeutor 异步调用的进程池的 Executor

2、ThreadPoolExecutor 对象 -- 线程

　　首先需要定义一个池的执行器对象，Executor类子类对象。

　　Future类

　　测试：IO 密集型测试，创建一个线城池，开启三个线程，因为此模块没有提供join方法，所以通过编程来实现主线程等待。

 1 import threading
 2 from concurrent import futures
 3 
 4 import logging
 5 import time
 6 
 7 FORMAT = '%(asctime)-15s\t  %(process)s %(threadName)s %(process)s %(message)s'
 8 logging.basicConfig(level=logging.INFO, format=FORMAT)
 9 
10 # 这是一个IO 密集型的函数，建议使用多线程
11 def worker(n):
12     logging.info('begin to work{}'.format(n))
13     time.sleep(5)
14     logging.info('finisdhed{}'.format(n))
15     return n
16 
17 # 创建线程池，容量为3个
18 executor = futures.ThreadPoolExecutor(max_workers=3)
19 
20 fs = []
21 for i in range(3):
22     # 提交任务
23     future = executor.submit(worker, i)
24     fs.append(future)
25 
26 
27 while True:
28     time.sleep(1)
29     logging.info(threading.enumerate())
30 
31     flag = True
32     for f in fs:
33         logging.info(f.done())
34         flag = flag and f.done()
35 
36     if flag:
37         for f in fs:
38             logging.info('the ans is {}'.format(f.result()))
39         executor.shutdown()
40         logging.info(threading.enumerate())
41         break
42 
43 logging.info('--------------------')

线程数=池容量

 1 2018-10-30 10:29:03,486      9176 ThreadPoolExecutor-0_0 9176 begin to work0
 2 2018-10-30 10:29:03,486      9176 ThreadPoolExecutor-0_1 9176 begin to work1
 3 2018-10-30 10:29:03,486      9176 ThreadPoolExecutor-0_2 9176 begin to work2
 4 2018-10-30 10:29:04,486      9176 MainThread 9176 [<_MainThread(MainThread, started 8332)>, <Thread(ThreadPoolExecutor-0_0, started daemon 5720)>, <Thread(ThreadPoolExecutor-0_1, started daemon 5512)>, <Thread(ThreadPoolExecutor-0_2, started daemon 5944)>]
 5 2018-10-30 10:29:04,486      9176 MainThread 9176 False
 6 2018-10-30 10:29:04,486      9176 MainThread 9176 False
 7 2018-10-30 10:29:04,486      9176 MainThread 9176 False
 8 2018-10-30 10:29:05,486      9176 MainThread 9176 [<_MainThread(MainThread, started 8332)>, <Thread(ThreadPoolExecutor-0_0, started daemon 5720)>, <Thread(ThreadPoolExecutor-0_1, started daemon 5512)>, <Thread(ThreadPoolExecutor-0_2, started daemon 5944)>]
 9 2018-10-30 10:29:05,486      9176 MainThread 9176 False
10 2018-10-30 10:29:05,486      9176 MainThread 9176 False
11 2018-10-30 10:29:05,487      9176 MainThread 9176 False
12 2018-10-30 10:29:06,488      9176 MainThread 9176 [<_MainThread(MainThread, started 8332)>, <Thread(ThreadPoolExecutor-0_0, started daemon 5720)>, <Thread(ThreadPoolExecutor-0_1, started daemon 5512)>, <Thread(ThreadPoolExecutor-0_2, started daemon 5944)>]
13 2018-10-30 10:29:06,488      9176 MainThread 9176 False
14 2018-10-30 10:29:06,488      9176 MainThread 9176 False
15 2018-10-30 10:29:06,488      9176 MainThread 9176 False
16 2018-10-30 10:29:07,488      9176 MainThread 9176 [<_MainThread(MainThread, started 8332)>, <Thread(ThreadPoolExecutor-0_0, started daemon 5720)>, <Thread(ThreadPoolExecutor-0_1, started daemon 5512)>, <Thread(ThreadPoolExecutor-0_2, started daemon 5944)>]
17 2018-10-30 10:29:07,488      9176 MainThread 9176 False
18 2018-10-30 10:29:07,488      9176 MainThread 9176 False
19 2018-10-30 10:29:07,489      9176 MainThread 9176 False
20 2018-10-30 10:29:08,487      9176 ThreadPoolExecutor-0_1 9176 finisdhed1
21 2018-10-30 10:29:08,487      9176 ThreadPoolExecutor-0_0 9176 finisdhed0
22 2018-10-30 10:29:08,487      9176 ThreadPoolExecutor-0_2 9176 finisdhed2
23 2018-10-30 10:29:08,489      9176 MainThread 9176 [<_MainThread(MainThread, started 8332)>, <Thread(ThreadPoolExecutor-0_0, started daemon 5720)>, <Thread(ThreadPoolExecutor-0_1, started daemon 5512)>, <Thread(ThreadPoolExecutor-0_2, started daemon 5944)>]
24 2018-10-30 10:29:08,489      9176 MainThread 9176 True
25 2018-10-30 10:29:08,489      9176 MainThread 9176 True
26 2018-10-30 10:29:08,489      9176 MainThread 9176 True
27 2018-10-30 10:29:08,489      9176 MainThread 9176 the ans is 0
28 2018-10-30 10:29:08,489      9176 MainThread 9176 the ans is 1
29 2018-10-30 10:29:08,489      9176 MainThread 9176 the ans is 2
30 2018-10-30 10:29:08,490      9176 MainThread 9176 [<_MainThread(MainThread, started 8332)>]
31 2018-10-30 10:29:08,490      9176 MainThread 9176 --------------------

得出：线程池关闭之后，就没有线程空闲线程存在了　

　　测试：IO 密集型测试，创建一个线城池，开启一个线程，因为此模块没有提供join方法，所以通过编程来实现主线程等待。

 1 import threading
 2 from concurrent import futures
 3 
 4 import logging
 5 import time
 6 
 7 FORMAT = '%(asctime)-15s\t  %(process)s %(threadName)s %(process)s %(message)s'
 8 logging.basicConfig(level=logging.INFO, format=FORMAT)
 9 
10 # 这是一个IO 密集型的函数，建议使用多线程
11 def worker(n):
12     logging.info('begin to work{}'.format(n))
13     time.sleep(5)
14     logging.info('finisdhed{}'.format(n))
15     return n
16 
17 # 创建线程池，容量为3个
18 executor = futures.ThreadPoolExecutor(max_workers=3)
19 
20 fs = []
21 for i in range(1):
22     # 提交任务
23     future = executor.submit(worker, i)
24     fs.append(future)
25 
26 
27 while True:
28     time.sleep(1)
29     logging.info(threading.enumerate())
30 
31     flag = True
32     for f in fs:
33         logging.info(f.done())
34         flag = flag and f.done()
35 
36     if flag:
37         for f in fs:
38             logging.info('the ans is {}'.format(f.result()))
39         # logging.info(threading.enumerate())
40         # logging.info('====================')
41         executor.shutdown()
42         logging.info(threading.enumerate())
43         break
44 
45 logging.info('--------------------')

线程数 < 池容量

 1 D:\python3.7\python.exe E:/code_pycharm/tt2.py
 2 2018-10-30 10:31:01,309      8408 ThreadPoolExecutor-0_0 8408 begin to work0
 3 2018-10-30 10:31:02,309      8408 MainThread 8408 [<_MainThread(MainThread, started 8852)>, <Thread(ThreadPoolExecutor-0_0, started daemon 9200)>]
 4 2018-10-30 10:31:02,309      8408 MainThread 8408 False
 5 2018-10-30 10:31:03,309      8408 MainThread 8408 [<_MainThread(MainThread, started 8852)>, <Thread(ThreadPoolExecutor-0_0, started daemon 9200)>]
 6 2018-10-30 10:31:03,309      8408 MainThread 8408 False
 7 2018-10-30 10:31:04,309      8408 MainThread 8408 [<_MainThread(MainThread, started 8852)>, <Thread(ThreadPoolExecutor-0_0, started daemon 9200)>]
 8 2018-10-30 10:31:04,309      8408 MainThread 8408 False
 9 2018-10-30 10:31:05,309      8408 MainThread 8408 [<_MainThread(MainThread, started 8852)>, <Thread(ThreadPoolExecutor-0_0, started daemon 9200)>]
10 2018-10-30 10:31:05,309      8408 MainThread 8408 False
11 2018-10-30 10:31:06,309      8408 ThreadPoolExecutor-0_0 8408 finisdhed0
12 2018-10-30 10:31:06,309      8408 MainThread 8408 [<_MainThread(MainThread, started 8852)>, <Thread(ThreadPoolExecutor-0_0, started daemon 9200)>]
13 2018-10-30 10:31:06,309      8408 MainThread 8408 True
14 2018-10-30 10:31:06,310      8408 MainThread 8408 the ans is 0
15 2018-10-30 10:31:06,310      8408 MainThread 8408 [<_MainThread(MainThread, started 8852)>]
16 2018-10-30 10:31:06,310      8408 MainThread 8408 --------------------
17 
18 Process finished with exit code 0

得出：线程池处于懒惰模式，提交一个线程函数，就开启一个线程

　　　测试：IO 密集型测试，创建一个线城池，开启六个线程，因为此模块没有提供join方法，所以通过编程来实现主线程等待。

 1 import threading
 2 from concurrent import futures
 3 
 4 import logging
 5 import time
 6 
 7 FORMAT = '%(asctime)-15s\t  %(process)s %(threadName)s %(process)s %(message)s'
 8 logging.basicConfig(level=logging.INFO, format=FORMAT)
 9 
10 # 这是一个IO 密集型的函数，建议使用多线程
11 def worker(n):
12     logging.info('begin to work{}'.format(n))
13     time.sleep(5)
14     logging.info('finisdhed{}'.format(n))
15     return n
16 
17 # 创建线程池，容量为3个
18 executor = futures.ThreadPoolExecutor(max_workers=3)
19 
20 fs = []
21 for i in range(6):
22     # 提交任务
23     future = executor.submit(worker, i)
24     fs.append(future)
25 
26 
27 while True:
28     time.sleep(1)
29     logging.info(threading.enumerate())
30 
31     flag = True
32     for f in fs:
33         logging.info(f.done())
34         flag = flag and f.done()
35 
36     if flag:
37         for f in fs:
38             logging.info('the ans is {}'.format(f.result()))
39         # logging.info(threading.enumerate())
40         # logging.info('====================')
41         executor.shutdown()
42         logging.info(threading.enumerate())
43         break
44 
45 logging.info('--------------------')

线程数 >池容量

 1 D:\python3.7\python.exe E:/code_pycharm/tt2.py
 2 2018-10-30 10:36:29,200      7524 ThreadPoolExecutor-0_0 7524 begin to work0
 3 2018-10-30 10:36:29,201      7524 ThreadPoolExecutor-0_1 7524 begin to work1
 4 2018-10-30 10:36:29,201      7524 ThreadPoolExecutor-0_2 7524 begin to work2
 5 2018-10-30 10:36:30,201      7524 MainThread 7524 [<_MainThread(MainThread, started 8852)>, <Thread(ThreadPoolExecutor-0_0, started daemon 8332)>, <Thread(ThreadPoolExecutor-0_1, started daemon 7056)>, <Thread(ThreadPoolExecutor-0_2, started daemon 5080)>]
 6 2018-10-30 10:36:30,201      7524 MainThread 7524 False
 7 2018-10-30 10:36:30,201      7524 MainThread 7524 False
 8 2018-10-30 10:36:30,202      7524 MainThread 7524 False
 9 2018-10-30 10:36:30,202      7524 MainThread 7524 False
10 2018-10-30 10:36:30,202      7524 MainThread 7524 False
11 2018-10-30 10:36:30,202      7524 MainThread 7524 False
12 2018-10-30 10:36:31,204      7524 MainThread 7524 [<_MainThread(MainThread, started 8852)>, <Thread(ThreadPoolExecutor-0_0, started daemon 8332)>, <Thread(ThreadPoolExecutor-0_1, started daemon 7056)>, <Thread(ThreadPoolExecutor-0_2, started daemon 5080)>]
13 2018-10-30 10:36:31,204      7524 MainThread 7524 False
14 2018-10-30 10:36:31,204      7524 MainThread 7524 False
15 2018-10-30 10:36:31,204      7524 MainThread 7524 False
16 2018-10-30 10:36:31,204      7524 MainThread 7524 False
17 2018-10-30 10:36:31,204      7524 MainThread 7524 False
18 2018-10-30 10:36:31,204      7524 MainThread 7524 False
19 2018-10-30 10:36:32,204      7524 MainThread 7524 [<_MainThread(MainThread, started 8852)>, <Thread(ThreadPoolExecutor-0_0, started daemon 8332)>, <Thread(ThreadPoolExecutor-0_1, started daemon 7056)>, <Thread(ThreadPoolExecutor-0_2, started daemon 5080)>]
20 2018-10-30 10:36:32,204      7524 MainThread 7524 False
21 2018-10-30 10:36:32,204      7524 MainThread 7524 False
22 2018-10-30 10:36:32,204      7524 MainThread 7524 False
23 2018-10-30 10:36:32,204      7524 MainThread 7524 False
24 2018-10-30 10:36:32,204      7524 MainThread 7524 False
25 2018-10-30 10:36:32,204      7524 MainThread 7524 False
26 2018-10-30 10:36:33,204      7524 MainThread 7524 [<_MainThread(MainThread, started 8852)>, <Thread(ThreadPoolExecutor-0_0, started daemon 8332)>, <Thread(ThreadPoolExecutor-0_1, started daemon 7056)>, <Thread(ThreadPoolExecutor-0_2, started daemon 5080)>]
27 2018-10-30 10:36:33,204      7524 MainThread 7524 False
28 2018-10-30 10:36:33,204      7524 MainThread 7524 False
29 2018-10-30 10:36:33,204      7524 MainThread 7524 False
30 2018-10-30 10:36:33,205      7524 MainThread 7524 False
31 2018-10-30 10:36:33,205      7524 MainThread 7524 False
32 2018-10-30 10:36:33,206      7524 MainThread 7524 False
33 2018-10-30 10:36:34,200      7524 ThreadPoolExecutor-0_0 7524 finisdhed0
34 2018-10-30 10:36:34,200      7524 ThreadPoolExecutor-0_0 7524 begin to work3
35 2018-10-30 10:36:34,201      7524 ThreadPoolExecutor-0_1 7524 finisdhed1
36 2018-10-30 10:36:34,201      7524 ThreadPoolExecutor-0_1 7524 begin to work4
37 2018-10-30 10:36:34,201      7524 ThreadPoolExecutor-0_2 7524 finisdhed2
38 2018-10-30 10:36:34,201      7524 ThreadPoolExecutor-0_2 7524 begin to work5
39 2018-10-30 10:36:34,206      7524 MainThread 7524 [<_MainThread(MainThread, started 8852)>, <Thread(ThreadPoolExecutor-0_0, started daemon 8332)>, <Thread(ThreadPoolExecutor-0_1, started daemon 7056)>, <Thread(ThreadPoolExecutor-0_2, started daemon 5080)>]
40 2018-10-30 10:36:34,206      7524 MainThread 7524 True
41 2018-10-30 10:36:34,206      7524 MainThread 7524 True
42 2018-10-30 10:36:34,206      7524 MainThread 7524 True
43 2018-10-30 10:36:34,206      7524 MainThread 7524 False
44 2018-10-30 10:36:34,206      7524 MainThread 7524 False
45 2018-10-30 10:36:34,206      7524 MainThread 7524 False
46 2018-10-30 10:36:35,206      7524 MainThread 7524 [<_MainThread(MainThread, started 8852)>, <Thread(ThreadPoolExecutor-0_0, started daemon 8332)>, <Thread(ThreadPoolExecutor-0_1, started daemon 7056)>, <Thread(ThreadPoolExecutor-0_2, started daemon 5080)>]
47 2018-10-30 10:36:35,206      7524 MainThread 7524 True
48 2018-10-30 10:36:35,206      7524 MainThread 7524 True
49 2018-10-30 10:36:35,206      7524 MainThread 7524 True
50 2018-10-30 10:36:35,206      7524 MainThread 7524 False
51 2018-10-30 10:36:35,206      7524 MainThread 7524 False
52 2018-10-30 10:36:35,206      7524 MainThread 7524 False
53 2018-10-30 10:36:36,206      7524 MainThread 7524 [<_MainThread(MainThread, started 8852)>, <Thread(ThreadPoolExecutor-0_0, started daemon 8332)>, <Thread(ThreadPoolExecutor-0_1, started daemon 7056)>, <Thread(ThreadPoolExecutor-0_2, started daemon 5080)>]
54 2018-10-30 10:36:36,206      7524 MainThread 7524 True
55 2018-10-30 10:36:36,206      7524 MainThread 7524 True
56 2018-10-30 10:36:36,206      7524 MainThread 7524 True
57 2018-10-30 10:36:36,206      7524 MainThread 7524 False
58 2018-10-30 10:36:36,206      7524 MainThread 7524 False
59 2018-10-30 10:36:36,206      7524 MainThread 7524 False
60 2018-10-30 10:36:37,206      7524 MainThread 7524 [<_MainThread(MainThread, started 8852)>, <Thread(ThreadPoolExecutor-0_0, started daemon 8332)>, <Thread(ThreadPoolExecutor-0_1, started daemon 7056)>, <Thread(ThreadPoolExecutor-0_2, started daemon 5080)>]
61 2018-10-30 10:36:37,206      7524 MainThread 7524 True
62 2018-10-30 10:36:37,206      7524 MainThread 7524 True
63 2018-10-30 10:36:37,206      7524 MainThread 7524 True
64 2018-10-30 10:36:37,207      7524 MainThread 7524 False
65 2018-10-30 10:36:37,207      7524 MainThread 7524 False
66 2018-10-30 10:36:37,207      7524 MainThread 7524 False
67 2018-10-30 10:36:38,207      7524 MainThread 7524 [<_MainThread(MainThread, started 8852)>, <Thread(ThreadPoolExecutor-0_0, started daemon 8332)>, <Thread(ThreadPoolExecutor-0_1, started daemon 7056)>, <Thread(ThreadPoolExecutor-0_2, started daemon 5080)>]
68 2018-10-30 10:36:38,207      7524 MainThread 7524 True
69 2018-10-30 10:36:38,209      7524 MainThread 7524 True
70 2018-10-30 10:36:38,210      7524 MainThread 7524 True
71 2018-10-30 10:36:38,210      7524 MainThread 7524 False
72 2018-10-30 10:36:38,210      7524 MainThread 7524 False
73 2018-10-30 10:36:38,210      7524 MainThread 7524 False
74 2018-10-30 10:36:39,200      7524 ThreadPoolExecutor-0_0 7524 finisdhed3
75 2018-10-30 10:36:39,201      7524 ThreadPoolExecutor-0_1 7524 finisdhed4
76 2018-10-30 10:36:39,201      7524 ThreadPoolExecutor-0_2 7524 finisdhed5
77 2018-10-30 10:36:39,210      7524 MainThread 7524 [<_MainThread(MainThread, started 8852)>, <Thread(ThreadPoolExecutor-0_0, started daemon 8332)>, <Thread(ThreadPoolExecutor-0_1, started daemon 7056)>, <Thread(ThreadPoolExecutor-0_2, started daemon 5080)>]
78 2018-10-30 10:36:39,210      7524 MainThread 7524 True
79 2018-10-30 10:36:39,210      7524 MainThread 7524 True
80 2018-10-30 10:36:39,211      7524 MainThread 7524 True
81 2018-10-30 10:36:39,211      7524 MainThread 7524 True
82 2018-10-30 10:36:39,211      7524 MainThread 7524 True
83 2018-10-30 10:36:39,211      7524 MainThread 7524 True
84 2018-10-30 10:36:39,211      7524 MainThread 7524 the ans is 0
85 2018-10-30 10:36:39,212      7524 MainThread 7524 the ans is 1
86 2018-10-30 10:36:39,212      7524 MainThread 7524 the ans is 2
87 2018-10-30 10:36:39,212      7524 MainThread 7524 the ans is 3
88 2018-10-30 10:36:39,212      7524 MainThread 7524 the ans is 4
89 2018-10-30 10:36:39,212      7524 MainThread 7524 the ans is 5
90 2018-10-30 10:36:39,213      7524 MainThread 7524 [<_MainThread(MainThread, started 8852)>]
91 2018-10-30 10:36:39,213      7524 MainThread 7524 --------------------
92 
93 Process finished with exit code 0

得出：三个线程，只要有一个结束，就开启第四个线程

3、ProcessPoolExecutor 对象 -- 进程

　　方法和线程一样，就是线程变为进程

 1 import threading
 2 from concurrent import futures
 3 
 4 import logging
 5 import time
 6 
 7 FORMAT = '%(asctime)-15s\t  %(process)s %(processName)s   %(message)s'
 8 logging.basicConfig(level=logging.INFO, format=FORMAT)
 9 
10 # 这是一个IO 密集型的函数，建议使用多线程
11 def worker(n):
12     logging.info('begin to work{}'.format(n))
13     time.sleep(5)
14     logging.info('finisdhed{}'.format(n))
15     return n
16 
17 # 创建线程池，容量为3个
18 # executor = futures.ThreadPoolExecutor(max_workers=3)
19 
20 if __name__ == '__main__':
21     executor = futures.ProcessPoolExecutor(max_workers=3)
22     fs = []
23     for i in range(3):
24         # 提交任务
25         future = executor.submit(worker, i)
26         fs.append(future)
27 
28 
29     while True:
30         time.sleep(1)
31         logging.info(threading.enumerate())
32 
33         flag = True
34         for f in fs:
35             logging.info(f.done())
36             flag = flag and f.done()
37 
38         if flag:
39             for f in fs:
40                 logging.info('the ans is {}'.format(f.result()))
41             # logging.info(threading.enumerate())
42             # logging.info('====================')
43             executor.shutdown()
44             logging.info(threading.enumerate())
45             break
46 
47     logging.info('--------------------')

进程测试

 1 D:\python3.7\python.exe E:/code_pycharm/tt2.py
 2 2018-10-30 10:47:48,933      3224 SpawnProcess-1   begin to work0
 3 2018-10-30 10:47:48,951      7484 SpawnProcess-2   begin to work1
 4 2018-10-30 10:47:48,974      8848 SpawnProcess-3   begin to work2
 5 2018-10-30 10:47:49,829      6220 MainProcess   [<_MainThread(MainThread, started 3604)>, <Thread(QueueManagerThread, started daemon 9156)>, <Thread(QueueFeederThread, started daemon 8476)>]
 6 2018-10-30 10:47:49,829      6220 MainProcess   False
 7 2018-10-30 10:47:49,829      6220 MainProcess   False
 8 2018-10-30 10:47:49,829      6220 MainProcess   False
 9 2018-10-30 10:47:50,829      6220 MainProcess   [<_MainThread(MainThread, started 3604)>, <Thread(QueueManagerThread, started daemon 9156)>, <Thread(QueueFeederThread, started daemon 8476)>]
10 2018-10-30 10:47:50,829      6220 MainProcess   False
11 2018-10-30 10:47:50,829      6220 MainProcess   False
12 2018-10-30 10:47:50,829      6220 MainProcess   False
13 2018-10-30 10:47:51,829      6220 MainProcess   [<_MainThread(MainThread, started 3604)>, <Thread(QueueManagerThread, started daemon 9156)>, <Thread(QueueFeederThread, started daemon 8476)>]
14 2018-10-30 10:47:51,829      6220 MainProcess   False
15 2018-10-30 10:47:51,829      6220 MainProcess   False
16 2018-10-30 10:47:51,829      6220 MainProcess   False
17 2018-10-30 10:47:52,829      6220 MainProcess   [<_MainThread(MainThread, started 3604)>, <Thread(QueueManagerThread, started daemon 9156)>, <Thread(QueueFeederThread, started daemon 8476)>]
18 2018-10-30 10:47:52,829      6220 MainProcess   False
19 2018-10-30 10:47:52,829      6220 MainProcess   False
20 2018-10-30 10:47:52,829      6220 MainProcess   False
21 2018-10-30 10:47:53,829      6220 MainProcess   [<_MainThread(MainThread, started 3604)>, <Thread(QueueManagerThread, started daemon 9156)>, <Thread(QueueFeederThread, started daemon 8476)>]
22 2018-10-30 10:47:53,829      6220 MainProcess   False
23 2018-10-30 10:47:53,829      6220 MainProcess   False
24 2018-10-30 10:47:53,829      6220 MainProcess   False
25 2018-10-30 10:47:53,934      3224 SpawnProcess-1   finisdhed0
26 2018-10-30 10:47:53,951      7484 SpawnProcess-2   finisdhed1
27 2018-10-30 10:47:53,974      8848 SpawnProcess-3   finisdhed2
28 2018-10-30 10:47:54,829      6220 MainProcess   [<_MainThread(MainThread, started 3604)>, <Thread(QueueManagerThread, started daemon 9156)>, <Thread(QueueFeederThread, started daemon 8476)>]
29 2018-10-30 10:47:54,829      6220 MainProcess   True
30 2018-10-30 10:47:54,829      6220 MainProcess   True
31 2018-10-30 10:47:54,829      6220 MainProcess   True
32 2018-10-30 10:47:54,829      6220 MainProcess   the ans is 0
33 2018-10-30 10:47:54,829      6220 MainProcess   the ans is 1
34 2018-10-30 10:47:54,829      6220 MainProcess   the ans is 2
35 2018-10-30 10:47:54,854      6220 MainProcess   [<_MainThread(MainThread, started 3604)>]
36 2018-10-30 10:47:54,854      6220 MainProcess   --------------------
37 
38 Process finished with exit code 0

查看 python进程数，占用三个CPU

4、支持上下文管理

 1 import threading
 2 from concurrent import futures
 3 
 4 import logging
 5 import time
 6 
 7 FORMAT = '%(asctime)-15s\t  %(process)s %(processName)s   %(message)s'
 8 logging.basicConfig(level=logging.INFO, format=FORMAT)
 9 
10 # 这是一个IO 密集型的函数，建议使用多线程
11 def worker(n):
12     logging.info('begin to work{}'.format(n))
13     time.sleep(5)
14     logging.info('finisdhed{}'.format(n))
15     return n
16 
17 # 创建线程池，容量为3个
18 # executor = futures.ThreadPoolExecutor(max_workers=3)
19 
20 if __name__ == '__main__':
21     executor = futures.ProcessPoolExecutor(max_workers=3)
22 
23     with executor:
24         fs = []
25         for i in range(4):
26             # 提交任务
27             future = executor.submit(worker, i)
28             fs.append(future)
29 
30 
31         while True:
32             time.sleep(1)
33             logging.info(threading.enumerate())
34 
35             flag = True
36             for f in fs:
37                 logging.info(f.done())
38                 flag = flag and f.done()
39 
40             if flag:
41                 for f in fs:
42                     logging.info('the ans is {}'.format(f.result()))
43                 # logging.info(threading.enumerate())
44                 # logging.info('====================')
45                 break
46 
47     logging.info('--------------------')

with一般放到刚创建池对象后面，最终自动清除池

 1 D:\python3.7\python.exe E:/code_pycharm/tt2.py
 2 2018-10-30 10:54:18,388      8244 SpawnProcess-1   begin to work0
 3 2018-10-30 10:54:18,404      5944 SpawnProcess-2   begin to work1
 4 2018-10-30 10:54:18,433      5608 SpawnProcess-3   begin to work2
 5 2018-10-30 10:54:19,294      8476 MainProcess   [<_MainThread(MainThread, started 8516)>, <Thread(QueueManagerThread, started daemon 5892)>, <Thread(QueueFeederThread, started daemon 8632)>]
 6 2018-10-30 10:54:19,294      8476 MainProcess   False
 7 2018-10-30 10:54:19,294      8476 MainProcess   False
 8 2018-10-30 10:54:19,294      8476 MainProcess   False
 9 2018-10-30 10:54:19,294      8476 MainProcess   False
10 2018-10-30 10:54:20,294      8476 MainProcess   [<_MainThread(MainThread, started 8516)>, <Thread(QueueManagerThread, started daemon 5892)>, <Thread(QueueFeederThread, started daemon 8632)>]
11 2018-10-30 10:54:20,294      8476 MainProcess   False
12 2018-10-30 10:54:20,294      8476 MainProcess   False
13 2018-10-30 10:54:20,294      8476 MainProcess   False
14 2018-10-30 10:54:20,294      8476 MainProcess   False
15 2018-10-30 10:54:21,294      8476 MainProcess   [<_MainThread(MainThread, started 8516)>, <Thread(QueueManagerThread, started daemon 5892)>, <Thread(QueueFeederThread, started daemon 8632)>]
16 2018-10-30 10:54:21,294      8476 MainProcess   False
17 2018-10-30 10:54:21,294      8476 MainProcess   False
18 2018-10-30 10:54:21,294      8476 MainProcess   False
19 2018-10-30 10:54:21,294      8476 MainProcess   False
20 2018-10-30 10:54:22,294      8476 MainProcess   [<_MainThread(MainThread, started 8516)>, <Thread(QueueManagerThread, started daemon 5892)>, <Thread(QueueFeederThread, started daemon 8632)>]
21 2018-10-30 10:54:22,294      8476 MainProcess   False
22 2018-10-30 10:54:22,294      8476 MainProcess   False
23 2018-10-30 10:54:22,294      8476 MainProcess   False
24 2018-10-30 10:54:22,294      8476 MainProcess   False
25 2018-10-30 10:54:23,294      8476 MainProcess   [<_MainThread(MainThread, started 8516)>, <Thread(QueueManagerThread, started daemon 5892)>, <Thread(QueueFeederThread, started daemon 8632)>]
26 2018-10-30 10:54:23,294      8476 MainProcess   False
27 2018-10-30 10:54:23,294      8476 MainProcess   False
28 2018-10-30 10:54:23,294      8476 MainProcess   False
29 2018-10-30 10:54:23,295      8476 MainProcess   False
30 2018-10-30 10:54:23,388      8244 SpawnProcess-1   finisdhed0
31 2018-10-30 10:54:23,388      8244 SpawnProcess-1   begin to work3
32 2018-10-30 10:54:23,404      5944 SpawnProcess-2   finisdhed1
33 2018-10-30 10:54:23,433      5608 SpawnProcess-3   finisdhed2
34 2018-10-30 10:54:24,295      8476 MainProcess   [<_MainThread(MainThread, started 8516)>, <Thread(QueueManagerThread, started daemon 5892)>, <Thread(QueueFeederThread, started daemon 8632)>]
35 2018-10-30 10:54:24,295      8476 MainProcess   True
36 2018-10-30 10:54:24,295      8476 MainProcess   True
37 2018-10-30 10:54:24,295      8476 MainProcess   True
38 2018-10-30 10:54:24,295      8476 MainProcess   False
39 2018-10-30 10:54:25,295      8476 MainProcess   [<_MainThread(MainThread, started 8516)>, <Thread(QueueManagerThread, started daemon 5892)>, <Thread(QueueFeederThread, started daemon 8632)>]
40 2018-10-30 10:54:25,295      8476 MainProcess   True
41 2018-10-30 10:54:25,295      8476 MainProcess   True
42 2018-10-30 10:54:25,295      8476 MainProcess   True
43 2018-10-30 10:54:25,295      8476 MainProcess   False
44 2018-10-30 10:54:26,295      8476 MainProcess   [<_MainThread(MainThread, started 8516)>, <Thread(QueueManagerThread, started daemon 5892)>, <Thread(QueueFeederThread, started daemon 8632)>]
45 2018-10-30 10:54:26,295      8476 MainProcess   True
46 2018-10-30 10:54:26,295      8476 MainProcess   True
47 2018-10-30 10:54:26,295      8476 MainProcess   True
48 2018-10-30 10:54:26,295      8476 MainProcess   False
49 2018-10-30 10:54:27,296      8476 MainProcess   [<_MainThread(MainThread, started 8516)>, <Thread(QueueManagerThread, started daemon 5892)>, <Thread(QueueFeederThread, started daemon 8632)>]
50 2018-10-30 10:54:27,296      8476 MainProcess   True
51 2018-10-30 10:54:27,296      8476 MainProcess   True
52 2018-10-30 10:54:27,296      8476 MainProcess   True
53 2018-10-30 10:54:27,296      8476 MainProcess   False
54 2018-10-30 10:54:28,296      8476 MainProcess   [<_MainThread(MainThread, started 8516)>, <Thread(QueueManagerThread, started daemon 5892)>, <Thread(QueueFeederThread, started daemon 8632)>]
55 2018-10-30 10:54:28,296      8476 MainProcess   True
56 2018-10-30 10:54:28,296      8476 MainProcess   True
57 2018-10-30 10:54:28,296      8476 MainProcess   True
58 2018-10-30 10:54:28,296      8476 MainProcess   False
59 2018-10-30 10:54:28,391      8244 SpawnProcess-1   finisdhed3
60 2018-10-30 10:54:29,296      8476 MainProcess   [<_MainThread(MainThread, started 8516)>, <Thread(QueueManagerThread, started daemon 5892)>, <Thread(QueueFeederThread, started daemon 8632)>]
61 2018-10-30 10:54:29,296      8476 MainProcess   True
62 2018-10-30 10:54:29,296      8476 MainProcess   True
63 2018-10-30 10:54:29,296      8476 MainProcess   True
64 2018-10-30 10:54:29,296      8476 MainProcess   True
65 2018-10-30 10:54:29,296      8476 MainProcess   the ans is 0
66 2018-10-30 10:54:29,296      8476 MainProcess   the ans is 1
67 2018-10-30 10:54:29,296      8476 MainProcess   the ans is 2
68 2018-10-30 10:54:29,296      8476 MainProcess   the ans is 3
69 2018-10-30 10:54:29,327      8476 MainProcess   --------------------
70 
71 Process finished with exit code 0

结果

 1 import threading
 2 from concurrent import futures
 3 
 4 import logging
 5 import time
 6 
 7 FORMAT = '%(asctime)-15s\t  %(process)s %(processName)s   %(message)s'
 8 logging.basicConfig(level=logging.INFO, format=FORMAT)
 9 
10 # 这是一个IO 密集型的函数，建议使用多线程
11 def worker(n):
12     logging.info('begin to work{}'.format(n))
13     time.sleep(5)
14     logging.info('finisdhed{}'.format(n))
15     return n
16 
17 # 创建线程池，容量为3个
18 
19 if __name__ == '__main__':
20     # executor = futures.ProcessPoolExecutor(max_workers=3)
21     executor = futures.ThreadPoolExecutor(max_workers=3)
22 
23     with executor:
24         fs = []
25         for i in range(4):
26             # 提交任务
27             future = executor.submit(worker, i)
28             fs.append(future)
29 
30 
31         while True:
32             time.sleep(1)
33             logging.info(threading.enumerate())
34 
35             flag = True
36             for f in fs:
37                 logging.info(f.done())
38                 flag = flag and f.done()
39 
40             if flag:
41                 for f in fs:
42                     logging.info('the ans is {}'.format(f.result()))
43                 # logging.info(threading.enumerate())
44                 # logging.info('====================')
45                 break
46     logging.info(threading.enumerate())
47 
48     logging.info('--------------------')

多线程测试

 1 D:\python3.7\python.exe E:/code_pycharm/tt2.py
 2 2018-10-30 10:58:12,862      5604 MainProcess   begin to work0
 3 2018-10-30 10:58:12,862      5604 MainProcess   begin to work1
 4 2018-10-30 10:58:12,862      5604 MainProcess   begin to work2
 5 2018-10-30 10:58:13,862      5604 MainProcess   [<_MainThread(MainThread, started 3956)>, <Thread(ThreadPoolExecutor-0_0, started daemon 4192)>, <Thread(ThreadPoolExecutor-0_1, started daemon 9128)>, <Thread(ThreadPoolExecutor-0_2, started daemon 9036)>]
 6 2018-10-30 10:58:13,862      5604 MainProcess   False
 7 2018-10-30 10:58:13,862      5604 MainProcess   False
 8 2018-10-30 10:58:13,862      5604 MainProcess   False
 9 2018-10-30 10:58:13,862      5604 MainProcess   False
10 2018-10-30 10:58:14,863      5604 MainProcess   [<_MainThread(MainThread, started 3956)>, <Thread(ThreadPoolExecutor-0_0, started daemon 4192)>, <Thread(ThreadPoolExecutor-0_1, started daemon 9128)>, <Thread(ThreadPoolExecutor-0_2, started daemon 9036)>]
11 2018-10-30 10:58:14,863      5604 MainProcess   False
12 2018-10-30 10:58:14,863      5604 MainProcess   False
13 2018-10-30 10:58:14,863      5604 MainProcess   False
14 2018-10-30 10:58:14,863      5604 MainProcess   False
15 2018-10-30 10:58:15,863      5604 MainProcess   [<_MainThread(MainThread, started 3956)>, <Thread(ThreadPoolExecutor-0_0, started daemon 4192)>, <Thread(ThreadPoolExecutor-0_1, started daemon 9128)>, <Thread(ThreadPoolExecutor-0_2, started daemon 9036)>]
16 2018-10-30 10:58:15,863      5604 MainProcess   False
17 2018-10-30 10:58:15,863      5604 MainProcess   False
18 2018-10-30 10:58:15,863      5604 MainProcess   False
19 2018-10-30 10:58:15,863      5604 MainProcess   False
20 2018-10-30 10:58:16,863      5604 MainProcess   [<_MainThread(MainThread, started 3956)>, <Thread(ThreadPoolExecutor-0_0, started daemon 4192)>, <Thread(ThreadPoolExecutor-0_1, started daemon 9128)>, <Thread(ThreadPoolExecutor-0_2, started daemon 9036)>]
21 2018-10-30 10:58:16,863      5604 MainProcess   False
22 2018-10-30 10:58:16,863      5604 MainProcess   False
23 2018-10-30 10:58:16,863      5604 MainProcess   False
24 2018-10-30 10:58:16,863      5604 MainProcess   False
25 2018-10-30 10:58:17,863      5604 MainProcess   finisdhed2
26 2018-10-30 10:58:17,863      5604 MainProcess   begin to work3
27 2018-10-30 10:58:17,863      5604 MainProcess   [<_MainThread(MainThread, started 3956)>, <Thread(ThreadPoolExecutor-0_0, started daemon 4192)>, <Thread(ThreadPoolExecutor-0_1, started daemon 9128)>, <Thread(ThreadPoolExecutor-0_2, started daemon 9036)>]
28 2018-10-30 10:58:17,863      5604 MainProcess   finisdhed1
29 2018-10-30 10:58:17,863      5604 MainProcess   finisdhed0
30 2018-10-30 10:58:17,863      5604 MainProcess   False
31 2018-10-30 10:58:17,864      5604 MainProcess   True
32 2018-10-30 10:58:17,864      5604 MainProcess   True
33 2018-10-30 10:58:17,864      5604 MainProcess   False
34 2018-10-30 10:58:18,864      5604 MainProcess   [<_MainThread(MainThread, started 3956)>, <Thread(ThreadPoolExecutor-0_0, started daemon 4192)>, <Thread(ThreadPoolExecutor-0_1, started daemon 9128)>, <Thread(ThreadPoolExecutor-0_2, started daemon 9036)>]
35 2018-10-30 10:58:18,864      5604 MainProcess   True
36 2018-10-30 10:58:18,864      5604 MainProcess   True
37 2018-10-30 10:58:18,864      5604 MainProcess   True
38 2018-10-30 10:58:18,864      5604 MainProcess   False
39 2018-10-30 10:58:19,864      5604 MainProcess   [<_MainThread(MainThread, started 3956)>, <Thread(ThreadPoolExecutor-0_0, started daemon 4192)>, <Thread(ThreadPoolExecutor-0_1, started daemon 9128)>, <Thread(ThreadPoolExecutor-0_2, started daemon 9036)>]
40 2018-10-30 10:58:19,864      5604 MainProcess   True
41 2018-10-30 10:58:19,864      5604 MainProcess   True
42 2018-10-30 10:58:19,864      5604 MainProcess   True
43 2018-10-30 10:58:19,864      5604 MainProcess   False
44 2018-10-30 10:58:20,864      5604 MainProcess   [<_MainThread(MainThread, started 3956)>, <Thread(ThreadPoolExecutor-0_0, started daemon 4192)>, <Thread(ThreadPoolExecutor-0_1, started daemon 9128)>, <Thread(ThreadPoolExecutor-0_2, started daemon 9036)>]
45 2018-10-30 10:58:20,864      5604 MainProcess   True
46 2018-10-30 10:58:20,864      5604 MainProcess   True
47 2018-10-30 10:58:20,864      5604 MainProcess   True
48 2018-10-30 10:58:20,864      5604 MainProcess   False
49 2018-10-30 10:58:21,864      5604 MainProcess   [<_MainThread(MainThread, started 3956)>, <Thread(ThreadPoolExecutor-0_0, started daemon 4192)>, <Thread(ThreadPoolExecutor-0_1, started daemon 9128)>, <Thread(ThreadPoolExecutor-0_2, started daemon 9036)>]
50 2018-10-30 10:58:21,864      5604 MainProcess   True
51 2018-10-30 10:58:21,864      5604 MainProcess   True
52 2018-10-30 10:58:21,864      5604 MainProcess   True
53 2018-10-30 10:58:21,864      5604 MainProcess   False
54 2018-10-30 10:58:22,863      5604 MainProcess   finisdhed3
55 2018-10-30 10:58:22,864      5604 MainProcess   [<_MainThread(MainThread, started 3956)>, <Thread(ThreadPoolExecutor-0_0, started daemon 4192)>, <Thread(ThreadPoolExecutor-0_1, started daemon 9128)>, <Thread(ThreadPoolExecutor-0_2, started daemon 9036)>]
56 2018-10-30 10:58:22,864      5604 MainProcess   True
57 2018-10-30 10:58:22,864      5604 MainProcess   True
58 2018-10-30 10:58:22,864      5604 MainProcess   True
59 2018-10-30 10:58:22,864      5604 MainProcess   True
60 2018-10-30 10:58:22,864      5604 MainProcess   the ans is 0
61 2018-10-30 10:58:22,864      5604 MainProcess   the ans is 1
62 2018-10-30 10:58:22,864      5604 MainProcess   the ans is 2
63 2018-10-30 10:58:22,864      5604 MainProcess   the ans is 3
64 2018-10-30 10:58:22,864      5604 MainProcess   [<_MainThread(MainThread, started 3956)>]
65 2018-10-30 10:58:22,864      5604 MainProcess   --------------------

多线程测试，最终线程都关闭了

总结：

　　该库统一了线程池，进程池的调用，简化了编程。

　　缺点是无法设置线程名。

posted @ 2018-10-29 22:29 JerryZao 阅读(1954) 评论(0) 编辑收藏举报

会员力量，点亮园子希望

刷新页面返回顶部

JerryZao

为什么坚持，想一想当初，每天进步一点点！

Python-多线程+多进程包（concurrent包，处理并发）

公告