Python多进程任务分解

不废话，直接上代码：

from multiprocessing import Process
import numpy as np

CORE_NUM = 4
MAT_ROW  = 10
MAT_COL  = 10

def routine(label, filename):
    mat = np.zeros((MAT_ROW, MAT_COL), dtype = np.float32)
    for r in range(0, mat.shape[0]):
        for c in range(0, mat.shape[1]):
            mat[r][c] = label
    np.save(filename, mat)

def map():
    # Multi-Process for acceleration
    processes = []

    for i in range(CORE_NUM):
        process = Process(target = routine,
                          args = (i, "TestMulti" + str(i) + ".cache")) 
        processes.append(process)
        
    for i in range(CORE_NUM):
        processes[i].start()

    for i in range(CORE_NUM):
        processes[i].join()
    return

def reduce():
    for i in range(CORE_NUM):
        ms = np.load("TestMulti" + str(i) + ".cache.npy")
        print(ms)
        
if __name__ == "__main__":
    map()
    reduce()

 

需要注意的有两点：

1、我首先在spyder里写的代码，结果运行失败，总提示错误信息：

AttributeError: module '__main__' has no attribute '__spec__'

网上搜索了才知道，代码本身没有问题，不用spyder，直接在命令行中运行就可以了。

2、创建子进程的代码必须放在if __name__ == "__main__": 代码块里面。原因是这段代码在子进程里面会重复执行。具体现象可去掉这个条件语句，再执行以下就看到了。