Python 多进程 joblib

Python 多进程 joblib

1. joblib package

• Website

2. 实例

• 实例

import time
import datetime
import math
from joblib import Parallel, parallel_config, delayed, 

# function
def func(i):
    time.sleep(1.)
    x = sqrt(i)
    y = i**2
    return i, x, y

# start time
t1 = time.time()

# Parallel computing: way 1
r = Parallel(backend='threading', n_jobs=10, verbose=1)(
    delayed(func)
    (i,) for i in range(10) )

# way 2:
with parallel_config(backend='threading', n_jobs=10, verbose=1):
    r = Parallel()(
        delayed(func)
        (i,) for i in range(10) )

# function reture
i, x, y = zip(*r)

print('Total running time:', time.time() - t1)

注意：

• Parallel() 返回的结果，是按照迭代的顺序

• 经过测试，并行计算的速度有时不如顺序执行的速度

  • 可能的原因：启动多个进程也需要时间开销

• sklearn 中 Cross-validation

parallel = Parallel(n_jobs=self.n_jobs, verbose=self.verbose, mpre_dispatch=self.pre_dispatch)

out = parallel(
    delayed(_fit_and_score)
    (clone(base_estimator), X, y, train=train, test=test, parameters=parameters, **fit_and_score_kwargs) 
        for parameters, (train, test) in product(candidate_params, cv.split(X, y, groups)) )

Reference

Python之并行--基于joblib, 知乎, site