pandas、spark计算相关性系数速度对比

pandas、spark计算相关性系数速度对比

相关性计算有三种算法:pearson、spearman,kenall。

在pandas库中,对一个Dataframe,可以直接计算这三个算法的相关系数correlation,方法为:data.corr()

底层是依赖scipy库的算法。

为了提升计算速度,使用spark平台来加速执行。

比较了pandas,spark并发scipy算法,spark mllib库的计算速度。

总体来说,spark mllib速度最快,其次是spark并发,pandas速度最慢。

corr执行速度测试结果

时间单位:秒

数据大小 corr算法 pandas spark + scipy spark mllib 备注
1000*3600 pearsonr 203 170 37 pyspark
1000*3600 pearsonr 203 50 没有计算 spark scipy计算一半
1000*3600 pearsonr 203 125 37 client模式
1000*3600 pearsonr 202 157 38 client模式
1000*3600 spearmanr 1386 6418 37 client模式
1000*3600 spearmanr 1327 6392 38 client模式
1000*3600 kendall 4326 398 无此算法 client模式
1000*3600 kendall 4239 346 无此算法 client模式
1000*1000 spearmanr 127 294 12 client 模式
1000*1000 spearmanr 98 513 5.55 client 模式
1000*360 spearmanr 13 150 没有计算 160秒,列表推导式 res = [st.spearmanr(data.iloc[:, i], data.iloc[:, j])[0] for i in range(N) for j in range(N)]
1000*360 kendall 40 45 无此算法 116秒,列表推导式 res = [st.kendall(data.iloc[:, i], data.iloc[:, j])[0] for i in range(N) for j in range(N)]

说明:spearmanr 算法在spark scipy组合下执行速度较慢,需要再对比分析,感觉存在问题的。


三种算法脚本如下:

pandas 脚本


import numpy as np
import pandas as pd
import time

C = 1000
N = 3600

data = pd.DataFrame(np.random.randn(C * N).reshape(C, -1))


print("============================ {}".format(data.shape))
print("start pandas corr ---{} ".format(time.time()))
start = time.time()
# {'pearson', 'kendall', 'spearman'}
res = data.corr(method='pearson')
end_1 = time.time()

res = data.corr(method='spearman')
end_2 = time.time()

res = data.corr(method='kendall')
end_3 = time.time()

print("pandas pearson count {} total cost : {}".format(len(res), end_1 - start))
print("pandas spearman count {} total cost : {}".format(len(res), end_2 - end_1))
print("pandas kendall count {} total cost : {}".format(len(res), end_3 - end_2))

spark scipy脚本

from pyspark import SparkContext
sc = SparkContext()
import numpy as np
import pandas as pd
from scipy import stats as st
import time

# t1 = st.kendalltau(x, y)
# t2 = st.spearmanr(x, y)
# t3 = st.pearsonr(x, y)

C = 1000
N = 3600

data = pd.DataFrame(np.random.randn(C * N).reshape(C, -1))


def pearsonr(n):
    x = data.iloc[:, n]
    res = [st.pearsonr(x, data.iloc[:, i])[0] for i in range(data.shape[1])]
    return res


def spearmanr(n):
    x = data.iloc[:, n]
    res = [st.spearmanr(x, data.iloc[:, i])[0] for i in range(data.shape[1])]
    return res


def kendalltau(n):
    x = data.iloc[:, n]
    res = [st.kendalltau(x, data.iloc[:, i])[0] for i in range(data.shape[1])]
    return res


start = time.time()
res = sc.parallelize(np.arange(N)).map(lambda x: pearsonr(x)).collect()
# res = sc.parallelize(np.arange(N)).map(lambda x: spearmanr(x)).collect()
# res = sc.parallelize(np.arange(N)).map(lambda x: kendalltau(x)).collect()
end = time.time()

print("pearsonr count {} total cost : {}".format(len(res), end - start))
print("spearmanr count {} total cost : {}".format(len(res), end - start))
print("kendalltau count {} total cost : {}".format(len(res), end - start))


# 纯python算法
s = time.time()
res = [st.spearmanr(data.iloc[:, i], data.iloc[:, j])[0] for i in range(N) for j in range(N)]
end = time.time()
print(end-s)

start = time.time()
dd = sc.parallelize(res).map(lambda x: st.spearmanr(data.iloc[:, x[0]], data.iloc[:, x[1]])).collect()
end = time.time()
print(end-start)

start = time.time()
dd = sc.parallelize(res).map(lambda x: st.kendalltau(data.iloc[:, x[0]], data.iloc[:, x[1]])).collect()
end = time.time()
print(end-start)

spark mllib脚本

from pyspark import SparkContext
sc = SparkContext()
from pyspark.mllib.stat import Statistics
import time
import numpy as np

L = 1000
N = 3600
t = [np.random.randn(N) for i in range(L)]

data = sc.parallelize(t)

start = time.time()
res = Statistics.corr(data, method="pearson")  # spearman  pearson
end = time.time()
print("pearson : ", end-start)


start = time.time()
res = Statistics.corr(data, method="spearman")  # spearman  pearson
end = time.time()
print("spearman: ", end-start)
posted @ 2020-07-02 15:55  MyStitch  阅读(2116)  评论(0编辑  收藏  举报