pyspark-Py4JError报错解决Py4JError: An error occurred while calling o35.isBarrier.

 

遇到如下报错:

Py4JError: An error occurred while calling o35.isBarrier. Trace:
py4j.Py4JException: Method isBarrier([]) does not exist
    at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318)
    at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326)
    at py4j.Gateway.invoke(Gateway.java:274)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.lang.Thread.run(Thread.java:748)

 

关键字分析：
Py4JError: An error occurred while calling o35.isBarrier. Trace:
py4j.Py4JException: Method isBarrier([]) does not exist

pyspark与spark报错原因:
版本不一致导致的源代码逻辑实现不一致，导致底层调用o35.isBarrier时出错。

pyspark版本验证:

spark版本:

解决方法：
安装对应版本的pyspark包:

install -v pyspark==2.3.3

 

或者用spark自带的pyspark包安装:

安装:
先执行 python3 setup.py build
再执行python3 setup.py install

最后，程序没有再报错: