spark 访问OSS java.lang.NoClassDefFoundError: com/aliyun/oss/OSSException 

 1 21/05/28 15:35:45 INFO [main] YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
 2 Exception in thread "main" java.lang.NoClassDefFoundError: com/aliyun/oss/OSSException
 3         at com.aliyun.fs.oss.nat.JetOssNativeFileSystemStore.initialize(JetOssNativeFileSystemStore.java:140)
 4         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 5         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 6         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 7         at java.lang.reflect.Method.invoke(Method.java:498)
 8         at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:411)
 9         at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
10         at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
11         at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
12         at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:348)
13         at com.sun.proxy.$Proxy29.initialize(Unknown Source)
14         at com.aliyun.fs.oss.nat.NativeOssFileSystem.initialize(NativeOssFileSystem.java:142)
15         at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2812)
16         at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100)
17         at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2849)
18         at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2831)
19         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389)
20         at com.offerslook.click.SparkStart.main(SparkStart.java:41)
21         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
22         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
23         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
24         at java.lang.reflect.Method.invoke(Method.java:498)
25         at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
26         at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:851)
27         at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
28         at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
29         at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
30         at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:926)
31         at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:935)
32         at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
33 Caused by: java.lang.ClassNotFoundException: com.aliyun.oss.OSSException
34         at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
35         at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
36         at java.lang.ClassLoader.loadClass(ClassLoader.java:351)

解决一：

 

解决二：

如果EMR-3.24.0 之前版本，采用一继续报错

./spark-submit --class com...Main --jars /data/aliyun_jars/aliyun-sdk-oss-3.0.0.jar /data/spark.jar

 

posted @ 2021-05-28 17:31  大裤衩子  阅读(574)  评论(0)    收藏  举报