每日随笔——使用Maven对Scala独立应用程序进行编译打包

通过之前的博客，我们已经下载好maven并配置好环境，且能实现

（一）使用sbt对Scala独立应用程序进行编译打包

（二）使用Maven对Java独立应用程序进行编译打包

现在进行最后一步：使用Maven对Scala独立应用程序进行编译打包

Java应用程序代码

cd /export/server
mkdir -p  sparkapp3/src/main/scala

在 ./sparkapp3/src/main/scala下建立一个名为 SimpleApp.java 的文件（vim ./sparkapp3/src/main/scala/SimpleApp.java），添加代码如下：

1. import org.apache.spark.SparkContext
2. import org.apache.spark.SparkContext._
3. import org.apache.spark.SparkConf
4.  
5. object SimpleApp {
6. def main(args: Array[String]) {
7. val logFile = "file:///export/server/spark/README.md" // Should be some file on your system
8. val conf = new SparkConf().setAppName("Simple Application")
9. val sc = new SparkContext(conf)
10. val logData = sc.textFile(logFile, 2).cache()
11. val numAs = logData.filter(line => line.contains("a")).count()
12. val numBs = logData.filter(line => line.contains("b")).count()
13. println("Lines with a: %s, Lines with b: %s".format(numAs, numBs))
14. }
15. }

该程序依赖Spark Java API,因此我们需要通过Maven进行编译打包。在./sparkapp3目录中新建文件pom.xml，命令如下：

cd /export/server/sparkapp3
vim pom.xml

在pom.xml中添加相关依赖

project>
    <groupId>cn.edu.xmu</groupId>
    <artifactId>simple-project</artifactId>
    <modelVersion>4.0.0</modelVersion>
    <name>Simple Project</name>
    <packaging>jar</packaging>
    <version>1.0</version>
    <repositories>
        <repository>
            <id>jboss</id>
            <name>JBoss Repository</name>
            <url>http://repository.jboss.com/maven2/</url>
        </repository>
    </repositories>
    <dependencies>
        <dependency> <!-- Spark dependency -->
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.12</artifactId>
            <version>3.4.0</version>
        </dependency>
    </dependencies>

  <build>
    <sourceDirectory>src/main/scala</sourceDirectory>
    <plugins>
      <plugin>
        <groupId>org.scala-tools</groupId>
        <artifactId>maven-scala-plugin</artifactId>
        <executions>
          <execution>
            <goals>
              <goal>compile</goal>
            </goals>
          </execution>
        </executions>
        <configuration>
          <scalaVersion>2.12.17</scalaVersion>
          <args>
            <arg>-target:jvm-1.8</arg>
          </args>
        </configuration>
    </plugin>
    </plugins>
</build>
</project>

打包

cd  sparkapp3    #一定把这个目录设置为当前目录
/export/server/maven/bin/mvn  package

 

通过 spark-submit 运行程序

 /export/server/spark/bin/spark-submit --class "SimpleApp" /export/server/sparkapp3/target/simple-project-1.0.jar 2>&1 | grep "Lines with a:"