hibench7.0编译安装

  引言 HiBench是一个大数据基准套件，可以帮助您评测不同大数据平台的性能、吞吐量和系统资源利用率。本文仅介绍如何对hadoop进行测试，其他大数据平台使用请参考官网https://github.com/intel-hadoop/HiBench 软件依赖 HiBench需要java环境，以及Maven管理。 安装java运行环境 新建目录/home/java 然后上传相应的jdk二进制文件到此目录并解压。（本例为aarch64为平台，x86平台一样的操作步骤，亲试） root@kylin1:/home/java# ll 总用量 71608 drwxr-xr-x 3 root root     4096 4月   8 11:59 ./ drwxr-xr-x 4 root root     4096 4月   8 11:28 ../ drwxr-xr-x 7 root root     4096 4月   8 11:44 jdk1.8.0_201/ -rw-r--r-- 1 root root 73312819 4月   8 11:43 jdk-8u201-linux-arm64-vfp-hflt.tar.gz 添加环境变量：/etc/profile 文件末尾增加如下行 export JAVA_HOME=/home/java/jdk1.8.0_201 export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar export PATH=$PATH:$JAVA_HOME/bin 使其生效 source /etc/profile 查看 root@kylin1:/home/java# java -version java version "1.8.0_201" Java(TM) SE Runtime Environment (build 1.8.0_201-b09) Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode)   安装Maven 下载Maven包 wget http://apache.fayea.com/maven/maven-3/3.6.0/binaries/apache-maven-3.6.0-bin.zip 解压缩 unzip apache-maven-3.6.0-bin.zip -d /usr/local/ 添加环境变量 cat /etc/profile,在文件末尾增加如下 export M3_HOME=/usr/local/apache-maven-3.6.0 export PATH=$M3_HOME/bin:$PATH   source /etc/profile 测试Maven环境 mvn -v 看到相应版本信息输出即表明配置正确： root@kylin3:/usr/local/HiBench-HiBench-7.0# mvn -v Apache Maven 3.6.0 (97c98ec64a1fdfee7767ce5ffb20918da4f719f3; 2018-10-25T02:41:47+08:00) Maven home: /usr/local/apache-maven-3.6.0 Java version: 1.8.0_201, vendor: Oracle Corporation, runtime: /home/java/jdk1.8.0_201/jre Default locale: zh_CN, platform encoding: UTF-8 OS name: "linux", version: "4.4.58-20180615.kylin.server.yun+-generic", arch: "aarch64", family: "unix"   下载HiBench git clone https://github.com/intel-hadoop/HiBench.git比较慢，建议直接网页下载zip包，然后解压到/usr/local目录下。 root@kylin3:/usr/local/HiBench-HiBench-7.0# pwd /usr/local/HiBench-HiBench-7.0   安装Hibench 切到HiBench下，执行对应的安装操作，可以选择自己想要安装的模块。以安装hadoop框架为例： root@kylin3:/usr/local/HiBench-HiBench-7.0# mvn -Phadoopbench -Dspark=2.1 -Dscala=2.11 clean package 更多安装方法参考https://github.com/Intel-bigdata/HiBench/blob/master/docs/build-hibench.md   因为网络原因，下次模块过程中经常出现暂停现象。当前解决办法暂停下载，ctrl+c 然后继续执行上面的命令。 遇到的问题： [WARNING] Could not get content org.apache.maven.wagon.TransferFailedException: Failed to transfer file http://archive.apache.org/dist/hive/hive-0.14.0/apache-hive-0.14.0-bin.tar.gz with status code 503 下载hive出错，解决： root@kylin3:/usr/local/HiBench-HiBench-7.0# pwd /usr/local/HiBench-HiBench-7.0 root@kylin3:/usr/local/HiBench-HiBench-7.0# vim hadoopbench/sql/pom.xml 修改该文件中，将http改为https即可 <properties>     <repo>https://archive.apache.org</repo> <file>dist/hive/hive-0.14.0/apache-hive-0.14.0-bin.tar.gz</file> </properties>     org.apache.maven.wagon.TransferFailedException: Failed to transfer file http://archive.apache.org/dist/nutch/apache-nutch-1.2-bin.tar.gz with status code 503 root@kylin3:/usr/local/HiBench-HiBench-7.0/hadoopbench/nutchindexing# pwd /usr/local/HiBench-HiBench-7.0/hadoopbench/nutchindexing vim pod.xml 修改如下文件由http改成https <configuration> <url>https://archive.apache.org/dist/nutch/apache-nutch-1.2-bin.tar.gz</url> </configuration>   最终结果 [INFO] Reactor Summary: [INFO] [INFO] hibench 7.0 ........................................ SUCCESS [  0.173 s] [INFO] hibench-common 7.0 ................................. SUCCESS [ 10.755 s] [INFO] HiBench data generation tools 7.0 .................. SUCCESS [ 12.696 s] [INFO] hadoopbench 7.0 .................................... SUCCESS [  0.004 s] [INFO] hadoopbench-sql 7.0 ................................ SUCCESS [  2.355 s] [INFO] mahout 7.0 ......................................... SUCCESS [  6.676 s] [INFO] PEGASUS: A Peta-Scale Graph Mining System 2.0-SNAPSHOT SUCCESS [  0.978 s] [INFO] nutchindexing 7.0 .................................. SUCCESS [ 41.968 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time:  01:15 min [INFO] Finished at: 2019-04-09T11:58:27+08:00 [INFO] ------------------------------------------------------------------------