centos6.4 编译hadoop 2.6

1.镜像配置
/etc/yum.repo.d/CentOS6-Base-163.repo

 

2./etc/resove.conf
添加内容：
nameserver 8.8.8.8
nameserver 8.8.4.4

3.下载Hadoop源码工程并解压

编辑文件：usr/local/hadoop-2.6.0-src/hadoop-common-project/hadoop-auth/pom.xml，第55行左右增加如下内容：

<dependency>
         <groupId>org.mortbay.jetty</groupId>
         <artifactId>jetty-util</artifactId>
         <scope>test</scope>
</dependency>

4.检查jdk（JDK 1.6+）

建议用jdk1.7

5.安装配置maven
版本：3.3.9

1)配置环境变量
/etc/profile
#MAVEN_HOME
export MAVEN_HOME=/opt/modules/apache-maven-3.3.9
export PATH=$MAVEN_HOME/bin:$PATH:$JAVA_HOME/bin

2)测试 mvn -version

3）conf/setting.xml

添加内容：

<mirror>
<id>alimaven</id>
<name>aliyun maven</name>
<url>http://maven.aliyun.com/nexus/content/groups/public/</url>
<mirrorOf>central</mirrorOf>
</mirror>

6.ProtocolBuffer 2.5.0安装

./configure --prefix=/usr/local/protoc/

./make

./make install

 安装完成后，配置环境变量，增加：export PROTOC_HOME=/usr/local/protoc/，和export PATH=..:$PROTOC_HOME/bin:..

验证：protoc --version

 

7.snappy安装

./configure

./make

./make install

8.CMake
sudo yum install cmake

 

9.hadoop编译
mvn clean package -Pdist,native -DskipTests -Dtar -Dbundle.snappy -Dsnappy.lib=/usr/local/lib

 

bin/yarn jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-cdh5.5.0.jar wordcount -Dmapreduce.output.fileoutputformat.compress=true -Dmapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.SnappyCodec /user/kfk/datas/ /user/kfk/newCompress3