龙芯平台下安装配置hadoop
一、 hadoop简介
Hadoop是一个由Apache基金会所开发的分布式系统基础架构。
用户可以在不了解分布式底层细节的情况下，开发分布式程序。充分利用集群的威力进行高速运算和存储。
Hadoop实现了一个分布式文件系统（Hadoop Distributed File System），简称HDFS。HDFS有高容错性的特点，并且设计用来部署在低廉的（low-cost）硬件上；而且它提供高吞吐量（high throughput）来访问应用程序的数据，适合那些有着超大数据集（large data set）的应用程序。HDFS放宽了（relax）POSIX的要求，可以以流的形式访问（streaming access）文件系统中的数据。
Hadoop的框架最核心的设计就是：HDFS和MapReduce。HDFS为海量的数据提供了存储，则MapReduce为海量的数据提供了计算。
二、 软件介质及安装环境
软件版本：hadoop-2.6.5.tar.gz
下载地址：
http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.6.5/hadoop-2.6.5.tar.gz
安装环境：
Master节点服务器 Sugon L620-G15（192.168.32.182）
Slave节点服务器 长城3A单路（192.168.32.153 ）
以下分别简称为master服务器和slave服务器
操作系统：iSoft Server OS 5.0 beta3 for mips
三、 安装配置
master服务器、slave服务器分别进行如下操作：
关闭防火墙
[hadoop@master hadoop]$ sudo service iptables stop[sudo] password for hadoop: iptables：清除防火墙规则：[确定]
iptables：将链设置为政策ACCEPT：filter [确定]
iptables：正在卸载模块：[确定]
[hadoop@master hadoop]$ sudo chkconfig iptables off
修改hostname分别为master和slave
Master服务器
[root@isoft182 ~]# cat /etc/sysconfig/networkNETWORKING=yesHOSTNAME=masterGATEWAY=192.168.32.1
Slave服务器
[root@test-153 ~]# cat /etc/sysconfig/networkNETWORKING=yesHOSTNAME=slaveGATEWAY=192.168.32.1
分别修改hosts文件添加内容如下：
192.168.32.182 master192.168.32.153 slave
新建hadoop用户
进入hadoop
设为无密码登录slave服务器
[hadoop@isoft182 ~]$ ssh-keygenGenerating public/private rsa key pair.Enter file in which to save the key (/home/hadoop/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/hadoop/.ssh/id_rsa.Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.The key fingerprint is:9b:e0:7e:bb:b4:7c:9b:66:e7:1f:5f:49:30:22:be:d0 hadoop@isoft182The key's randomart image is:+--[ RSA 2048]----+| || || . . o || o . . o || ..SE . || . ..o. . .|| . +. ...|| . o..+.. o.|| ..=*o+... .| [hadoop@isoft182 ~]$ ssh-copy-id slavehadoop@slave's password: Now try logging into the machine, with "ssh 'slave'", and check in: .ssh/authorized_keys to make sure we haven't added extra keys that you weren't expecting.
配置java环境
~/.bashrc文件最后加入
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk然后
source ~./bashrc是设置生效。
确认java版本信息
[hadoop@isoft182 ~]$ java -versionopenjdk version "1.8.0_25"OpenJDK Runtime Environment (build 1.8.0_25-rc19-b17)OpenJDK 64-Bit Server VM (build 25.25-b02, mixed mode)
安装hadoop
sudo tar -zxf hadoop-2.6.5.tar.gz -C /usr/local #解压到/usr/local中
cd /usr/local/sudo mv ./hadoop-2.6.5/ ./hadoop #将文件夹名改为hadoop
sudo chown -R hadoop:hadoop ./hadoop #修改文件权限
检查版本信息
[hadoop@isoft182 local]$ ./hadoop/bin/hadoop versionHadoop 2.6.5Subversion https://github.com/apache/hadoop.git -r e8c9fe0b4c252caf2ebf1464220599650f119997Compiled by sjlee on 2016-10-02T23:43ZCompiled with protoc 2.5.0From source with checksum f05c9fa095a395faa9db9f7ba5d754This command was run using /usr/local/hadoop/share/hadoop/common/hadoop-common-2.6.5.jar
配置hadoop环境变量
~/.bashrc文件下添加
export PATH=$PATH:/usr/local/hadoop/bin:/usr/local/hadoop/sbinsource .bashrc使其生效。
修改hadoop配置文件
进入目录/usr/local/hadoop/etc/hadoop
修改slaves文件
[hadoop@master hadoop]$ cat slavesslave
修改core-site.xml
<configuration> <property> <name>fs.defaultFS</name> <value>hdfs://master:9000</value> </property> <property> <name>hadoop.tmp.dir</name> <value>file:/usr/local/hadoop/tmp</value> <description>Abase for other temporary directories.</description> </property></configuration>
修改hdfs-site.xml
[hadoop@master hadoop]$ cat hdfs-site.xml<?xml version="1.0" encoding="UTF-8"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file.--> <!-- Put site-specific property overrides in this file. --> <configuration> <property> <name>dfs.namenode.secondary.http-address</name> <value>master:50090</value> </property> <property> <name>dfs.replication</name> <value>1</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>file:/usr/local/hadoop/tmp/dfs/name</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>file:/usr/local/hadoop/tmp/dfs/data</value> </property></configuration>
将mapred-site.xml.template修改mapred-site.xml，内容如下：
[hadoop@master hadoop]$ cat mapred-site.xml<?xml version="1.0"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file.--> <!-- Put site-specific property overrides in this file. --> <configuration><property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>mapreduce.jobhistory.address</name> <value>master:10020</value> </property> <property> <name>mapreduce.jobhistory.webapp.address</name> <value>master:19888</value> </property></configuration>
修改文件 yarn-site.xml
[hadoop@master hadoop]$ cat yarn-site.xml<?xml version="1.0"?><!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file.--><configuration> <property> <name>yarn.resourcemanager.hostname</name> <value>master</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property><!-- Site specific YARN configuration properties --> </configuration>
将master服务器的/usr/local/hadoop文件复制到slave服务器相应目录下
并执行
sudo chown -R hadoop:hadoop hadoop/
master服务器初始化
Master 节点执行 NameNode 的格式化：
[hadoop@master ~]$ hdfs namenode -format启动hadoop
start-dfs.shstart-yarn.shmr-jobhistory-daemon.sh start historyserver
通过命令 jps 可以查看各个节点所启动的进程。在 Master 节点上可以看到 NameNode、ResourceManager、SecondrryNameNode、JobHistoryServer 进程，如下图所示：
[hadoop@master ~]$ jps5176 NameNode5883 Jps5836 JobHistoryServer5548 ResourceManager5341 SecondaryNameNode
在 Slave 节点可以看到 DataNode 和 NodeManager 进程，如下图所示：
[hadoop@slave ~]$ jps5302 Jps5179 NodeManager5070 DataNode
另外还需要在 Master 节点上通过命令 hdfs dfsadmin -report 查看 DataNode 是否正常启动，如果 Live datanodes 不为 0 ，则说明集群启动成功。
[hadoop@master ~]$ hdfs dfsadmin -report17/09/01 14:06:58 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicableConfigured Capacity: 51471126528 (47.94 GB)Present Capacity: 43123650560 (40.16 GB)DFS Remaining: 43123625984 (40.16 GB)DFS Used: 24576 (24 KB)DFS Used%: 0.00%Under replicated blocks: 0Blocks with corrupt replicas: 0Missing blocks: 0 -------------------------------------------------Live datanodes (1): Name: 192.168.32.153:50010 (slave)Hostname: slaveDecommission Status : NormalConfigured Capacity: 51471126528 (47.94 GB)DFS Used: 24576 (24 KB)Non DFS Used: 8347475968 (7.77 GB)DFS Remaining: 43123625984 (40.16 GB)DFS Used%: 0.00%DFS Remaining%: 83.78%Configured Cache Capacity: 0 (0 B)Cache Used: 0 (0 B)Cache Remaining: 0 (0 B)Cache Used%: 100.00%Cache Remaining%: 0.00%Xceivers: 1Last contact: Fri Sep 01 14:07:06 CST 2017
四、 执行分布式实例
执行WordCount实例
进入/usr/local/hadoop目录
创建文件夹data_input，并新建两个文本文件，内容任意
mkdir data_inputtouch file1.txt touch file2.txt
执行以下命令
./bin/hadoop fs -mkdir /data./bin/hadoop fs -put -f ./data_input/* /data
执行WordCount命令，并查看结果：
./bin/hadoop jar ./share/hadoop/mapreduce/sources/hadoop-mapreduce-examples-2..6.5-sources.jar org.apache.hadoop.examples.WordCount /data /output17/09/02 12:06:01 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable17/09/02 12:06:08 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.32.182:803217/09/02 12:06:13 INFO input.FileInputFormat: Total input paths to process : 217/09/02 12:06:13 INFO mapreduce.JobSubmitter: number of splits:217/09/02 12:06:15 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1504324771910_000217/09/02 12:06:18 INFO impl.YarnClientImpl: Submitted application application_1504324771910_000217/09/02 12:06:18 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1504324771910_0002/17/09/02 12:06:18 INFO mapreduce.Job: Running job: job_1504324771910_000217/09/02 12:07:09 INFO mapreduce.Job: Job job_1504324771910_0002 running in uber mode : false17/09/02 12:07:09 INFO mapreduce.Job: map 0% reduce 0%17/09/02 12:07:44 INFO mapreduce.Job: map 100% reduce 0%17/09/02 12:08:15 INFO mapreduce.Job: map 100% reduce 100%17/09/02 12:08:16 INFO mapreduce.Job: Job job_1504324771910_0002 completed successfully17/09/02 12:08:18 INFO mapreduce.Job: Counters: 49 File System Counters FILE: Number of bytes read=122 FILE: Number of bytes written=322095 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=278 HDFS: Number of bytes written=81 HDFS: Number of read operations=9 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Launched map tasks=2 Launched reduce tasks=1 Data-local map tasks=2 Total time spent by all maps in occupied slots (ms)=63108 Total time spent by all reduces in occupied slots (ms)=27632 Total time spent by all map tasks (ms)=63108 Total time spent by all reduce tasks (ms)=27632 Total vcore-milliseconds taken by all map tasks=63108 Total vcore-milliseconds taken by all reduce tasks=27632 Total megabyte-milliseconds taken by all map tasks=64622592 Total megabyte-milliseconds taken by all reduce tasks=28295168 Map-Reduce Framework Map input records=10 Map output records=8 Map output bytes=112 Map output materialized bytes=128 Input split bytes=196 Combine input records=8 Combine output records=7 Reduce input groups=6 Reduce shuffle bytes=128 Reduce input records=7 Reduce output records=6 Spilled Records=14 Shuffled Maps =2 Failed Shuffles=0 Merged Map outputs=2 GC time elapsed (ms)=1846 CPU time spent (ms)=20900 Physical memory (bytes) snapshot=699613184 Virtual memory (bytes) snapshot=5563170816 Total committed heap usage (bytes)=603979776 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=82 File Output Format Counters Bytes Written=81
上面的日志显示出了wordCount的详细情况，然后执行查看结果命令查看统计结果：
[hadoop@master hadoop]$ ./bin/hadoop fs -cat /output/part-r-0000017/09/02 12:11:19 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable--file1.txt 1--file2.txt 1dddd 2ddddd 2dddeeeeeeeeeeeee 1dddkkkkkkkkkkkkk 1
wordCount统计结果完成。
关闭 Hadoop 集群也是在 Master 节点上执行的：
stop-yarn.shstop-dfs.shmr-jobhistory-daemon.sh stop historyserver
五、 问题与解决方法
1) 启动hadoop服务时会提示：17/09/01 14:42:01 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
解决办法：原来系统预装的glibc库是2.12版本，而hadoop期望是2.14版本，所以打印警告信息。
方法一：重新编译glibc.2.14版本，安装后专门给hadoop使用。
方法二：直接在log4j日志中去除告警信息。在etc/hadoop/log4j.properties文件中添加
log4j.logger.org.apache.hadoop.util.NativeCodeLoader=ERROR第一种方法风险比较大，本文档使用第二种方法。
2) Master 节点执行 NameNode 的格式化后，在slave节点无法启动DataNode服务
解决方法：这是由于格式化后，会导致master节点的tmp/dfs/name/current/VERSION文件中的clusterID与slave节点下的tmp/dfs/data/current/VERSION文件中的clusterID不一致，所以DataNode无法启动。将slave下的clusterID替换为master下的clusterID即可解决问题。
3) 执行WordCount实例时，有超时现象，remote无法到达，而且使用hdfs dfsadmin –report命令，可以看到datanodes的hostname为一个ip地址，172.16.0.1
解决方法：这是由于之前有同事在slave节点服务器上安装了docker服务，使用过网桥设备，使得系统中多了一个无用的网卡设备，关闭docker服务重启主机即可。
