Linux环境下安装hadoop分布式集群+问题总结

一、环境

  • Red Hat 4.8.5
  • Hadoop 2.7.6
  • JDK1.8.0
  • 集群环境:两台主机,一台作为Master,主机名为:master;一台作为Slave,主机名为:slave1

二、准备工作:配置ssh免密码登录

  请参考博客:配置集群模式下的ssh免密码登录

三、安装步骤:

  在master和slave1上分别安装JAVA:下载jdk压缩包,添加环境变量

  在master上下载hadoop压缩包,添加环境变量

export JAVA_HOME=/home/meng/pp/jdk
export CLASS_HOME=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export HADOOP_HOME=/home/meng/app/hadoop
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin

  修改Hadoop目录下/etc/hadoop文件夹中的配置文件

  所需要修改的配置文件在解压缩后的hadoop目录下的/etc/hadoop文件夹

  这些文件分别是:

  hadoop-env.sh

  core-site.xml

  hdfs-site.xml

  mapred-site.xml

  yarn-site.xml

  slaves

  master

注意:在本文中统一使用master作为主机名配置地址信息,实际操作中可以使用IP或者本机主机名代替

  若想修改主机名,可以通过修改/etc/hostname文件,重启系统生效

hadoop-env.sh文件:

  用于配置运行环境,需要添加java路径

# The java implementation to use.
export JAVA_HOME=/home/meng/app/jdk

core-site.xml文件:

  定义hdfs中namenode的地址,形式是主机名+端口号

  tmp文件夹地址可自行设置,必须是有效路径,tmp文件夹本身系统会自动创建

<configuration>
<!-- 指定HADOOP所使用的文件系统schema(URI),HDFS的老大(NAMENODE)的地址  -->
<property>
        <name>fs.defaultFS</name>
        <value>hdfs://master:9000</value>
</property>
<property>
        <name>hadoop.tmp.dir</name>
        <value>file:/home/meng/app/hadoop/tmp</value>
</property>
<property>
        <name>io.file.buffer.size</name>
        <value>131702</value>
</property>
</configuration>

hdfs-site.xml文件:

  指定namenode和datanode的地址,必须是有效路径

  指定hdfs副本的数量,自行设置

  指定secondary namenode的地址,主机名或者IP+端口

<configuration>
<!-- 指定namenode路径  -->
<property>
        <name>dfs.namenode.name.dir</name>
        <value>/home/meng/app/hadoop/namenode</value>
</property>
<!-- 指定datanode路径 -->
<property>
        <name>dfs.datanode.name.dir</name>
        <value>/home/meng/app/hadoop/datanode</value>
</property>
<!-- 指定hdfs副本的数量 -->
<property>
        <name>dfs.replication</name>
        <value>2</value>
</property>
<property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>master:9001</value>
</property>
<property>
        <name>dfs.webhdfs.enabled</name>
        <value>true</value>
</property>
</configuration>

mapred-site.xml:

  若文件夹下没有该文件,可将mapred-site.xml.template复制重命名为mapred-site.xml

cp mapred-site.xml.template mapred-site.xml
<configuration>
<!-- 指定mapreduce 运行在yarn上  -->
<property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
        <final>true</final>
</property>
</configuration>

yarn-site.xml:

  设置yarn的主机地址,使用IP或者主机名

<configuration>
<!-- Site specific YARN configuration properties -->
<!-- 设置yarn的主机地址 -->
<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>master</value>
</property>
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property
</configuration>

master:

  加入master的IP或者主机名

hadoop

slaves:

  加入slaves的IP或者主机名,伪分布式可填写master的IP或者主机名

slave1
slave2

  第五步:启动hadoop

  首次启动,需要格式化namenode

hdfs namenode -format

  在hadoop安装目录下有一个sbin文件夹

derby.log           hdfs-config.sh  metastore_db        start-all.sh         start-yarn.sh     stop-secure-dns.sh
distribute-exclude.sh  httpfs.sh       mr-jobhistory-daemon.sh    start-balancer.sh    stop-all.sh       stop-yarn.sh
hadoop-daemon.sh       kms.sh           refresh-namenodes.sh    start-dfs.sh         stop-balancer.sh  yarn-daemon.sh
hadoop-daemons.sh      Linux           slaves.sh        start-secure-dns.sh  stop-dfs.sh       yarn-daemons.sh

   执行start-all.sh文件即可。

  在完成上述设置后,笔者的datanode始终运行不了,后来几经修改尝试,根据core-site.xml中配置的tmp路径,找到对应的tmp文件夹,将其清空之后,便可运行成功。

  而在这些节点报错时,我们需要如何看错误日志呢?当我们执行start-all.sh文件文件时,控制台会打印这些信息,而下划线标注的路径即时log日志保存的路径。

This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
18/12/06 21:27:37 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [192.168.72.128]
192.168.72.128: starting namenode, logging to /home/meng/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-meng-namenode-hadoop.out
192.168.72.128: starting datanode, logging to /home/meng/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-meng-datanode-hadoop.out
Starting secondary namenodes [hadoop]
hadoop: starting secondarynamenode, logging to /home/meng/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-meng-secondarynamenode-hadoop.out
18/12/06 21:27:55 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
starting yarn daemons
starting resourcemanager, logging to /home/meng/app/hadoop-2.6.0-cdh5.7.0/logs/yarn-meng-resourcemanager-hadoop.out
192.168.72.128: starting nodemanager, logging to /home/meng/app/hadoop-2.6.0-cdh5.7.0/logs/yarn-meng-nodemanager-hadoop.out

  datanode的日志文件报错如下:

192.168.72.128:9000 starting to offer service
2018-12-06 04:49:08,925 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2018-12-06 04:49:08,925 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 50020: starting
2018-12-06 04:49:09,530 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/meng/app/tmp/dfs/data/in_use.lock acquired by nodename 21391@localhost
2018-12-06 04:49:09,541 WARN org.apache.hadoop.hdfs.server.common.Storage: java.io.IOException: Incompatible clusterIDs in /home/meng/app/tmp/dfs/data: namenode clusterID = CID-c050c6d2-b27f-4db1-acfc-cfde956261a2; datanode clusterID = CID-0a0802fc-33a1-4cb0-b179-04c30ec901a5
2018-12-06 04:49:09,542 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for Block pool <registering> (Datanode Uuid unassigned) service to /192.168.72.128:9000. Exiting. 
java.io.IOException: All specified directories are failed to load.
    at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:478)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1394)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1355)
    at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:317)
    at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:228)
    at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:829)
    at java.lang.Thread.run(Thread.java:748)
2018-12-06 04:49:09,543 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service for: Block pool <registering> (Datanode Uuid unassigned) service to /192.168.72.128:9000
2018-12-06 04:49:09,545 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Removed Block pool <registering> (Datanode Uuid unassigned)
2018-12-06 04:49:11,546 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Exiting Datanode
2018-12-06 04:49:11,548 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 0
2018-12-06 04:49:11,566 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at localhost/127.0.0.1
************************************************************/

  通过阅读上述日志可了解大意:namenode和datanode的簇ID不相容,而这些簇ID是保存在tmp文件夹下的,想必是由于笔者多次修改配置文件导致的一些问题,并不是一定会出现的,虽然不了解簇ID的生成机制,但是通过情况tmp文件应该会使其重新生成,问题便可被解决。

posted @ 2018-12-07 13:20 头发马上掉光光 阅读(...) 评论(...) 编辑 收藏