hadoop hdfs简单集群安装

hadoop集群三个节点

hadoop-master1, hadoop-slave1,hadoop-slave2

1、安装jdk 1.8

2、免密码登录，配置hosts解析 所有机器上都要配置一致，包括自身host解析

3、下载hadoop2.9 解压到 /usr/local/

4、配置环境变量

#java环境变量
export JAVA_HOME=/usr/local/jdk1.8.0_131
export JRE_HOME=/usr/local/jdk1.8.0_131/jre
export PATH=$PATH:/usr/local/jdk1.8.0_131/bin:/usr/local/jdk1.8.0_131/jre/bin


#hadoop环境变量
export HADOOP_HOME=/usr/local/hadoop 
export HADOOP_MAPRED_HOME=$HADOOP_HOME 
export HADOOP_COMMON_HOME=$HADOOP_HOME 
export HADOOP_HDFS_HOME=$HADOOP_HOME 
export YARN_HOME=$HADOOP_HOME 
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native 
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin 
export HADOOP_INSTALL=$HADOOP_HOME

5、编辑hadoop配置文件  /usr/local/hadoop/etc/hadoop/ 目录下

   hadoop-env.sh, core-site.xml, hdfs-site.xml, mapred-site.xml, yarn-site.xml ， masters， slaves

hadoop-enf.sh

#加入JAVA_HOME路径
export JAVA_HOME=/usr/local/jdk1.8.0_131

core-site.xml

<configuration>
    <property>  
        <name>fs.defaultFS</name>  
        <value>hdfs://hadoop-master1:9000</value>  
    </property>  
    <property>  
        <name>hadoop.tmp.dir</name>  
        <value>file:/opt/hadoop/tmp</value>  
        <description>Abase for other temporary   directories.</description>  
    </property>
</configuration>

hdfs-site.xml

<configuration>
     <property>  
              <name>dfs.namenode.secondary.http-address</name>  
              <value>master:9001</value>  
     </property>  
     <property>  
             <name>dfs.namenode.name.dir</name>  
             <value>file:/opt/hadoop/dfs/name</value>  
      </property>  
      <property>  
              <name>dfs.datanode.data.dir</name>  
              <value>file:/opt/hadoop/dfs/data</value>  
      </property>  
      <property>  
               <name>dfs.replication</name>  
               <value>2</value>  
      </property>  
      <property>  
               <name>dfs.webhdfs.enabled</name>  
               <value>true</value>  
      </property> 
</configuration>

mapred-site.xml

<configuration>
          <property>                                                                    
　　　　　　　　<name>mapreduce.framework.name</name>  
                <value>yarn</value>  
           </property>
</configuration>

yarn-site.xml

  

<configuration>

<!-- Site specific YARN configuration properties -->

        <property>  
               <name>yarn.nodemanager.aux-services</name>  
               <value>mapreduce_shuffle</value>  
        </property> 
</configuration>

masters

hadoop-master1

 

slaves

hadoop-salve1
hadoop-slave2

 

 

6、hadoop可执行文件  /usr/local/hadoop/[bin,sbin]

因为hadoop路径已经加入环境变量

格式化 hadoop namenode -format

启动 start-all.sh

关闭stop-all.sh

查看是否启动master和slaves上分别执行  jps

查看hdfs集群  hdfs dfsadmin -report

 

7、常见问题：保持 slave上datanode(目录)/current/VERSION 与 master上namenode(目录)/current/VERSION中

slave datanode 与master namenode 保持一致

clusterID=CID-3d8485ee-025a-4ef2-8533-7faa94cf8d5a

 

8、 所有master，slave中hadoop配置文件完全相同，只需要配置一遍然后拷贝到其他机器中即可

存放namenode 和datanode的路径需要指定(hdfs-site.xml)，并手动创建目录