在CentOS7上搭建hadoop集群

hadoop官方下载地址https://hadoop.apache.org/releases.html
用到的命令/root/hadoop-3.4.0/bin/hdfs namenode -format ，start-all.sh， stop-all.sh
1.三台虚拟机分别设置静态ip
2.设置免密登录 （更改主机名、添加映射）
3.配置hadoop （添加环境变量）
core-site.xml hdfs-site.xml mapred-site.xml yarn-site.xml workes

点击查看代码

  <property>
        <name>fs.default.name</name>
        <value>hdfs://master:9000</value>
  </property>
  <property>
        <name>hadoop.tmp.dir</name>
        <value>/root/hadoop-3.4.0/tmp</value>
  </property>

点击查看代码

      <property>
        <name>dfs.replication</name>
        <value>2</value>
      </property>
      <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:/root/hadoop-3.4.0/dfs/name</value>
      </property>
      <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:/root/hadoop-3.4.0/dfs/data</value>
      </property>
#新增一条，web管理
<property>
<name>dfs.http.address</name>
<value>0.0.0.0:50070 </value>
</property>

点击查看代码

  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
  <property>
    <name>mapreduce.jobhistory.address</name>
    <value>Master:10020</value>
  </property>
  <property>
    <name>mapreduce.jobhistory.webapp.address</name>
    <value>Master:19888</value>
  </property>

点击查看代码

   <!-- Site specific YARN configuration properties -->
      <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
      </property>
      <property>
        <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
      </property>
      <property>
        <name>yarn.resourcemanager.address</name>
        <value>master:8032</value>
      </property>
      <property>
    <name>yarn.resourcemanager.scheduler.address</name>
        <value>master:8030</value>
      </property>
      <property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>master:8031</value>
      </property>
      <property>
        <name>yarn.resourcemanager.admin.address</name>
        <value>master:8033</value>
      </property>
      <property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>master:8088</value>
# 新增内容②，hive测试，value通过hadoop classpath查询
      <property>
        <name>yarn.application.classpath</name>
        <value>/root/hadoop-3.4.0/etc/hadoop:/root/hadoop-3.4.0/share/hadoop/common/lib/*:/root/hadoop-3.4.0/share/hadoop/common/*:/root/hadoop-3.4.0/share/hadoop/hdfs:/root/hadoop-3.4.0/share/hadoop/hdfs/lib/*:/root/hadoop-3.4.0/share/hadoop/hdfs/*:/root/hadoop-3.4.0/share/hadoop/mapreduce/*:/root/hadoop-3.4.0/share/hadoop/yarn:/root/hadoop-3.4.0/share/hadoop/yarn/lib/*:/root/hadoop-3.4.0/share/hadoop/yarn/*</value>   
      </property>

      </property>

点击查看代码

master
clone1
clone2

4.初始化、启动与关闭

更改主机名用于区分，添加映射以便ssh时用主机名代替ip
在linux里添加hadoop的环境变量，在hadoop里添加java的环境变量。

点击查看代码

[root@master ~]# vim /etc/profile

//省略前文
#Java
export JAVA_HOME=/root/jdk1.8.0_401
export CLASSPATH=.:$JAVA_HOME/lib/tools.jar:JAVA_HOME/lib/dt.jar
export PATH=$PATH:$JAVA_HOME/bin
#Hadoop
export HADOOP_HOME=/root/hadoop-3.4.0
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HADOOP_HOME/libexec
export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root


[root@master hadoop-3.4.0]# vim  /root/had*/etc/hadoop/hadoop-env.sh

//省略前文
#  JAVA_HOME=/usr/java/testing hdfs dfs -ls
JAVA_HOME=/root/jdk1.8.0_401
#
# Therefore, the vast majority (BUT NOT ALL!) of these defaults
# are configured for substitution and not append.  If append
# is preferable, modify this file accordingly.
//省略后文





[root@master hadoop-3.4.0]# vim  /root/had*/etc/hadoop/yarn-env.sh

//省略前文
###

# YARN Services parameters
###
# Directory containing service examples
# export YARN_SERVICE_EXAMPLES_DIR = $HADOOP_YARN_HOME/share/hadoop/yarn/yarn-service-examples
# export YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE=true
JAVA_HOME=/root/jdk1.8.0_401

用scp在启动前把配置好的文件和hadoop发送过去 内存需要大一点，2g会卡住。