centos7.9上hadoop-2.7.2伪分布式部署

1.安装jdk

1.1 在Oracle官网上现在jdk1.8 ，然后上传到Linux服务器中

1.2  安装jdk 

rpm -ivh jdk-8u371-linux-x64.rpm

2 创建部署用户 hadoop

useradd -d /hadoop hadoop

echo 123 |passwd --stdin hadoop

3 修改/etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.100.10 node1

　　

4 使用Hadoop用户上传hadoop安装包hadoop-2.7.2.tar.gz 并解压

tar -zxf hadoop-2.7.2.tar.gz -C /hadoop/

5.设置软链接

cd /hadoop 
ln -s hadoop-2.7.2 hadoop

6.修改/etc/profile

export JAVA_HOME=/usr/lib/jvm/jdk-1.8-oracle-x64
export HADOOP_HOME=/hadoop/hadoop

export PATH=$PATH:$JAVA_HOME:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

7.配置namenode服务

7.1 修改配置文件

cd hadoop/etc/hadoop/
vim core-site.xml
<configuration>

  <property>
    <name>fs.defaultFS</name>
        <value>hdfs://node1:9000</value>
  </property>

  <property>
    <name>hadoop.tmp.dir</name>
        <value>/hadoop/hadoop/data</value>
  </property>

</configuration>

7.2  格式化namenode

hadoop namenode -format

　　当输出日志中出现NFO common.Storage: Storage directory /hadoop/hadoop/data/dfs/name has been successfully formatted.时，说明格式化成功.

7.3 启动namenode服务

7.4  在浏览器中输入服务器IP:50070   注：core-site 中配置的9000端口为rpc通信端口，50070为http端口

 

 8.启动datanode

-bash-4.2$ hadoop-daemon.sh  start datanode
starting datanode, logging to /hadoop/hadoop-2.7.2/logs/hadoop-hadoop-datanode-node1.out
-bash-4.2$ jps
11638 NameNode
13016 Jps
12940 DataNode
-bash-4.2$ 

 9.启动yarn

9.1 配置yarn 

cd hadoop/etc/hadoop/
vim mapred-site.xml  注：此处为告诉hadoop运算框架为yarn，默认为local
<configuration>
        <property>
          <name>mapreduce.framework.name</name>
          <value>yarn</value>
        </property>
</configuration>

vim yarn-site.xml 注：此处为告诉hadoop yarn运行在哪个机器上,mapreduce_shuffle为Hadoop在reduce阶段如何获取数据，shuffle为从map结果中拉取
  <property>
    <name>yarn.resourcemanager.hostname</name>
        <value>node1</value>
  </property>

  <property>
    <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
  </property>

9.2 启动yarn

-bash-4.2$ yarn-daemon.sh start resourcemanager 
starting resourcemanager, logging to /hadoop/hadoop-2.7.2/logs/yarn-hadoop-resourcemanager-node1.out
-bash-4.2$ yarn-daemon.sh start nodemanager
starting nodemanager, logging to /hadoop/hadoop-2.7.2/logs/yarn-hadoop-nodemanager-node1.out
-bash-4.2$ jps
15073 NodeManager
15108 Jps
11638 NameNode
14823 ResourceManager
12940 DataNode
-bash-4.2$