Hadoop分布式集群搭建

1、Hadoop集群规划

（1）基本

HDFS:  NN（name node）、DN（data node）

YARN: RM（resource manage）、NM（node manage）

（2）对于集群，如何分布以上的节点呢？

通过文件，打开hadoop000文件中的hadoop000.vmx

点击：开启此虚拟机

搭建OOTB环境

第一台机器hadoop001：192.168.131.102 （NameNnode、DataNode、ResourceManager、NodeManager）

第二台机器hadoop002：192.168.131.103 （DataNode、NodeManager）

第三台机器hadoop005：192.168.131.98 （DataNode、NodeManager）

C:\Users\jieqiong>ssh hadoop@192.168.131.101
hadoop@192.168.131.101's password:
Last login: Wed Sep 15 21:58:09 2021 from 192.168.131.1

（3）/etc/hostname：修改hostname（hadoop000/hadoop001/hadoop002）

[hadoop@hadoop000 ~]$ cat /etc/hostname

（4）/etc/host：ip和hostname的映射关系（每台）

[hadoop@hadoop002 ~]$ cat /etc/hosts

192.168.131.102 hadoop001
192.168.131.103 hadoop002
192.168.131.98 hadoop005
192.168.131.102 localhost

 

2、前置安装

（1）ssh免密码登录：ssh-keygen -t rsa

 需要在每一台host上进行操作

（2）在hadoop001机器上

[root@hadoop001 ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop001
[root@hadoop001 ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop002
[root@hadoop001 ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop003

 

（3）测试是否ok

[hadoop@hadoop001 ~]$ ssh hadoop@hadoop001
Last login: Mon Oct 11 20:11:09 2021 from 192.168.131.1
[hadoop@hadoop001 ~]$ exit
logout
Connection to hadoop001 closed.
[hadoop@hadoop001 ~]$ ssh hadoop@hadoop002
Last login: Mon Oct 11 19:55:42 2021 from 192.168.131.1
[hadoop@hadoop002 ~]$ exit
logout
Connection to hadoop002 closed.
[hadoop@hadoop001 ~]$ ssh hadoop@hadoop005
Last login: Mon Oct 11 19:55:48 2021 from 192.168.131.1
[hadoop@hadoop000 ~]$

 

 3、JDK安装

（1）下载

先把jdk-8u91-linux-x64.tar.gz下载到本地，再传到服务器上

在本地命令提示符中，拷贝本地软件包至服务器：scp jdk-8u91-linux-x64.tar.gz hadoop@192.168.131.101:~/software/

在software中显示jdk-8u91-linux-x64.tar.gz，表示成功上传至服务器

[hadoop@hadoop000 ~]$ cd software/
[hadoop@hadoop000 software]$ ls
jdk-8u91-linux-x64.tar.gz

 

（2）解压

解压jdk到服务器~/app/：tar -zxvf jdk-8u91-linux-x64.tar.gz -C ~/app/

进入app文件：cd app/

显示是否已解压：ls

显示jdk1.8.0._91后，表示已经解压

[hadoop@hadoop000 software]$ tar -zxvf jdk-8u91-linux-x64.tar.gz -C ~/app/
[hadoop@hadoop000 software]$ cd
[hadoop@hadoop000 ~]$ cd app/
[hadoop@hadoop000 app]$ ls
jdk1.8.0_91

 （3）添加配置至环境变量中

进入app文件：cd app/

进入jdk1.8.0._91文件：cd jdk1.8.0_91/

显示当前文件地址：pwd

复制当前目录地址：/home/hadoop/app/jdk1.8.0_91

退出至根目录：cd

输入：vi .bash_profile

配置：

export JAVA_HOME=/home/hadoop/app/jdk1.8.0_91

export PATH=$JAVA_HOME/bin:$PATH

 （4）使配置生效

source .bash_profile

 （5）验证

查看JAVA_HOME是否存在：

echo $JAVA_HOME

若显示/home/hadoop/app/jdk1.8.0_91，则表示存在

测试一下：java -version

显示：java version "1.8.0_91"

 （6）目前在一个节点上hadoop001部署完成jdk，并配置了环境变量

 （7）将hadoop001的jdk拷贝到其他节点上hadoop002、hadoop005

[hadoop@hadoop001 app]$ scp -r jdk1.8.0_91 hadoop@hadoop002:~/app/
[hadoop@hadoop001 app]$ scp -r jdk1.8.0_91 hadoop@hadoop005:~/app/

 （8）将hadoop001配置的环境变量拷贝到其他节点上hadoop002、hadoop005

[hadoop@hadoop001 app]$ scp ~/.bash_profile hadoop@hadoop002:~/
[hadoop@hadoop001 app]$ scp ~/.bash_profile hadoop@hadoop005:~/

 （9）配置生效

[hadoop@hadoop002 ~]$ source ~/.bash_profile
[hadoop@hadoop005 ~]$ source ~/.bash_profile

 

4、Hadoop集群部署

 （1）HDFS安装

●下载

先把hadoop-2.6.0-cdh5.15.1.tar.gz下载到本地

●从本地上传至服务器

在本地命令提示符界面先进入到压缩包所在的文件地址中

输入：scp hadoop-2.6.0-cdh5.15.1.tar.gz hadoop@192.168.131.101:~/software/

●解压

在进入到software/：cd software/

查看文件内存大小：ll -lh

解压到服务器~/app/：tar -zxvf hadoop-2.6.0-cdh5.15.1.tar.gz -C ~/app/

在app文件中显示hadoop-2.6.0-cdh5.15.1即ok

 ●hadoop-env.sh

配置JAVA_HOME

进入：

/home/hadoop/app/hadoop-2.6.0-cdh5.15.1/etc/hadoop 目录下

输入：vi hadoop-env.sh 

配置

将原本的

export JAVA_HOME=${JAVA_HOME}

改为：

#export JAVA_HOME=${JAVA_HOME}

export JAVA_HOME=/home/hadoop/app/jdk1.8.0_91

其中：以下不需要修改

export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop"}

●core-site.xml

配置文件系统的地址

进入目录：/home/hadoop/app/hadoop-2.6.0-cdh5.15.1/etc/hadoop

输入：vi core-site.xml

配置文件系统一个namespace的主节点hadoop001

<property>

    <name>fs.defaultFS</name>

    <value>hdfs://hadoop001:8020</value>

</property>

指默认的文件系统指向hadoop000的主机，端口号为8020。

●创建tmp文件夹

进入目录：/home/hadoop/app

创建tmp文件夹：mkdir tmp

此时tmp中为空，之后所有hadoop相关的都放至此处tmp文件夹中。

●hdfs-site.xml 

配置路径存放地址以及副本系数

输入：vi hdfs-site.xml 

配置副本系数DataNode

<property>
    <name>dfs.replication</name>
    <value>3</value>
</property>

多个节点时配置

<property>
    <name>dfs.namenode.name.dir</name>
    <value>/home/hadoop/app/tmp/dfs/name</value>
</property>

<property>
    <name>dfs.datanode.data.dir</name>
    <value>/home/hadoop/app/tmp/dfs/data</value>
</property>

一个节点时配置：

<property>

    <name>hadoop.tmp.dir</name>

    <value>/home/hadoop/app/tmp</value>

</property>

●yarn-site.xml

配置resourcemanager

<property>
  <name>yarn.nodemanager.aux-services</name>
  <value>mapreduce_shuffle</value>
</property>

<property>
  <name>yarn.resourcemanager.hostname</name>
  <value>hadoop001</value>
</property>

●mapred-site.xml

首先根据模板，先拷贝模板

[hadoop@hadoop000 hadoop]$ cp mapred-site.xml.template mapred-site.xml

然后在mapred-site.xml中进行配置

<property>
  <name>mapreduce.framework.name</name>
  <value>yarn</value>
</property>

●slaves

配置从节点

进入目录：/home/hadoop/app/hadoop-2.6.0-cdh5.15.1/etc/hadoop

输入：vi slaves

将localhost 改为

hadoop001
hadoop002
hadoop005

●分发hadoop至其他机器

[hadoop@hadoop001 app]$ scp -r hadoop-2.6.0-cdh5.15.1 hadoop@hadoop002:~/app/
[hadoop@hadoop002 app]$ scp -r hadoop-2.6.0-cdh5.15.1 hadoop@hadoop005:~/app/

●配置环境变量

进入目录：/home/hadoop/app/hadoop-2.6.0-cdh5.15.1/etc/hadoop

输入：vi ~/.bash_profile

配置

export HADOOP_HOME=/home/hadoop/app/hadoop-2.6.0-cdh5.15.1
export PATH=$HADOOP_HOME/bin:$PATH

执行配置：source ~/.bash_profile

●拷贝配置文件

[hadoop@hadoop001 app]$ scp ~/.bash_profile hadoop@hadoop002:~/
[hadoop@hadoop001 app]$ scp ~/.bash_profile hadoop@hadoop005:~/

 

●NameNode格式化HDFS

只在001上执行即可

此时/home/hadoop/app/tmp文件夹中为空

第一次执行的时候，一定要格式化文件系统，后面不要重复执行。

在目录下： $HADOOP_HOME/bin

执行格式化：hdfs namenode -format

看到Storage directory /home/hadoop/app/tmp/dfs/name has been successfully formatted.表示执行成功

●启动HDFS

启动集群（单机）

进入目录：/home/hadoop/app/hadoop-2.6.0-cdh5.15.1/sbin

启动输入：./start-dfs.sh 

验证是否启动成功输入：jps

●重点，导致问题的根本原因，在/etc/hosts文件中

在目录下/home/hadoop/app/hadoop-2.6.0-cdh5.15.1/etc/hadoop

输入：sudo vi /etc/hosts

将内容更改为以下：

即：把原本的127.0.0.1删除，再增添以下内容

127.0.0.1 localhost localhost.localdomain localhost4.localdomain4

::1       localhost localhost.localdomain localhost6.localdomain6

●成功启动dfs

进入网址：http://192.168.131.102:50070/

[hadoop@hadoop001 sbin]$ ./start-dfs.sh
Starting namenodes on [hadoop001]
hadoop001: starting namenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.15.1/logs/hadoop-hadoop-namenode-hadoop001.out
hadoop001: starting datanode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.15.1/logs/hadoop-hadoop-datanode-hadoop001.out
hadoop002: starting datanode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.15.1/logs/hadoop-hadoop-datanode-hadoop002.out
hadoop005: starting datanode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.15.1/logs/hadoop-hadoop-datanode-hadoop005.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.15.1/logs/hadoop-hadoop-secondarynamenode-hadoop001.out
[hadoop@hadoop001 sbin]$ jps
10592 Jps
10149 NameNode
10472 SecondaryNameNode
2853 DataNode
[hadoop@hadoop002 hadoop]$ jps
7120 Jps
7027 DataNode

[hadoop@hadoop005 ~]$ jps
2537 DataNode
2633 Jps

 

●成功启动yarn

进入网址：http://192.168.131.102:8088/

[hadoop@hadoop001 sbin]$ ./start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.15.1/logs/yarn-hadoop-resourcemanager-hadoop001.out
hadoop001: starting nodemanager, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.15.1/logs/yarn-hadoop-nodemanager-hadoop001.out
hadoop005: starting nodemanager, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.15.1/logs/yarn-hadoop-nodemanager-hadoop005.out
hadoop002: starting nodemanager, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.15.1/logs/yarn-hadoop-nodemanager-hadoop002.out
[hadoop@hadoop001 sbin]$ jps
10664 ResourceManager
10892 NodeManager

[hadoop@hadoop002 hadoop]$ jps
7173 NodeManager

[hadoop@hadoop005 ~]$ jps
2698 NodeManager

 

5、作业提交到Hadoop集群上运行

[hadoop@hadoop002 mapreduce]$ pwd
/home/hadoop/app/hadoop-2.6.0-cdh5.15.1/share/hadoop/mapreduce
[hadoop@hadoop002 mapreduce]$ hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.15.1.jar pi 2 3

异常一：block丢失

[hadoop@hadoop001 sbin]$ hadoop dfsadmin -report

原因datanode配置错误，改为以下：

[hadoop@hadoop001 sbin]$ cd /home/hadoop/app/hadoop-2.6.0-cdh5.15.1/etc/hadoop
[hadoop@hadoop001 hadoop]$ vi hdfs-site.xml

<property>
    <name>dfs.datanode.data.dir</name>
    <value>/home/hadoop/app/tmp/dfs/data</value>
</property>


[hadoop@hadoop002 sbin]$ cd /home/hadoop/app/hadoop-2.6.0-cdh5.15.1/etc/hadoop
[hadoop@hadoop002 hadoop]$ vi hdfs-site.xml

<property>
    <name>dfs.datanode.data.dir</name>
    <value>/home/hadoop/app/tmp/dfs/data/node1</value>
</property>


[hadoop@hadoop005 ~]$ cd /home/hadoop/app/hadoop-2.6.0-cdh5.15.1/etc/hadoop
[hadoop@hadoop005 hadoop]$ vi hdfs-site.xml

<property>
    <name>dfs.datanode.data.dir</name>
    <value>/home/hadoop/app/tmp/dfs/data/node2</value>
</property>

 

6、解决上述问题后，运行jar包，虽然出结果，但是报错。 

posted @ 2021-10-12 17:12  酱汁怪兽  阅读(210)  评论(0)    收藏  举报