【Hadoop】单机、伪分布式、完全分布式集群搭建

搭建Hadoop本地模式

本地模式就是单机装hadoop。

安装hadoop

上传hadoop包

通过winSCP上传hadoop包到/opt/soft/文件夹下

[root@bigdata111 soft]# ls
hadoop-2.8.4.tar.gz  jdk-8u144-linux-x64.tar.gz

解压hadoop

解压hadoop到/opt/module/下

[root@bigdata111 module]# tar -zvxf hadoop-2.8.4.tar.gz -C /opt/module/
[root@bigdata111 soft]# cd /opt/module/
[root@bigdata111 module]# ls
hadoop-2.8.4  jdk1.8.0_144

设置hadoop环境变量

[root@bigdata111 module]# vi /etc/profile

末尾添加如下配置，保存退出：

export HADOOP_HOME=/opt/module/hadoop-2.8.4
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

刷新配置文件

[root@bigdata111 module]# source /etc/profile

查看hadoop是否安装成功

[root@bigdata111 module]# hadoop
Usage: hadoop [--config confdir] [COMMAND | CLASSNAME]
  CLASSNAME            run the class named CLASSNAME
 or
  where COMMAND is one of:
  fs                   run a generic filesystem user client
  version              print the version
  jar <jar>            run a jar file
                       note: please use "yarn jar" to launch
                             YARN applications, not this command.
  checknative [-a|-h]  check native hadoop and compression libraries availability
  distcp <srcurl> <desturl> copy file or directories recursively
  archive -archiveName NAME -p <parent path> <src>* <dest> create a hadoop archive
  classpath            prints the class path needed to get the
                       Hadoop jar and the required libraries
  credential           interact with credential providers
  daemonlog            get/set the log level for each daemon
  trace                view and modify Hadoop tracing settings

Most commands print help when invoked w/o parameters.

测试hadoop实例

创建测试文件

在module目录下新建testdoc文件，输入文本：

[root@bigdata111 module]# cd /opt/module
[root@bigdata111 module]# touch testdoc
[root@bigdata111 module]# vi testdoc
[root@bigdata111 module]# cat testdoc
this is a test page!
chinese is the best country
this is a ceshi page!
i love china
listen to the music
and son on

切换jar包目录

切换到hadoop的jar包执行目录：

[root@bigdata111 module]# cd /opt/module/hadoop-2.8.4/share/hadoop/mapreduce/
[root@bigdata111 mapreduce]# ls
hadoop-mapreduce-client-app-2.8.4.jar     hadoop-mapreduce-client-core-2.8.4.jar  hadoop-mapreduce-client-hs-plugins-2.8.4.jar  hadoop-mapreduce-client-jobclient-2.8.4-tests.jar  hadoop-mapreduce-examples-2.8.4.jar  lib           sources
hadoop-mapreduce-client-common-2.8.4.jar  hadoop-mapreduce-client-hs-2.8.4.jar    hadoop-mapreduce-client-jobclient-2.8.4.jar   hadoop-mapreduce-client-shuffle-2.8.4.jar          jdiff                                lib-examples

执行wordcount程序

[root@bigdata111 mapreduce]# hadoop jar hadoop-mapreduce-examples-2.8.4.jar wordcount /opt/module/testdoc /opt/module/out
[root@bigdata111 mapreduce]# ls /opt/module/out
part-r-00000  _SUCCESS
[root@bigdata111 mapreduce]# cat /opt/module/out/part-r-00000
a	2
and	1
best	1
ceshi	1
china	1
chinese	1
country	1
i	1
is	3
listen	1
love	1
music	1
on	1
page!	2
son	1
test	1
the	2
this	2
to	1

搭建Hadoop伪分布式

伪分布式就是在单台机器上配置分布式操作。

查看hadoop可执行文件

[root@bigdata111 mapreduce]# cd /opt/module/hadoop-2.8.4/
[root@bigdata111 hadoop-2.8.4]# ls
bin  etc  include  lib  libexec  LICENSE.txt  NOTICE.txt  README.txt  sbin  share
[root@bigdata111 hadoop-2.8.4]# cd bin
[root@bigdata111 bin]# ls
container-executor  hadoop  hadoop.cmd  hdfs  hdfs.cmd  mapred  mapred.cmd  rcc  test-container-executor  yarn  yarn.cmd
[root@bigdata111 bin]# cd ..
[root@bigdata111 hadoop-2.8.4]# cd sbin
[root@bigdata111 sbin]# ls
distribute-exclude.sh  hadoop-daemons.sh  hdfs-config.sh  kms.sh                   refresh-namenodes.sh  start-all.cmd  start-balancer.sh  start-dfs.sh         start-yarn.cmd  stop-all.cmd  stop-balancer.sh  stop-dfs.sh         stop-yarn.cmd  yarn-daemon.sh
hadoop-daemon.sh       hdfs-config.cmd    httpfs.sh       mr-jobhistory-daemon.sh  slaves.sh             start-all.sh   start-dfs.cmd      start-secure-dns.sh  start-yarn.sh   stop-all.sh   stop-dfs.cmd      stop-secure-dns.sh  stop-yarn.sh   yarn-daemons.sh

切换配置文件目录

进入到hadoop设置/opt/module/hadoop-2.8.4/etc/hadoop/目录：

[root@bigdata111 hadoop]# cd /opt/module/hadoop-2.8.4/etc/hadoop/
[root@bigdata111 hadoop]# ls
capacity-scheduler.xml  core-site.xml   hadoop-metrics2.properties  hdfs-site.xml            httpfs-signature.secret  kms-env.sh            log4j.properties  mapred-queues.xml.template  ssl-client.xml.example  yarn-env.sh
configuration.xsl       hadoop-env.cmd  hadoop-metrics.properties   httpfs-env.sh            httpfs-site.xml          kms-log4j.properties  mapred-env.cmd    mapred-site.xml.template    ssl-server.xml.example  yarn-site.xml
container-executor.cfg  hadoop-env.sh   hadoop-policy.xml           httpfs-log4j.properties  kms-acls.xml             kms-site.xml          mapred-env.sh     slaves                      yarn-env.cmd

配置core-site.xml

[root@bigdata111 hadoop]# vi core-site.xml

<configuration>

<!-- 指定HDFS中NameNode的地址 -->
<property>
<name>fs.defaultFS</name>
<value>hdfs://bigdata111:9000</value>
</property>

<!-- 指定hadoop运行时产生文件的存储目录 -->
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/module/hadoop-2.8.4/data/tmp</value>
</property>

</configuration>

配置hdfs-site.xml

[root@bigdata111 hadoop]# vi hdfs-site.xml

<configuration>

<!--数据冗余数-->
<property>
<name>dfs.replication</name>
<value>1</value>
</property>

</configuration>

配置yarn-site.xml

[root@bigdata111 hadoop]# vi yarn-site.xml

<configuration>

<!-- Site specific YARN configuration properties -->

<!-- reducer获取数据的方式 -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>

<!-- 指定YARN的ResourceManager的地址 -->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>bigdata111</value>
</property>

<!-- 日志聚集功能使能 -->
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>

<!-- 日志保留时间设置7天(秒) -->
<property>
<name>yarn.log-aggregation.retain-seconds</name>
<value>604800</value>
</property>


</configuration>

配置mapred-site.xml

重命名mapred-site.xml.template为mapred-site.xml，配置内容

[root@bigdata111 hadoop]# mv mapred-site.xml.template mapred-site.xml
[root@bigdata111 hadoop]# ls
capacity-scheduler.xml  core-site.xml   hadoop-metrics2.properties  hdfs-site.xml            httpfs-signature.secret  kms-env.sh            log4j.properties  mapred-queues.xml.template  ssl-client.xml.example  yarn-env.sh
configuration.xsl       hadoop-env.cmd  hadoop-metrics.properties   httpfs-env.sh            httpfs-site.xml          kms-log4j.properties  mapred-env.cmd    mapred-site.xml             ssl-server.xml.example  yarn-site.xml
container-executor.cfg  hadoop-env.sh   hadoop-policy.xml           httpfs-log4j.properties  kms-acls.xml             kms-site.xml          mapred-env.sh     slaves                      yarn-env.cmd
[root@bigdata111 hadoop]# vi mapred-site.xml

<configuration>

<!-- 指定mr运行在yarn上-->
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>

<!--历史服务器的地址-->
<property>
<name>mapreduce.jobhistory.address</name>
<value>bigdata111:10020</value>
</property>

<!--历史服务器页面的地址-->
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>bigdata111:19888</value>
</property>


</configuration>

配置hadoop-env.sh

修改java_home为绝对路径，保存退出：

[root@bigdata111 hadoop]# vi hadoop-env.sh

export JAVA_HOME=/opt/module/jdk1.8.0_144

格式化namenode

配置完毕，格式化namenode(仅第一次格式化文件)

[root@bigdata111 hadoop]# hadoop namenode -format

为什么要格式化？

NameNode主要被用来管理整个分布式文件系统的命名空间(实际上就是目录和文件)的元数据信息，同时为了保证数据的可靠性，还加入了操作日志，所以，NameNode会持久化这些数据(保存到本地的文件系统中)。对于第一次使用HDFS，在启动NameNode时，需要先执行-format命令，然后才能正常启动NameNode节点的服务。

格式化做了哪些事情？

在NameNode节点上，有两个最重要的路径，分别被用来存储元数据信息和操作日志，而这两个路径来自于配置文件，它们对应的属性分别是dfs.name.dir和dfs.name.edits.dir，同时，它们默认的路径均是/tmp/hadoop/dfs/name。格式化时，NameNode会清空两个目录下的所有文件，之后，会在目录dfs.name.dir下创建文件

hadoop.tmp.dir 这个配置，会让dfs.name.dir和dfs.name.edits.dir会让两个目录的文件生成在一个目录里

开启hdfs和yarn服务

当namenode和resourcemanager在一台机器时，使用如下命令：

[root@bigdata111 hadoop]# start-all.sh

当二者不为一台机器时，使用如下命令：

[root@bigdata111 hadoop]# start-dfs.sh
[root@bigdata111 hadoop]# start-yarn.sh

访问hdfs 的web页面

默认端口：50070

http://192.168.1.111:50070

访问yarn的web页面

默认端口：8088

http://192.168.1.111:8088

搭建Hadoop完全分布式集群

采用VMvare克隆模式，以111机器为模板，克隆另外两台机器。

修改主机名和IP

修改克隆的两台机器的hostname和IP地址，方便xshell连接：

[root@bigdata112 ~]# vi /etc/hostname
[root@bigdata112 ~]# vi /etc/sysconfig/network-scripts/ifcfg-eno16777736
[root@bigdata112 ~]# service network restart
[root@bigdata112 ~]# ip addr

TYPE=Ethernet
BOOTPROTO=static
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_FAILURE_FATAL=no
NAME=eno16777736
UUID=24bbe130-f59a-4b25-9df6-cf5857c89699
DEVICE=eno16777736
ONBOOT=yes
IPADDR=192.168.1.112
GATEWAY=192.168.1.2
DNS1=8.8.8.8

删除data目录

删除/opt/module/hadoop-2.8.4的data目录，目的配置分布式集群。

[root@bigdata111 hadoop-2.8.4]# cd /opt/module/hadoop-2.8.4/
[root@bigdata111 hadoop-2.8.4]# rm -rf data/

配置hosts

配置hosts的IP和主机名对应关系

[root@bigdata111 hadoop-2.8.4]# vi /etc/hosts

192.168.1.111 bigdata111
192.168.1.112 bigdata112
192.168.1.113 bigdata113

SCP发送其他机器

将第一台配置好的hosts文件发送到其他两台机器：

[root@bigdata111 hadoop-2.8.4]# scp /etc/hosts root@bigdata112:/etc/
[root@bigdata111 hadoop-2.8.4]# scp /etc/hosts root@bigdata113:/etc/

配置SSH免密登录

利用Xshell的发送键输入到所有会话功能，在三台机器生成秘钥

[root@bigdata111 hadoop-2.8.4]# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
cc:47:37:5a:93:0f:77:38:53:af:a3:57:47:55:27:59 root@bigdata111
The key's randomart image is:
+--[ RSA 2048]----+
|              .oE|
|             ..++|
|          . B = +|
|       o . + * * |
|        S o   + o|
|         .   . o.|
|            . .  |
|             .   |
|                 |
+-----------------+

利用Xshell的发送键输入到所有会话功能，将秘钥添加到集群中各个机器的秘钥库中

[root@bigdata111 hadoop-2.8.4]# ssh-copy-id bigdata111
[root@bigdata111 hadoop-2.8.4]# ssh-copy-id bigdata112
[root@bigdata111 hadoop-2.8.4]# ssh-copy-id bigdata113

查看秘钥库是否存在

[root@bigdata111 .ssh]# cd /root/.ssh
[root@bigdata111 .ssh]# ls
authorized_keys  id_rsa  id_rsa.pub  known_hosts
[root@bigdata111 .ssh]# cat authorized_keys 
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7cSXZDdNJ0Cg+1wyVoCn4pWEAxy/13/ekg//YVkGwEsR6HO4XaYxxstVBij5JoTEEjSDNmz2HifTZDB098py3x882ZLVHJllJWzXYX4gVof/tmdmk5AJbhIlX3SoauTrrrzFiMtuXKdu6slvzhs9IbDp68xCUNiVI06OnWFSuhQc8Td+tekwlFPfm+v3W/PqUUgQAd+OAqOUC2vEjjnACQNw/wgGvF/lqrXDv5ZIFmYCBlB7YxwP9RykOvAzEe7w2W7TOt0K8V8oKKTui4aZuahWDbsGwlD7TAQRkilXkG59XG48AWOQoU/XFxph+XECqJzjmdxYedzY8inYW/Lfx root@bigdata111
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYyMVfLaL9w9sGz5hQG96ksUN5ih2RHdwsiXBpL/ZRG7LasKS+OQcszmc61TJfV0Vjad7kuL9wlg2YqlVIQJvaIUQCw4+5BrO0vCy4JBrz/FiDjzxKx0Ba+ILziuMxl35RxDCVGph17i2jpYfy6jGLejYK9kpJH4ueIj8mm+4LTKabRZTcjdNNI0kYM+Tr08wEIuQ45adqVU9MpZc/j6i1FIr4R/RabyuO1FhEh0+Oc5Xbm3jSAYH0MgEvK1cuG9wmX7SaB/opO00Ts+nW/P4umeZQUy51IQSRdUF6BlMrshnCSlKHnuLv2eSCx9yv3QuQMWHnL/SOXUgTnIuzbrv9 root@bigdata112
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoBOAT/n1QCnaVJtRS1Q9GeoP665gIayWxpSWbjEFus4DL4as5S9jAIhBQWrTnvZzm+Skb4dxGPgdPYLaMFX9tdDYPPsnnRR92sLpRw9gwvG5ROL5XPpV2X+Yxl6yACmlMT0JP1uk+Ekm623n6wtBSBP1BDtJ/fhXkRX6bo2kuXs4BvmP76cikdGBDygKNIEMPTcs6p2lfOnuVdQLSCGm+Q9NswKSBVElNyywNl5J9L/5kIzGXnoGtwhQtdrOjZ+c1tyiwhCz42I3c4z0Sb/zH3OFtHCvRG7cF72uDFxe1QwVJ4h1hJ1dmtwVCckNMbmmgK72PsN8Zg4Y8XtBXgX8n root@bigdata113

验证SSH免密码登录是否配置成功

[root@bigdata111 .ssh]# ssh bigdata112
Last login: Mon Aug  5 09:23:11 2019 from bigdata112
[root@bigdata112 ~]# ssh bigdata111
Last login: Mon Aug  5 09:09:23 2019 from 192.168.1.1

部署jdk和hadoop

去除勾选“发送键输入到所有会话”，从bigdata111发送module文件夹到另外两台机器/opt/文件夹下：

[root@bigdata111 module]# scp -r /opt/module/ root@bigdata112:/opt/
[root@bigdata111 module]# scp -r /opt/module/ root@bigdata113:/opt/

将环境变量/etc/profile发送到另外两台机器：

[root@bigdata111 module]# scp -r /etc/profile root@bigdata112:/etc/
[root@bigdata111 module]# scp -r /etc/profile root@bigdata113:/etc/

切换到另外两台机器，刷新环境变量：

[root@bigdata112 module]# source /etc/profile
[root@bigdata112 module]# jps
2775 Jps
[root@bigdata113 module]# source /etc/profile
[root@bigdata113 module]# jps
2820 Jps

配置集群xml

勾选“发送键输入到所有会话”，配置hdfs-site,yarn-site,mapred-site的xml文件：

hdfs-site.xml配置如下（SecondaryNameNode配置在113上）：

<configuration>

<!--数据冗余数-->
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<!--secondary的地址-->
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>bigdata113:50090</value>
</property>
<!--关闭权限-->
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>

</configuration>

yarn-site.xml配置如下（yarn配置在112上）：

<configuration>

<!-- Site specific YARN configuration properties -->
<!-- reducer获取数据的方式 -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>

<!-- 指定YARN的ResourceManager的地址 -->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>bigdata112</value>
</property>

<!-- 日志聚集功能使能 -->
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>

<!-- 日志保留时间设置7天(秒) -->
<property>
<name>yarn.log-aggregation.retain-seconds</name>
<value>604800</value>
</property>

</configuration>

mapred-site.xml配置如下：

<configuration>

<!-- 指定mr运行在yarn上-->
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>

<!--历史服务器的地址-->
<property>
<name>mapreduce.jobhistory.address</name>
<value>bigdata112:10020</value>
</property>

<!--历史服务器页面的地址-->
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>bigdata112:19888</value>
</property>

</configuration>

配置slaves的datanode

[root@bigdata111 ~]# cd /opt/module/hadoop-2.8.4/etc/hadoop/
[root@bigdata111 hadoop]# ls
capacity-scheduler.xml  core-site.xml   hadoop-metrics2.properties  hdfs-site.xml            httpfs-signature.secret  kms-env.sh            log4j.properties  mapred-queues.xml.template  ssl-client.xml.example  yarn-env.sh
configuration.xsl       hadoop-env.cmd  hadoop-metrics.properties   httpfs-env.sh            httpfs-site.xml          kms-log4j.properties  mapred-env.cmd    mapred-site.xml             ssl-server.xml.example  yarn-site.xml
container-executor.cfg  hadoop-env.sh   hadoop-policy.xml           httpfs-log4j.properties  kms-acls.xml             kms-site.xml          mapred-env.sh     slaves                      yarn-env.cmd
[root@bigdata111 hadoop]# vi slaves

bigdata111
bigdata112
bigdata113

格式化namenode

利用xshell的“发送键输入到所有会话”功能，格式化namenode

[root@bigdata111 hadoop]# hadoop namenode -format
[root@bigdata112 hadoop]# hadoop namenode -format
[root@bigdata113 hadoop]# hadoop namenode -format

启动111的hdfs

[root@bigdata111 hadoop]# start-dfs.sh

启动112的yarn

[root@bigdata112 hadoop]# start-yarn.sh

输出三台机器的jps进程

[root@bigdata111 hadoop]# jps
2512 DataNode
2758 NodeManager
2377 NameNode
2894 Jps

[root@bigdata112 ~]# jps
2528 NodeManager
2850 Jps
2294 DataNode
2413 ResourceManager

[root@bigdata113 ~]# jps
2465 NodeManager
2598 Jps
2296 DataNode
2398 SecondaryNameNode

posted @ 2019-08-10 17:44 ShadowFiend 阅读(593) 评论(0) 编辑收藏举报

会员力量，点亮园子希望

刷新页面返回顶部

ShadowFiend

You cannot improve your past, but you can improve your future. Once time is wasted, life is wasted.

【Hadoop】单机、伪分布式、完全分布式集群搭建

搭建Hadoop本地模式

安装hadoop

上传hadoop包

解压hadoop

设置hadoop环境变量

刷新配置文件

查看hadoop是否安装成功

测试hadoop实例

创建测试文件

切换jar包目录

执行wordcount程序

搭建Hadoop伪分布式

查看hadoop可执行文件

切换配置文件目录

配置core-site.xml

配置hdfs-site.xml

配置yarn-site.xml

配置mapred-site.xml

配置hadoop-env.sh

格式化namenode

开启hdfs和yarn服务

访问hdfs 的web页面

访问yarn的web页面

搭建Hadoop完全分布式集群

修改主机名和IP

删除data目录

配置hosts

SCP发送其他机器

配置SSH免密登录

部署jdk和hadoop

配置集群xml

配置slaves的datanode

格式化namenode

启动111的hdfs

启动112的yarn

输出三台机器的jps进程

公告