Hadoop 集群-完全分布式模式(Fully-Distributed Mode)
Hadoop 集群-完全分布式模式(Fully-Distributed Mode)
作者:尹正杰
版权声明:原创作品,谢绝转载!否则将追究法律责任。
一.查看官方文档
1>.查看对应Apache Hadoop版本的官方文档
博主推荐阅读: http://hadoop.apache.org/docs/r2.10.0/
2>.查看Hadoop设置单节点群集的文档
博主推荐阅读: http://hadoop.apache.org/docs/r2.10.0/hadoop-project-dist/hadoop-common/ClusterSetup.html
3>.Hadoop配置文件说明
Hadoop配置文件分两类:默认配置文件和自定义配置文件,只有用户想修改某一默认配置值时,才需要修改自定义配置文件,更改相应属性值。 默认配置文件: core-default.xml: 默认配置文件存放在"${HADOOP_HOME}/share/doc/hadoop/hadoop-project-dist/hadoop-common/"路径下。 博主推荐阅读: http://hadoop.apache.org/docs/r2.10.0/hadoop-project-dist/hadoop-common/core-default.xml hdfs-default.xml: 默认配置文件存放在"${HADOOP_HOME}/share/doc/hadoop/hadoop-project-dist/hadoop-hdfs/"路径下。 博主推荐阅读: http://hadoop.apache.org/docs/r2.10.0/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml yarn-default.xml: 默认配置文件存放在"/share/doc/hadoop/hadoop-yarn/hadoop-yarn-common/"路径下。 博主推荐阅读: http://hadoop.apache.org/docs/r2.10.0/hadoop-yarn/hadoop-yarn-common/yarn-default.xml mapred-default.xml: 默认配置文件存放在""路径下。 博主推荐阅读: http://hadoop.apache.org/docs/r2.10.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml 自定义配置文件: core-site.xml,hdfs-site.xml,yarn-site.xml,mapred-site.xml这四个配置文件存放默认存放在"$HADOOP_HOME/etc/hadoop"这个路径下。 用户可以根据项目需求重新进行修改配置,自定义的值会覆盖上面对应默认配置文件的值哟。 博主推荐阅读: http://hadoop.apache.org/docs/r2.10.0/hadoop-project-dist/hadoop-common/DeprecatedProperties.html
[root@hadoop101.yinzhengjie.org.cn ~]# ll ${HADOOP_HOME}/share/doc/hadoop/hadoop-project-dist/hadoop-common/core-default.xml -rw-r--r-- 1 root root 104408 Mar 12 03:27 /yinzhengjie/softwares/hadoop-2.10.0/share/doc/hadoop/hadoop-project-dist/hadoop-common/core-default.xml [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# cat ${HADOOP_HOME}/share/doc/hadoop/hadoop-project-dist/hadoop-common/core-default.xml | wc -l [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ll ${HADOOP_HOME}/share/doc/hadoop/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml -rw-r--r-- 1 root root 160516 Mar 12 03:27 /yinzhengjie/softwares/hadoop-2.10.0/share/doc/hadoop/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# cat ${HADOOP_HOME}/share/doc/hadoop/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml | wc -l [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ll ${HADOOP_HOME}/share/doc/hadoop/hadoop-yarn/hadoop-yarn-common/yarn-default.xml -rw-r--r-- 1 root root 132138 Mar 12 03:27 /yinzhengjie/softwares/hadoop-2.10.0/share/doc/hadoop/hadoop-yarn/hadoop-yarn-common/yarn-default.xml [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# cat ${HADOOP_HOME}/share/doc/hadoop/hadoop-yarn/hadoop-yarn-common/yarn-default.xml | wc -l [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ll ${HADOOP_HOME}/share/doc/hadoop/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml -rw-r--r-- 1 root root 74650 Mar 12 03:27 /yinzhengjie/softwares/hadoop-2.10.0/share/doc/hadoop/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# cat ${HADOOP_HOME}/share/doc/hadoop/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml | wc -l [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]#
二.完全分布式准备环境
1>.实验环境说明
本篇博客采用6台Linux发行版本为"CentOS Linux release 7.6.1810 (Core)"操作系统部署Hadoop完全分布式集群。 本篇博客介绍了如何安装和配置Hadoop集群,范围从几个节点到具有数千个节点的超大型集群。要使用Hadoop,你可能需要将其安装在单台机器上,可参考我之前的笔记,链接如下: https://www.cnblogs.com/yinzhengjie2020/p/12422758.html 各节点Hadoop角色分配如下,当然你也可以将下面的角色自定义进行合并到一台主机,虽然博主不建议你这样干,但是如果你的物理机实在内存不足建议至少保留2台虚拟机,将下面的所有的角色分配到2个节点,但是对你的体验可能较差。 hadoop101.yinzhengjie.org.cn: 部署NameNode角色 hadoop102.yinzhengjie.org.cn: 部署DataNode,NodeManager角色 hadoop103.yinzhengjie.org.cn: 部署DataNode,NodeManager角色 hadoop104.yinzhengjie.org.cn: 部署DataNode,NodeManager角色 hadoop105.yinzhengjie.org.cn: 部署SecondNameNode hadoop106.yinzhengjie.org.cn: 部署ResourceManager角色
2>.配置集群时间同步
温馨提示:
身为运维人员,只要你搭建的服务是以集群的方式工作,那第一件事情就是应该让所有节点的时间保持一致。
如果服务不依赖时间同步,如果你没有配置集群时间同步那倒无所谓,比如Hadoop服务如果各节点相差10分钟还是可以运行的,但是这对开发以后排错会产生困扰,因此我强烈建议大家配置集群时间同步。
不仅如此,因为在使用集群搭建后期的HBase,Kudu等服务,若时间不同会直接导致你集群无法启动哟~
博主推荐阅读: https://www.cnblogs.com/yinzhengjie/p/12292549.html
3>.hadoop101.yinzhengjie.org.cn节点配置和集群各个节点实现免密登录(为管理hdfs集群和ansible环境做准备)
[root@hadoop101.yinzhengjie.org.cn ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 #Hadoop cluster 172.200.4.101 hadoop101.yinzhengjie.org.cn 172.200.4.102 hadoop102.yinzhengjie.org.cn 172.200.4.103 hadoop103.yinzhengjie.org.cn 172.200.4.104 hadoop104.yinzhengjie.org.cn 172.200.4.105 hadoop105.yinzhengjie.org.cn [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa #快速生成私钥和公钥 Generating public/private rsa key pair. Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: SHA256:+xTXq08c66RwzI70WB+HXCmPY1ml2f/g4UfAEp8We/g root@hadoop101.yinzhengjie.org.cn The key's randomart image is: +---[RSA 2048]----+ | | | . . | | + =.| | ..O++| | S . .+=*o| | . = o**E| | . + ==%oo| | + O.X.=o| | + =.*..| +----[SHA256]-----+ [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# ll ~/.ssh/ total 8 -rw------- 1 root root 1679 Mar 11 18:00 id_rsa #私钥 -rw-r--r-- 1 root root 415 Mar 11 18:00 id_rsa.pub #公钥 [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ssh-copy-id root@hadoop101.yinzhengjie.org.cn /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub" The authenticity of host 'hadoop101.yinzhengjie.org.cn (172.200.4.101)' can't be established. ECDSA key fingerprint is SHA256:y6iS5ipSyWSGRmgcjivbWhd78pKfrcuQHeBPd5H9/U8. ECDSA key fingerprint is MD5:da:0f:2a:93:c0:d4:6e:7e:13:16:61:f1:93:a7:38:01. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@hadoop101.yinzhengjie.org.cn's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'root@hadoop101.yinzhengjie.org.cn'" and check to make sure that only the key(s) you wanted were added. [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# who root pts/0 2020-03-11 17:58 (172.200.0.1) [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# ssh root@hadoop101.yinzhengjie.org.cn Last login: Wed Mar 11 17:58:12 2020 from 172.200.0.1 [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# who root pts/0 2020-03-11 17:58 (172.200.0.1) root pts/1 2020-03-11 18:04 (172.200.4.101) [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# exit logout Connection to hadoop101.yinzhengjie.org.cn closed. [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ssh-copy-id root@hadoop102.yinzhengjie.org.cn /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub" The authenticity of host 'hadoop102.yinzhengjie.org.cn (172.200.4.102)' can't be established. ECDSA key fingerprint is SHA256:y6iS5ipSyWSGRmgcjivbWhd78pKfrcuQHeBPd5H9/U8. ECDSA key fingerprint is MD5:da:0f:2a:93:c0:d4:6e:7e:13:16:61:f1:93:a7:38:01. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@hadoop102.yinzhengjie.org.cn's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'root@hadoop102.yinzhengjie.org.cn'" and check to make sure that only the key(s) you wanted were added. [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# who root pts/0 2020-03-11 17:58 (172.200.0.1) [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# ssh root@hadoop102.yinzhengjie.org.cn Last login: Wed Mar 11 13:01:52 2020 from 172.200.0.1 [root@hadoop102.yinzhengjie.org.cn ~]# [root@hadoop102.yinzhengjie.org.cn ~]# who root pts/0 2020-03-11 18:06 (172.200.4.101) [root@hadoop102.yinzhengjie.org.cn ~]# [root@hadoop102.yinzhengjie.org.cn ~]# exit logout Connection to hadoop102.yinzhengjie.org.cn closed. [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# who root pts/0 2020-03-11 17:58 (172.200.0.1) [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ssh-copy-id root@hadoop103.yinzhengjie.org.cn /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub" The authenticity of host 'hadoop103.yinzhengjie.org.cn (172.200.4.103)' can't be established. ECDSA key fingerprint is SHA256:y6iS5ipSyWSGRmgcjivbWhd78pKfrcuQHeBPd5H9/U8. ECDSA key fingerprint is MD5:da:0f:2a:93:c0:d4:6e:7e:13:16:61:f1:93:a7:38:01. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@hadoop103.yinzhengjie.org.cn's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'root@hadoop103.yinzhengjie.org.cn'" and check to make sure that only the key(s) you wanted were added. [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# who root pts/0 2020-03-11 17:58 (172.200.0.1) [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# ssh root@hadoop103.yinzhengjie.org.cn Last login: Tue Mar 10 06:37:08 2020 from 172.200.0.1 [root@hadoop103.yinzhengjie.org.cn ~]# [root@hadoop103.yinzhengjie.org.cn ~]# who root pts/0 2020-03-11 18:10 (172.200.4.101) [root@hadoop103.yinzhengjie.org.cn ~]# [root@hadoop103.yinzhengjie.org.cn ~]# exit logout Connection to hadoop103.yinzhengjie.org.cn closed. [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# who root pts/0 2020-03-11 17:58 (172.200.0.1) [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ssh-copy-id root@hadoop104.yinzhengjie.org.cn /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub" The authenticity of host 'hadoop104.yinzhengjie.org.cn (172.200.4.104)' can't be established. ECDSA key fingerprint is SHA256:y6iS5ipSyWSGRmgcjivbWhd78pKfrcuQHeBPd5H9/U8. ECDSA key fingerprint is MD5:da:0f:2a:93:c0:d4:6e:7e:13:16:61:f1:93:a7:38:01. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@hadoop104.yinzhengjie.org.cn's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'root@hadoop104.yinzhengjie.org.cn'" and check to make sure that only the key(s) you wanted were added. [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# who root pts/0 2020-03-11 17:58 (172.200.0.1) [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# ssh root@hadoop104.yinzhengjie.org.cn Last login: Tue Mar 10 06:37:30 2020 from 172.200.0.1 [root@hadoop104.yinzhengjie.org.cn ~]# [root@hadoop104.yinzhengjie.org.cn ~]# who root pts/0 2020-03-11 18:11 (172.200.4.101) [root@hadoop104.yinzhengjie.org.cn ~]# [root@hadoop104.yinzhengjie.org.cn ~]# exit logout Connection to hadoop104.yinzhengjie.org.cn closed. [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# who root pts/0 2020-03-11 17:58 (172.200.0.1) [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ssh-copy-id root@hadoop105.yinzhengjie.org.cn /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub" The authenticity of host 'hadoop105.yinzhengjie.org.cn (172.200.4.105)' can't be established. ECDSA key fingerprint is SHA256:y6iS5ipSyWSGRmgcjivbWhd78pKfrcuQHeBPd5H9/U8. ECDSA key fingerprint is MD5:da:0f:2a:93:c0:d4:6e:7e:13:16:61:f1:93:a7:38:01. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@hadoop105.yinzhengjie.org.cn's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'root@hadoop105.yinzhengjie.org.cn'" and check to make sure that only the key(s) you wanted were added. [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# who root pts/0 2020-03-11 17:58 (172.200.0.1) [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# ssh root@hadoop105.yinzhengjie.org.cn Last login: Tue Mar 10 06:37:53 2020 from 172.200.0.1 [root@hadoop105.yinzhengjie.org.cn ~]# [root@hadoop105.yinzhengjie.org.cn ~]# who root pts/0 2020-03-11 18:12 (172.200.4.101) [root@hadoop105.yinzhengjie.org.cn ~]# [root@hadoop105.yinzhengjie.org.cn ~]# exit logout Connection to hadoop105.yinzhengjie.org.cn closed. [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# who root pts/0 2020-03-11 17:58 (172.200.0.1) [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ssh-copy-id root@hadoop106.yinzhengjie.org.cn /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub" The authenticity of host 'hadoop106.yinzhengjie.org.cn (172.200.4.106)' can't be established. ECDSA key fingerprint is SHA256:y6iS5ipSyWSGRmgcjivbWhd78pKfrcuQHeBPd5H9/U8. ECDSA key fingerprint is MD5:da:0f:2a:93:c0:d4:6e:7e:13:16:61:f1:93:a7:38:01. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@hadoop106.yinzhengjie.org.cn's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'root@hadoop106.yinzhengjie.org.cn'" and check to make sure that only the key(s) you wanted were added. [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# who root pts/0 2020-03-11 17:58 (172.200.0.1) root pts/1 2020-03-11 20:13 (172.200.0.1) [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# ssh root@hadoop106.yinzhengjie.org.cn Last login: Wed Mar 11 21:13:39 2020 from 172.200.0.1 [root@hadoop106.yinzhengjie.org.cn ~]# [root@hadoop106.yinzhengjie.org.cn ~]# who root pts/0 2020-03-11 21:13 (172.200.0.1) root pts/1 2020-03-11 21:14 (172.200.4.101) [root@hadoop106.yinzhengjie.org.cn ~]# [root@hadoop106.yinzhengjie.org.cn ~]# exit logout Connection to hadoop106.yinzhengjie.org.cn closed. [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# who root pts/0 2020-03-11 17:58 (172.200.0.1) root pts/1 2020-03-11 20:13 (172.200.0.1) [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]#
4>.hadoop106.yinzhengjie.org.cn节点仅和yarn集群配置免密码登录
[root@hadoop106.yinzhengjie.org.cn ~]# ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa Generating public/private rsa key pair. Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: SHA256:8KEEDjqqHLpGPK9HOz/j88WHLZDXXGLYKqTApi8HN7Q root@hadoop106.yinzhengjie.org.cn The key's randomart image is: +---[RSA 2048]----+ | . . | | ..o . o | |o =. o... + . | |..+ o.o+..= o | |o+ E ..+So o | |++=.. = o | |+o+o. = o | | oo= + . o | |o.o +o=. | +----[SHA256]-----+ [root@hadoop106.yinzhengjie.org.cn ~]#
[root@hadoop106.yinzhengjie.org.cn ~]# ssh-copy-id root@hadoop102.yinzhengjie.org.cn /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub" The authenticity of host 'hadoop102.yinzhengjie.org.cn (172.200.4.102)' can't be established. ECDSA key fingerprint is SHA256:y6iS5ipSyWSGRmgcjivbWhd78pKfrcuQHeBPd5H9/U8. ECDSA key fingerprint is MD5:da:0f:2a:93:c0:d4:6e:7e:13:16:61:f1:93:a7:38:01. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@hadoop102.yinzhengjie.org.cn's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'root@hadoop102.yinzhengjie.org.cn'" and check to make sure that only the key(s) you wanted were added. [root@hadoop106.yinzhengjie.org.cn ~]# [root@hadoop106.yinzhengjie.org.cn ~]# who root pts/0 2020-03-12 06:49 (172.200.0.1) root pts/1 2020-03-12 07:31 (172.200.0.1) [root@hadoop106.yinzhengjie.org.cn ~]# [root@hadoop106.yinzhengjie.org.cn ~]# ssh root@hadoop102.yinzhengjie.org.cn Last login: Thu Mar 12 07:35:49 2020 from 172.200.4.101 [root@hadoop102.yinzhengjie.org.cn ~]# [root@hadoop102.yinzhengjie.org.cn ~]# who root pts/0 2020-03-12 07:41 (172.200.4.106) [root@hadoop102.yinzhengjie.org.cn ~]# [root@hadoop102.yinzhengjie.org.cn ~]# exit logout Connection to hadoop102.yinzhengjie.org.cn closed. [root@hadoop106.yinzhengjie.org.cn ~]# [root@hadoop106.yinzhengjie.org.cn ~]# who root pts/0 2020-03-12 06:49 (172.200.0.1) root pts/1 2020-03-12 07:31 (172.200.0.1) [root@hadoop106.yinzhengjie.org.cn ~]#
[root@hadoop106.yinzhengjie.org.cn ~]# ssh-copy-id root@hadoop103.yinzhengjie.org.cn /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub" The authenticity of host 'hadoop103.yinzhengjie.org.cn (172.200.4.103)' can't be established. ECDSA key fingerprint is SHA256:y6iS5ipSyWSGRmgcjivbWhd78pKfrcuQHeBPd5H9/U8. ECDSA key fingerprint is MD5:da:0f:2a:93:c0:d4:6e:7e:13:16:61:f1:93:a7:38:01. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@hadoop103.yinzhengjie.org.cn's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'root@hadoop103.yinzhengjie.org.cn'" and check to make sure that only the key(s) you wanted were added. [root@hadoop106.yinzhengjie.org.cn ~]# [root@hadoop106.yinzhengjie.org.cn ~]# who root pts/0 2020-03-12 06:49 (172.200.0.1) [root@hadoop106.yinzhengjie.org.cn ~]# [root@hadoop106.yinzhengjie.org.cn ~]# ssh root@hadoop103.yinzhengjie.org.cn Last login: Thu Mar 12 07:20:02 2020 from 172.200.0.1 [root@hadoop103.yinzhengjie.org.cn ~]# [root@hadoop103.yinzhengjie.org.cn ~]# who root pts/0 2020-03-12 07:20 (172.200.0.1) root pts/1 2020-03-12 07:28 (172.200.4.106) [root@hadoop103.yinzhengjie.org.cn ~]# [root@hadoop103.yinzhengjie.org.cn ~]# exit logout Connection to hadoop103.yinzhengjie.org.cn closed. [root@hadoop106.yinzhengjie.org.cn ~]#
[root@hadoop106.yinzhengjie.org.cn ~]# ssh-copy-id root@hadoop104.yinzhengjie.org.cn /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub" The authenticity of host 'hadoop104.yinzhengjie.org.cn (172.200.4.104)' can't be established. ECDSA key fingerprint is SHA256:y6iS5ipSyWSGRmgcjivbWhd78pKfrcuQHeBPd5H9/U8. ECDSA key fingerprint is MD5:da:0f:2a:93:c0:d4:6e:7e:13:16:61:f1:93:a7:38:01. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@hadoop104.yinzhengjie.org.cn's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'root@hadoop104.yinzhengjie.org.cn'" and check to make sure that only the key(s) you wanted were added. [root@hadoop106.yinzhengjie.org.cn ~]# [root@hadoop106.yinzhengjie.org.cn ~]# who root pts/0 2020-03-12 06:49 (172.200.0.1) [root@hadoop106.yinzhengjie.org.cn ~]# [root@hadoop106.yinzhengjie.org.cn ~]# ssh root@hadoop104.yinzhengjie.org.cn Last login: Thu Mar 12 06:55:07 2020 from 172.200.4.101 [root@hadoop104.yinzhengjie.org.cn ~]# [root@hadoop104.yinzhengjie.org.cn ~]# who root pts/0 2020-03-12 07:29 (172.200.4.106) [root@hadoop104.yinzhengjie.org.cn ~]# [root@hadoop104.yinzhengjie.org.cn ~]# exit logout Connection to hadoop104.yinzhengjie.org.cn closed. [root@hadoop106.yinzhengjie.org.cn ~]#
[root@hadoop106.yinzhengjie.org.cn ~]# ssh-copy-id root@hadoop105.yinzhengjie.org.cn /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub" The authenticity of host 'hadoop105.yinzhengjie.org.cn (172.200.4.105)' can't be established. ECDSA key fingerprint is SHA256:y6iS5ipSyWSGRmgcjivbWhd78pKfrcuQHeBPd5H9/U8. ECDSA key fingerprint is MD5:da:0f:2a:93:c0:d4:6e:7e:13:16:61:f1:93:a7:38:01. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@hadoop105.yinzhengjie.org.cn's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'root@hadoop105.yinzhengjie.org.cn'" and check to make sure that only the key(s) you wanted were added. [root@hadoop106.yinzhengjie.org.cn ~]# [root@hadoop106.yinzhengjie.org.cn ~]# who root pts/0 2020-03-12 06:49 (172.200.0.1) [root@hadoop106.yinzhengjie.org.cn ~]# [root@hadoop106.yinzhengjie.org.cn ~]# ssh root@hadoop105.yinzhengjie.org.cn Last login: Thu Mar 12 06:55:07 2020 from 172.200.4.101 [root@hadoop105.yinzhengjie.org.cn ~]# [root@hadoop105.yinzhengjie.org.cn ~]# who root pts/0 2020-03-12 07:29 (172.200.4.106) [root@hadoop105.yinzhengjie.org.cn ~]# [root@hadoop105.yinzhengjie.org.cn ~]# exit logout Connection to hadoop105.yinzhengjie.org.cn closed. [root@hadoop106.yinzhengjie.org.cn ~]#
5>.安装ansible并配置hadoop主机清单
[root@hadoop101.yinzhengjie.org.cn ~]# yum list ansible Loaded plugins: fastestmirror Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast Determining fastest mirrors * base: mirror.bit.edu.cn * extras: mirror.bit.edu.cn * updates: mirrors.aliyun.com Available Packages ansible.noarch 2.4.2.0-2.el7 extras [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# yum -y install ansible Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirror.bit.edu.cn * extras: mirror.bit.edu.cn * updates: mirrors.aliyun.com base | 3.6 kB 00:00:00 extras | 2.9 kB 00:00:00 updates | 2.9 kB 00:00:00 (1/2): extras/7/x86_64/primary_db | 164 kB 00:00:00 (2/2): updates/7/x86_64/primary_db | 6.7 MB 00:00:01 Resolving Dependencies --> Running transaction check ---> Package ansible.noarch 0:2.4.2.0-2.el7 will be installed --> Processing Dependency: sshpass for package: ansible-2.4.2.0-2.el7.noarch --> Processing Dependency: python2-jmespath for package: ansible-2.4.2.0-2.el7.noarch --> Processing Dependency: python-six for package: ansible-2.4.2.0-2.el7.noarch --> Processing Dependency: python-setuptools for package: ansible-2.4.2.0-2.el7.noarch --> Processing Dependency: python-passlib for package: ansible-2.4.2.0-2.el7.noarch --> Processing Dependency: python-paramiko for package: ansible-2.4.2.0-2.el7.noarch --> Processing Dependency: python-jinja2 for package: ansible-2.4.2.0-2.el7.noarch --> Processing Dependency: python-httplib2 for package: ansible-2.4.2.0-2.el7.noarch --> Processing Dependency: python-cryptography for package: ansible-2.4.2.0-2.el7.noarch --> Processing Dependency: PyYAML for package: ansible-2.4.2.0-2.el7.noarch --> Running transaction check ---> Package PyYAML.x86_64 0:3.10-11.el7 will be installed --> Processing Dependency: libyaml-0.so.2()(64bit) for package: PyYAML-3.10-11.el7.x86_64 ---> Package python-httplib2.noarch 0:0.9.2-1.el7 will be installed ---> Package python-jinja2.noarch 0:2.7.2-4.el7 will be installed --> Processing Dependency: python-babel >= 0.8 for package: python-jinja2-2.7.2-4.el7.noarch --> Processing Dependency: python-markupsafe for package: python-jinja2-2.7.2-4.el7.noarch ---> Package python-paramiko.noarch 0:2.1.1-9.el7 will be installed --> Processing Dependency: python2-pyasn1 for package: python-paramiko-2.1.1-9.el7.noarch ---> Package python-passlib.noarch 0:1.6.5-2.el7 will be installed ---> Package python-setuptools.noarch 0:0.9.8-7.el7 will be installed --> Processing Dependency: python-backports-ssl_match_hostname for package: python-setuptools-0.9.8-7.el7.noarch ---> Package python-six.noarch 0:1.9.0-2.el7 will be installed ---> Package python2-cryptography.x86_64 0:1.7.2-2.el7 will be installed --> Processing Dependency: python-idna >= 2.0 for package: python2-cryptography-1.7.2-2.el7.x86_64 --> Processing Dependency: python-cffi >= 1.4.1 for package: python2-cryptography-1.7.2-2.el7.x86_64 --> Processing Dependency: python-ipaddress for package: python2-cryptography-1.7.2-2.el7.x86_64 --> Processing Dependency: python-enum34 for package: python2-cryptography-1.7.2-2.el7.x86_64 ---> Package python2-jmespath.noarch 0:0.9.0-3.el7 will be installed ---> Package sshpass.x86_64 0:1.06-2.el7 will be installed --> Running transaction check ---> Package libyaml.x86_64 0:0.1.4-11.el7_0 will be installed ---> Package python-babel.noarch 0:0.9.6-8.el7 will be installed ---> Package python-backports-ssl_match_hostname.noarch 0:3.5.0.1-1.el7 will be installed --> Processing Dependency: python-backports for package: python-backports-ssl_match_hostname-3.5.0.1-1.el7.noarch ---> Package python-cffi.x86_64 0:1.6.0-5.el7 will be installed --> Processing Dependency: python-pycparser for package: python-cffi-1.6.0-5.el7.x86_64 ---> Package python-enum34.noarch 0:1.0.4-1.el7 will be installed ---> Package python-idna.noarch 0:2.4-1.el7 will be installed ---> Package python-ipaddress.noarch 0:1.0.16-2.el7 will be installed ---> Package python-markupsafe.x86_64 0:0.11-10.el7 will be installed ---> Package python2-pyasn1.noarch 0:0.1.9-7.el7 will be installed --> Running transaction check ---> Package python-backports.x86_64 0:1.0-8.el7 will be installed ---> Package python-pycparser.noarch 0:2.14-1.el7 will be installed --> Processing Dependency: python-ply for package: python-pycparser-2.14-1.el7.noarch --> Running transaction check ---> Package python-ply.noarch 0:3.4-11.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================================================================================================================================================================================== Package Arch Version Repository Size ============================================================================================================================================================================================================================================================================== Installing: ansible noarch 2.4.2.0-2.el7 extras 7.6 M Installing for dependencies: PyYAML x86_64 3.10-11.el7 base 153 k libyaml x86_64 0.1.4-11.el7_0 base 55 k python-babel noarch 0.9.6-8.el7 base 1.4 M python-backports x86_64 1.0-8.el7 base 5.8 k python-backports-ssl_match_hostname noarch 3.5.0.1-1.el7 base 13 k python-cffi x86_64 1.6.0-5.el7 base 218 k python-enum34 noarch 1.0.4-1.el7 base 52 k python-httplib2 noarch 0.9.2-1.el7 extras 115 k python-idna noarch 2.4-1.el7 base 94 k python-ipaddress noarch 1.0.16-2.el7 base 34 k python-jinja2 noarch 2.7.2-4.el7 base 519 k python-markupsafe x86_64 0.11-10.el7 base 25 k python-paramiko noarch 2.1.1-9.el7 base 269 k python-passlib noarch 1.6.5-2.el7 extras 488 k python-ply noarch 3.4-11.el7 base 123 k python-pycparser noarch 2.14-1.el7 base 104 k python-setuptools noarch 0.9.8-7.el7 base 397 k python-six noarch 1.9.0-2.el7 base 29 k python2-cryptography x86_64 1.7.2-2.el7 base 502 k python2-jmespath noarch 0.9.0-3.el7 extras 39 k python2-pyasn1 noarch 0.1.9-7.el7 base 100 k sshpass x86_64 1.06-2.el7 extras 21 k Transaction Summary ============================================================================================================================================================================================================================================================================== Install 1 Package (+22 Dependent packages) Total download size: 12 M Installed size: 60 M Downloading packages: (1/23): PyYAML-3.10-11.el7.x86_64.rpm | 153 kB 00:00:00 (2/23): libyaml-0.1.4-11.el7_0.x86_64.rpm | 55 kB 00:00:00 (3/23): python-backports-1.0-8.el7.x86_64.rpm | 5.8 kB 00:00:00 (4/23): python-backports-ssl_match_hostname-3.5.0.1-1.el7.noarch.rpm | 13 kB 00:00:00 (5/23): python-cffi-1.6.0-5.el7.x86_64.rpm | 218 kB 00:00:00 (6/23): python-enum34-1.0.4-1.el7.noarch.rpm | 52 kB 00:00:00 (7/23): python-idna-2.4-1.el7.noarch.rpm | 94 kB 00:00:00 (8/23): python-ipaddress-1.0.16-2.el7.noarch.rpm | 34 kB 00:00:00 (9/23): python-babel-0.9.6-8.el7.noarch.rpm | 1.4 MB 00:00:00 (10/23): python-markupsafe-0.11-10.el7.x86_64.rpm | 25 kB 00:00:00 (11/23): python-httplib2-0.9.2-1.el7.noarch.rpm | 115 kB 00:00:00 (12/23): python-jinja2-2.7.2-4.el7.noarch.rpm | 519 kB 00:00:00 (13/23): python-ply-3.4-11.el7.noarch.rpm | 123 kB 00:00:00 (14/23): python-paramiko-2.1.1-9.el7.noarch.rpm | 269 kB 00:00:00 (15/23): python-pycparser-2.14-1.el7.noarch.rpm | 104 kB 00:00:00 (16/23): python-six-1.9.0-2.el7.noarch.rpm | 29 kB 00:00:00 (17/23): python-passlib-1.6.5-2.el7.noarch.rpm | 488 kB 00:00:00 (18/23): python-setuptools-0.9.8-7.el7.noarch.rpm | 397 kB 00:00:00 (19/23): python2-jmespath-0.9.0-3.el7.noarch.rpm | 39 kB 00:00:00 (20/23): sshpass-1.06-2.el7.x86_64.rpm | 21 kB 00:00:00 (21/23): python2-pyasn1-0.1.9-7.el7.noarch.rpm | 100 kB 00:00:00 (22/23): python2-cryptography-1.7.2-2.el7.x86_64.rpm | 502 kB 00:00:00 (23/23): ansible-2.4.2.0-2.el7.noarch.rpm | 7.6 MB 00:00:01 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Total 7.8 MB/s | 12 MB 00:00:01 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : python2-pyasn1-0.1.9-7.el7.noarch 1/23 Installing : python-ipaddress-1.0.16-2.el7.noarch 2/23 Installing : python-six-1.9.0-2.el7.noarch 3/23 Installing : python-httplib2-0.9.2-1.el7.noarch 4/23 Installing : sshpass-1.06-2.el7.x86_64 5/23 Installing : libyaml-0.1.4-11.el7_0.x86_64 6/23 Installing : PyYAML-3.10-11.el7.x86_64 7/23 Installing : python-backports-1.0-8.el7.x86_64 8/23 Installing : python-backports-ssl_match_hostname-3.5.0.1-1.el7.noarch 9/23 Installing : python-setuptools-0.9.8-7.el7.noarch 10/23 Installing : python-babel-0.9.6-8.el7.noarch 11/23 Installing : python-passlib-1.6.5-2.el7.noarch 12/23 Installing : python-ply-3.4-11.el7.noarch 13/23 Installing : python-pycparser-2.14-1.el7.noarch 14/23 Installing : python-cffi-1.6.0-5.el7.x86_64 15/23 Installing : python-markupsafe-0.11-10.el7.x86_64 16/23 Installing : python-jinja2-2.7.2-4.el7.noarch 17/23 Installing : python-idna-2.4-1.el7.noarch 18/23 Installing : python-enum34-1.0.4-1.el7.noarch 19/23 Installing : python2-cryptography-1.7.2-2.el7.x86_64 20/23 Installing : python-paramiko-2.1.1-9.el7.noarch 21/23 Installing : python2-jmespath-0.9.0-3.el7.noarch 22/23 Installing : ansible-2.4.2.0-2.el7.noarch 23/23 Verifying : python-backports-ssl_match_hostname-3.5.0.1-1.el7.noarch 1/23 Verifying : python2-jmespath-0.9.0-3.el7.noarch 2/23 Verifying : python-enum34-1.0.4-1.el7.noarch 3/23 Verifying : python-setuptools-0.9.8-7.el7.noarch 4/23 Verifying : python-jinja2-2.7.2-4.el7.noarch 5/23 Verifying : python-six-1.9.0-2.el7.noarch 6/23 Verifying : python-idna-2.4-1.el7.noarch 7/23 Verifying : python-markupsafe-0.11-10.el7.x86_64 8/23 Verifying : python-ply-3.4-11.el7.noarch 9/23 Verifying : python-passlib-1.6.5-2.el7.noarch 10/23 Verifying : python-paramiko-2.1.1-9.el7.noarch 11/23 Verifying : python-babel-0.9.6-8.el7.noarch 12/23 Verifying : python-backports-1.0-8.el7.x86_64 13/23 Verifying : python-cffi-1.6.0-5.el7.x86_64 14/23 Verifying : python-pycparser-2.14-1.el7.noarch 15/23 Verifying : libyaml-0.1.4-11.el7_0.x86_64 16/23 Verifying : ansible-2.4.2.0-2.el7.noarch 17/23 Verifying : python-ipaddress-1.0.16-2.el7.noarch 18/23 Verifying : sshpass-1.06-2.el7.x86_64 19/23 Verifying : python-httplib2-0.9.2-1.el7.noarch 20/23 Verifying : python2-pyasn1-0.1.9-7.el7.noarch 21/23 Verifying : PyYAML-3.10-11.el7.x86_64 22/23 Verifying : python2-cryptography-1.7.2-2.el7.x86_64 23/23 Installed: ansible.noarch 0:2.4.2.0-2.el7 Dependency Installed: PyYAML.x86_64 0:3.10-11.el7 libyaml.x86_64 0:0.1.4-11.el7_0 python-babel.noarch 0:0.9.6-8.el7 python-backports.x86_64 0:1.0-8.el7 python-backports-ssl_match_hostname.noarch 0:3.5.0.1-1.el7 python-cffi.x86_64 0:1.6.0-5.el7 python-enum34.noarch 0:1.0.4-1.el7 python-httplib2.noarch 0:0.9.2-1.el7 python-idna.noarch 0:2.4-1.el7 python-ipaddress.noarch 0:1.0.16-2.el7 python-jinja2.noarch 0:2.7.2-4.el7 python-markupsafe.x86_64 0:0.11-10.el7 python-paramiko.noarch 0:2.1.1-9.el7 python-passlib.noarch 0:1.6.5-2.el7 python-ply.noarch 0:3.4-11.el7 python-pycparser.noarch 0:2.14-1.el7 python-setuptools.noarch 0:0.9.8-7.el7 python-six.noarch 0:1.9.0-2.el7 python2-cryptography.x86_64 0:1.7.2-2.el7 python2-jmespath.noarch 0:0.9.0-3.el7 python2-pyasn1.noarch 0:0.1.9-7.el7 sshpass.x86_64 0:1.06-2.el7 Complete! [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# tail -13 /etc/ansible/hosts #Add by yinzhengjie for hadoop hosts #Namenode [nn] hadoop101.yinzhengjie.org.cn #SecondNamenode [2nn] hadoop105.yinzhengjie.org.cn #datanode [dn] hadoop[102:104].yinzhengjie.org.cn #ResourceManager [rm] hadoop106.yinzhengjie.org.cn [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible nn -m ping hadoop101.yinzhengjie.org.cn | SUCCESS => { "changed": false, "ping": "pong" } [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible 2nn -m ping hadoop105.yinzhengjie.org.cn | SUCCESS => { "changed": false, "ping": "pong" } [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible dn -m ping hadoop102.yinzhengjie.org.cn | SUCCESS => { "changed": false, "ping": "pong" } hadoop103.yinzhengjie.org.cn | SUCCESS => { "changed": false, "ping": "pong" } hadoop104.yinzhengjie.org.cn | SUCCESS => { "changed": false, "ping": "pong" } [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible rm -m ping hadoop106.yinzhengjie.org.cn | SUCCESS => { "changed": false, "ping": "pong" } [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m ping hadoop105.yinzhengjie.org.cn | SUCCESS => { "changed": false, "ping": "pong" } hadoop101.yinzhengjie.org.cn | SUCCESS => { "changed": false, "ping": "pong" } hadoop103.yinzhengjie.org.cn | SUCCESS => { "changed": false, "ping": "pong" } hadoop102.yinzhengjie.org.cn | SUCCESS => { "changed": false, "ping": "pong" } hadoop104.yinzhengjie.org.cn | SUCCESS => { "changed": false, "ping": "pong" } hadoop106.yinzhengjie.org.cn | SUCCESS => { "changed": false, "ping": "pong" } [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]#
6>.使用ansible工具将"/etc/hosts"文件拷贝到其它节点
[root@hadoop101.yinzhengjie.org.cn ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 #Hadoop cluster 172.200.4.101 hadoop101.yinzhengjie.org.cn 172.200.4.102 hadoop102.yinzhengjie.org.cn 172.200.4.103 hadoop103.yinzhengjie.org.cn 172.200.4.104 hadoop104.yinzhengjie.org.cn 172.200.4.105 hadoop105.yinzhengjie.org.cn 172.200.4.106 hadoop106.yinzhengjie.org.cn [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m copy -a 'src=/etc/hosts dest=/etc/hosts' hadoop101.yinzhengjie.org.cn | SUCCESS => { "changed": false, "checksum": "7aa42f0af4417ce6c188ed4cad34287dfa3570ea", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/hosts", "size": 433, "state": "file", "uid": 0 } hadoop103.yinzhengjie.org.cn | SUCCESS => { "changed": true, "checksum": "7aa42f0af4417ce6c188ed4cad34287dfa3570ea", "dest": "/etc/hosts", "gid": 0, "group": "root", "md5sum": "769e0a0d41e5be699399b271cea8a377", "mode": "0644", "owner": "root", "size": 433, "src": "/root/.ansible/tmp/ansible-tmp-1583932891.99-144884840221439/source", "state": "file", "uid": 0 } hadoop102.yinzhengjie.org.cn | SUCCESS => { "changed": true, "checksum": "7aa42f0af4417ce6c188ed4cad34287dfa3570ea", "dest": "/etc/hosts", "gid": 0, "group": "root", "md5sum": "769e0a0d41e5be699399b271cea8a377", "mode": "0644", "owner": "root", "size": 433, "src": "/root/.ansible/tmp/ansible-tmp-1583932891.96-167534829416060/source", "state": "file", "uid": 0 } hadoop105.yinzhengjie.org.cn | SUCCESS => { "changed": true, "checksum": "7aa42f0af4417ce6c188ed4cad34287dfa3570ea", "dest": "/etc/hosts", "gid": 0, "group": "root", "md5sum": "769e0a0d41e5be699399b271cea8a377", "mode": "0644", "owner": "root", "size": 433, "src": "/root/.ansible/tmp/ansible-tmp-1583932892.02-30034905247840/source", "state": "file", "uid": 0 } hadoop104.yinzhengjie.org.cn | SUCCESS => { "changed": true, "checksum": "7aa42f0af4417ce6c188ed4cad34287dfa3570ea", "dest": "/etc/hosts", "gid": 0, "group": "root", "md5sum": "769e0a0d41e5be699399b271cea8a377", "mode": "0644", "owner": "root", "size": 433, "src": "/root/.ansible/tmp/ansible-tmp-1583932892.04-75312091987102/source", "state": "file", "uid": 0 } hadoop106.yinzhengjie.org.cn | SUCCESS => { "changed": true, "checksum": "7aa42f0af4417ce6c188ed4cad34287dfa3570ea", "dest": "/etc/hosts", "gid": 0, "group": "root", "md5sum": "769e0a0d41e5be699399b271cea8a377", "mode": "0644", "owner": "root", "size": 433, "src": "/root/.ansible/tmp/ansible-tmp-1583932894.24-77039419077594/source", "state": "file", "uid": 0 } [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'cat /etc/hosts' hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >> 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 #Hadoop cluster 172.200.4.101 hadoop101.yinzhengjie.org.cn 172.200.4.102 hadoop102.yinzhengjie.org.cn 172.200.4.103 hadoop103.yinzhengjie.org.cn 172.200.4.104 hadoop104.yinzhengjie.org.cn 172.200.4.105 hadoop105.yinzhengjie.org.cn 172.200.4.106 hadoop106.yinzhengjie.org.cn hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >> 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 #Hadoop cluster 172.200.4.101 hadoop101.yinzhengjie.org.cn 172.200.4.102 hadoop102.yinzhengjie.org.cn 172.200.4.103 hadoop103.yinzhengjie.org.cn 172.200.4.104 hadoop104.yinzhengjie.org.cn 172.200.4.105 hadoop105.yinzhengjie.org.cn 172.200.4.106 hadoop106.yinzhengjie.org.cn hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >> 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 #Hadoop cluster 172.200.4.101 hadoop101.yinzhengjie.org.cn 172.200.4.102 hadoop102.yinzhengjie.org.cn 172.200.4.103 hadoop103.yinzhengjie.org.cn 172.200.4.104 hadoop104.yinzhengjie.org.cn 172.200.4.105 hadoop105.yinzhengjie.org.cn 172.200.4.106 hadoop106.yinzhengjie.org.cn hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >> 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 #Hadoop cluster 172.200.4.101 hadoop101.yinzhengjie.org.cn 172.200.4.102 hadoop102.yinzhengjie.org.cn 172.200.4.103 hadoop103.yinzhengjie.org.cn 172.200.4.104 hadoop104.yinzhengjie.org.cn 172.200.4.105 hadoop105.yinzhengjie.org.cn 172.200.4.106 hadoop106.yinzhengjie.org.cn hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >> 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 #Hadoop cluster 172.200.4.101 hadoop101.yinzhengjie.org.cn 172.200.4.102 hadoop102.yinzhengjie.org.cn 172.200.4.103 hadoop103.yinzhengjie.org.cn 172.200.4.104 hadoop104.yinzhengjie.org.cn 172.200.4.105 hadoop105.yinzhengjie.org.cn 172.200.4.106 hadoop106.yinzhengjie.org.cn hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >> 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 #Hadoop cluster 172.200.4.101 hadoop101.yinzhengjie.org.cn 172.200.4.102 hadoop102.yinzhengjie.org.cn 172.200.4.103 hadoop103.yinzhengjie.org.cn 172.200.4.104 hadoop104.yinzhengjie.org.cn 172.200.4.105 hadoop105.yinzhengjie.org.cn 172.200.4.106 hadoop106.yinzhengjie.org.cn [root@hadoop101.yinzhengjie.org.cn ~]#
博主推荐阅读: https://www.cnblogs.com/yinzhengjie/p/10447587.html
7>.使用ansible工具在所有节点批量安装rsync服务并编写集群分发脚本
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'yum -y install rsync' #安装rsync服务 [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# vim /usr/local/bin/rsync-hadoop.sh #编写rsync同步脚本 [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# cat /usr/local/bin/rsync-hadoop.sh #!/bin/bash #@author :yinzhengjie #blog:http://www.cnblogs.com/yinzhengjie #EMAIL:y1053419035@qq.com #判断用户是否传参 if [ $# -lt 1 ];then echo "请输入参数"; exit fi #获取文件路径 file=$@ #获取子路径 filename=`basename $file` #获取父路径 dirpath=`dirname $file` #获取完整路径 cd $dirpath fullpath=`pwd -P` #同步文件到DataNode for (( hostId=102;hostId<=106;hostId++ )) do #使终端变绿色 tput setaf 2 echo "******* [hadoop${hostId}.yinzhengjie.org.cn] node starts synchronizing [${file}] *******" #使终端变回原来的颜色,即白灰色 tput setaf 7 #远程执行命令 rsync -lr $filename `whoami`@hadoop${hostId}.yinzhengjie.org.cn:${fullpath} #判断命令是否执行成功 if [ $? == 0 ];then echo "命令执行成功" fi done [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ll /usr/local/bin/rsync-hadoop.sh -rw-r--r-- 1 root root 903 Mar 11 20:30 /usr/local/bin/rsync-hadoop.sh [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# chmod +x /usr/local/bin/rsync-hadoop.sh [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# ll /usr/local/bin/rsync-hadoop.sh -rwxr-xr-x 1 root root 903 Mar 11 20:30 /usr/local/bin/rsync-hadoop.sh [root@hadoop101.yinzhengjie.org.cn ~]#
温馨提示:
虽然我们可以不用rsync服务,使用ansible的确是可以帮我们完成文件拷贝的,但ansible依旧时基于shutil.copy方法进行拷贝的,它并不会判断源文件和目标文件的差异,而是直接将源文件原封不动的拷贝到目标文件中。
而使用rsync做文件的复制要比ansible的copy速度快,原因是rsync只对差异文件做更新。因此生成环境中使用rsync进行同步的方式很常见,比如NFS+rsync结合使用的案例。
三.完全分布式集群配置实操案例
1>.在node101.yinzhengjie.org.cn主机安装Hadoop运行环境
此步骤只需要安装JDK,Hadoop软件并配置相应的环境变量即可。
博主推荐阅读: https://www.cnblogs.com/yinzhengjie2020/p/12422758.html
2>.修改core-site.xml
[root@hadoop101.yinzhengjie.org.cn ~]# vim ${HADOOP_HOME}/etc/hadoop/core-site.xml [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# cat ${HADOOP_HOME}/etc/hadoop/core-site.xml <?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> <!-- Put site-specific property overrides in this file. --> <configuration> <property> <name>fs.defaultFS</name> <value>hdfs://hadoop101.yinzhengjie.org.cn:9000</value> <description>指定HDFS中NameNode的RPC地址</description> </property> <property> <name>hadoop.tmp.dir</name> <value>/yinzhengjie/softwares/hadoop-2.10.0/data/tmp</value> <description>指定Hadoop运行时产生文件的存储目录</description> </property> </configuration> [root@hadoop101.yinzhengjie.org.cn ~]#
3>.修改hdfs-site.xml
[root@hadoop101.yinzhengjie.org.cn ~]# vim ${HADOOP_HOME}/etc/hadoop/hdfs-site.xml [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# cat ${HADOOP_HOME}/etc/hadoop/hdfs-site.xml <?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> <!-- Put site-specific property overrides in this file. --> <configuration> <property> <name>dfs.replication</name> <value>3</value> <description>指定HDFS副本的数量</description> </property> <property> <name>dfs.namenode.secondary.http-address</name> <value>hadoop105.yinzhengjie.org.cn:50090</value> <description>指定Hadoop辅助名称节点主机配置</description> </property> </configuration> [root@hadoop101.yinzhengjie.org.cn ~]#
4>.修改yarn-site.xml
[root@hadoop101.yinzhengjie.org.cn ~]# vim ${HADOOP_HOME}/etc/hadoop/yarn-site.xml [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# cat ${HADOOP_HOME}/etc/hadoop/yarn-site.xml <?xml version="1.0"?> <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> <configuration> <!-- Site specific YARN configuration properties --> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> <description>Reducer获取数据的方式</description> </property> <property> <name>yarn.resourcemanager.hostname</name> <value>hadoop106.yinzhengjie.org.cn</value> <description>指定YARN的ResourceManager的地址</description> </property> </configuration> [root@hadoop101.yinzhengjie.org.cn ~]#
5>.创建mapred-site.xml文件
[root@hadoop101.yinzhengjie.org.cn ~]# cp ${HADOOP_HOME}/etc/hadoop/mapred-site.xml.template ${HADOOP_HOME}/etc/hadoop/mapred-site.xml #需要手动创建该文件 [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# vim ${HADOOP_HOME}/etc/hadoop/mapred-site.xml [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# cat ${HADOOP_HOME}/etc/hadoop/mapred-site.xml <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> <!-- Put site-specific property overrides in this file. --> <configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> <description>指定MR运行在YARN上</description> </property> </configuration> [root@hadoop101.yinzhengjie.org.cn ~]#
6>.修改hadoop-env.sh
[root@hadoop101.yinzhengjie.org.cn ~]# grep ^export ${HADOOP_HOME}/etc/hadoop/hadoop-env.sh | grep JAVA_HOME export JAVA_HOME=${JAVA_HOME} [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# echo ${JAVA_HOME} /yinzhengjie/softwares/jdk1.8.0_201 [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# vim ${HADOOP_HOME}/etc/hadoop/hadoop-env.sh [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# grep ^export ${HADOOP_HOME}/etc/hadoop/hadoop-env.sh | grep JAVA_HOME export JAVA_HOME=/yinzhengjie/softwares/jdk1.8.0_201 [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]#
7>.将node101.yinzhengjie.org.cn服务器的配置同步到其它节点上(可能耗时5-10分钟,如果在服务器上的话不到30秒搞定)
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'mkdir -pv /yinzhengjie/softwares/' [WARNING]: Consider using file module with state=directory rather than running mkdir hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >> mkdir: created directory ‘/yinzhengjie/softwares/’ hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >> mkdir: created directory ‘/yinzhengjie/softwares/’ hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >> mkdir: created directory ‘/yinzhengjie/softwares/’ hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >> mkdir: created directory ‘/yinzhengjie/softwares/’ hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >> hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >> mkdir: created directory ‘/yinzhengjie/softwares/’ [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# scp -r /yinzhengjie/softwares/hadoop-2.10.0/ hadoop102.yinzhengjie.org.cn:/yinzhengjie/softwares/ #我们第一次拷贝数据使用scp命令拷贝,从第二次开始同步数据则需要使用rsync拷贝 [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# scp -r /yinzhengjie/softwares/hadoop-2.10.0/ hadoop103.yinzhengjie.org.cn:/yinzhengjie/softwares/ [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# scp -r /yinzhengjie/softwares/hadoop-2.10.0/ hadoop104.yinzhengjie.org.cn:/yinzhengjie/softwares/ [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# scp -r /yinzhengjie/softwares/hadoop-2.10.0/ hadoop105.yinzhengjie.org.cn:/yinzhengjie/softwares/ [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# scp -r /yinzhengjie/softwares/hadoop-2.10.0/ hadoop106.yinzhengjie.org.cn:/yinzhengjie/softwares/ 温馨提示,以上五条命令可以简写为一条,如下所示: [root@hadoop101.yinzhengjie.org.cn ~]# for hostId in `seq 102 106`; do scp -r /yinzhengjie/softwares/hadoop-2.10.0/ hadoop${hostId}.yinzhengjie.org.cn:/yinzhengjie/softwares/; done 拷贝完Hadoop软件包后,需要拷贝JDK的软件包,具体操作如下: [root@hadoop101.yinzhengjie.org.cn ~]# for hostId in `seq 102 106`; do scp -r /yinzhengjie/softwares/jdk1.8.0_201/ hadoop${hostId}.yinzhengjie.org.cn:/yinzhengjie/softwares/; done 拷贝完JDK和Hadoop之后,千万别忘记拷贝我们写的环境变量文件哟,拷贝较小的文件或者同步文件的话可以考虑使用rsync脚本来拷贝(温馨提示:宿主机不建议大家对过大文件拷贝使用rsync,有可能物理操作系统程序崩溃,尤其是笔记本),如下所示。
[root@hadoop101.yinzhengjie.org.cn ~]# rsync-hadoop.sh /etc/profile.d/jdk.sh ******* [hadoop102.yinzhengjie.org.cn] node starts synchronizing [/etc/profile.d/jdk.sh] ******* 命令执行成功 ******* [hadoop103.yinzhengjie.org.cn] node starts synchronizing [/etc/profile.d/jdk.sh] ******* 命令执行成功 ******* [hadoop104.yinzhengjie.org.cn] node starts synchronizing [/etc/profile.d/jdk.sh] ******* 命令执行成功 ******* [hadoop105.yinzhengjie.org.cn] node starts synchronizing [/etc/profile.d/jdk.sh] ******* 命令执行成功 ******* [hadoop106.yinzhengjie.org.cn] node starts synchronizing [/etc/profile.d/jdk.sh] ******* 命令执行成功 [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# rsync-hadoop.sh /etc/profile.d/hadoop.sh ******* [hadoop102.yinzhengjie.org.cn] node starts synchronizing [/etc/profile.d/hadoop.sh] ******* 命令执行成功 ******* [hadoop103.yinzhengjie.org.cn] node starts synchronizing [/etc/profile.d/hadoop.sh] ******* 命令执行成功 ******* [hadoop104.yinzhengjie.org.cn] node starts synchronizing [/etc/profile.d/hadoop.sh] ******* 命令执行成功 ******* [hadoop105.yinzhengjie.org.cn] node starts synchronizing [/etc/profile.d/hadoop.sh] ******* 命令执行成功 ******* [hadoop106.yinzhengjie.org.cn] node starts synchronizing [/etc/profile.d/hadoop.sh] ******* 命令执行成功 [root@hadoop101.yinzhengjie.org.cn ~]#
8>.格式化namenode节点
[root@hadoop101.yinzhengjie.org.cn ~]# hdfs namenode -format
9>.启动namenode
[root@hadoop101.yinzhengjie.org.cn ~]# jps 5798 Jps [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# hadoop-daemon.sh start namenode starting namenode, logging to /yinzhengjie/softwares/fully-mode/logs/hadoop-root-namenode-hadoop101.yinzhengjie.org.cn.out [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# jps 5908 Jps 5830 NameNode [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# ss -ntl State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 128 172.200.4.101:9000 *:* LISTEN 0 128 *:50070 *:* LISTEN 0 128 *:22 *:* LISTEN 0 128 :::22 :::* [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]#
10>.启动datanode
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps' hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >> 4271 Jps hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >> 4282 Jps hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >> 4300 Jps hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >> 5532 Jps hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >> 6064 Jps 5830 NameNode hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >> 5483 Jps [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible dn -m shell -a 'hadoop-daemon.sh start datanode' hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >> starting datanode, logging to /yinzhengjie/softwares/hadoop-2.10.0/logs/hadoop-root-datanode-hadoop104.yinzhengjie.org.cn.out hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >> starting datanode, logging to /yinzhengjie/softwares/hadoop-2.10.0/logs/hadoop-root-datanode-hadoop102.yinzhengjie.org.cn.out hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >> starting datanode, logging to /yinzhengjie/softwares/hadoop-2.10.0/logs/hadoop-root-datanode-hadoop103.yinzhengjie.org.cn.out [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps' hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >> 5915 Jps hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >> 4809 Jps 4682 DataNode hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >> 4827 Jps 4700 DataNode hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >> 4672 DataNode 4799 Jps hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >> 5830 NameNode 6969 Jps hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >> 5886 Jps [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]#
11>.启动Secondarynamenode
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps' hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >> 5354 Jps 5055 DataNode hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >> 6264 Jps hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >> 5072 DataNode 5371 Jps hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >> 8054 Jps 7727 NameNode hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >> 5344 Jps 5046 DataNode hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >> 6222 Jps [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible 2nn -m shell -a 'hadoop-daemon.sh start secondarynamenode' hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >> starting secondarynamenode, logging to /yinzhengjie/softwares/hadoop-2.10.0/logs/hadoop-root-secondarynamenode-hadoop105.yinzhengjie.org.cn.out [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps' hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >> 6418 SecondaryNameNode 6506 Jps hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >> 5072 DataNode 5429 Jps hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >> 5046 DataNode 5402 Jps hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >> 8227 Jps 7727 NameNode hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >> 5413 Jps 5055 DataNode hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >> 6281 Jps [root@hadoop101.yinzhengjie.org.cn ~]#
12>.启动ResourceManager节点
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps' hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >> 5072 DataNode 5499 Jps hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >> 8376 Jps 7727 NameNode hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >> 5480 Jps 5055 DataNode hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >> 6640 Jps 6418 SecondaryNameNode hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >> 5046 DataNode 5469 Jps hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >> 6342 Jps [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible rm -m shell -a 'yarn-daemon.sh start resourcemanager' hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >> starting resourcemanager, logging to /yinzhengjie/softwares/hadoop-2.10.0/logs/yarn-root-resourcemanager-hadoop106.yinzhengjie.org.cn.out [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps' hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >> 5046 DataNode 5530 Jps hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >> 5543 Jps 5055 DataNode hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >> 8535 Jps 7727 NameNode hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >> 5072 DataNode 5557 Jps hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >> 6418 SecondaryNameNode 6701 Jps hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >> 6691 Jps 6415 ResourceManager [root@hadoop101.yinzhengjie.org.cn ~]#
13>.启动NodeManager节点
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps' hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >> 5601 Jps 5055 DataNode hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >> 5072 DataNode 5617 Jps hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >> 5046 DataNode 5590 Jps hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >> 6768 Jps 6418 SecondaryNameNode hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >> 8758 Jps 7727 NameNode hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >> 6749 Jps 6415 ResourceManager [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible nm -m shell -a 'yarn-daemon.sh start nodemanager' hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >> starting nodemanager, logging to /yinzhengjie/softwares/hadoop-2.10.0/logs/yarn-root-nodemanager-hadoop102.yinzhengjie.org.cn.out hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >> starting nodemanager, logging to /yinzhengjie/softwares/hadoop-2.10.0/logs/yarn-root-nodemanager-hadoop103.yinzhengjie.org.cn.out hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >> starting nodemanager, logging to /yinzhengjie/softwares/hadoop-2.10.0/logs/yarn-root-nodemanager-hadoop104.yinzhengjie.org.cn.out [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps' hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >> 6418 SecondaryNameNode 6823 Jps hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >> 8915 Jps 7727 NameNode hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >> 5833 Jps 5674 NodeManager 5055 DataNode hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >> 5046 DataNode 5822 Jps 5663 NodeManager hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >> 5072 DataNode 5850 Jps 5691 NodeManager hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >> 6806 Jps 6415 ResourceManager [root@hadoop101.yinzhengjie.org.cn ~]#
14>.配置slaves(在hadoop101.yinzhengjie.org.cn节点通过slave文件,使用ssh命令启动slave节点的进程,因此最好配置免密登录~)
root@hadoop101.yinzhengjie.org.cn ~]# echo ${HADOOP_HOME} /yinzhengjie/softwares/hadoop-2.10.0 [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# vim ${HADOOP_HOME}/etc/hadoop/slaves [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# cat ${HADOOP_HOME}/etc/hadoop/slaves hadoop102.yinzhengjie.org.cn hadoop103.yinzhengjie.org.cn hadoop104.yinzhengjie.org.cn [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# rsync-hadoop.sh /yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/slaves ******* [hadoop102.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/slaves] ******* 命令执行成功 ******* [hadoop103.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/slaves] ******* 命令执行成功 ******* [hadoop104.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/slaves] ******* 命令执行成功 ******* [hadoop105.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/slaves] ******* 命令执行成功 ******* [hadoop106.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/slaves] ******* 命令执行成功 [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'cat /yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/slaves' hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >> hadoop102.yinzhengjie.org.cn hadoop103.yinzhengjie.org.cn hadoop104.yinzhengjie.org.cn hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >> hadoop102.yinzhengjie.org.cn hadoop103.yinzhengjie.org.cn hadoop104.yinzhengjie.org.cn hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >> hadoop102.yinzhengjie.org.cn hadoop103.yinzhengjie.org.cn hadoop104.yinzhengjie.org.cn hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >> hadoop102.yinzhengjie.org.cn hadoop103.yinzhengjie.org.cn hadoop104.yinzhengjie.org.cn hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >> hadoop102.yinzhengjie.org.cn hadoop103.yinzhengjie.org.cn hadoop104.yinzhengjie.org.cn hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >> hadoop102.yinzhengjie.org.cn hadoop103.yinzhengjie.org.cn hadoop104.yinzhengjie.org.cn [root@hadoop101.yinzhengjie.org.cn ~]#
15>.停止hadoop集群
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps' hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >> 5351 SecondaryNameNode 6282 Jps hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >> 9096 Jps 5531 NameNode hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >> 4096 DataNode 6089 NodeManager 6267 Jps hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >> 6146 NodeManager 6324 Jps 4089 DataNode hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >> 6257 Jps 4084 DataNode 6079 NodeManager hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >> 7555 ResourceManager 7866 Jps [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible nn -m shell -a 'stop-dfs.sh' hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >> Stopping namenodes on [hadoop101.yinzhengjie.org.cn] hadoop101.yinzhengjie.org.cn: stopping namenode hadoop102.yinzhengjie.org.cn: stopping datanode hadoop103.yinzhengjie.org.cn: stopping datanode hadoop104.yinzhengjie.org.cn: stopping datanode Stopping secondary namenodes [hadoop105.yinzhengjie.org.cn] hadoop105.yinzhengjie.org.cn: stopping secondarynamenode [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps' hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >> 9583 Jps hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >> 6377 Jps hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >> 6146 NodeManager 6414 Jps hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >> 6357 Jps 6089 NodeManager hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >> 6347 Jps 6079 NodeManager hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >> 7555 ResourceManager 7925 Jps [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible rm -m shell -a 'stop-yarn.sh' hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >> stopping yarn daemons stopping resourcemanager hadoop103.yinzhengjie.org.cn: stopping nodemanager hadoop103.yinzhengjie.org.cn: nodemanager did not stop gracefully after 5 seconds: killing with kill -9 hadoop102.yinzhengjie.org.cn: stopping nodemanager hadoop102.yinzhengjie.org.cn: nodemanager did not stop gracefully after 5 seconds: killing with kill -9 hadoop104.yinzhengjie.org.cn: stopping nodemanager hadoop104.yinzhengjie.org.cn: nodemanager did not stop gracefully after 5 seconds: killing with kill -9 no proxyserver to stop [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps' hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >> 6765 Jps hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >> 10312 Jps hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >> 6756 Jps hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >> 6548 Jps hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >> 6823 Jps hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >> 8239 Jps [root@hadoop101.yinzhengjie.org.cn ~]#
16>.修改脚本,将hadoop的pid文件放在指定目录中,默认放在"/tmp"目录。
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'mkdir -pv /yinzhengjie/softwares/hadoop-2.10.0/pid' [WARNING]: Consider using file module with state=directory rather than running mkdir hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >> mkdir: created directory ‘/yinzhengjie/softwares/hadoop-2.10.0/pid’ hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >> mkdir: created directory ‘/yinzhengjie/softwares/hadoop-2.10.0/pid’ hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >> mkdir: created directory ‘/yinzhengjie/softwares/hadoop-2.10.0/pid’ hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >> mkdir: created directory ‘/yinzhengjie/softwares/hadoop-2.10.0/pid’ hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >> mkdir: created directory ‘/yinzhengjie/softwares/hadoop-2.10.0/pid’ hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >> mkdir: created directory ‘/yinzhengjie/softwares/hadoop-2.10.0/pid’ [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# grep HADOOP_PID_DIR ${HADOOP_HOME}/etc/hadoop/hadoop-env.sh export HADOOP_PID_DIR=${HADOOP_PID_DIR} export HADOOP_SECURE_DN_PID_DIR=${HADOOP_PID_DIR} [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# vim ${HADOOP_HOME}/etc/hadoop/hadoop-env.sh [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# grep HADOOP_PID_DIR ${HADOOP_HOME}/etc/hadoop/hadoop-env.sh #export HADOOP_PID_DIR=${HADOOP_PID_DIR} export HADOOP_PID_DIR=/yinzhengjie/softwares/hadoop-2.10.0/pid export HADOOP_SECURE_DN_PID_DIR=${HADOOP_PID_DIR} [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# grep HADOOP_MAPRED_PID_DIR ${HADOOP_HOME}/etc/hadoop/mapred-env.sh #export HADOOP_MAPRED_PID_DIR= # The pid files are stored. /tmp by default. [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# vim ${HADOOP_HOME}/etc/hadoop/mapred-env.sh [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# grep HADOOP_MAPRED_PID_DIR ${HADOOP_HOME}/etc/hadoop/mapred-env.sh #export HADOOP_MAPRED_PID_DIR= # The pid files are stored. /tmp by default. export HADOOP_MAPRED_PID_DIR=/yinzhengjie/softwares/hadoop-2.10.0/pid [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# grep YARN_PID_DIR ${HADOOP_HOME}/etc/hadoop/yarn-env.sh [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# vim ${HADOOP_HOME}/etc/hadoop/yarn-env.sh [root@hadoop101.yinzhengjie.org.cn ~]# [root@hadoop101.yinzhengjie.org.cn ~]# grep YARN_PID_DIR ${HADOOP_HOME}/etc/hadoop/yarn-env.sh export YARN_PID_DIR=/yinzhengjie/softwares/hadoop-2.10.0/pid [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# rsync-hadoop.sh ${HADOOP_HOME}/etc/hadoop/hadoop-env.sh ******* [hadoop102.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/hadoop-env.sh] ******* 命令执行成功 ******* [hadoop103.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/hadoop-env.sh] ******* 命令执行成功 ******* [hadoop104.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/hadoop-env.sh] ******* 命令执行成功 ******* [hadoop105.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/hadoop-env.sh] ******* 命令执行成功 ******* [hadoop106.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/hadoop-env.sh] ******* 命令执行成功 [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# rsync-hadoop.sh ${HADOOP_HOME}/etc/hadoop/mapred-env.sh ******* [hadoop102.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/mapred-env.sh] ******* 命令执行成功 ******* [hadoop103.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/mapred-env.sh] ******* 命令执行成功 ******* [hadoop104.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/mapred-env.sh] ******* 命令执行成功 ******* [hadoop105.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/mapred-env.sh] ******* 命令执行成功 ******* [hadoop106.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/mapred-env.sh] ******* 命令执行成功 [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# rsync-hadoop.sh ${HADOOP_HOME}/etc/hadoop/yarn-env.sh ******* [hadoop102.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/yarn-env.sh] ******* 命令执行成功 ******* [hadoop103.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/yarn-env.sh] ******* 命令执行成功 ******* [hadoop104.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/yarn-env.sh] ******* 命令执行成功 ******* [hadoop105.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/yarn-env.sh] ******* 命令执行成功 ******* [hadoop106.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/yarn-env.sh] ******* 命令执行成功 [root@hadoop101.yinzhengjie.org.cn ~]#
17>.启动hadoop集群
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps' hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >> 6879 Jps hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >> 6943 Jps hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >> 6886 Jps hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >> 10580 Jps hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >> 6674 Jps hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >> 8361 Jps [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible nn -m shell -a 'start-dfs.sh' hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >> Starting namenodes on [hadoop101.yinzhengjie.org.cn] hadoop101.yinzhengjie.org.cn: starting namenode, logging to /yinzhengjie/softwares/hadoop-2.10.0/logs/hadoop-root-namenode-hadoop101.yinzhengjie.org.cn.out hadoop102.yinzhengjie.org.cn: starting datanode, logging to /yinzhengjie/softwares/hadoop-2.10.0/logs/hadoop-root-datanode-hadoop102.yinzhengjie.org.cn.out hadoop104.yinzhengjie.org.cn: starting datanode, logging to /yinzhengjie/softwares/hadoop-2.10.0/logs/hadoop-root-datanode-hadoop104.yinzhengjie.org.cn.out hadoop103.yinzhengjie.org.cn: starting datanode, logging to /yinzhengjie/softwares/hadoop-2.10.0/logs/hadoop-root-datanode-hadoop103.yinzhengjie.org.cn.out Starting secondary namenodes [hadoop105.yinzhengjie.org.cn] hadoop105.yinzhengjie.org.cn: starting secondarynamenode, logging to /yinzhengjie/softwares/hadoop-2.10.0/logs/hadoop-root-secondarynamenode-hadoop105.yinzhengjie.org.cn.out [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps' hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >> 11121 Jps 10774 NameNode hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >> 6987 DataNode 7118 Jps hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >> 6923 DataNode 7054 Jps hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >> 6806 Jps 6717 SecondaryNameNode hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >> 6930 DataNode 7061 Jps hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >> 8419 Jps [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible rm -m shell -a 'start-yarn.sh' hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >> starting yarn daemons starting resourcemanager, logging to /yinzhengjie/softwares/hadoop-2.10.0/logs/yarn-root-resourcemanager-hadoop106.yinzhengjie.org.cn.out hadoop103.yinzhengjie.org.cn: starting nodemanager, logging to /yinzhengjie/softwares/hadoop-2.10.0/logs/yarn-root-nodemanager-hadoop103.yinzhengjie.org.cn.out hadoop104.yinzhengjie.org.cn: starting nodemanager, logging to /yinzhengjie/softwares/hadoop-2.10.0/logs/yarn-root-nodemanager-hadoop104.yinzhengjie.org.cn.out hadoop102.yinzhengjie.org.cn: starting nodemanager, logging to /yinzhengjie/softwares/hadoop-2.10.0/logs/yarn-root-nodemanager-hadoop102.yinzhengjie.org.cn.out [root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps' hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >> 6922 Jps 6717 SecondaryNameNode hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >> 10774 NameNode 11383 Jps hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >> 7221 NodeManager 7399 Jps 6987 DataNode hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >> 7157 NodeManager 7335 Jps 6923 DataNode hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >> 6930 DataNode 7164 NodeManager 7342 Jps hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >> 8885 Jps 8574 ResourceManager [root@hadoop101.yinzhengjie.org.cn ~]#
18>.查看hadoop集群的pid文件
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'ls -l /yinzhengjie/softwares/hadoop-2.10.0/pid/' hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >> total 8 -rw-r--r-- 1 root root 6 Mar 12 08:40 hadoop-root-namenode.pid hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >> total 8 -rw-r--r-- 1 root root 5 Mar 12 08:40 hadoop-root-datanode.pid -rw-r--r-- 1 root root 5 Mar 12 08:41 yarn-root-nodemanager.pid hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >> total 4 -rw-r--r-- 1 root root 5 Mar 12 08:40 hadoop-root-secondarynamenode.pid hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >> total 8 -rw-r--r-- 1 root root 5 Mar 12 08:40 hadoop-root-datanode.pid -rw-r--r-- 1 root root 5 Mar 12 08:41 yarn-root-nodemanager.pid hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >> total 8 -rw-r--r-- 1 root root 5 Mar 12 08:40 hadoop-root-datanode.pid -rw-r--r-- 1 root root 5 Mar 12 08:41 yarn-root-nodemanager.pid hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >> total 4 -rw-r--r-- 1 root root 5 Mar 12 08:41 yarn-root-resourcemanager.pid [root@hadoop101.yinzhengjie.org.cn ~]#