大数据开发平台搭建

一、JDK安装

1、删除系统自带的jdk

查看系统自带的jdk

[root@hadoop-master ~]# rpm -qa | grep jdk
copy-jdk-configs-3.3-10.el7_5.noarch
java-1.8.0-openjdk-headless-1.8.0.242.b08-0.el7_7.x86_64

删除系统自带的jdk

[root@hadoop-master ~]# rpm -e --nodeps java-1.8.0-openjdk-headless-1.8.0.242.b08-0.el7_7.x86_64

2、安装

创建文件夹

[root@hadoop-master ~]# mkdir /usr/java

上传安装包

将所需要的安装包放到该用户的家目录：

解压安装包

将该安装包解压到/usr/java包下：

[root@hadoop-master java]# tar -zxvf jdk-7u67-linux-x64.tar.gz  -C /usr/java

如下：

[root@hadoop-master java]# ls
jdk1.7.0_67

3、配置环境变量

在/etc/profile最后面加入下面的内容：

...
...
export JAVA_HOME=/usr/java/jdk1.7.0_67
export JRE_HOME=/usr/java/jdk1.7.0_67/jre
export PATH=$PATH:/usr/java/jdk1.7.0_67/bin
export CLASSPATH=./:/usr/java/jdk1.7.0_67/lib:/usr/java/jdk1.7.0_67/jre/lib

4、配置立即生效

[root@hadoop-master java]# source /etc/profile

5、查看是否安装成功

[root@hadoop-master java]# java -version
java version "1.7.0_67"
Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)

一、配置免密登录

1、master节点操作

生成秘匙

进入到master节点上，生成公匙：

[root@hadoop-master ~]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:FIyiPOh1VOOYrcLHPHzkymcY3gWuCqHd/Jui+eY/ogk root@hadoop-master.com
The key's randomart image is:
+---[RSA 2048]----+
|      .=.        |
|    ..* o.       |
| o ..+ =.        |
|. =.+.=..        |
|...+.O +S.       |
|.o.o= O .        |
|E.. o* +         |
| ..o=.+.         |
|  =Bo+=o         |
+----[SHA256]-----+

可以看到当前用户目录下多出的秘匙文件：

[root@hadoop-master ~]# cd .ssh
[root@hadoop-master .ssh]# ls
id_rsa  id_rsa.pub

复制公匙文件

[root@hadoop-master .ssh]#  cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

复制后的结果是：

[root@hadoop-master .ssh]# ls
authorized_keys  id_rsa  id_rsa.pub
[root@hadoop-master .ssh]# cat authorized_keys 
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJv30jkhGWQ+iYM+FhV3gCGN6d/GunU42+jXs9MVShm9CDE7SwAjqh88/OtkJ9sI/+L4F81z71BzsZKi0/+ACFZ+WG
ZayS/nSL9JYRvqbx0XUDssJ2zgniw/HDAF8fIjV3KP7pm2cKefG+anHlh4yYZfAg55C4EkmSqQLrHJ0O89GzqYzAMkkI8Iv4eH/BL9yXQgrlM45PEPpDZ+kMOYRkmiTJzJR
COQ6C9DpIt49UiHWnHxAFGA9e0g8WjL0cQJhfontc7BYOtsigzm0gyFhFGm+6WIexIPGnkE/Yxa62BcKlam0o/f4j2WZHD8L7SbzosVC3yFvxxfJ8x1dq9GL/ root@hadoop
-master.com

3、slave1节点操作

生成秘匙

登录上从节点slave1，从节点slave1生成秘匙，并且将其发送给主节点：

[root@hadoop-slave1 ~]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:dRlbekhUSaSifnY6Qc+0Tg3kj+/Y2oBmJPrgQd4G2cM root@hadoop-slave1.com
The key's randomart image is:
+---[RSA 2048]----+
|           .++=. |
|           ..O.  |
|          ooB .  |
|       + o.o+.   |
|      + E..+ *   |
|     o = +..* o  |
|      = + *+o.   |
|     . = =.o.=.  |
|      . . ..oo+  |
+----[SHA256]-----+

复制秘匙

复制秘匙文件，并且将其发送给master节点：

[root@hadoop-slave1 ~]#  cp ~/.ssh/id_rsa.pub ~/.ssh/slave1-id_rsa.pub
[root@hadoop-slave1 ~]# scp ~/.ssh/slave1-id_rsa.pub root@192.168.0.110:~/
[root@hadoop-slave1 ~]# rm -rf ~/.ssh/slave1-id_rsa.pub

4、slave2节点操作

生成秘匙

登录上从节点slave2，从节点slave2生成秘匙，并且将其发送给主节点：

[root@hadoop-slave2 ~]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:KOQGdDzAoSSUEcQENaT6/udACXeCxDtocDeDRadySSo root@hadoop-slave2.com
The key's randomart image is:
+---[RSA 2048]----+
|O%%B= . |
|+B=**+ |
|Eo*oB+. |
|oooO + . |
|o .* . S |
| . o . |
| . . |
| . .. |
| ...o. |
+----[SHA256]-----+

复制秘匙

复制秘匙文件，并且将其发送给master节点：

[root@hadoop-slave1 ~]#  cp ~/.ssh/id_rsa.pub ~/.ssh/slave2-id_rsa.pub
[root@hadoop-slave1 ~]# scp ~/.ssh/slave2-id_rsa.pub root@192.168.0.110:~/
[root@hadoop-slave1 ~]# rm -rf ~/.ssh/slave2-id_rsa.pub

5、master节点操作

复制从节点的秘钥文件，追加到authorized_keys文件中

[root@hadoop-master ~]# cat ~/slave1-id_rsa.pub >> ~/.ssh/authorized_keys 
[root@hadoop-master ~]# cat ~/slave2-id_rsa.pub >> ~/.ssh/authorized_keys

修改authorized_keys文件权限，并且复制到从节点slave1,slave2上：

[root@hadoop-master ~]# chmod 600 ~/.ssh/authorized_keys 
[root@hadoop-master ~]# scp ~/.ssh/authorized_keys root@192.168.0.111:~/
[root@hadoop-master ~]# scp ~/.ssh/authorized_keys root@192.168.0.112:~/

6、slave1节点操作

修改authorized_keys 权限，并且拷贝到.ssh文件夹中：

[root@hadoop-slave1 ~]# chmod 600 ~/authorized_keys 
[root@hadoop-slave1 ~]# cp ~/authorized_keys ~/.ssh/

7、slave2节点操作

修改authorized_keys 权限，并且拷贝到.ssh文件夹中：

[root@hadoop-slave1 ~]# chmod 600 ~/authorized_keys 
[root@hadoop-slave1 ~]# cp ~/authorized_keys ~/.ssh/

8、测试

[root@hadoop-master ~]# ssh 192.168.0.111
Last login: Sun Feb 23 09:04:58 2020 from 192.168.0.107
[root@hadoop-slave1 ~]# exit
logout
Connection to 192.168.0.111 closed.
[root@hadoop-master ~]# ssh 192.168.0.112
Last login: Sun Feb 23 09:05:02 2020 from 192.168.0.107
[root@hadoop-slave2 ~]# exit
logout
Connection to 192.168.0.112 closed.

如果在/etc/hosts文件中进行了hostname与ip的映射，那么可以使用ip进行登录。

二、hadoop安装

（一）安装前准备工作

1、hosts文件编写

将master与slave的ip与hostname分别写入到三台master与slave的/etc/hosts文件中（所有的机器都一样）

192.168.0.110 hadoop-master
192.168.0.111 hadoop-slave1
192.168.0.112 hadoop-slave2

1、安装包上传

将所需要的俺炸U那个包上传到~/hadoop/software目录中：

[root@hadoop-master software]# ls
apache-hive-1.2.2-bin.tar.gz  jdk-7u67-linux-x64.tar.gz
hadoop-2.6.1.tar.gz           scala-2.11.8.tgz

2、slave节点上安装JDK准备

之前在master节点上安装过JDK，但是从节点也需要安装，不过因为已经免秘登录了，所以现在可以先拷贝到从节点，然后直接在master节点上登录slave节点进行安装即可：

拷贝JDK到slave节点

分别在slave1、slave2上建立两个文件夹，一个用于放置安装包，一个用于放置解压后的安装程序，这里以slave1为例：

[root@hadoop-slave1 ~]# mkdir -p ~/hadoop/software
[root@hadoop-slave1 ~]# mkdir /usr/java

然后将JDK从master中远程传递给slave节点上：

[root@hadoop-master software]# scp jdk-7u67-linux-x64.tar.gz root@192.168.0.111:~/hadoop/software
jdk-7u67-linux-x64.tar.gz                               100%  136MB  29.0MB/s   00:04

3、slave节点上安装JDK

这与master节点上安装方式相同，可参考master节点上安装过程。

（二）安装hadoop

仅在master节点进行该项配置

1、解压hadoop安装包

将安装包解压到/root/hadoop目录下

[root@hadoop-master ~]# tar -xzvf hadoop-2.6.1.tar.gz -C  hadoop/

解压后的结果如下：

2、配置hadoop

创建存放数据文件夹

在/root/hadoop下创建存放数据的多个文件夹：

[root@hadoop-master hadoop]# mkdir temp
[root@hadoop-master hadoop]# mkdir -p hdfs/data
[root@hadoop-master hadoop]# mkdir hdfs/name

配置hadoop环境变量

vim /etc/profile,加入如下内容，红色字体即是：

export JAVA_HOME=/usr/java/jdk1.7.0_67
export HADOOP_HOME=/root/hadoop/hadoop-2.6.1
export JRE_HOME=/usr/java/jdk1.7.0_67/jre
export PATH=$PATH:/usr/java/jdk1.7.0_67/bin:$HADOOP_HOME/bin
export CLASSPATH=./:/usr/java/jdk1.7.0_67/lib:/usr/java/jdk1.7.0_67/jre/lib

然后使其立即生效：

[root@hadoop-master hdfs]# source /etc/profile

然后进行验证是否安装成功：

[root@hadoop-master ~]# hadoop
Usage: hadoop [--config confdir] COMMAND
       where COMMAND is one of:
  fs                   run a generic filesystem user client
  version              print the version
  jar <jar>            run a jar file
  checknative [-a|-h]  check native hadoop and compression libraries availability
  distcp <srcurl> <desturl> copy file or directories recursively
  archive -archiveName NAME -p <parent path> <src>* <dest> create a hadoop archive
  classpath            prints the class path needed to get the
  credential           interact with credential providers
                       Hadoop jar and the required libraries
  daemonlog            get/set the log level for each daemon
  trace                view and modify Hadoop tracing settings
 or
  CLASSNAME            run the class named CLASSNAME

Most commands print help when invoked w/o parameters.

如果出现上面这样的，就是安装成功了。

配置hadoop-env.sh

该文件位于上述解压后目录中的hadoop-2.6.1/etc/hadoop/hadoop-env.sh，在该文件中对jdk进行配置：

将hadoop-env.sh文件中export JAVA_HOME=${JAVA_HOME}修改为当前JDK的路径：

export JAVA_HOME=/usr/java/jdk1.7.0_67

配置 yarn-env.sh

该文件位于上述解压后目录中的hadoop-2.6.1/etc/hadoop/yarn-env.sh，在该文件中对jdk进行配置：

将yarn-env.sh文件中export JAVA_HOME=/home/y/libexec/jdk1.6.0/修改为当前JDK的路径：

export JAVA_HOME=/usr/java/jdk1.7.0_67/

配置核心组件core.site.xml

该文件位于上述解压后目录中的hadoop-2.6.1/etc/hadoop/core-site.xml，配置成为以下文本：

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
<!-- Put site-specific property overrides in this file. -->

<configuration>
 <configuration>
  <property>
     <name>fs.defaultFS</name>
     <value>hdfs://hadoop-master:9000</value>
  </property>
  <property>
     <name>hadoop.tmp.dir</name>
     <value>file:/home/hadoop/tmp</value>
  </property>
  <property>
     <name>io.file.buffer.size</name>
     <value>131702</value>
  </property>
</configuration>

配置 hdfs-site.xml

该文件位于上述解压后目录中的hadoop-2.6.1/etc/hadoop/ hdfs-site.xm，配置成为以下文本：

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
<configuration>
<configuration>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:/home/hadoop/hdfs/name</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:/home/hadoop/hdfs/data</value>
    </property>
    <property>
        <name>dfs.replication</name>
        <value>2</value>
    </property>
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>hadoop-master:9001</value>
    </property>
    <property>
        <name>dfs.webhdfs.enabled</name>
        <value>true</value>
    </property>
</configuration>
</configuration>

配置 yarn-site.xml

该文件位于上述解压后目录中的hadoop-2.6.1/etc/hadoop/ yarn-site.xm，配置成为以下文本：

<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<configuration>

<!-- Site specific YARN configuration properties -->
 <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
    <property>
        <name>yarn.resourcemanager.address</name>
        <value>hadoop-master:8032</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>hadoop-master:8030</value>
    </property>
    <property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>hadoop-master:8031</value>
    </property>
    <property>
        <name>yarn.resourcemanager.admin.address</name>
        <value>hadoop-master:8033</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>hadoop-master:8088</value>
    </property>
    <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>2048</value>
    </property>
    <property>
        <name>yarn.scheduler.maximum-allocation-mb</name>
        <value>2048</value>
    </property>
</configuration>

配置 mapred-site.xml

　　该文件在配置目录中并没有，但是在上述解压后目录中的hadoop-2.6.1/etc/hadoop/中存在mapred-site.xm.template这个文件，可以通过vim mapred-site.xml直接创建和写入遮掩一个文件，这是计算框架配置文件，配置成为以下文本：

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.address</name>
        <value>hadoop-master:10020</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>hadoop-master:19888</value>
    </property>
</configuration>

配置slaves文件

该文件位于上述解压后目录中的hadoop-2.6.1/etc/hadoop/ slaves，讲该文件中的内容：

localhost

替换为如下slave节点的ip或者hostname：

192.168.0.111
192.168.0.112

复制到slave节点

将上述配置好的hadoop程序复制到slave节点：

[root@hadoop-master ~]# scp -r /root/hadoop root@hadoop-slave1:/root/
[root@hadoop-master ~]# scp -r /root/hadoop root@hadoop-slave2:/root/

将整个hadoop目录拷贝到用户的家目录，注意这里/root就是用户的家目录（~）。

启动hadoop服务器

在master节点上启动hadoop服务器，slave节点会自动启动进入到/root/hadoop/hadoop-2.6.1目录：

（1）初始化

在master上，进入到/root/hadoop/hadoop-2.6.1目录下，执行格式化HDFS文件系统命令：

[root@hadoop-master hadoop-2.6.1]# bin/hdfs namenode -format

出现以下信息，表示初始化成功：

[root@hadoop-master hadoop-2.6.1]# bin/hdfs namenode -format
20/02/23 18:06:14 INFO namenode.NameNode: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = hadoop-master/192.168.0.110
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 2.6.1
STARTUP_MSG:   classpath = /root/hadoop/hadoop-2.6.1/etc/hadoop:/root/hadoop/hadoop-2.6.1/share
/hadoop/common/lib/xmlenc-0.52.jar:/root/hadoop/hadoop-2.6.1/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:
/root/hadoop/hadoop-2.6.1/share/hadoop/common/lib/commons-httpclient-3.1.jar:/root/hadoop/hadoop-2.6.1
/share/hadoop/common/lib/jetty-6.1.26.jar:/root/hadoop/hadoop-2.6.1/share/hadoop/common/lib
/curator-recipes-2.6.0.jar:/root/hadoop/hadoop-2.6.1/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:
/root/hadoop/hadoop-2.6.1/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/root/hadoop
/hadoop-2.6.1/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/root/hadoop/hadoop-2.6.1/share/hadoop
/common/lib/log4j-1.2.17.jar:/root/hadoop/hadoop-2.6.1/share/hadoop/common/lib/commons-math3-3.1.1.jar:
/root/hadoop/hadoop-2.6.1/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/root/hadoop/hadoop-2.6.1
/share/hadoop/common/lib/jettison-1.1.jar:/root/hadoop/hadoop-2.6.1/share/hadoop/common/lib/curator-client-2.6.0.jar:
/root/hadoop/hadoop-2.6.1/share/hadoop/common/lib/xz-1.0.jar:/root/hadoop/hadoop-2.6.1/share/hadoop/common/lib
/stax-api-1.0-2.jar:/root/hadoop/hadoop-2.6.1/share/hadoop/common/lib/jets3t-0.9.0.jar:/root/hadoop/hadoop-2.6.1/
share/hadoop/common/lib/httpcore-4.2.5.jar:/root/hadoop/hadoop-2.6.1/share/hadoop/common/lib/httpclient-4.2.5.jar:/root/
hadoop/hadoop-2.6.1/share/hadoop/common/lib/hadoop-auth-2.6.1.jar:/root/hadoop/hadoop-2.6.1/share/hadoop/common/lib/
commons-logging-1.1.3.jar:/root/hadoop/hadoop-2.6.1/share/hadoop/common/lib/commons-el-1.0.jar:/root/hadoop
/hadoop-2.6.1/share/hadoop/common/lib/mockito-all-1.8.5.jar:/root/hadoop/hadoop-2.6.1/share/hadoop/common/lib
/jaxb-api-2.2.2.jar:/root/hadoop/hadoop-2.6.1/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/root/hadoop
/hadoop-2.6.1/share/hadoop/common/lib/guava-11.0.2.jar:/root/hadoop/hadoop-2.6.1/share/hadoop/common/lib
/gson-2.2.4.jar:/root/hadoop/hadoop-2.6.1/share/hadoop/common/lib/hadoop-annotations-2.6.1.jar:/root/hadoop
/hadoop-2.6.1/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/root/hadoop/hadoop-2.6.1/share/hadoop
/common/lib/commons-beanutils-1.7.0.jar:/root/hadoop/hadoop-2.6.1/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:
/root/hadoop/hadoop-2.6.1/share/hadoop/common/lib/jetty-util-6.1.26.jar:/root/hadoop/hadoop-2.6.1/share/hadoop/common
/lib/zookeeper-3.4.6.jar:/root/hadoop/hadoop-2.6.1/share/hadoop/common/lib/commons-configuration-1.6.jar:/root/hadoop
/hadoop-2.6.1/share/hadoop/common/lib/paranamer-2.3.jar:/root/hadoop/hadoop-2.6.1/share/hadoop/common/lib/jersey-server-1.9.jar:
/root/hadoop/hadoop-2.6.1/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/root/hadoop/hadoop-2.6.1/share/hadoop/common/lib
/jackson-jaxrs-1.9.13.jar:/root/hadoop/hadoop-2.6.1/share/hadoop/common/lib/commons-cli-1.2.jar:/root/hadoop/hadoop-2.6.1
/share/hadoop/common/lib/asm-3.2.jar:/root/hadoop/hadoop-2.6.1/share/hadoop/common/lib/jsr305-1.3.9.jar:/root/hadoop/hadoop-2.6.1
/share/hadoop/common/lib/avro-1.7.4.jar:/root/hadoop/hadoop-2.6.1/share/hadoop/common/lib/junit-4.11.jar:/root/hadoop/hadoop-2.6.1
/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/root/hadoop/hadoop-2.6.1/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/root/hadoop
/hadoop-2.6.1/share/hadoop/common/lib/curator-framework-2.6.0.jar:/root/hadoop/hadoop-2.6.1/share/hadoop/common/lib/
jackson-mapper-asl-1.9.13.jar:/root/hadoop/hadoop-2.6.1/share/hadoop/common/lib/jersey-json-1.9.jar:/root/hadoop/hadoop-2.6.1
/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/root/hadoop/hadoop-2.6.1/share/hadoop/common/lib/commons-net-3.1.jar:/
root/hadoop/hadoop-2.6.1/share/hadoop/common/lib/commons-compress-1.4.1.jar:/root/hadoop/hadoop-2.6.1/share/hadoop/common/
lib/jersey-core-1.9.jar:/root/hadoop/hadoop-2.6.1/share/hadoop/common/lib/servlet-api-2.5.jar:/root/hadoop/hadoop-2.6.1
/share/hadoop/common/lib/jsp-api-2.1.jar:/root/hadoop/hadoop-2.6.1/share/hadoop/common/lib/activation-1.1.jar:/root/hadoop
/hadoop-2.6.1/share/hadoop/common/lib/jsch-0.1.42.jar:/root/hadoop/hadoop-2.6.1/share/hadoop/common/lib/api-util-1.0.0-M20.
...
...
************************************************************/
20/02/23 18:06:14 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
20/02/23 18:06:14 INFO namenode.NameNode: createNameNode [-format]
Formatting using clusterid: CID-14737f4a-c23a-41d2-b8aa-7d77c8371674
20/02/23 18:06:15 INFO namenode.FSNamesystem: No KeyProvider found.
20/02/23 18:06:15 INFO namenode.FSNamesystem: fsLock is fair:true
20/02/23 18:06:15 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
20/02/23 18:06:15 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
20/02/23 18:06:15 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
20/02/23 18:06:15 INFO blockmanagement.BlockManager: The block deletion will start around 2020 Feb 23 18:06:15
20/02/23 18:06:15 INFO util.GSet: Computing capacity for map BlocksMap
20/02/23 18:06:15 INFO util.GSet: VM type       = 64-bit
20/02/23 18:06:15 INFO util.GSet: 2.0% max memory 966.7 MB = 19.3 MB
20/02/23 18:06:15 INFO util.GSet: capacity      = 2^21 = 2097152 entries
20/02/23 18:06:15 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
20/02/23 18:06:15 INFO blockmanagement.BlockManager: defaultReplication         = 2
20/02/23 18:06:15 INFO blockmanagement.BlockManager: maxReplication             = 512
20/02/23 18:06:15 INFO blockmanagement.BlockManager: minReplication             = 1
20/02/23 18:06:15 INFO blockmanagement.BlockManager: maxReplicationStreams      = 2
20/02/23 18:06:15 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks  = false
20/02/23 18:06:15 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
20/02/23 18:06:15 INFO blockmanagement.BlockManager: encryptDataTransfer        = false
20/02/23 18:06:15 INFO blockmanagement.BlockManager: maxNumBlocksToLog          = 1000
20/02/23 18:06:15 INFO namenode.FSNamesystem: fsOwner             = root (auth:SIMPLE)
20/02/23 18:06:15 INFO namenode.FSNamesystem: supergroup          = supergroup
20/02/23 18:06:15 INFO namenode.FSNamesystem: isPermissionEnabled = true
20/02/23 18:06:15 INFO namenode.FSNamesystem: HA Enabled: false
20/02/23 18:06:15 INFO namenode.FSNamesystem: Append Enabled: true
20/02/23 18:06:16 INFO util.GSet: Computing capacity for map INodeMap
20/02/23 18:06:16 INFO util.GSet: VM type       = 64-bit
20/02/23 18:06:16 INFO util.GSet: 1.0% max memory 966.7 MB = 9.7 MB
20/02/23 18:06:16 INFO util.GSet: capacity      = 2^20 = 1048576 entries
20/02/23 18:06:16 INFO namenode.NameNode: Caching file names occuring more than 10 times
20/02/23 18:06:16 INFO util.GSet: Computing capacity for map cachedBlocks
20/02/23 18:06:16 INFO util.GSet: VM type       = 64-bit
20/02/23 18:06:16 INFO util.GSet: 0.25% max memory 966.7 MB = 2.4 MB
20/02/23 18:06:16 INFO util.GSet: capacity      = 2^18 = 262144 entries
20/02/23 18:06:16 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
20/02/23 18:06:16 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
20/02/23 18:06:16 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension     = 30000
20/02/23 18:06:16 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
20/02/23 18:06:16 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
20/02/23 18:06:16 INFO util.GSet: Computing capacity for map NameNodeRetryCache
20/02/23 18:06:16 INFO util.GSet: VM type       = 64-bit
20/02/23 18:06:16 INFO util.GSet: 0.029999999329447746% max memory 966.7 MB = 297.0 KB
20/02/23 18:06:16 INFO util.GSet: capacity      = 2^15 = 32768 entries
20/02/23 18:06:16 INFO namenode.NNConf: ACLs enabled? false
20/02/23 18:06:16 INFO namenode.NNConf: XAttrs enabled? true
20/02/23 18:06:16 INFO namenode.NNConf: Maximum size of an xattr: 16384
20/02/23 18:06:16 INFO namenode.FSImage: Allocated new BlockPoolId: BP-937003390-192.168.0.110-1582452376310
20/02/23 18:06:16 INFO common.Storage: Storage directory /home/hadoop/hdfs/name has been successfully formatted.
20/02/23 18:06:16 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
20/02/23 18:06:16 INFO util.ExitUtil: Exiting with status 0
20/02/23 18:06:16 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at hadoop-master/192.168.0.110
************************************************************/

（2）启动hadoop集群

[root@hadoop-master hadoop-2.6.1]# sbin/start-all.sh

出现以下信息表示启动成功：

This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [hadoop-master]
The authenticity of host 'hadoop-master (192.168.0.110)' can't be established.
ECDSA key fingerprint is SHA256:X07vjxCDQHRhz9OZXtIYZzjTvvYVkaDVuko6bVdzJ90.
ECDSA key fingerprint is MD5:5e:b1:67:be:57:28:b1:13:e8:d3:9d:fa:2a:4b:10:49.
Are you sure you want to continue connecting (yes/no)? yes
hadoop-master: Warning: Permanently added 'hadoop-master,192.168.0.110' (ECDSA) to the list of known hosts.
hadoop-master: starting namenode, logging to /root/hadoop/hadoop-2.6.1/logs/hadoop-root-namenode-hadoop-master.out
192.168.0.112: starting datanode, logging to /root/hadoop/hadoop-2.6.1/logs/hadoop-root-datanode-hadoop-slave2.out
192.168.0.111: starting datanode, logging to /root/hadoop/hadoop-2.6.1/logs/hadoop-root-datanode-hadoop-slave1.out
Starting secondary namenodes [hadoop-master]
hadoop-master: starting secondarynamenode, logging to /root/hadoop/hadoop-2.6.1/logs/hadoop-root-secondarynamenode-hadoop-master.out
starting yarn daemons
starting resourcemanager, logging to /root/hadoop/hadoop-2.6.1/logs/yarn-root-resourcemanager-hadoop-master.out
192.168.0.111: starting nodemanager, logging to /root/hadoop/hadoop-2.6.1/logs/yarn-root-nodemanager-hadoop-slave1.out
192.168.0.112: starting nodemanager, logging to /root/hadoop/hadoop-2.6.1/logs/yarn-root-nodemanager-hadoop-slave2.out

当然你也可以分步骤来启动通过以下命令：

[root@hadoop-master hadoop-2.6.1]# sbin/start-dfs.sh
[root@hadoop-master hadoop-2.6.1]# sbin/start-yarn.sh

（3）停止集群

[root@hadoop-slave1 hadoop-2.6.1]# sbin/stop-all.sh

（4）查看相关信息

[root@hadoop-master hadoop-2.6.1]# jps
3118 ResourceManager
3403 Jps
2980 SecondaryNameNode
2807 NameNode

访问hadoop web

通过访问master的ip地址加端口即可，如：

192.168.0.110:8088
192.168.0.110:50070

注意，请先关闭或者永久关闭防火墙：

system stop firewalld.service #临时关闭
systemctl disable firewalld.service #永久关闭

下面是访问master8088端口的界面：

posted @ 2020-03-15 17:27 iveBoy 阅读(653) 评论(0) 收藏举报

刷新页面返回顶部

iveBoy