大数据,十有八九挂在装环境上(4)
安装zookeeper
把安装包 zookeeper-3.4.14.tar.gz 放到software目录里
解压
tar -zxvf zookeeper-3.4.14.tar.gz -C ../servers/
创建data和log目录
#创建zk存储数据目录 mkdir -p /opt/lagou/servers/zookeeper-3.4.14/data #创建zk日志文件目录 mkdir -p /opt/lagou/servers/zookeeper-3.4.14/data/logs #修改zk配置文件 cd /opt/lagou/servers/zookeeper-3.4.14/conf #文件改名 mv zoo_sample.cfg zoo.cfg vim zoo.cfg #更新datadir dataDir=/opt/lagou/servers/zookeeper-3.4.14/data #增加logdir dataLogDir=/opt/lagou/servers/zookeeper-3.4.14/data/logs #增加集群配置 ##server.服务器ID=服务器IP地址:服务器之间通信端口:服务器之间投票选举端口 server.1=centos7-1:2888:3888 server.2=centos7-2:2888:3888 server.3=centos7-3:2888:3888 #打开注释 #ZK提供了自动清理事务日志和快照文件的功能,这个参数指定了清理频率,单位是小时 autopurge.purgeInterval=1
添加myid配置
cd /opt/lagou/servers/zookeeper-3.4.14/data echo 1 > myid
分发到其他节点上
cd /opt/lagou/servers rsync-script /opt/lagou/servers/zookeeper-3.4.14
修改其他节点上的myid
测试启动
/opt/lagou/servers/zookeeper-3.4.14/bin/zkServer.sh start
查看状况
/opt/lagou/servers/zookeeper-3.4.14/bin/zkServer.sh status
启动/停止脚本
cd ~
mkdir shell
cd shell
vim zk.sh #!/bin/sh echo "start zookeeper server..." if(($#==0));then echo "no params"; exit; fi hosts="cenots7-1 centos7-2 centos7-3" for host in $hosts do ssh $host "source /etc/profile; /opt/lagou/servers/zookeeper-3.4.14/bin/zkServer.sh $1" done
chmod 777 zk.sh
搭建HA
集群规划
centos7-1 | centos7-2 | centos7-3 |
NameNode |
NameNode | |
JournalNode | JournalNode | JournalNode |
DataNode | DataNode | DataNode |
ZK | ZK | ZK |
ResourceManager | ResourceManager | |
NodeManager | NodeManager | NodeManager |
配置HDFS-HA集群
1. 停止原来的HDFS集群
stop-dfs.sh
2. 每个节点都创建ha文件夹
mkdir /opt/lagou/servers/ha
3. 节点centos7-1复制hadoop到ha里
cp -r hadoop-2.9.2/ ha/
4. 删除ha里面的data
rm -rf /opt/lagou/servers/ha/hadoop-2.9.2/data
5. 配置hdfs-site.xml
cd /opt/lagou/servers/ha/hadoop-2.9.2/etc/hadoop/
<property> <name>dfs.nameservices</name> <value>lagoucluster</value> </property> <property> <name>dfs.ha.namenodes.lagoucluster</name> <value>nn1,nn2</value> </property> <property> <name>dfs.namenode.rpc-address.lagoucluster.nn1</name> <value>centos7-1:9000</value> </property> <property> <name>dfs.namenode.rpc-address.lagoucluster.nn2</name> <value>centos7-2:9000</value> </property> <property> <name>dfs.namenode.http-address.lagoucluster.nn1</name> <value>centos7-1:50070</value> </property> <property> <name>dfs.namenode.http-address.lagoucluster.nn2</name> <value>centos7-2:50070</value> </property> <property> <name>dfs.namenode.shared.edits.dir</name> <value>qjournal://centos7-1:8485;centos7-2:8485;centos7-3:8485/lagou</value> </property> <property> <name>dfs.client.failover.proxy.provider.lagoucluster</name> <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value> </property> <property> <name>dfs.ha.fencing.methods</name> <value>sshfence</value> </property> <property> <name>dfs.ha.fencing.ssh.private-key-files</name> <value>/root/.ssh/id_rsa</value> </property> <property> <name>dfs.journalnode.edits.dir</name> <value>/opt/journalnode</value> </property> <property> <name>dfs.ha.automatic-failover.enabled</name> <value>true</value> </property>
6. 配置core-site.xml
<property> <name>fs.defaultFS</name> <value>hdfs://lagoucluster</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/opt/lagou/servers/ha/hadoop-2.9.2/data/tmp</value> </property> <property> <name>ha.zookeeper.quorum</name> <value>centos7-1:2181,centos7-2:2181,centos7-3:2181</value> </property>
7. 分发刚才的到每个节点
rsync-script /opt/lagou/servers/ha/hadoop-2.9.2/
完成
测试集群
1. 在各集群执行命令,不要使用环境中的命令
/opt/lagou/servers/ha/hadoop-2.9.2/sbin/hadoop-daemon.sh start journalnode
每个节点jps会出现 JournalNode 表示成功
2. 在nn1节点(centos7-1)上,对其进行格式化,并启动
/opt/lagou/servers/ha/hadoop-2.9.2/bin/hdfs namenode -format /opt/lagou/servers/ha/hadoop-2.9.2/sbin/hadoop-daemon.sh start namenode
3. 在nn2节点(centos7-2)上,同步nn1的元数据信息
/opt/lagou/servers/ha/hadoop-2.9.2/bin/hdfs namenode -bootstrapStandby
4. 在nn1节点上初始化zkfc
/opt/lagou/servers/ha/hadoop-2.9.2/bin/hdfs zkfc -formatZK
5. nn1启动集群
/opt/lagou/servers/ha/hadoop-2.9.2/sbin/start-dfs.sh
http://centos7-1:50070/dfshealth.html#tab-overview
http://centos7-2:50070/dfshealth.html#tab-overview
可以查看成果
配置yarn-HA集群
1. yarn-site.xml
<property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <!--启用resourcemanager ha--> <property> <name>yarn.resourcemanager.ha.enabled</name> <value>true</value> </property> <!--声明两台resourcemanager的地址--> <property> <name>yarn.resourcemanager.cluster-id</name> <value>cluster-yarn</value> </property> <property> <name>yarn.resourcemanager.ha.rm-ids</name> <value>rm1,rm2</value> </property> <property> <name>yarn.resourcemanager.hostname.rm1</name> <value>centos7-2</value> </property> <property> <name>yarn.resourcemanager.hostname.rm2</name> <value>centos7-3</value> </property> <!--指定zookeeper集群的地址--> <property> <name>yarn.resourcemanager.zk-address</name> <value>centos7-1:2181,centos7-2:2181,centos7-3:2181</value> </property> <!--启用自动恢复--> <property> <name>yarn.resourcemanager.recovery.enabled</name> <value>true</value> </property> <!--指定resourcemanager的状态信息存储在zookeeper集群--> <property> <name>yarn.resourcemanager.store.class</name> <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value> </property>
2. 分发刚才的到每个节点
rsync-script yarn-site.xml
3. 启动yarn
/opt/lagou/servers/ha/hadoop-2.9.2/sbin/start-yarn.sh
4. centos7-2执行
/opt/lagou/servers/ha/hadoop-2.9.2/sbin/yarn-daemon.sh start resourcemanager
http://centos7-3:8088/cluster
http://centos7-2:8088/cluster
可以查看成果
硬菜HBase
1. 下载安装包
http://archive.apache.org/dist/hbase/1.3.1/
hbase-1.3.1-bin.tar.gz
2. 把安装包放置在
/opt/lagou/software/
3. 上传安装包
4. 解压安装包
tar -zxvf hbase-1.3.1-bin.tar.gz -C /opt/lagou/servers
5. 修改配置文件
需要把hadoop中的配置core-site.xml 、hdfs-site.xml拷贝到hbase安装目录下的conf文件夹中
ln -s /opt/lagou/servers/hadoop-2.9.2/etc/hadoop/core-site.xml /opt/lagou/servers/hbase-1.3.1/conf/core-site.xml ln -s /opt/lagou/servers/hadoop-2.9.2/etc/hadoop/hdfs-site.xml /opt/lagou/servers/hbase-1.3.1/conf/hdfs-site.xml
修改conf目录下配置文件
a. 修改hbase-env.sh
#添加java环境变量 (echo $JAVA_HOME) export JAVA_HOME=/opt/module/jdk1.8.0_231 #指定使用外部的zk集群 export HBASE_MANAGES_ZK=FALSE
b. 修改 hbase-site.xml
<configuration> <!-- 指定hbase在HDFS上存储的路径 --> <property> <name>hbase.rootdir</name> <value>hdfs://centos7-1:9000/hbase</value> </property> <!-- 指定hbase是分布式的 --> <property> <name>hbase.cluster.distributed</name> <value>true</value> </property> <!-- 指定zk的地址,多个用“,”分割 --> <property> <name>hbase.zookeeper.quorum</name> <value>centos7-1:2181,centos7-2:2181,centos7-3:2181</value> </property> </configuration>
c. 修改regionservers文件
#指定regionserver节点 centos7-1 centos7-2 centos7-3
d. hbase的conf目录下创建文件backup-masters (Standby Master) vi backup-masters centos7-2
6. 配置hbase的环境变量
vi /etc/profile #Hbase export HBASE_HOME=/opt/lagou/servers/hbase-1.3.1 export PATH=$PATH:$HBASE_HOME/bin
7. 分发配置到其他节点
rsync-script /opt/lagou/servers/hbase-1.3.1
rsync-script /etc/profile
所有节点 source
source /etc/profile
HBase部署完成
前提条件:先启动hadoop和zk集群
启动HBase:start-hbase.sh
停止HBase:stop-hbase.sh
可视化验证
启动好HBase集群之后,可以访问地址:centos7-1:16010