Hadoop-2.7.2集群部署步骤
Hadoop-2.7.2集群部署步骤
1、namenode主机信息
|序号|IP|Hostname|操作系统|CPU|内存|硬盘|环境及服务|备注|
|:--😐
|1、|10.10.40.10|master1|CentOS 7.6 64位|24c|96G|1T|java1.8+zookeeper+namenode|
|2、|10.10.40.11|master2|CentOS 7.6 64位|24c|96G|1T|java1.8+zookeeper+namenode|
|3、|10.10.40.12|master3|CentOS 7.6 64位|24c|96G|1T|java1.8+zookeeper+namenode|
2、datanode主机信息(裸盘)
|序号|IP|Hostname|操作系统|CPU|内存|硬盘|服务|备注|
|:--😐
|1、|10.10.40.13-42|datanode1-30|CentOS 7.6 64位|32c|128G|5.5T*12|java1.8+datanode|
3、集群部署java环境
-
下载安装包：jdk-8u151-linux-x64.tar.gz
-
上传到各个节点
-
解压到执行位置：tar -xf jdk-8u151-linux-x64.tar.gz -C /data1/xinsrv
-
添加环境变量：source /etc/profile.d/java.sh
export JAVA_HOME=/data1/xinsrv/jdk1.8.0_151/
PATH=$PATH:$HOME/bin:$JAVA_HOME/bin
export PATH
-
版本验证：java -version
4、部署zookeeper集群
部署主机：10.10.40.10-10.10.40.11（可多台建议zk集群主机个数为奇数）
-
下载安装包：zookeeper-3.4.8.tar.gz
-
上传到主机
-
解压到安装位置：tar -xf zookeeper-3.4.8.tar.gz -C /data1/xinsrv/ && cd /data1/xinsrv/ && mv zookeeper-3.4.8 zookeeper
-
创建相关依赖目录：mkdir -p /data1/data/zookeeper && mkdir -p /data1/logs/app/zookeeper
-
配置myid：
10.10.40.10 echo "1" >/data1/data/zookeeper/myid
10.10.40.11 echo "2" >/data1/data/zookeeper/myid
10.10.40.12 echo "3" >/data1/data/zookeeper/myid -
配置conf文件：cat /data1/xinsrv/zookeeper/conf/zoo.cfg |egrep -v "#|$"
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data1/data/zookeeper
dataLogDir=/data1/logs/app/zookeeper
clientPort=2181
maxClientCnxns=3000
server.1=10.10.40.10:3181:4181
server.2=10.10.40.11:3181:4181
server.3=10.10.40.12:3181:4181
4.1、服务启动：cd /etc/init.d/ && ./zookeeper_2181 start
cat zookeeper_2181
!/bin/bash
zookeeper Startup script for zookeeper
chkconfig: - 93 19
processname: zookeeper
. /etc/init.d/functions
bin=/data1/xinsrv/zookeeper/bin/zkServer.sh
conf=/data1/xinsrv/zookeeper/conf/zoo.cfg
logdir=/data1/logs/xinsrv/zookeeper
[ ! -d $logdir ] && mkdir -p $logdir && cd $logdir
service_count=ps -ef |grep /data1/xinsrv/zookeeper/conf/zoo.cfg|grep -v grep|wc -l
start(){
if [ $service_count -gt 1 ];then
echo "the service of zookeeper is running..."
exit
fi
$bin start $conf
if [ $? -ne 0 ];then
action "starting zookeeper..." /bin/false
exit
else
action "starting zookeeper..." /bin/true
fi
}
stop(){
if [ $service_count -lt 1 ];then
echo "the service of zookeeper is not running..."
exit
fi
$bin stop $conf
if [ $? -ne 0 ];then
action "stopping zookeeper..." /bin/false
exit
else
action "stopping zookeeper..." /bin/true
fi
}
restart(){
$bin restart $conf
}
status(){
$bin status $conf
}
main () {
case "$1" in
start)
start
;;
stop)
stop
;;
restart)
restart
;;
status)
status
;;
)
echo $"Usage: $0 {start|stop|status|restart}"
exit 1
esac
}
main $
5、部署hadoop集群
5.1、基础环境调整
-
所有节点
-
安装依赖：yum install python-devel libevent-devel libmcrypt libmcrypt-devel -y
-
集群添加：host解析
10.10.40.10 master1
10.10.40.11 master2
10.10.40.12 master3
10.10.40.13 datanode1
10.10.40.14 datanode2
10.10.40.15 datanode3 -
创建相关目录及授权：
- mkdir -p /data1/cache/hadoop /data1/cache/hadoop/dfs/data /data1/cache/hadoop/dfs/name /data1/cache/hadoop/tmp /var/run/hadoop /var/run/spark /data1/xinsrv/hadoop-2.7.2 /data1/xinsrv/tez-0.8.4
- chown hadoop.hadoop -R /data1/cache/hadoop /data1/cache/hadoop/dfs/data /data1/cache/hadoop/dfs/name /data1/cache/hadoop/tmp /var/run/hadoop /var/run/spark /data1/xinsrv/hadoop-2.7.2 /data1/xinsrv/tez-0.8.4
5.2、hadoop安装
-
所有节点
-
下载安装包：hadoop-2.7.2.tar.gz tez-0.8.4.tar.gz scala-2.11.8.tgz
-
上传到主机
-
安装tez：tar xf tez-0.8.4.tar.gz -C /data1/xinsrv
-
安装scala：tar xf scala-2.11.8.tgz -C /data1/xinsrv
-
安装hadoop：useradd hadoop && tar xf hadoop-2.7.2.tar.gz -C /data1/xinsrv && chown -R hadoop.hadoop /data1/xinsrv/hadoop-2.7.2
-
datanode裸盘格式化及挂载，挂载盘按disk1-12命名
-
datanode依赖目录创建：
df -h |grep disk &>/dev/null && for n in
df -h |grep disk|awk '{print $NF}'|grep "^/disk";do mkdir -p $n/cache/hadoop/dfs/data/ && chown -R hadoop.hadoop $n/cache/hadoop/dfs/data/ -
安装tez：tar xf tez-0.8.4.tar.gz -C /data1/xinsrv/
-
添加免秘钥登录：10.10.40.10 ，sudo su - hadoop 然后执行ssh-keygen -t rsa生成秘钥，其他节点添加相关公钥。
5.3、hadoop-env.sh（配置hadoop集群启动环境）
cat hadoop-env.sh |egrep -v "#|$"
export JAVA_HOME=/data1/xinsrv/jdk1.8.0_151
export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/data1/xinsrv/hadoop-2.7.2/etc/hadoop"}
for f in $HADOOP_HOME/contrib/capacity-scheduler/.jar; do
if [ "$HADOOP_CLASSPATH" ]; then
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$f
else
export HADOOP_CLASSPATH=$f
fi
done
export HADOOP_HEAPSIZE=8192
export HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true"
export HADOOP_NAMENODE_OPTS="-Xms30g -Xmx30g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled -XX:+PrintTenuringDistribution -Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_NAMENODE_OPTS"
export HADOOP_DATANODE_OPTS="-Dhadoop.security.logger=ERROR,RFAS $HADOOP_DATANODE_OPTS"
export HADOOP_SECONDARYNAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_SECONDARYNAMENODE_OPTS"
export HADOOP_NFS3_OPTS="$HADOOP_NFS3_OPTS"
export HADOOP_PORTMAP_OPTS="-Xmx1024m $HADOOP_PORTMAP_OPTS"
export HADOOP_CLIENT_OPTS="-Xmx2048m $HADOOP_CLIENT_OPTS"
export HADOOP_SECURE_DN_USER=${HADOOP_SECURE_DN_USER}
export HADOOP_SECURE_DN_LOG_DIR=${HADOOP_LOG_DIR}/${HADOOP_HDFS_USER}
export HADOOP_PID_DIR=/var/run/hadoop
export HADOOP_SECURE_DN_PID_DIR=${HADOOP_PID_DIR}
export HADOOP_IDENT_STRING=$USER
export TEZ_CONF_DIR=/data1/xinsrv/hadoop-2.7.2/etc/hadoop/tez-site.xml
export TEZ_JARS=/data1/xinsrv/tez-0.8.4
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:${TEZ_CONF_DIR}😒{TEZ_JARS}/😒{TEZ_JARS}/lib/*
5.4、hdfs-site.xml（namenode和datanode相关配置）
cat hdfs-site.xml
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
5.5、core-site.xml
cat core-site.xml
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
provided in the Aliyun OSS Documentation.
5.6、mapred-site.xml
cat mapred-site.xml
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
5.7、cat yarn-site.xml
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
5.8、slaves（添加datanode节点IP）
cat slaves
10.10.40.13
10.10.40.14
10.10.40.15
10.10.40.16
6、hadoop集群启动
6.1、datanode环境检查
主要用于检查相关组件和目录是否存在及权限是否正常。
cat hadoop_check.sh
#!/bin/bash
#######################################################
# $Name: hadoop_check.sh
# $Version: v1.0
# $Function: 检查hadoop集群新增节点环境
# $Author: liuxiaopeng
# $Create Date: 2018年07月09日18:57:20
# $Description:
#######################################################
# Shell Env
SHELL_NAME=basename $0
SHELL_DIR="/tmp"
SHELL_LOG="${SHELL_DIR}/${SHELL_NAME}.log"
LOCK_FILE="/tmp/${SHELL_NAME}.lock"
mains
Write Log
shell_log(){
LOG_INFO=$1
echo "$(date "+%Y-%m-%d") $(date "+%H-%M-%S") : ${SHELL_NAME} : ${LOG_INFO}" >> ${SHELL_LOG}
}
red_color(){
CONTENT=$1
echo -e "\033[31m $CONTENT \033[0m"
}
green_color(){
CONTENT=$1
echo -e "\033[32m $CONTENT \033[0m"
}
# Shell Usage
#shell_usage(){
# echo $"Usage: $0 {backup}"
#}
shell_lock(){
touch ${LOCK_FILE}
}
shell_unlock(){
rm -f ${LOCK_FILE}
}
TAG=$(/sbin/ip a l |grep "inet "|awk '{print $2}'|awk -F '/' '{print $1}'|grep -v '127.0.0.1')
check_hosts(){
linename1=cat /etc/hosts|grep $TAG
linenum1=cat /etc/hosts|grep $TAG|wc -l
if [ $linenum1 -ne 1 ];then
red_color "第一项：HOSTS配置【异常】: $linename1"
else
green_color "第一项：HOSTS配置【正常】"
fi
}
check_slaves(){
linenum2=cat /data1/xinsrv/hadoop-2.7.2/etc/hadoop/slaves|grep $TAG|wc -l
if [ $linenum2 -ne 1 ];then
red_color "第二项：slaves配置【异常】:没有配置该IP"
else
green_color "第二项：slaves配置【正常】"
fi
}
check_dir_cache(){
linename3=ls -ld /data1/cache/hadoop|awk '{print $3}'
[ -d /data1/cache/hadoop ] && [ $linename3 == "hadoop" ] && linenum3=1
[ -d /data1/cache/hadoop ] && [ $linename3 == "hadoop" ] || linenum3=2
if [ $linenum3 -ne 1 ];then
red_color "第三项：/data1/cache/hadoop目录及属性配置【异常】:目录或者属性异常"
else
green_color "第三项：/data1/cache/hadoop目录及属性配置【正常】"
fi
}
check_dir_run(){
linename4=ls -ld /var/run/hadoop|awk '{print $3}'
[ -d /var/run/hadoop ] && [ $linename4 == "hadoop" ] && linenum4=1
[ -d /var/run/hadoop ] && [ $linename4 == "hadoop" ] || linenum4=2
if [ $linenum4 -ne 1 ];then
red_color "第四项：/var/run/hadoop目录及属性配置【异常】:目录或者属性异常"
else
green_color "第四项：/var/run/hadoop目录及属性配置【正常】"
fi
}
check_scala(){
linename5=which scala
[ $linename5 == "/data1/xinsrv/scala-2.11.8/bin/scala" ] && linenum5=1
[ $linename5 == "/data1/xinsrv/scala-2.11.8/bin/scala" ] || linenum5=2
if [ $linenum5 -ne 1 ];then
red_color "第五项：scala未部署或者profile配置【异常】"
else
green_color "第五项：scala配置【正常】"
fi
}
check_jdk(){
linename6=$(ls -l $(ls -l which java|awk '{print $NF}')|awk '{print $NF}')
[ $linename6 == "/data1/xinsrv/jdk1.8.0_151/bin/java" ] && linenum6=1
[ $linename6 == "/data1/xinsrv/jdk1.8.0_151/bin/java" ] || linenum6=2
if [ $linenum6 -ne 1 ];then
red_color "第六项：JDK未部署或者profile配置【异常】"
else
green_color "第六项：JDK配置【正常】"
fi
}
check_numpy(){
linenum7=pip2.7 list|grep numpy|wc -l
if [ $linenum7 -ne 1 ];then
red_color "第七项：python2.7 numpy模块未安装【异常】"
else
green_color "第七项：python2.7 numpy模块【正常】"
fi
}
check_pandas(){
linenum8=pip2.7 list|grep pandas|wc -l
if [ $linenum8 -ne 1 ];then
red_color "第八项：python2.7 pandas模块未安装【异常】"
else
green_color "第八项：python2.7 pandas模块【正常】"
fi
}
check_hadoop(){
linename9=ls -l /data1/xinsrv/hadoop-2.7.2/bin/yarn|awk '{print $3}'
[ $linename9 == "hadoop" ] && linenum9=1
[ $linename9 == "hadoop" ] || linenum9=2
if [ $linenum9 -ne 1 ];then
red_color "第九项：hadoop-2.7.2未安装或者权限非hadoop【异常】"
else
green_color "第九项：hadoop-2.7.2安装及属性【正常】"
fi
}
check_tez(){
linename10=ls -l /data1/xinsrv/tez-0.8.4/lib/jetty-6.1.26.jar|awk '{print $3}'
[ $linename10 == "hadoop" ] && linenum10=1
[ $linename10 == "hadoop" ] || linenum10=2
if [ $linenum10 -ne 1 ];then
red_color "第十项：tez-0.8.4未安装或者权限非hadoop【异常】"
else
green_color "第十项：tez-0.8.4安装及属性【正常】"
fi
}
check_disk(){
linename11=ls -ld /disk1/cache/hadoop/dfs/data/|awk '{print $3}'
[ $linename11 == "hadoop" ] && linenum11=1
[ $linename11 == "hadoop" ] || linenum11=2
if [ $linenum11 -ne 1 ];then
red_color "第十一项：/disk1/cache/hadoop/dfs/data/未创建或者权限非hadoop【异常】"
else
green_color "第十一项：/disk1/cache/hadoop/dfs/data/创建及属性【正常】"
fi
}
check_hostname(){
linename12=cat /etc/hosts|grep $TAG|awk '{print $2}'
hostname $linename12
hostname=hostname
[ $linename12 == $hostname ] && linenum12=1
[ $linename12 == $hostname ] || linenum12=2
if [ $linenum12 -ne 1 ];then
red_color "第十二项：hostname 不为$linename12【异常】"
else
green_color "第十二项：hostname 为$linename12【正常】"
fi
}
check_hdfs_site(){
linename13=cat /data1/xinsrv/hadoop-2.7.2/etc/hadoop/hdfs-site.xml |grep /disk1/cache/hadoop/dfs/data|wc -l
[ $linename13 -eq 1 ] && linenum13=1
[ $linename13 -eq 1 ] || linenum13=2
if [ $linenum13 -ne 1 ];then
red_color "第十三项：/data1/xinsrv/hadoop-2.7.2/etc/hadoop/hdfs-site.xml 未配置disk1目录【异常】"
else
green_color "第十三项：/data1/xinsrv/hadoop-2.7.2/etc/hadoop/hdfs-site.xml 配置【正常】"
fi
}
check_disk_total_num(){
linename14=df -h|grep disk|wc -l
[ $linename14 -eq 16 -o $linename14 -eq 6 -o $linename14 -eq 12 ] && linenum14=1
[ $linename14 -eq 16 -o $linename14 -eq 6 -o $linename14 -eq 12 ] || linenum14=2
if [ $linenum14 -ne 1 ];then
red_color "第十四项： 主机/disk盘数为：$linename14 【异常】"
else
green_color "第十四项：主机/disk盘数为：$linename14【正常】"
fi
}
main(){
check_hosts
check_slaves
check_dir_cache
check_dir_run
check_scala
check_jdk
check_numpy
check_pandas
check_hadoop
check_tez
check_disk
check_hostname
check_hdfs_site
check_disk_total_num
green_color "如果以上均正常，请在${TAG}主机执行:sudo su - hadoop; /data1/xinsrv/hadoop-2.7.2/sbin/yarn-daemon.sh start nodemanager;/data1/xinsrv/hadoop-2.7.2/sbin/hadoop-daemon.sh start datanode; 检查命令：jps"
}
main $*
6.2、启动hadoop
6.2.1、集群初始化namenode
/data1/xinsrv/hadoop-2.7.2/sbin/start-dfs.sh
hadoop namenode -format（仅第一次初始化集群使用！！！！）
6.2.2、启动集群服务
/data1/xinsrv/hadoop-2.7.2/sbin/start-all.sh
6.2.3、 FATAL org.apache.hadoop.ha.ZKFailoverController: Unable to start failover controller. Parent znode does not exist.
hdfs zkfc -formatZK
6.2.4、集群出现:Incompatible namespaceID for journal Storage Directory
修改另外一个namenode VERSION文件
删除datanode节点数据
for n in /disk1/cache/hadoop/dfs/data/ /disk2/cache/hadoop/dfs/data/ /disk3/cache/hadoop/dfs/data/ /disk4/cache/hadoop/dfs/data/ /disk5/cache/hadoop/dfs/data/ /disk6/cache/hadoop/dfs/data/ /disk7/cache/hadoop/dfs/data/ /disk8/cache/hadoop/dfs/data/ /disk9/cache/hadoop/dfs/data/ /disk10/cache/hadoop/dfs/data/ /disk11/cache/hadoop/dfs/data/ /disk12/cache/hadoop/dfs/data/;do cd $n && rm -rf ./*;done
再次启动：/data1/xinsrv/hadoop-2.7.2/sbin/start-all.sh
7、启动后服务验证
本地电脑添加hadoop集群hosts解析：
1、hadoop集群状态：http://10.10.40.11:50070/dfshealth.html#tab-overview
2、yarn：http://master1:8088/cluster/scheduler
浙公网安备 33010602011771号