CDH6.3.1搭建(至少3台机器 配置server节点16G起步 agent节点4G起步 server节点内存过小会导致机器在启动该进程的时候机器会卡死)

> 网盘地址:
> 链接：https://pan.baidu.com/s/1iM6jooaNu3XqEDdRckXS6A
> 提取码：QWER

1.所有节点配置host文件:
echo “192.168.15.151 cm1”>> /etc/hosts
echo “192.168.15.152 cm2”>> /etc/hosts
echo “192.168.15.153 cm3”>> /etc/hosts

2.关闭防火墙清空清空防护墙规则(根据企业网络安全需要可以自定义防火墙规则开启指定的服务端口)
systemctl stop firewalld
systemctl disable firewalld
iptables -F

3.关闭所有节点selinux
vi /etc/selinux/config
将SELINUX=enforcing改为SELINUX=disabled

4.设置集群NTP时间同步(这个必须要配置不然启动agent进程会报错)
所有节点 yum install -y ntp
集群节点统一注释掉:server 0 - 3.centos.pool.ntp.org.iburst
server节点添加:
server 127.127.1.0
fudge 127.127.1.0 stratum 5
agent 节点添加：
server 192.168.15.151 (同步server节点的ip)
集群所有节点重启ntp服务
systemctl restart ntpd
systemctl status ntpd
所有节点检查ntp同步情况 ntpq -p
(这里暂时先配置server节点同步自己的本地时间,agent节点指向server节点同步时间)

5.集群所有节点安装JDK (安装在/usr/java 这个目录)
mkdir -p /usr/java
export JAVA_HOME=/usr/java/jdk1.8.0_201
export PATH=.:$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

离线安装mysql-5.7.25-linux-glibc2.12-x86_64
1.查看系统中是否存的mariadb，如果存在，将其删除，即可以如下操作：
rpm -qa | grep mariadb
rpm -e --nodeps mariadb-libs-5.5.56-2.el7.x86_64

2.将mysql-5.7.25-linux-glibc2.12-x86_64.tar安装包上传到opt目录下，并依次执行如下的解压命令：
tar -xvf mysql-5.7.25-linux-glibc2.12-x86_64.tar

3.执行上面的命令后，可以得到mysql-5.7.25-linux-glibc2.12-x86_64.tar.gz文件，然后再解压该文件
tar -zxvf mysql-5.7.25-linux-glibc2.12-x86_64.tar.gz

4.那么此时在opt目录就会mysql-5.7.25-linux-glibc2.12-x86_64目录，将该目录修改成mysql-5.7.25，即：
mv mysql-5.7.25-linux-glibc2.12-x86_64 mysql-5.7.25

5.创建mysql用户和组 :
groupadd mysql
useradd -g mysql mysql
passwd mysql

6.创建数据库的数据目录、日志目录和临时目录:
mkdir /opt/mysql-5.7.25/data
mkdir /opt/mysql-5.7.25/log
mkdir /opt/mysql-5.7.25/tmp

7.将mysql-5.7.25目录及子目录的所属用户及组都设置为mysql
chown -R mysql:mysql /opt/mysql-5.7.25

8.在etc目录下创建my.cnf配置文件
touch /etc/my.cnf

[client]
port = 3306
socket = /opt/mysql-5.7.25/tmp/mysql.sock
[mysqld]
port = 3306
basedir = /opt/mysql-5.7.25
datadir = /opt/mysql-5.7.25/dat
tmpdir = /opt/mysql-5.7.25/tmp
socket = /opt/mysql-5.7.25/tmp/mysql.sock
pid-file = /opt/mysql-5.7.25/log/mysql.pid
log_error = /opt/mysql-5.7.25/log/error.log
slow_query_log_file = /opt/mysql-5.7.25/log/slow.log

9.初始化数据库:
/opt/mysql-5.7.25/bin/mysqld --initialize --user=mysql

10.设置开机自启动mysql服务
cp /opt/mysql-5.7.25/support-files/mysql.server /etc/rc.d/init.d/mysqld
chmod +x /etc/rc.d/init.d/mysqld
chkconfig --add mysqld
切换mysql用户:
su mysql
启动mysql服务:
service mysqld start

11.获取log目录下error.log文件生成的随机密码
cat error.log |grep root
2019-04-15T20:48:49.436053Z 1 [Note] A temporary password is generated
for root@localhost: 0-7YeFpwhf>a 那么初始密码就是：0-7YeFpwhf>a
切换root用户登陆Mysql重新设置密码:
use mysql;
set password for root@localhost=password(“000”);
修改 user 表，把 Host 表内容修改为%
update user set host=’%’ where host=‘localhost’;

12.创建CDH元数据库,元数据库用户,aom服务检测数据,aom服务检测数据库用户,以及用到该组件的库,用户
create database cmf DEFAULT CHARACTER SET utf8;
grant all on cmf. TO 'cmf'@'%' IDENTIFIED BY 'cmf';*

create database amon DEFAULT CHARACTER SET utf8;
grant all on amon. TO 'amon'@'%' IDENTIFIED BY 'amon';*

create database hive DEFAULT CHARACTER SET utf8;
grant all on hive. TO 'hive'@'%' IDENTIFIED BY 'hive';*

create database oozie DEFAULT CHARACTER SET utf8;
grant all on oozie. TO 'oozie'@'%' IDENTIFIED BY 'oozie';*

create database hue DEFAULT CHARACTER SET utf8;
grant all on hue. TO 'hue'@'%' IDENTIFIED BY 'hue';*

flush privileges;

CDH部署:
1.server节点配置(server节点同样也是agent节点用来启动各个组件的角色进程)

mkdir -p /usr/share/java/
不要版本号
cp mysql-connector-java-5.1.47.jar /usr/share/java/mysql-connector-java.jar

tar -xzvf cm6.3.1-redhat7.tar.gz
cd /cm6.3.1/RPMS/x86_64/

rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm --nodeps --force
rpm -ivh cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm --nodeps --force
rpm -ivh cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm --nodeps --force

vi /etc/cloudera-scm-server/db.properties
com.cloudera.cmf.db.type=mysql
com.cloudera.cmf.db.host=cm1:3306
com.cloudera.cmf.db.name=cmf
com.cloudera.cmf.db.user=cmf
com.cloudera.cmf.db.password=cmf
com.cloudera.cmf.db.setupType=EXTERNAL

安装httpd:
yum install -y httpd
mkdir /var/www/html/cdh6_parcel

cp CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel /var/www/html/cdh6_parcel
cp CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel.sha1 /var/www/html/cdh6_parcel/CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel.sha
cp manifest.json /var/www/html/cdh6_parcel

启动httpd:
service httpd start

查看httpd是否配置成功

http://192.168.15.151/cdh6_parcel/

将agent节点需要的rpm包传送到该节点
scp cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm cm2:/opt/
scp cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm cm3:/opt/

scp cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm cm2:/opt/
scp cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm cm3:/opt/

2.所有agent节点安装:(server节点不需要在进行安装)
rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm --nodeps --force
rpm -ivh cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm --nodeps --force

所有agent节点配置文件修改指向server(agent与server之间会保持有心跳连接所以IP不能是自己)
vi /etc/cloudera-scm-agent/config.ini
修改
server_host=cm1

3.启动CDH
server节点启动: service cloudera-scm-server start
启动成功后等待2分钟左右查看netstat -ntlp |grep 7180 出现此端口代表server启动成功
查看server启动日志
tail -F /var/log/cloudera-scm-server/cloudera-scm-server.log

所有agent节点启动:service cloudera-scm-agent start
查看agent启动日志
tail -F /var/log/cloudera-scm-agent/cloudera-scm-agent.log

192.168.15.151:7180

页面:Parcel存储库设置,远程Parcel存储库URL:http://192.168.15.151/cdh6_parcel