如何在Redhat7.6中安装CDH7
文档编写目的
CDP DC7.0.3是Cloudera与Hortonworks合并后，第一个融合CDH和HDP所有组件的on-premise版本，CDP Data Center主要由Cloudera Runtime构成，Cloudera Runtime由超过35个开源项目组成，当然CDP Data Center还包括其它功能如管理功能Cloudera Manager，Key Management，专业支持等，如下图所示：
Cloudera Runtime的主要组件版本如下图所示：
|
Component |
Apache Version |
|---|---|
|
Atlas |
2.0.0 |
|
Avro |
1.8 |
|
DAS |
1.4 |
|
Hadoop |
3.1 |
|
HBase |
2.2 |
|
Hive |
3.1 |
|
Hive Warehouse Connector |
1.0 (Technical Preview) |
|
Hue |
4.3 |
|
Impala |
3.3 |
|
Kafka |
2.3 |
|
Kudu |
1.11 |
|
Oozie |
5.1 |
|
ORC |
1.5 |
|
Ozone |
0.4.0 (Technical Preview) |
|
Parquet |
1.1 |
|
Phoenix |
5 |
|
Ranger |
2 |
|
Solr |
7.4 |
|
Spark |
2.4 |
|
Sqoop |
1.4 |
|
Tez |
0.9 |
|
Zookeeper |
3.5.5 |
本文档Fayson主要描述如何在Redhat7.6安装CDP DC7.0.3。CDP DC7与CDH5/6的安装步骤一致，主要包括以下四部分：
1.安全前置准备，包括安装操作系统、关闭防火墙、同步服务器时钟等；
2.外部数据库如MySQL安装
3.安装Cloudera Manager；
4.安装CDH集群；
- 测试环境
1.CM和CDH版本为7.0.3
2.Redhat7.6
3.OpenJDK1.8.0_232
4.MariaDB-5.5.64
5.root用户安装
前置准备
2.1 hostname及hosts配置
集群中各个节点之间能互相通信使用静态IP地址。IP地址和主机名通过/etc/hosts配置，主机名通过/etc/hostname进行配置。
以cm节点（172.31.2.24）为例：
1.hostname配置，建议你将hostname配置为有意义的字母或单词组合，比如cdp01，请不要使用localhost作为主机名，也不要使用下划线或者其它特殊字符作为主机名。/etc/hostname文件示例如下：
[root@ip-172-31-2-24 ~]# cat /etc/hostname
ip-172-31-2-24.ap-southeast-1.compute.internal
2.hosts配置需要将集群的所有主机都加入进来，包括客户端节点，管理节点和工作节点。/etc/hosts文件示例如下:
[root@ip-172-31-2-24 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
172.31.2.24 ip-172-31-2-24.ap-southeast-1.compute.internal
172.31.15.60 ip-172-31-15-60.ap-southeast-1.compute.internal
172.31.8.194 ip-172-31-8-194.ap-southeast-1.compute.internal
172.31.13.81 ip-172-31-13-81.ap-southeast-1.compute.internal
以上两步操作，在集群中其它节点做相同配置。确认CM的hosts文件已经同步到集群所有节点：
2.2 禁用SELinux
在所有节点执行setenforce 0 命令，此处使用批处理shell执行：
[root@ip-172-31-2-24 shell]# sh ssh_do_all.sh node.list "setenforce 0"
集群所有节点修改/etc/selinux/config文件如下：
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three two values:
# targeted - Targeted processes are protected,
# minimum - Modification of targeted policy. Only selected processes are protected.
# mls - Multi Level Security protection.
SELINUXTYPE=targeted
2.3 关闭防火墙
集群所有节点执行 systemctl stop命令，此处通过shell批量执行命令如下：
[root@ip-172-31-2-24 shell]# sh ssh_do_all.sh node.list "systemctl stop firewalld"
[root@ip-172-31-2-24 shell]# sh ssh_do_all.sh node.list "systemctl disable firewalld"
[root@ip-172-31-2-24 shell]# sh ssh_do_all.sh node.list "systemctl status firewalld"
2.4 集群时钟同步
在Redhat7.x的操作系统上，已经默认的安装了chrony，我们这里先卸载chrony，然后安装ntp。使用ntp来配置各台机器的时钟同步，将cm（172.31.2.24）服务作为本地ntp服务器，其它3台服务器与其保持同步。
1.所有机器卸载chrony
[root@ip-172-31-2-24 shell]# sh ssh_do_all.sh node.list "yum -y remove chrony"
2.所有机器安装ntp
[root@ip-172-31-2-24 shell]# sh ssh_do_all.sh node.list "yum -y install ntp"
3.cm机器配置时钟与自己同步
[root@ ip-172-31-2-24 shell]# vim /etc/ntp.conf
#server 0.rhel.pool.ntp.org iburst
#server 1.rhel.pool.ntp.org iburst
#server 2.rhel.pool.ntp.org iburst
#server 3.rhel.pool.ntp.org iburst
server 127.127.1.0 # local clock
fudge 127.127.1.0 stratum 10
4.集群其它节点，配置找cm机器去同步
[root@ip-172-31-11-232 ~]# vim /etc/ntp.conf
#server 0.rhel.pool.ntp.org iburst
#server 1.rhel.pool.ntp.org iburst
#server 2.rhel.pool.ntp.org iburst
#server 3.rhel.pool.ntp.org iburst
server 172.31.2.24
5.重启所有机器的ntp服务
[root@ ip-172-31-2-24 shell]# sh ssh_do_all.sh node.list "systemctl restart ntpd"
[root@ ip-172-31-2-24 shell]# sh ssh_do_all.sh node.list "systemctl enable ntpd"
[root@ ip-172-31-2-24 shell]# sh ssh_do_all.sh node.list "systemctl status ntpd"
6.验证始终同步，在所有节点执行ntpq -p命令，如下使用脚本批量执行
[root@ip-172-31-2-24 shell]# sh ssh_do_all.sh node.list "ntpq -p"
左边出现*号表示同步成功。
2.5 设置swap
1.在所有机器执行以下命令以临时设置swap为1，并即时生效
sh ssh_do_all.sh node.list "sysctl -a | grep vm.swappiness"
sh ssh_do_all.sh node.list "echo 1 > /proc/sys/vm/swappiness"
sh ssh_do_all.sh node.list "sysctl -a | grep vm.swappiness"
确保最后swap打印应该都为1
2.为所有机器永久设置swap为1，修改/etc/sysctl.conf中vm.swappiness为1，没有则新增。
[root@ip-172-31-2-24 ~]# vim /etc/sysctl.conf
vm.swappiness = 1
将/etc/sysctl.conf文件同步到集群所有机器
2.6 设置透明大页面
1.所有节点执行以下命令关闭透明大页面，并即时生效
sh ssh_do_all.sh node.list "echo never > /sys/kernel/mm/transparent_hugepage/defrag"
sh ssh_do_all.sh node.list "echo never > /sys/kernel/mm/transparent_hugepage/enabled"
sh ssh_do_all.sh node.list "cat /sys/kernel/mm/transparent_hugepage/enabled"
sh ssh_do_all.sh node.list "cat /sys/kernel/mm/transparent_hugepage/defrag"
2.修改所有节点的/etc/rc.d/rc.local文件的权限以实现开机执行
[root@ip-172-31-2-24 shell]# sh ssh_do_all.sh node.list "chmod +x /etc/rc.d/rc.local"
[root@ip-172-31-2-24 shell]# sh ssh_do_all.sh node.list "ls -l /etc/rc.d/rc.local"
3.在所有节点的/etc/rc.d/rc.local文件中新增如下内容，以实现开机自动关闭透明大页面。
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then echo never > /sys/kernel/mm/transparent_hugepage/enabled fi if test -f /sys/kernel/mm/transparent_hugepage/defrag; then echo never > /sys/kernel/mm/transparent_hugepage/defrag fi
将该文件同步到所有节点
2.7 配置操作系统repo
Fayson用的是AWS的环境，这步是可以省略的，放在这里供物理机部署的兄弟们参考。
1.挂载操作系统iso文件
[root@ip-172-31-2-24 ~]$ sudo mkdir /media/DVD1
[root@ip-172-31-2-24 ~]$ sudo mount -o loop
CentOS-7-x86_64-DVD-1611.iso /media/DVD1/
2.配置操作系统repo
[root@ip-172-31-2-24 ~]$ sudo vim /etc/yum.repos.d/local_os.repo
[local_iso]
name=CentOS-$releasever - Media
baseurl=file:///media/DVD1
gpgcheck=0
enabled=1
[root@ip-172-31-2-24 ~]$ sudo yum repolist
2.8 安装httpd服务
1.安装httpd服务
[root@ip-172-31-2-24 shell]# yum -y install httpd
2.启动httpd服务
[root@ip-172-31-2-24 shell]# systemctl start httpd
3.安装完httpd后，重新制作操作系统repo，换成http的方式方便其它服务器也可以访问
[root@ip-172-31-2-24 ~]$ mkdir /var/www/html/iso
[root@ip-172-31-2-24 ~]$ scp -r /media/DVD1/* /var/www/html/iso/
[root@ip-172-31-2-24 ~]$ vim /etc/yum.repos.d/os.repo
[osrepo]
name=os_repo
baseurl=http:// 172.31.2.24/iso/
enabled=true
gpgcheck=false
[root@ip-172-31-2-24 ~]$ yum repolist
4.修改/etc/httpd/conf/httpd.conf配置文件，在<IfModule mime_module>中修改以下内容
AddType application/x-gzip .gz .tgz .parcel
5.保存httpd.conf的修改，并重启httpd服务
[root@ip-172-31-2-24 java]# systemctl restart httpd
2.9 安装MariaDB
1.安装MariaDB
[root@ip-172-31-2-24 ~]# yum -y install mariadb
[root@ip-172-31-2-24 ~]# yum -y install mariadb-server
2.启动并配置MariaDB
[root@ip-172-31-2-24 ~]# systemctl start mariadb
[root@ip-172-31-2-24 ~]# systemctl enable mariadb
Created symlink from /etc/systemd/system/multi-user.target.wants/mariadb.service to /usr/lib/systemd/system/mariadb.service.
[root@ip-172-31-2-24 ~]# /usr/bin/mysql_secure_installation
NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
SERVERS IN PRODUCTION USE! PLEASE READ EACH STEP CAREFULLY!
In order to log into MariaDB to secure it, we'll need the current
password for the root user. If you've just installed MariaDB, and
you haven't set the root password yet, the password will be blank,
so you should just press enter here.
Enter current password for root (enter for none):
OK, successfully used password, moving on...
Setting the root password ensures that nobody can log into the MariaDB
root user without the proper authorisation.
Set root password? [Y/n] Y
New password:
Re-enter new password:
Password updated successfully!
Reloading privilege tables..
... Success!
By default, a MariaDB installation has an anonymous user, allowing anyone
to log into MariaDB without having to have a user account created for
them. This is intended only for testing, and to make the installation
go a bit smoother. You should remove them before moving into a
production environment.
Remove anonymous users? [Y/n] Y
... Success!
Normally, root should only be allowed to connect from 'localhost'. This
ensures that someone cannot guess at the root password from the network.
Disallow root login remotely? [Y/n] n
... skipping.
By default, MariaDB comes with a database named 'test' that anyone can
access. This is also intended only for testing, and should be removed
before moving into a production environment.
Remove test database and access to it? [Y/n] Y
- Dropping test database...
... Success!
- Removing privileges on test database...
... Success!
Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.
Reload privilege tables now? [Y/n] Y
... Success!
Cleaning up...
All done! If you've completed all of the above steps, your MariaDB
installation should now be secure.
Thanks for using MariaDB!
3.建立CM，Hive等需要的表
[root@ip-172-31-2-24 shell]# mysql -u root -p
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 9
Server version: 5.5.64-MariaDB MariaDB Server
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> create database metastore default character set utf8;
Query OK, 1 row affected (0.00 sec)
MariaDB [(none)]> CREATE USER 'hive'@'%' IDENTIFIED BY 'password';
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]> GRANT ALL PRIVILEGES ON metastore. * TO 'hive'@'%';
Query OK, 0 rows affected (0.01 sec)
MariaDB [(none)]> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]> create database cm default character set utf8;
Query OK, 1 row affected (0.00 sec)
MariaDB [(none)]> CREATE USER 'cm'@'%' IDENTIFIED BY 'password';
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]> GRANT ALL PRIVILEGES ON cm. * TO 'cm'@'%';
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]> create database rm default character set utf8;
Query OK, 1 row affected (0.00 sec)
MariaDB [(none)]> CREATE USER 'rm'@'%' IDENTIFIED BY 'password';
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]> GRANT ALL PRIVILEGES ON rm. * TO 'rm'@'%';
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]> create database hue default character set utf8;
Query OK, 1 row affected (0.00 sec)
MariaDB [(none)]> CREATE USER 'hue'@'%' IDENTIFIED BY 'password';
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]> GRANT ALL PRIVILEGES ON hue. * TO 'hue'@'%';
Query OK,