基于yum的方式安装Cloudera Manager Server（使用Mysql 8.0版本）
基于yum的方式安装Cloudera Manager Server（使用Mysql 8.0版本）
作者：尹正杰
版权声明：原创作品，谢绝转载！否则将追究法律责任。
一.安装和配置元数据库
1>.部署MySQL数据库
我们应启动的5台服务器中选出来2台服务做MySQL主从同步。我们把mysql服务安装在node101.yinzhengjie.org.cn上。至于MySQL的部署方式并不是本片博客的重点，不过可参考我之前的笔记：使用二进制安装包的方式单机部署MySQL8.0.13。
由于这个元数据库还是比较重要的，在生产环境使用中，建议使用MySQL的复制机制部署一个主从复制。或者使用Keepalived+MySQL部署一个主主双活的数据库，以提升其可用性。
2>.配置元数据库
创建所需的表和账号。首先我们修改root的密码。接着创建scm,rman,hive,oozie,hue,sentry等6套库和用户，分别用户Cloudera Manager，Reports Manager，hive，oozie，hue，sentry存储数据。
[root@node101 ~]# mysql -uroot -pyinzhengjie mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 13 Server version: 8.0.15 MySQL Community Server - GPL Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> mysql> CREATE DATABASE scm DEFAULT CHARACTER SET UTF8; Query OK, 1 row affected, 1 warning (0.01 sec) mysql> mysql> CREATE USER scm@'%' IDENTIFIED WITH mysql_native_password BY 'yinzhengjie'; Query OK, 0 rows affected (0.01 sec) mysql> mysql> GRANT ALL PRIVILEGES ON scm.* TO scm@'%'; Query OK, 0 rows affected (0.01 sec) mysql>
mysql> CREATE DATABASE rman DEFAULT CHARACTER SET UTF8; Query OK, 1 row affected, 1 warning (0.01 sec) mysql> CREATE USER rman@'%' IDENTIFIED WITH mysql_native_password BY 'yinzhengjie'; Query OK, 0 rows affected (0.00 sec) mysql> GRANT ALL PRIVILEGES ON rman.* TO rman@'%'; Query OK, 0 rows affected (0.00 sec) mysql>
mysql> CREATE DATABASE hive DEFAULT CHARACTER SET UTF8; Query OK, 1 row affected, 1 warning (0.01 sec) mysql> CREATE USER hive@'%' IDENTIFIED WITH mysql_native_password BY 'yinzhengjie'; Query OK, 0 rows affected (0.00 sec) mysql> GRANT ALL PRIVILEGES ON hive.* TO hive@'%'; Query OK, 0 rows affected (0.00 sec) mysql>
mysql> CREATE DATABASE oozie DEFAULT CHARACTER SET UTF8; Query OK, 1 row affected, 1 warning (0.00 sec) mysql> CREATE USER oozie@'%' IDENTIFIED WITH mysql_native_password BY 'yinzhengjie'; Query OK, 0 rows affected (0.01 sec) mysql> GRANT ALL PRIVILEGES ON oozie.* TO oozie@'%'; Query OK, 0 rows affected (0.00 sec) mysql>
mysql> CREATE DATABASE hue DEFAULT CHARACTER SET UTF8; Query OK, 1 row affected, 1 warning (0.00 sec) mysql> CREATE USER hue@'%' IDENTIFIED WITH mysql_native_password BY 'yinzhengjie'; Query OK, 0 rows affected (0.01 sec) mysql> GRANT ALL PRIVILEGES ON hue.* TO hue@'%'; Query OK, 0 rows affected (0.01 sec) mysql>
mysql> CREATE DATABASE sentry DEFAULT CHARACTER SET UTF8; Query OK, 1 row affected, 1 warning (0.00 sec) mysql> CREATE USER sentry@'%' IDENTIFIED WITH mysql_native_password BY 'yinzhengjie'; Query OK, 0 rows affected (0.01 sec) mysql> GRANT ALL PRIVILEGES ON sentry.* TO sentry@'%'; Query OK, 0 rows affected (0.00 sec) mysql>
mysql> FLUSH PRIVILEGES; Query OK, 0 rows affected (0.01 sec) mysql> mysql> SELECT user,host,plugin FROM mysql.user; +------------------+-----------+-----------------------+ | user | host | plugin | +------------------+-----------+-----------------------+ | hive | % | mysql_native_password | | hue | % | mysql_native_password | | oozie | % | mysql_native_password | | rman | % | mysql_native_password | | scm | % | mysql_native_password | | sentry | % | mysql_native_password | | mysql.infoschema | localhost | caching_sha2_password | | mysql.session | localhost | caching_sha2_password | | mysql.sys | localhost | caching_sha2_password | | root | localhost | caching_sha2_password | +------------------+-----------+-----------------------+ 10 rows in set (0.00 sec) mysql>
[root@node101 ~]# mysql -uscm -pyinzhengjie #测试是否可以正常登陆 mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 24 Server version: 8.0.15 MySQL Community Server - GPL Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> quit Bye [root@node101 ~]# [root@node101 ~]# mysql -h node101.yinzhengjie.org.cn -uscm -pyinzhengjie mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 25 Server version: 8.0.15 MySQL Community Server - GPL Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> quit Bye [root@node101 ~]#
二.Cloudera Manager 部署方式
1>.yum方式安装方式
yum安装方式一：（下载Cloudera Manager的源，修改配置文件并执行安装，这种方式是通过网络的方法安装，效率较低！）
下载地址：https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.16.1/。
[root@node101 ~]# ll /etc/yum.repos.d/ | grep cloudera-manager.repo [root@node101 ~]# [root@node101 ~]# wget -P /etc/yum.repos.d/ https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/cloudera-manager.repo --2019-02-21 11:47:07-- https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/cloudera-manager.repo Resolving archive.cloudera.com (archive.cloudera.com)... 151.101.108.167 Connecting to archive.cloudera.com (archive.cloudera.com)|151.101.108.167|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 290 [binary/octet-stream] Saving to: ‘/etc/yum.repos.d/cloudera-manager.repo’ 100%[========================================================================================================================================================================>] 290 --.-K/s in 0s 2019-02-21 11:47:07 (48.4 MB/s) - ‘/etc/yum.repos.d/cloudera-manager.repo’ saved [290/290] [root@node101 ~]# [root@node101 ~]# ll /etc/yum.repos.d/ | grep cloudera-manager.repo -rw-r--r-- 1 root root 290 Dec 14 00:32 cloudera-manager.repo [root@node101 ~]# [root@node101 ~]#
[root@node101 ~]# grep baseurl /etc/yum.repos.d/cloudera-manager.repo baseurl=https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5/ [root@node101 ~]# [root@node101 ~]# sed -i s'#baseurl=https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5/#baseurl=https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.16.1/#' /etc/yum.repos.d/cloudera-manager.repo #修改为你想要安装的CM版本 [root@node101 ~]# [root@node101 ~]# grep baseurl /etc/yum.repos.d/cloudera-manager.repo #我这里改为5.16.1版本 baseurl=https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.16.1/ [root@node101 ~]# [root@node101 ~]# yum -y install cloudera-manager-daemons cloudera-manager-server #
按照上面的方式按照是可以按照成功，但是网速想我这样慢的小伙伴就不推荐使用这种方式，我们可以提前把rpm包下载下来，然后手动安装即可。 
yum安装方式二：（如果你发现下载速度很慢，可以在自己电脑上下载安装包在上传到虚拟机内，本片博客采用的就是这种方式！大家也看到了，上面的下载速度太慢了！）
下载地址：https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.16.1/RPMS/x86_64/。
[root@node101 ~]# ll -lht total 754M -rwxr-xr-x 1 root root 11K Feb 21 12:21 cloudera-manager-server-db-2-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm -rwxr-xr-x 1 root root 8.5K Feb 21 12:21 cloudera-manager-server-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm -rwxr-xr-x 1 root root 754M Feb 21 12:21 cloudera-manager-daemons-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm [root@node101 ~]#
[root@node101 ~]# ll -lht total 754M -rwxr-xr-x 1 root root 11K Feb 21 12:21 cloudera-manager-server-db-2-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm -rwxr-xr-x 1 root root 8.5K Feb 21 12:21 cloudera-manager-server-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm -rwxr-xr-x 1 root root 754M Feb 21 12:21 cloudera-manager-daemons-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm [root@node101 ~]# [root@node101 ~]# yum -y install cloudera-manager-*.rpm Loaded plugins: fastestmirror Examining cloudera-manager-daemons-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm: cloudera-manager-daemons-5.16.1-1.cm5161.p0.1.el7.x86_64 Marking cloudera-manager-daemons-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm to be installed Examining cloudera-manager-server-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm: cloudera-manager-server-5.16.1-1.cm5161.p0.1.el7.x86_64 Marking cloudera-manager-server-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm to be installed Examining cloudera-manager-server-db-2-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm: cloudera-manager-server-db-2-5.16.1-1.cm5161.p0.1.el7.x86_64 Marking cloudera-manager-server-db-2-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm to be installed Resolving Dependencies --> Running transaction check ---> Package cloudera-manager-daemons.x86_64 0:5.16.1-1.cm5161.p0.1.el7 will be installed ---> Package cloudera-manager-server.x86_64 0:5.16.1-1.cm5161.p0.1.el7 will be installed ---> Package cloudera-manager-server-db-2.x86_64 0:5.16.1-1.cm5161.p0.1.el7 will be installed --> Processing Dependency: postgresql-server >= 8.4 for package: cloudera-manager-server-db-2-5.16.1-1.cm5161.p0.1.el7.x86_64 Loading mirror speeds from cached hostfile * base: mirrors.shu.edu.cn * extras: mirror.jdcloud.com * updates: centos.cs.nctu.edu.tw --> Running transaction check ---> Package postgresql-server.x86_64 0:9.2.24-1.el7_5 will be installed --> Processing Dependency: postgresql-libs(x86-64) = 9.2.24-1.el7_5 for package: postgresql-server-9.2.24-1.el7_5.x86_64 --> Processing Dependency: postgresql(x86-64) = 9.2.24-1.el7_5 for package: postgresql-server-9.2.24-1.el7_5.x86_64 --> Processing Dependency: libpq.so.5()(64bit) for package: postgresql-server-9.2.24-1.el7_5.x86_64 --> Running transaction check ---> Package postgresql.x86_64 0:9.2.24-1.el7_5 will be installed ---> Package postgresql-libs.x86_64 0:9.2.24-1.el7_5 will be installed --> Finished Dependency Resolution Dependencies Resolved ====================================================================================================================================================================================================================================================== Package Arch Version Repository Size ====================================================================================================================================================================================================================================================== Installing: cloudera-manager-daemons x86_64 5.16.1-1.cm5161.p0.1.el7 /cloudera-manager-daemons-5.16.1-1.cm5161.p0.1.el7.x86_64 935 M cloudera-manager-server x86_64 5.16.1-1.cm5161.p0.1.el7 /cloudera-manager-server-5.16.1-1.cm5161.p0.1.el7.x86_64 13 k cloudera-manager-server-db-2 x86_64 5.16.1-1.cm5161.p0.1.el7 /cloudera-manager-server-db-2-5.16.1-1.cm5161.p0.1.el7.x86_64 20 k Installing for dependencies: postgresql x86_64 9.2.24-1.el7_5 base 3.0 M postgresql-libs x86_64 9.2.24-1.el7_5 base 234 k postgresql-server x86_64 9.2.24-1.el7_5 base 3.8 M Transaction Summary ====================================================================================================================================================================================================================================================== Install 3 Packages (+3 Dependent packages) Total size: 942 M Total download size: 7.1 M Installed size: 968 M Downloading packages: postgresql-server-9.2.24-1.el7 FAILED http://mirror.lzu.edu.cn/centos/7.6.1810/os/x86_64/Packages/postgresql-server-9.2.24-1.el7_5.x86_64.rpm: [Errno 14] curl#56 - "Recv failure: Connection reset by peer" ] 0.0 B/s | 0 B --:--:-- ETA Trying other mirror. (1/3): postgresql-9.2.24-1.el7_5.x86_64.rpm | 3.0 MB 00:00:03 (2/3): postgresql-server-9.2.24-1.el7_5.x86_64.rpm | 3.8 MB 00:00:05 (3/3): postgresql-libs-9.2.24-1.el7_5.x86_64.rpm | 234 kB 00:00:07 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Total 962 kB/s | 7.1 MB 00:00:07 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : postgresql-libs-9.2.24-1.el7_5.x86_64 1/6 Installing : postgresql-9.2.24-1.el7_5.x86_64 2/6 Installing : postgresql-server-9.2.24-1.el7_5.x86_64 3/6 Installing : cloudera-manager-daemons-5.16.1-1.cm5161.p0.1.el7.x86_64 4/6 Installing : cloudera-manager-server-5.16.1-1.cm5161.p0.1.el7.x86_64 5/6 Installing : cloudera-manager-server-db-2-5.16.1-1.cm5161.p0.1.el7.x86_64 6/6 Verifying : cloudera-manager-server-5.16.1-1.cm5161.p0.1.el7.x86_64 1/6 Verifying : cloudera-manager-daemons-5.16.1-1.cm5161.p0.1.el7.x86_64 2/6 Verifying : cloudera-manager-server-db-2-5.16.1-1.cm5161.p0.1.el7.x86_64 3/6 Verifying : postgresql-server-9.2.24-1.el7_5.x86_64 4/6 Verifying : postgresql-libs-9.2.24-1.el7_5.x86_64 5/6 Verifying : postgresql-9.2.24-1.el7_5.x86_64 6/6 Installed: cloudera-manager-daemons.x86_64 0:5.16.1-1.cm5161.p0.1.el7 cloudera-manager-server.x86_64 0:5.16.1-1.cm5161.p0.1.el7 cloudera-manager-server-db-2.x86_64 0:5.16.1-1.cm5161.p0.1.el7 Dependency Installed: postgresql.x86_64 0:9.2.24-1.el7_5 postgresql-libs.x86_64 0:9.2.24-1.el7_5 postgresql-server.x86_64 0:9.2.24-1.el7_5 Complete! [root@node101 ~]#
2>..离线方式安装
详情请参考：离线方式部署Cloudera Manager5.15.1。
三.配置并启动Cloudera Manager
1>.配置Cloudera Manager
[root@node101 ~]# grep CMF_JAVA_OPTS /etc/default/cloudera-scm-server #Cloudera Manager默认分配的JVM堆内存是2G，某些情况下会出现OOM的情况，我们可以适当将其调大。 export CMF_JAVA_OPTS="-Xmx2G -XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp" [root@node101 ~]# [root@node101 ~]# sed -i s'#Xmx2G#Xmx4G#' /etc/default/cloudera-scm-server #我们将Cloudera Manager的堆内存调大到4G。 [root@node101 ~]# [root@node101 ~]# grep CMF_JAVA_OPTS /etc/default/cloudera-scm-server export CMF_JAVA_OPTS="-Xmx4G -XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp" [root@node101 ~]# [root@node101 ~]#
2>.初始化数据库配置
Cloudera已经提供了一个sh脚本用户配置Cloudera Manager连接元数据库，官方文档参考：https://www.cloudera.com/documentation/enterprise/5-14-x/topics/cm_ig_installing_configuring_dbs.html#concept_i2r_m3m_hn。
我们的安装规划下，元数据库和Cloudera Manager运行于同一个节点，且已经预先创建了scm库和用户，因此，命令很简单：
[root@node101 ~]# /usr/share/cmf/schema/scm_prepare_database.sh mysql scm scm yinzhengjie JAVA_HOME=/usr/java/jdk1.8 Verifying that we can write to /etc/cloudera-scm-server Creating SCM configuration file in /etc/cloudera-scm-server Executing: /usr/java/jdk1.8/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/usr/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db. Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary. [ main] DbCommandExecutor INFO Successfully connected to database. All done, your SCM database is configured correctly! #注意，如果你看到了这一行提示，说明配置完成，否则需要根据提示解决相应的错误！ [root@node101 ~]#
3>.启动Cloudera Manager
[root@node101 ~]# systemctl status cloudera-scm-server ● cloudera-scm-server.service - LSB: Cloudera SCM Server Loaded: loaded (/etc/rc.d/init.d/cloudera-scm-server; bad; vendor preset: disabled) Active: inactive (dead) Docs: man:systemd-sysv-generator(8) Feb 21 12:29:08 node101.yinzhengjie.org.cn systemd[1]: [/run/systemd/generator.late/cloudera-scm-server.service:13] Failed to add dependency on +cloudera-scm-server-db.service, ignoring: Invalid argument Feb 21 12:29:08 node101.yinzhengjie.org.cn systemd[1]: [/run/systemd/generator.late/cloudera-scm-server.service:13] Failed to add dependency on +cloudera-scm-server-db.service, ignoring: Invalid argument [root@node101 ~]# [root@node101 ~]# systemctl start cloudera-scm-server #启动Cloudera Manager [root@node101 ~]# [root@node101 ~]# systemctl status cloudera-scm-server ● cloudera-scm-server.service - LSB: Cloudera SCM Server Loaded: loaded (/etc/rc.d/init.d/cloudera-scm-server; bad; vendor preset: disabled) Active: active (exited) since Thu 2019-02-21 12:53:21 CST; 2min 38s ago Docs: man:systemd-sysv-generator(8) Process: 7253 ExecStart=/etc/rc.d/init.d/cloudera-scm-server start (code=exited, status=0/SUCCESS) Feb 21 12:53:16 node101.yinzhengjie.org.cn systemd[1]: Starting LSB: Cloudera SCM Server... Feb 21 12:53:16 node101.yinzhengjie.org.cn su[7277]: PAM unable to dlopen(/usr/lib64/security/pam_fprintd.so): /usr/lib64/security/pam_fprintd.so: cannot open shared object file: No such file or directory Feb 21 12:53:16 node101.yinzhengjie.org.cn su[7277]: PAM adding faulty module: /usr/lib64/security/pam_fprintd.so Feb 21 12:53:16 node101.yinzhengjie.org.cn su[7277]: (to cloudera-scm) root on none Feb 21 12:53:21 node101.yinzhengjie.org.cn cloudera-scm-server[7253]: Starting cloudera-scm-server: [ OK ] Feb 21 12:53:21 node101.yinzhengjie.org.cn systemd[1]: Started LSB: Cloudera SCM Server. [root@node101 ~]#
4>.查看Cloudera Manager的日志
[root@node101 ~]# tail -10f /var/log/cloudera-scm-server/cloudera-scm-server.log #查看Cloudera Manager的服务的日志 2019-02-21 12:54:38,541 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Initializing SearchTemplateManager:2019-02-21T04:54:38.540Z 2019-02-21 12:54:38,673 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Generating entities:2019-02-21T04:54:38.673Z 2019-02-21 12:54:38,721 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Num entities:208 2019-02-21 12:54:38,721 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Generating documents:2019-02-21T04:54:38.721Z 2019-02-21 12:54:38,795 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Num docs:221 2019-02-21 12:54:38,823 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Constructing repo:2019-02-21T04:54:38.823Z 2019-02-21 12:54:39,541 INFO WebServerImpl:org.mortbay.log: jetty-6.1.26.cloudera.4 2019-02-21 12:54:39,631 INFO WebServerImpl:org.mortbay.log: Started SelectChannelConnector@0.0.0.0:7180 #注意，如果出现这一行提示，说明你的服务启动成功啦！默认端口为7180，下面有WebServerImpl：Started Jetty server提示！ 2019-02-21 12:54:39,631 INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server. 2019-02-21 12:54:39,827 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Finished constructing repo:2019-02-21T04:54:39.827Z
5>.访问Cloudera Manager 的WebUI界面
打开浏览器，推荐使用Chrome并将英语设置为首选语言，访问：http://node101.yinzhengjie.org.cn:7180/cmf/login
当你的才华还撑不起你的野心的时候，你就应该静下心来学习。当你的能力还驾驭不了你的目标的时候，你就应该沉下心来历练。问问自己，想要怎样的人生。
欢迎加入基础架构自动化运维：598432640，大数据SRE进阶之路：959042252，DevOps进阶之路：526991186
