cdh-完善版1-centos版
https://docs.cloudera.com/documentation/enterprise/5-13-x/topics/cdh_ig_jdk_installation.html
https://docs.cloudera.com/documentation/enterprise/5-13-x/topics/installation.html
支持的jdk
Only 64 bit JDKs are supported. Oracle JDK 7 is supported across all versions of Cloudera Manager 5 and CDH 5. Oracle JDK 8 is supported in Cloudera Enterprise 5.3.x and higher.
| Oracle JDK Version | Notes |
|---|---|
| 1.8u181 | Recommended / Latest version tested |
| 1.8u162 | Recommended / Latest version tested |
| 1.8u144 | Recommended |
| 1.8u131 | Recommended |
| 1.8u121 | Recommended |
| 1.8u111 | Recommended |
| 1.8u102 | Recommended |
| 1.8u91 | Recommended |
| 1.8u74 | Recommended |
| 1.8u31 | Minimum required |
cdh5.13.x 使用 https://repo.huaweicloud.com/java/jdk/8u181-b13/jdk-8u181-linux-x64.rpm
操作系统
CDH and Cloudera Manager 5.16.x Supported Operating Systems
| Operating System | Version (bold=new) |
|---|---|
| Red Hat Enterprise Linux-compatible | |
| RHEL / CentOS / Oracle Linux RHCK | 7.8, 7.7,7.6, 7.5, 7.4, 7.3, 7.2, 7.16.10, 6.9 , 6.8, 6.7, 6.6, 6.5, 6.45.11, 5.10, 5.7 |
CDH and Cloudera Manager 5.13.x Supported Operating Systems
| Red Hat Enterprise Linux-compatible | |
|---|---|
| RHEL / CentOS / Oracle Linux RHCKMax SE Linux support: 7.2 | 7.6, 7.5, 7.4, 7.3, 7.2, 7.16.10, 6.9 , 6.8, 6.7, 6.6, 6.5, 6.45.11, 5.10, 5.7 |
databases
| MySQL | Cloudera Manager/CDH 5.10 - 5.16 | Cloudera Manager/CDH 5.1 - 5.9 | Cloudera Manager/CDH 5.0 |
|---|---|---|---|
| 5.1 |  |
|
|
| 5.5 | |
|
|
| 5.6 | |
|
|
| 5.7 | |
本地源仓库
简历本地源仓库使用yuｍ 安装，比较简单方便
如果操作系统是最小话安装的，需要安装一些必要依赖
公司的repo： http://192.168.1.220/repo/parcel/
需要下载的包,这些包解压后就是一个repo源, 用http 挂载之后就可以作为离线源安装使用, 注意版本要一致
cdh 包: https://archive.cloudera.com/cdh5/repo-as-tarball/5.13.3/
cm 管理包: https://archive.cloudera.com/cm5/repo-as-tarball/5.13.3/
还需要parcels: https://archive.cloudera.com/cdh5/parcels/5.13.3/
需要java 数据库连接包 jdk-connector
搭建http repos 源 还应该包括ntp, vim net-tools 等
仓库源repo配置
[cloudera-cm]
name=cloudera-repo
baseurl=http://<web_server>/cm/5
enabled=1
gpgcheck=0
[cloudera-cdh]
name=cloudera-repo
baseurl=http://<web_server>/cdh/5
enabled=1
gpgcheck=0
sudo chmod -R ugo+rX /var/www/html/cm # http目录
## 需要为下载的cdh parcel重新校验写入文件内部
sha1sum CDH-5.14.0-1.cdh5.14.0.p0.24-el7.parcel | awk '{ print $1 }' > CDH-5.14.0-1.cdh5.14.0.p0.24-el7.parcel.sha
服务器环境操作
操作系统为CentOS Linux release 7.4.1708 (Core)
1，环境
主机名
/etc/hosts
## 修改主机名, 配置hosts文件
hostnamectl set-hostname cdh-1
hostnamectl set-hostname cdh-2
####
cat > /etc/hosts << EOF
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.103 cdh-1
192.168.1.102 cdh-2
192.168.1.100 cdh-3
EOF
selinux
sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
getenforce
防火墙
systemctl stop firewalld && systemctl disable firewalld && systemctl status firewalld
内核参数
cat >> /etc/sysctl.conf <<EOF
vm.swappiness = 0
EOF
sysctl -p
##
已启用透明大页面压缩，可能会导致重大性能问题。请运行“echo never > /sys/kernel/mm/transparent_hugepage/defrag”和“echo never > /sys/kernel/mm/transparent_hugepage/enabled”以禁用此设置，然后将同一命令添加到 /etc/rc.local 等初始化脚本中，以便在系统重启时予以设置。以下主机将受到影响:
## 安装前会有一个检测服务器的动作,可以在此之后执行
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
ip 地址固定死,防止变化
soft repo 源
soft 源安装一些常用的软件,比如ntp ,vim 数据库需要的net-tools , mysql5.7 httpd 以及cdh客户端需要的一些依赖
## download 下的一些软件
yumdownloader ntp --resolve --destdir=./
yumdownloader http --resolve --destdir=./
yumdownloader httpd --resolve --destdir=./
yumdownloader vim --resolve --destdir=./
yumdownloader perl --resolve --destdir=./
yumdownloader net-tools --resolve --destdir=./
yumdownloader fuse psmisc cyrus-sasl-gssapi bind-utils libxslt openssl-devel python-psycopg2 cyrus-sasl-plain portmap mod_ssl MySQL-python fuse-libs --resolve --destdir=./
yumdownloader redhat-lsb --resolve --destdir=./
repo配置
#其他的repo 删除掉
rm -rf /etc/yum.repos.d/*
cat > /etc/yum.repos.d/local.repo <<EOF
[local-yum]
name=local-yum
baseurl=http://192.168.1.100/soft/
enabled=1
gpgcheck=0
EOF
## cdh 和 cm源
cat > /etc/yum.repos.d/cdh-cm.repo <<EOF
[cloudera-cm-my]
name=cloudera-repo
baseurl=http://192.168.1.100/cm/5
enabled=1
gpgcheck=0
[cloudera-cdh-my]
name=cloudera-repo
baseurl=http://192.168.1.100/cdh/5
enabled=1
gpgcheck=0
EOF
yum clean all
yum makecache
yum repolist
yum -y install vim ntp
ssh 免密
ssh-keygen -t rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
#在一台机器上自己ssh 自己，成功后将.ssh目录拷贝到其他机器
scp -r .ssh root@cdh-2:~/
# 互相验证登录
ntp
时间服务器
yum install ntp
## 日期应该在安装时候就选择好了时区,安装完成后就不用在重置了
date -s "2020-08-13 17:29"
#写入硬件
hwclock -w
##主节点操作, 设置主节点为ntp 服务器
grep -v "^#" /etc/ntp.conf | grep -v "^$"
cat > /etc/ntp.conf <<EOF
driftfile /var/lib/ntp/drift
restrict default nomodify notrap nopeer noquery
restrict 127.0.0.1
restrict ::1
restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap # 192.168.2.0 使用ntp的内网网段
server 127.127.1.0
Fudge 127.127.1.0 stratum 1
includefile /etc/ntp/crypto/pw
keys /etc/ntp/keys
disable monitor
EOF
##客户端配置
grep -v "^#" /etc/ntp.conf | grep -v "^$"
cat > /etc/ntp.conf <<EOF
driftfile /var/lib/ntp/drift
restrict default nomodify notrap nopeer noquery
####restrict 192.168.1.135 nomodify notrap noquery 这个不知道要不要
restrict 127.0.0.1
restrict ::1
restrict 10.30.10.0 mask 255.255.255.0 nomodify notrap
server 192.168.1.100 profer ## ntp 服务器的ip
Fudge 192.168.1.100 stratum 10 ## ntp 服务器的ip
includefile /etc/ntp/crypto/pw
keys /etc/ntp/keys
disable monitor
EOF
##开机自动启动
systemctl enable ntpd.service && systemctl start ntpd.service
#手动同步
ntpdate -u cdh-01
# 查看同步状态
ntpq -p
# 查看ntp运行状态, 大概5 分钟后显示机器处于同步状态
ntpstat
jdk（所有节点）
#卸载自带的jdk-open
rpm -qa | grep java
rpm -qa | grep jdk
yum -y remove xxjdk #删除所有的jdk
yum remove -y *jdk*
yum remove -y *java*
rpm -ivh jdk-8u202-linux-x64.rpm
yum -y install jdk1.8.x86_64
数据库安装
#卸载原有的mariadb
rpm -qa | grep mariadb #查看安装信息
rpm --nodeps -e `rpm -qa | grep mariadb`
##yum 安装
yum install -y mysql mysql-server ## 自动安装net-tools
##启动查看密码
systemctl start mysqld
cat /var/log/mysqld.log |grep password
# 为了加强安全性，MySQL5.7为root用户随机生成了一个密码，在error log中，关于error log的位置，如果安装的是RPM包，则默认是/var/log/mysqld.log。
cat /var/log/mysqld.log |grep password
###登录mysql数据库
mysql -uroot –p密码 # 密码为上一步看到的
##mysql5.7 权限策略问题导致密码薄弱会更新密码失败
###首先，修改validate_password_policy参数的值
## 临时降低密码安全策略
set global validate_password_policy=0;
set global validate_password_length=2;
set global validate_password_special_char_count=0;
set global validate_password_mixed_case_count=0;
set global validate_password_number_count=0;
set password = password('123123');
flush privileges ;
## 查看密码策略:
## SHOW VARIABLES LIKE 'validate_password%';
##授权用户远程连接
grant all privileges on *.* to 'root'@'localhost' identified by '123123' with grant option;
grant all privileges on *.* to 'root'@'127.0.0.1' identified by '123123' with grant option;
grant all privileges on *.* to 'root'@'%' identified by '123123' with grant option;
grant all privileges on *.* to 'hive'@'localhost' identified by '123123' with grant option;
grant all privileges on *.* to 'hive'@'127.0.0.1' identified by '123123' with grant option;
grant all privileges on *.* to 'hive'@'%' identified by '123123' with grant option;
grant all privileges on *.* to 'hue'@'localhost' identified by '123123' with grant option;
grant all privileges on *.* to 'hue'@'127.0.0.1' identified by '123123' with grant option;
grant all privileges on *.* to 'hue'@'%' identified by '123123' with grant option;
grant all privileges on *.* to 'oozie'@'localhost' identified by '123123' with grant option;
grant all privileges on *.* to 'oozie'@'127.0.0.1' identified by '123123' with grant option;
grant all privileges on *.* to 'oozie'@'%' identified by '123123' with grant option;
grant all privileges on *.* to 'cdh'@'localhost' identified by '123123' with grant option;
grant all privileges on *.* to 'cdh'@'127.0.0.1' identified by '123123' with grant option;
grant all privileges on *.* to 'cdh'@'%' identified by '123123' with grant option;
flush privileges ;
grant all privileges on *.* to 'scm'@'localhost' identified by '123123' with grant option;
grant all privileges on *.* to 'scm'@'127.0.0.1' identified by '123123' with grant option;
grant all privileges on *.* to 'scm'@'%' identified by '123123' with grant option;
##查看用户权限: mysql> show grants for test;
##创建需要的数据库：
create database metastore default character set utf8 DEFAULT COLLATE utf8_general_ci;
create database amon default character set utf8 DEFAULT COLLATE utf8_general_ci;
create database rmon default character set utf8 DEFAULT COLLATE utf8_general_ci;
create database hue default character set utf8 DEFAULT COLLATE utf8_general_ci;
create database sentry default character set utf8 DEFAULT COLLATE utf8_general_ci;
create database nav default character set utf8 DEFAULT COLLATE utf8_general_ci;
create database navms default character set utf8 DEFAULT COLLATE utf8_general_ci;
create database oozie default character set utf8 DEFAULT COLLATE utf8_general_ci;
create database hive default character set utf8 DEFAULT COLLATE utf8_general_ci;
#create database scm default character set utf8 DEFAULT COLLATE utf8_general_ci; # 不创建,初始化时自动创建
CM安装
# install the Cloudera Manager packages.
# 在管理节点安装cloudera-manager-daemons cloudera-manager-server
# RHEL, if you have a yum repo configured
sudo yum install cloudera-manager-daemons cloudera-manager-server -y
sudo yum install cloudera-manager-agent -y
#On all hosts, run the following command to install the Cloudera Manager agent:
#在子节点，安装管理客户端
sudo yum install cloudera-manager-daemons cloudera-manager-agent -y
## *** 在所有节点修改管理代理连接管理服务器的主机名
# On all hosts, configure the Cloudera Manager Agent to point to the Cloudera Manager Server by setting the following properties in the /etc/cloudera-scm-agent/config.ini configuration file:
# server_host Name of the host where Cloudera Manager Server is running.
# server_port Port on the host where Cloudera Manager Server is running.
sed -i 's/server_host=localhost/server_host=cdh-1/' /etc/cloudera-scm-agent/config.ini
##数据库连接程序
##初始化 yum安装的scm_prepare_database.sh 位置
mkdir /usr/share/java/ && cd /usr/share/java/
## 需要解压,解压后获取 mysql-connector-java-5.1.46-bin.jar
curl -LO http://192.168.1.100/soft/mysql-connector-java-5.1.46.tar.gz
tar xvf mysql-connector-java-5.1.46.tar.gz
##进入子一级目录下找到 mysql-connector-java-5.1.46-bin.jar
cp mysql-connector-java-5.1.46/mysql-connector-java-5.1.46-bin.jar .
cp mysql-connector-java-5.1.46-bin.jar mysql-connector-java.jar
chmod 755 mysql-connector-java-5.1.46-bin.jar mysql-connector-java.jar
cp mysql-connector-java-5.1.46-bin.jar mysql-connector-java.jar /usr/share/cmf/lib/
##初始化数据库, 这里的scm 数据库没有预先创建
# 使用同一个账号进行初始化,初始化前临时更改数据库的密码权限策略
set global validate_password_policy=0;
set global validate_password_length=2;
set global validate_password_special_char_count=0;
set global validate_password_mixed_case_count=0;
set global validate_password_number_count=0;
set global validate_password_policy=0;
set global validate_password_length=1;
/usr/share/cmf/schema/scm_prepare_database.sh mysql cm -hlocalhost -uroot -p123123 --scm-host cdh-1 scm 123123
######################初始化成功
/usr/share/cmf/schema/scm_prepare_database.sh mysql cm -hcdh-1 -ucdh -p123123 --scm-host cdh-1 scm cdh 123123 ##可能有问题,还是预先创建scm用户,设置密码,用其初始化
JAVA_HOME=/usr/java/jdk1.8.0_181-amd64
Verifying that we can write to /etc/cloudera-scm-server
Sat Aug 15 18:45:05 CST 2020 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Creating SCM configuration file in /etc/cloudera-scm-server
Executing: /usr/java/jdk1.8.0_181-amd64/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
Sat Aug 15 18:45:05 CST 2020 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
[ main] DbCommandExecutor INFO Successfully connected to database.
################
### /usr/share/cmf/schema/scm_prepare_database.sh mysql cm -hlocalhost -uroot -p123123 --scm-host cdh-1 scm scm scm
##初始化参考https://docs.cloudera.com/documentation/enterprise/latest/topics/prepare_cm_database.html
##CDH 初始化数据库 报错ERROR Exception when creating/dropping database with user 'root' and jdbc url 'jdbc:mys
# 如果数据库里有cm和scm表，需要先删除在重新初始化。
drop database cm ;
drop database scm ;
####所有节点运行[管理代理] scm-agent
####管理节点
systemctl start cloudera-scm-server ### 启动成功后执行下一步骤
systemctl start cloudera-scm-agent
##判断是否启动成功需要一段时间
## 验证
find / -name cloudera-scm-server.log
tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log
grep "Started Jetty server" cloudera-scm-server.log
####所有子节点
systemctl start cloudera-scm-agent
##################################
######### 访问主节点的端口,继续安装测试 admin admin
http://10.30.10.115:7180/cmf/login
添加主机
安装---->免费的许可--->继续--->继续-->搜索主机: 填写主机name cdh-1 cdh-2 需要安装的节点, cdh-1 是在主节点也安装了--->搜索主机,选择主机
如果主机无法选择:报错
BEGIN python -c 'import socket; import sys; s = socket.socket(socket.AF_INET); s.settimeout(5.0); s.connect((sys.argv[1], int(sys.argv[2]))); s.close();' localhost 7182
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "<string>", line 1, in connect
socket.error: [Errno 111] Connection refused
https://www.cnblogs.com/tmeily/p/4578553.html
sudo mv /usr/bin/host /usr/bin/host.bak 网页操作界面回退重新操作,重新搜索主机安装, 之前的初始化scm 有些问题
-->群集安装,设置parcels 地址,我们是远程的.设置远程地址
###安装后节点也需要安装
cd /usr/share/cmf/lib/
cp /usr/share/java/mysql-connector-java-5.1.46-bin.jar .
###cdh 也需要驱动
scp mysql-connector-java-5.1.46-bin.jar foo-2:/opt/cloudera/parcels/CDH-5.13.3-1.cdh5.13.3.p0.2/lib
##hive 数据库也需要 默认的, 以下的各自目录都重定向到了/usr/share/cmf/lib, 里面有驱动, 所以不用再复制了
scp mysql-connector-java-5.1.46-bin.jar foo-2:/opt/cloudera/parcels/CDH-5.13.3-1.cdh5.13.3.p0.2/lib/hive/lib
scp mysql-connector-java-5.1.46-bin.jar foo-2:/opt/cloudera/parcels/CDH-5.13.3-1.cdh5.13.3.p0.2/lib
scp mysql-connector-java-5.1.46-bin.jar foo-2:/opt/cloudera/parcels/CDH-5.13.3-1.cdh5.13.3.p0.2/lib/hive/lib
scp mysql-connector-java-5.1.46-bin.jar foo-2:/opt/cloudera/parcels/CDH-5.13.3-1.cdh5.13.3.p0.2/lib/oozie/lib
scp mysql-connector-java-5.1.46-bin.jar foo-2:/opt/cloudera/parcels/CDH-5.13.3-1.cdh5.13.3.p0.2/lib/hue/lib
scp mysql-connector-java-5.1.46-bin.jar foo-2:/var/lib/oozie
scp mysql-connector-java-5.1.46-bin.jar foo-2:/var/lib/hue
http://192.168.2.150:7180/cmf/clusters/1/express-add-services/index#step=commandDetailsStep
cdh-1 主节点操作步骤:
1 ip a s
2 nmtui
3 init 0
4 ip a s
5 cat /etc/redhat-release
6 hostnamectl set-hostname cdh-1
7 ip a s
8 sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
9 systemctl stop firewalld
10 systemctl disable firewalld
11 systemctl status firewalld
12 cat /etc/hosts
13 reboot
14 vim /etc/sysctl.conf
15 vi /etc/sysctl.conf
16 cat >> /etc/sysctl.conf <<EOF
17 vm.swappiness = 0
18 EOF
19 sysctl -p
20 systemctl stop firewalld
21 systemctl disable firewalld
22 systemctl status firewalld
23 cat >> /etc/hosts << EOF
24 192.168.0.166 cdh-1
25 192.168.0.102 cdh-2
26 192.168.0.181 cdh-3
27 EOF
28 cd /etc/yum.repos.d/
29 ll
30 rm -rf *
31 cat > /etc/yum.repos.d/local.repo <<EOF
32 [local-yum]
33 name=local-yum
34 baseurl=http://192.168.0.181/soft/
35 enabled=1
36 gpgcheck=0
37 EOF
38 yum clean all
39 yum makecache
40 yum -y install vim ntp
41 ssh-keygen -t rsa
42 cd
43 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
44 ssh-keygen -t rsa
45 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
46 chmod 600 ~/.ssh/authorized_keys
47 ssh cdh-1
48 scp -r .ssh root@cdh-2:~/
49 scp -r .ssh root@cdh-3:~/
50 ssh cdh-2
51 ssh cdh-3
52 ip a s
53 cat > /etc/ntp.conf <<EOF
54 driftfile /var/lib/ntp/drift
55 restrict default nomodify notrap nopeer noquery
56 restrict 127.0.0.1
57 restrict ::1
58 restrict 192.168.0.0 mask 255.255.255.0 nomodify notrap
59 server 192.168.0.181 profer
60 Fudge 192.168.0.181 stratum 10
61 includefile /etc/ntp/crypto/pw
62 keys /etc/ntp/keys
63 disable monitor
64 EOF
65 systemctl enable ntpd.service
66 systemctl start ntpd.service
67 ntpdate -u cdh-3
68 ntpq -p
69 ntpstat
70 rpm -qa | grep java
71 yum remove -y *java*
72 yum -y instal jdk
73 yum -y install jdk
74 yum makecache
75 yum search mysql
76 yum install mysql -y
77 rpm -qa | grep mariadb
78 systemctl start mysql
79 systemctl start mysqld
80 yum install mysql-server
81 history
82 ll
83 cd
84 systemctl start mysql
85 systemctl start mysqld
86 cat /var/log/mysqld.log |grep password\
87 cat /var/log/mysqld.log |grep password
88 mysql_secure_installation
89 mysql -uroot -pu<NWQkIMj2rI
90 mysql -uroot -p
91 mysql -uroot -p123123
92 cd /etc/yum.repos.d/
93 pwd
94 cat >> /etc/yum.repos.d/cloudera-manager.repo <<EOF
95 [clouderamanager]
96 name=Cloudera Manager
97 baseurl=http://192.168.0.181/cm/5/
98 gpgkeyhttp://192.168.0.181/cm/RPM-GPG-KEY-cloudera
99 enabled = 1
100 gpgcheck = 0
101 EOF
102 yum makecache
103 cd
104 install the Cloudera Manager packages.
105 sudo yum install cloudera-manager-daemons cloudera-manager-server
106 sudo yum install cloudera-manager-agent
107 yum makecache
108 sudo yum install cloudera-manager-agent
109 yum makecache
110 sudo yum install cloudera-manager-agent
111 sed -i 's/server_host=localhost/server_host=cdh-1/' /etc/cloudera-scm-agent/config.ini
112 cat /etc/cloudera-scm-agent/config.ini
113 mkdir /usr/share/java/ && cd /usr/share/java/
114 curl -LO http://192.168.0.181/soft/mysql-connector-java-5.1.46.tar.gz
115 cp mysql-connector-java-5.1.46.tar.gz mysql-connector-java.jar
116 chmod 755 mysql-connector-java-5.1.46.jar mysql-connector-java.jar
117 chmod 755 mysql-connector-java.jar mysql-connector-java.jar
118 chmod 755 mysql-connector-java-5.1.46.tar.gz mysql-connector-java.jar
119 ll /usr/share/cmf/lib/
120 ll
121 rm -rf *
122 ll /usr/share/cmf/schema/scm_prepare_database.sh
123 /usr/share/cmf/schema/scm_prepare_database.sh mysql cm -hlocalhost -uroot -p123123 --scm-host cdh-1 scm scm scm
124 mysql -uroot -p123123
125 /usr/share/cmf/schema/scm_prepare_database.sh mysql cm -hlocalhost -uroot -p123123 --scm-host cdh-1 scm scm scm
126 mysql -uroot -p123123
127 /usr/share/cmf/schema/scm_prepare_database.sh mysql cm -hlocalhost -uroot -p123123 --scm-host cdh-1 scm scm scm
128 systemctl start cloudera-scm-server
129 systemctl start cloudera-scm-agent
130 find / -name cloudera-scm-server.log
131 grep "Started Jetty server" /var/log/cloudera-scm-server/cloudera-scm-server.log
132 systemctl restart cloudera-scm-server
133 tailf /var/log/cloudera-scm-server/cloudera-scm-server.log
134 /usr/share/cmf/lib
135 cd /usr/share/cmf/lib
136 ll
137 ll mysql-connector-java-5.1.46-bin.jar
138 cp /usr/share/java/mysql-connector-java-5.1.46-bin.jar .
139 ll mysql-connector-java-5.1.46-bin.jar
140 ll /opt/cloudera/parcels/CDH-5.13.3-1.cdh5.13.3.p0.2/lib
141 ip a s
142 echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
143 echo never > /sys/kernel/mm/transparent_hugepage/defrag
144 echo never > /sys/kernel/mm/transparent_hugepage/enabled
cdh-2 节点操作:
1 nmtui
2 init 0
3 ip a s
4 cat /etc/redhat-release
5 hostnamectl set-hostname cdh-2
6 vi /etc/hosts
7 ip a s
8 sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
9 systemctl stop firewalld
10 systemctl disable firewalld
11 systemctl status firewalld
12 reboot
13 sysctl -p
14 cat >> /etc/sysctl.conf <<EOF
15 vm.swappiness = 0
16 EOF
17 sysctl -p
18 echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
19 cat >> /etc/hosts << EOF
20 192.168.0.166 cdh-1
21 192.168.0.102 cdh-2
22 192.168.0.181 cdh-3
23 EOF
24 HIS
25 history
26 cd /etc/yum.repos.d/
27 ll
28 rm -rf *
29 pwd
30 cat > /etc/yum.repos.d/local.repo <<EOF
31 [local-yum]
32 name=local-yum
33 baseurl=http://192.168.0.181/soft/
34 enabled=1
35 gpgcheck=0
36 EOF
37 yum clean all
38 yum makecache
39 yum -y install vim ntp
40 cat > /etc/ntp.conf <<EOF
41 driftfile /var/lib/ntp/drift
42 restrict default nomodify notrap nopeer noquery
43 restrict 127.0.0.1
44 restrict ::1
45 restrict 192.168.0.0 mask 255.255.255.0 nomodify notrap
46 server 192.168.0.181 profer
47 Fudge 192.168.0.181 stratum 10
48 includefile /etc/ntp/crypto/pw
49 keys /etc/ntp/keys
50 disable monitor
51 EOF
52 cd
53 systemctl enable ntpd.service
54 systemctl start ntpd.service
55 ntpdate -u cdh-3
56 ntpq -p
57 ntpstat
58 yum -y install jdk
59 yum install jdk
60 cat >> /etc/yum.repos.d/cloudera-manager.repo <<EOF
61 [clouderamanager]
62 name=Cloudera Manager
63 baseurl=http://192.168.0.181/cm/5/
64 gpgkeyhttp://192.168.0.181/cm/RPM-GPG-KEY-cloudera
65 enabled = 1
66 gpgcheck = 0
67 EOF
68 yum clean all
69 yum makecache
70 sudo yum install cloudera-manager-agent
71 yum clean all
72 cat /etc/yum.repos.d/cloudera-manager.repo
73 yum clean all
74 yum makecache
75 yum clean all
76 yum makecache
77 sudo yum install cloudera-manager-agent
78 sed -i 's/server_host=localhost/server_host=cdh-1/' /etc/cloudera-scm-agent/config.ini
79 mkdir /usr/share/java/ && cd /usr/share/java/
80 curl -LO http://192.168.0.181/soft/mysql-connector-java-5.1.46.tar.gz
81 cp mysql-connector-java-5.1.46-bin.jar mysql-connector-java.jar
82 LL
83 ll
84 cp mysql-connector-java-5.1.46.tar.gz mysql-connector-java.jar
85 ll /usr/share/cmf/lib/
86 cp mysql-connector-java-5.1.46.jar mysql-connector-java.jar /usr/share/cmf/lib/
87 ll
88 tar xvf mysql-connector-java-5.1.46.tar.gz
89 ll
90 rm -rf mysql-connector-java-5.1.46
91 ll
92 tar -xvf mysql-connector-java-5.1.46.tar.gz
93 cd mysql-connector-java
94 cd mysql-connector-java-5.1.46
95 ll
96 mv mysql-connector-java-5.1.46-bin.jar ../
97 cd ../
98 ll
99 rm -rf mysql-connector-java-5.1.46.tar.gz mysql-connector-java.jar
100 ll
101 pwd
102 ll
103 cp mysql-connector-java-5.1.46-bin.jar mysql-connector-java.jar
104 chmod 755 mysql-connector-java-5.1.46-bin.jar mysql-connector-java.jar
105 ll
106 scp mysql-connector-java-5.1.46-bin.jar cdh-1:/usr/share/java
107 scp mysql-connector-java.jar cdh-1:/usr/share/java
108 systemctl start cloudera-scm-agent
109 cd /usr/share/cmf/lib/
110 ll
111 ll mysql-connector-java-5.1.46-bin.jar
112 cp /usr/share/java/mysql-connector-java-5.1.46-bin.jar .
113 ll mysql-connector-java-5.1.46-bin.jar
114 pwd
115 echo never > /sys/kernel/mm/transparent_hugepage/defrag
116 echo never > /sys/kernel/mm/transparent_hugepage/enabled
问题
据说5.7版本数据开启GTID会导致cdh初始化数据库失败
在线开启步骤：
1、要求：
（1）必须是5.7.6版本以上的mysql
（2）GTID状态为OFF
2、开启步骤：
（1）：SET GLOBAL ENFORCE_GTID_CONSISTENCY = 'WARN';
（2）：SET GLOBAL ENFORCE_GTID_CONSISTENCY = 'ON';
（3）：SET GLOBAL GTID_MODE = 'OFF_PERMISSIVE';
（4）：SET GLOBAL GTID_MODE = 'ON_PERMISSIVE';
（5）：SET GLOBAL GTID_MODE = 'ON';
3、要永久启用，在my.cnf配置文件中添加参数：
gtid-mode=ON
enforce-gtid-consistency
在线关闭步骤：
1、要求：
（1）必须是5.7.6版本以上的mysql
（2）GTID状态为OFF
2、关闭步骤：
（1）：stop slave;
（2）：SET GLOBAL GTID_MODE = 'ON_PERMISSIVE';
（3）：SET GLOBAL GTID_MODE = 'OFF_PERMISSIVE';
（4）：SET GLOBAL GTID_MODE = 'OFF';
注：
每次开启和关闭时，都是这样一个过程：
打开-->过度模式-->完全打开
停止-->过度模式-->完全关闭
浙公网安备 33010602011771号