CDH-6.2.1 集群安装

1、修改ip 
vi /etc/sysconfig/network-scripts/ifcfg-ens33  
修改以下几个值：
BOOTPROTO=static
IPV6INIT=no
IPV6_AUTOCONF=no
IPV6_DEFROUTE=no
ONBOOT=yes
新增以下几个值：
IPADDR=10.10.10.21
NETMASK=255.255.220.0
GATEWAY=10.10.10.2
DNS1=8.8.8.8
DNS2=114.114.114.114


2、修改主机名和 /etc/hosts
hostnamectl set-hostname  hadoop01
hostnamectl set-hostname  hadoop02
hostnamectl set-hostname  hadoop03
hostnamectl set-hostname  hadoop04

修改/etc/hosts
cat >> /etc/hosts  << EOF
10.10.10.21 hadoop01
10.10.10.22 hadoop02
10.10.10.23 hadoop03
10.10.10.24 hadoop04
EOF

3、配置互信
生成互信文件，输入如下命令，一路回车就行
ssh-keygen
cd ~/.ssh
cat id_rsa.pub > authorized_keys 
chmod 600 *
cd ~/
scp -r .ssh 10.10.10.22:~/
scp -r .ssh 10.10.10.23:~/
scp -r .ssh 10.10.10.24:~/

最后每个节点都要验证
ssh  hadoop01 date
ssh  hadoop02 date
ssh  hadoop03 date
ssh  hadoop04 date


4、将虚拟内存的交换设为最小，但不禁用  /etc/sysctl.conf中包含了vm.swappiness=1或0）
sysctl -w vm.swappiness=1


echo "vm.swappiness=1" >> /etc/sysctl.conf

保存后使其生效
sysctl -p


5、关闭透明大页
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

cat >> /etc/rc.d/rc.local << EOF
if test -f /sys/kernel/mm/transparent_hugepage/enabled;then
echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi

if test -f /sys/kernel/mm/transparent_hugepage/defrag;then
echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi
EOF

chmod +x /etc/rc.d/rc.local
 

6、关闭SELinux 
setenforce 0
sed -i 's/^SELINUX=.*$/SELINUX=disabled/g' /etc/selinux/config


7、禁用防火墙 
systemctl disable firewalld


8、重启主机
reboot


9、配置chrony服务 
公有云有自己的时钟同步方案，因此这步不需要。如果是使用物理机的内网集群，需要修改
/etc/chrony.conf 指向内网ntp服务器。


10、安装依赖包
yum install -y cyrus-sasl-plain cyrus-sasl-gssapi portmap fuse-libs bind-utils libxslt fuse
yum install -y /lib/lsb/init-functions createrepo deltarpm python-deltarpm
yum install -y mod_ssl openssl-devel python-psycopg2 MySQL-python
    
    
11、安装httpd(只需要在部署本地yum源的机器上安装即可，不用三台全部安装)
yum install -y  httpd
yum install -y  createrepo
systemctl start httpd.service
    
cd /var/www/html/
mkdir -p cm6/6.2.1/redhat7/yum/RPMS/x86_64/

上传文件到此目录
上传allkeys.asc文件到/var/www/html/cm6/6.2.1/目录下

访问测试：http://10.10.10.21/cm6/6.2.1/redhat7/yum/RPMS/x86_64/

生成repodata目录
cd /var/www/html/cm6/6.2.1/redhat7/yum
createrepo .


12、配置本地yum源
cat > /etc/yum.repos.d/cloudera-manager.repo << EOF
[cloudera-manager]
name=Cloudera Manager
baseurl=http://10.10.10.21/cm6/6.2.1/redhat7/yum/
gpgcheck=0
enabled=1
EOF


执行命令：
yum clean all && yum makecache
yum list | grep cloudera


13、创建cloudera-scm用户（centos7要求必须有，centos6没有要求）
useradd cloudera-scm
echo "123456"|passwd --stdin cloudera-scm

cloudera-scm用户配置免密钥登录
echo "cloudera-scm ALL=(root)NOPASSWD:ALL" >> /etc/sudoers
su - cloudera-scm
exit


14、安装MySQL，步骤省略，创建 scm 数据库和账号
CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY '123456';


15、安装Server服务
yum install -y  oracle-j2sdk1.8-1.8.0+update181-1.x86_64
yum install -y  enterprise-debuginfo-6.2.1-1426065.el7.x86_64
yum install -y  cloudera-manager-server-6.2.1-1426065.el7.x86_64
yum install -y  cloudera-manager-server-db-2-6.2.1-1426065.el7.x86_64


16、设置元数据库为mysql
设置元数据库为mysql--前提是安装 ClouderaManager Server服务

/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm "123456"


JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
Verifying that we can write to /etc/cloudera-scm-server
Mon Nov 29 06:25:52 PST 2021 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Creating SCM configuration file in /etc/cloudera-scm-server
Executing:  /usr/java/jdk1.8.0_181-cloudera/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
Mon Nov 29 06:25:53 PST 2021 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
[                          main] DbCommandExecutor              INFO  Successfully connected to database.
All done, your SCM database is configured correctly!


提示ERROR JDBC Driver com.mysql.jdbc.Driver not found.
需要上传mysql驱动包jar到 /opt/cloudera/cm/lib目录下。

查看配置是否OK
cat /etc/cloudera-scm-server/db.properties


17、启动server服务进程（查看7180端口）
systemctl start cloudera-scm-server

查看启动日志：
tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

查看启动状态：
systemctl status cloudera-scm-server

确定端口号已经启动：
netstat -tlnp | grep 7180

查看 MySQL数据库中 scm 是否有表，如果没有表，则把 scm 进程杀掉，重新执行scm_prepare_database脚本。


18、配置本地parcel包
注意：必须要在数据库初始化后，将parcel文件
CDH-6.2.1-1.cdh6.2.1.p0.1425774-el7.parcel
CDH-6.2.1-1.cdh6.2.1.p0.1425774-el7.parcel.sha1
CDH-6.2.1-1.cdh6.2.1.p0.1425774-el7.parcel.sha256
放置到/opt/cloudera/parcel-repo目录中

重命名密钥文件名
mv CDH-6.2.1-1.cdh6.2.1.p0.1425774-el7.parcel.sha1 CDH-6.2.1-1.cdh6.2.1.p0.1425774-el7.parcel.sha


19、开始安装
访问Server：http://10.10.10.21:7180/cmf/login
账号密码均为admin

设置Parcel 更新频率为1分钟
自定义存储库：  http://10.10.10.21/cm6/6.2.1/


20、安装 Cloudera Management Service

mkdir -p /var/lib/cloudera-host-monitor
mkdir /var/lib/cloudera-service-monitor

chown -R cloudera-scm:cloudera-scm /var/lib/cloudera-host-monitor
chown -R cloudera-scm:cloudera-scm /var/lib/cloudera-service-monitor/

创建 Activity Monitor 数据库
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY '123456';


21、添加HDFS服务
安装好之后， HDFS => 配置 => def.permissions  中 HDFS(服务范围)  不要勾选，改为false 。
这个选项需要重启hdfs服务才能生效。
否则会因为权限问题不能在hdfs中创建目录和文件。


关闭safemode模型
查看当前的模型是否开启
[root@hadoop01 ~]# hdfs dfsadmin -safemode get
Safe mode is OFF

关闭模型
[root@hadoop01 ~]# hdfs dfsadmin -safemode leave
Safe mode is OFF


测试HDFS：
hdfs dfs -mkdir /test


22、添加Yarn服务

测试:
cat >> /tmp/testiinput.txt   << EOF
I love BeiJing
I love China
Beijing is the capital of China
EOF


创建HDFS目录并上传文件
hadoop fs -ls /
hadoop fs -put /tmp/testiinput.txt /input

运行wordcount
yarn jar /opt/cloudera/parcels/CDH/jars/hadoop-mapreduce-examples-3.0.0-cdh6.2.1.jar wordcount /input /output

完成后查看输出目录
hadoop fs -ls /output  

查看输出结果
hadoop fs -cat /output/part-r-00000   

 
23、添加ZooKeeper服务


24、添加Hive服务

Select Dependencies：
Hive 服务不依赖ZooKeeper，所以选择的时候选择另外一个没有Zookeeper依赖的选项。

提示ERROR JDBC Driver com.mysql.jdbc.Driver not found.
需要上传mysql驱动包jar到/opt/cloudera/cm/lib目录下。

mysql -uroot -p
create database hive;

提示：java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
cp /opt/cloudera/cm/lib/mysql-connector-java-5.1.40.jar /opt/cloudera/parcels/CDH/lib/hive/lib/


验证：
hive
select from_unixtime(1583781807, 'yyyy-MM-dd');


25、添加ooozie服务

Select Dependencies：
选择Hive、YARN、ZooKeeper 都有的选项

创建oozie所用的数据库
create database oozie;


提示：java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
cp /opt/cloudera/cm/lib/mysql-connector-java-5.1.40.jar /var/lib/oozie/

org.apache.oozie.action.ActionExecutorException: JA009: Invalid resource request! Cannot allocate containers as requested resource is greater than maximum allowed allocation. Requested resource type=[memory-mb], Requested resource=<memory:2048, vCores:1>, maximum allowed allocation=<memory:1024, vCores:4>, please note that maximum allowed allocation is calculated by scheduler based on maximum resource of registered NodeManagers, which might be less than configured maximum allocation=<memory:1024, vCores:4>


26、添加Sqoop服务

Sqoop 只有在运行的时候才消耗CPU和内存资源，所以可以在所有节点都安装，因为只是占用磁盘空间而已。

Sqoop 测试：
mysql -uroot -p
create schema test;
create table test
(
    id bigint auto_increment,
    name varchar(127) not null,
    age bigint null,
    class varchar(255) null,
    primary key (id)
);


vim text.txt

leo|12|class1
leo1|13|class2
leo2|14|class3
leo3|15|class4
leo4|16|class5
leo5|17|class6

cp /opt/cloudera/cm/lib/mysql-connector-java-5.1.40.jar /opt/cloudera/parcels/CDH/lib/sqoop/lib/

sqoop export \
--connect jdbc:mysql://hadoop01:3306/test \
--username root \
--password 123456 \
--table test \
--columns 'name,age,class' \
--export-dir /data/test \
--input-fields-terminated-by '|' \
--num-mappers 1


27、添加HBase服务
hbase shell

建立只有三个column family的表
create 't1', {NAME => 'f1', VERSIONS => 1}, {NAME => 'f2', VERSIONS => 1}, {NAME => 'f3', VERSIONS => 1}

插入没有指定column的名字
put 't1', 'r1', 'f1', 'v1'
put 't1', 'r2', 'f2', 'v2'
put 't1', 'r3', 'f3', 'v3'


插入指定column的名字
put 't1', 'r4', 'f1:c1', 'v1'
put 't1', 'r5', 'f2:c2', 'v2'
put 't1', 'r6', 'f3:c2', 'v3'

查询数据
scan 't1'


28、添加Spark服务

Select Dependencies：
不需要依赖HBase。


29、添加Hue服务
mysql -uroot -p
create database hue default character set utf8 collate utf8_general_ci;

Select Dependencies：
不依赖HBase，但是依赖HDFS、Hive、Oozie、YARN、ZooKeeper


提示：java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
cp /opt/cloudera/cm/lib/mysql-connector-java-5.1.40.jar /opt/cloudera/parcels/CDH/lib/oozie/lib/

WEB登录地址： 10.10.10.22:8889