同步工具-腾讯EMR表治理工具安装使用
一、安装
1、root用户上传文件
cd wang
rz -bey luoshu-1.0-bin.tar.gz
2、解压文件到服务目录
重新安装洛书需执行：rm -rf /usr/local/service/luoshu
mkdir /usr/local/service/luoshu
tar -zxf luoshu-1.0-bin.tar.gz -C /usr/local/service/luoshu
3、（仅初次安装洛书执行）连接hive metastore,执行初始化脚本。脚本会新建数据库luoshu_db
mysql -h 127.0.0.1 -u账号 -p'密码' < config/luoshu-mysql-init.sql
4、（仅初次安装洛书执行）查看本地有无jdk-17.0.6版本，没有则上传。解压到/usr/local/jdk-17.0.6
rz -bey jdk-17.0.6.tar.gz
tar -zxf jdk-17.0.6.tar.gz -C /usr/local/
5、修改luoshu/config下面的luoshu-env.sh
#!/usr/bin/env bash
# Options for luoshu server
# - JAVA_HOME, to set java home path
#export JAVA_HOME=
export JAVA_HOME=/usr/local/jdk-17.0.6
# - JAVA_EXTRA_OPTS, set java extra opts for luoshu server
#export JAVA_OPTS="-server -Xms512m -Xmx512m"
export JAVA_OPTS="-server -Xms512m -Xmx512m -Djava.security.krb5.conf=/etc/krb5.conf --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED"
# - HADOOP_CONF_DIR, to point Spark towards Hadoop configuration files
#export HADOOP_CONF_DIR=
export HADOOP_CONF_DIR=/usr/local/service/hadoop/etc/hadoop
# - HIVE_CONF_DIR, to point Spark towards Hive configuration files
#export HIVE_CONF_DIR=
export HIVE_CONF_DIR=/usr/local/service/hive/conf
# - LUOSHU_LOG_DIR, to set where log files are stored. (Default: ${LUOSHU_HOME}/logs)
#export LUOSHU_LOG_DIR=
# - SERVER_PORT, to set luoshu web server listen port. (Default: 8080)
#export SERVER_PORT=8080
export SERVER_PORT=9090
# - DB_DRIVER, to set datastore engine driver class name. (Default: com.mysql.cj.jdbc.Driver)
#export DB_DRIVER=com.mysql.cj.jdbc.Driver
export DB_DRIVER=com.mysql.cj.jdbc.Driver
# - DB_URL, to set datastore engine connection url
#export DB_URL="jdbc:mysql://localhost:3306/db?useSSL=false&characterEncoding=UTF-8"
export DB_URL="jdbc:mysql://localhost:3306/luoshu_db?useSSL=false&characterEncoding=UTF-8"
# - DB_USER, to set datastore engine connection username
#export DB_USER=
export DB_USER=账号
# - DB_PASS, to set datastore engine connection password
#export DB_PASS=
export DB_PASS=密码6、配置luoshu-config.properties新增如下配置
#本地ip和spark做rpc通信，不能用127.0，用本机ip
thrift.server.host=10.10.10.10
thrift.server.port=9091
7、修改spark配置
/usr/local/service/spark/conf/spark-env.sh
新增：export JAVA_HOME=/usr/local/jdk
8、krb5.conf增加配置
原因：洛书使用了jdk17，不支持低版本加密方式，需要开启兼容
vim /etc/krb5.conf
[libdefaults]下面新增
allow_weak_crypto = true
9、增加jar包
/usr/local/service/hadoop/share/hadoop/common/lib下面拷贝包
cp chdfs_hadoop_plugin_network-2.7.jar /usr/local/service/luoshu/lib/
cp gson-2.2.4.jar /usr/local/service/luoshu/lib/
cp goosefs-1.1.0-SNAPSHOT-client.jar /usr/local/service/luoshu/lib/
10、修改文件夹权限
chown hadoop:hadoop luoshu -R
11、多询问一下，有没有数据库变更
有的更新可能有数据库变更，询问有没有数据库字段变更，确保启动正常
12、启动luoshu（hadoop用户）
sudo su - hadoop
cd /usr/local/service/luoshu/bin
sh start.sh
13、验证服务
（1）查看启动日志
tail -f /usr/local/service/luoshu/logs/luoshu.log -n 500
（2）ps -ef |grep -i luoshu
查看是否是hadoop登录
（3）登录洛书页面，新增任务，查看是否验证正常
data_lake_ods.order_info1
14、附加-luoshu定时提交任务
默认是创建时间，每天一次全表优化
优化命令：
/usr/local/service/spark/bin/spark-submit --master yarn --deploy-mode cluster --queue default --class com.tencent.luoshu.optimizer.SparkBatchOptimizer --conf spark.yarn.submit.waitAppCompletion=false --conf spark.yarn.dist.files=/usr/local/service/luoshu/config/luoshu-config.properties --conf spark.app.name=spark-luoshu --conf spark.yarn.tags=spark-20240103164142 --conf spark.yarn.applicationType=spark-20240103164142 /usr/local/service/luoshu/spark/luoshu-spark-1.0-jar-with-dependencies.jar -1
15、附加-测试kerberos连接工具
/usr/local/jdk-17.0.6/bin/java -Djava.security.krb5.conf=/etc/krb5.conf --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED -jar kerberos-test-1.0-jar-with-dependencies2.jar
二、使用
1、页面管理地址
127.0.0.1:9090/luoshu
2、配置定时清理任务
