CDH5.16安装flink1.10.0

1.采用parcels包的方式来安装flink

编译parcels请参考：制作Flink的Parcel包和csd文件

CDH5.16.1 集成 Flink-1.10.0

cdh5.15.1集成flink说明

2.将parcels包拷贝到对应目录

lintong@master:/opt/cloudera/parcel-repo$ ls | grep FLINK
FLINK-1.10.0-BIN-SCALA_2.11-xenial.parcel
FLINK-1.10.0-BIN-SCALA_2.11-xenial.parcel.sha

csd包

lintong@master:/opt/cloudera/csd$ ls
FLINK_ON_YARN-1.10.0.jar

3.安装

配置主机

配置flink集群参数

部署成功

flink集群部署后会在YARN上启动一个以flink用户运行的flink session，注意此时在此flink session中是不能认证成其他用户的，只能是flink用户，可以参考：Flink on Yarn with Kerberos

比如使用flink-scala-shell就会运行在此flink session中，而在里面认证是不会生效

部署成功

参考：Cloudera Manager中安装部署Flink服务

 

4.遇到的一些问题：

其中kerberos相关的参数如果集群没有启用kerberos的话可以不填写

如果有开启kerberos的话，需要在首先在os上添加flink用户

sudo groupadd flink
sudo useradd flink -g flink -r --no-log-init -d /var/lib/flink
sudo mkdir /var/lib/flink
sudo chown flink:flink /var/lib/flink

然后创建生成flink用户的keytab

如果集成了ldap，需要在ldap中创建flink Group和People

如果重启flink集群的时候遇到

security.UserGroupInformation: PriviledgedActionException as:flink (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

需要先使用flink用户进行kinit认证

sudo -u flink kinit -kt /etc/keytab/flink.keytab flink/master@HADOOP.COM

启动flink-scala-shell验证

lintong@master:/opt/cloudera/parcels/FLINK/lib/flink/bin$ ./start-scala-shell.sh yarn

如果遇到

Exception in thread "main" java.lang.NoSuchMethodError: jline.console.completer.CandidateListCompletionHandler.setPrintSpaceAfterFullCompletion(Z

请下载 jline-2.14.3.jar包 到flink的lib目录

lintong@master:/opt/cloudera/parcels/FLINK/lib/flink/lib$ wget https://repo1.maven.org/maven2/jline/jline/2.14.3/jline-2.14.3.jar

如果遇到

Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/exceptions/YarnException
    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
    at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
    at java.lang.Class.getMethod0(Class.java:3018)
    at java.lang.Class.getMethod(Class.java:1784)
    at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
    at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.yarn.exceptions.YarnException
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 7 more

请下载 flink-shaded-hadoop-2-uber-2.7.5-7.0.jar包，放到flink的lib目录下

lintong@master:/opt/cloudera/parcels/FLINK/lib/flink/lib$ wget https://repo1.maven.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.7.5-7.0/flink-shaded-hadoop-2-uber-2.7.5-7.0.jar

读取文件并打印

scala> val dataSet = benv.readTextFile("hdfs:///user/hive/warehouse/test/000000_0")
scala> dataSet.print()