flink集群使用minio做元数据共享存储

环境:
OS:ubuntu26
flink:1.17.1

1.部署minio

步骤省略

可以参考如下连接

https://www.cnblogs.com/hxlasky/p/18643317 

 

2.flink端下载jar包
在一个节点上下载后,scp到另外的节点,我这里是在192.168.1.134节点上下载
cd /opt/flink-1.17.1/lib
wget https://repo1.maven.org/maven2/org/apache/flink/flink-s3-fs-hadoop/1.17.1/flink-s3-fs-hadoop-1.17.1.jar
wget https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/3.3.4/hadoop-aws-3.3.4.jar

scp flink-s3-fs-hadoop-1.17.1.jar root@192.168.1.135:/opt/flink-1.17.1/lib/
scp hadoop-aws-3.3.4.jar root@192.168.1.135:/opt/flink-1.17.1/lib/

scp flink-s3-fs-hadoop-1.17.1.jar root@192.168.1.113:/opt/flink-1.17.1/lib/
scp hadoop-aws-3.3.4.jar root@192.168.1.113:/opt/flink-1.17.1/lib/

 

3.修改配置
vi /opt/flink-1.17.1/conf/flink-conf.yaml

#重点:s3路径,格式s3://bucket名/子路径
high-availability.storageDir: s3://flink-ha/ha-storage/

# MinIO S3认证参数
s3.endpoint: http://192.168.12.87:9000
s3.access-key: admin
s3.secret-key: admin@123
s3.path-style-access: true

 

4.minio创建bucket
flink-ha

 

4.重启集群
/opt/flink-1.17.1/bin/stop-cluster.sh
/opt/flink-1.17.1/bin/start-cluster.sh

若无法启动的话新删除掉zookeeper里的数据
/opt/zookeeper-3.7.0/bin/zkCli.sh -server 192.168.1.134:2182,192.168.1.135:2182,192.168.1.113:2182
[zk: 192.168.1.134:2182,192.168.1.135:2182,192.168.1.113:2182(CONNECTED) 2] ls /
[flink, flink17, test_zk, zookeeper]
[zk: 192.168.1.134:2182,192.168.1.135:2182,192.168.1.113:2182(CONNECTED) 3] deleteall /flink17