flink-1.20.5高可用部署
环境:
OS:Centos 7
flink:flink-1.20.5-bin-scala_2.12.tgz
zookeeper:3.7.0
架构:3 台机器,1 主2备 JobManager + 3 台同时做TaskManager
节点规划(你可以替换主机名/IP)
node1(192.168.1.134):JobManager(active) + zk
node2(192.168.1.135):JobManager(standby) + zk
node3(192.168.1.113):JobManager (standby) + zk
#####################################配置ssh免密###################################
不配置的话好像不能使用start-cluster.sh启动整个集群
1.修改主机名称
# 192.168.1.134
hostnamectl set-hostname flink-134
# 192.168.1.135
hostnamectl set-hostname flink-135
# 192.168.1.113
hostnamectl set-hostname flink-113
2.配置/etc/hosts
vi /etc/hosts
192.168.1.134 flink-134
192.168.1.135 flink-135
192.168.1.113 flink-113
3.配置免密
3.1.机器A:192.168.1.134 上操作
su - root
cd ~
ssh-keygen -t rsa
ssh-copy-id -i root@192.168.1.134
ssh-copy-id -i root@192.168.1.135
ssh-copy-id -i root@192.168.1.113
ssh 192.168.1.134 date
ssh 192.168.1.135 date
ssh 192.168.1.113 date
3.2.机器B:192.168.1.135 上操作
su - root
cd ~
ssh-keygen -t rsa
ssh-copy-id root@192.168.1.135
ssh-copy-id root@192.168.1.134
ssh-copy-id root@192.168.1.113
ssh 192.168.1.134 date
ssh 192.168.1.135 date
ssh 192.168.1.113 date
3.3.机器C:192.168.1.113 上操作
su - root
cd ~
ssh-keygen -t rsa
ssh-copy-id -i root@192.168.1.113
ssh-copy-id -i root@192.168.1.134
ssh-copy-id -i root@192.168.1.135
ssh 192.168.1.134 date
ssh 192.168.1.135 date
ssh 192.168.1.113 date
#####################################部署zookeeper#################################
高可用需要使用到zookeeper
部署参考如下链接
https://www.cnblogs.com/hxlasky/p/23477829
#####################################部署flink集群#################################
1.安装java环境
[root@localhost conf]# java -version
java version "17.0.20.1" 2026-08-18 LTS
Java(TM) SE Runtime Environment (build 17.0.20.1+1-LTS-4)
Java HotSpot(TM) 64-Bit Server VM (build 17.0.20.1+1-LTS-4, mixed mode, sharing)
2.下载flink
下载地址
https://archive.apache.org/dist/flink
3.安装
在其中一个节点上安装,我这里是在192.168.1.134机器上安装
[root@localhost soft]#cd /soft
[root@localhost soft]#tar -xvf flink-1.20.5-bin-scala_2.12.tgz
[root@localhost soft]#mv flink-1.20.5 /opt/
4.采用minio做元数据存储
4.1 下载依赖包
cd /opt/flink-1.20.5/lib
wget https://repo1.maven.org/maven2/org/apache/flink/flink-s3-fs-hadoop/1.20.5/flink-s3-fs-hadoop-1.20.5.jar
wget https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/3.3.4/hadoop-aws-3.3.4.jar
scp flink-s3-fs-hadoop-1.20.5.jar root@192.168.1.135:/opt/flink-1.20.5/lib/
scp flink-s3-fs-hadoop-1.20.5.jar root@192.168.1.113:/opt/flink-1.20.5/lib/
scp hadoop-aws-3.3.4.jar root@192.168.1.135:/opt/flink-1.20.5/lib/
scp hadoop-aws-3.3.4.jar root@192.168.1.113:/opt/flink-1.20.5/lib/
4.2 需要部署minio
可参考如下部署链接
https://www.cnblogs.com/hxlasky/p/18643317
4.3 minio创建bucket
flink-ha-20
5.修改配置
vi /opt/flink-1.20.5/conf/config.yaml
该文件注释的配置项太多,只修改如下配置项即可
下面以节点1(192.168.1.134)为例子,其他节点需要修改相应的IP
s3.endpoint: http://192.168.12.87:9000
s3.access-key: admin
s3.secret-key: admin@123
s3.path-style-access: true
jobmanager:
bind-host: 0.0.0.0
rpc:
address: 192.168.1.134
port: 6123
memory:
process:
size: 1600m
execution:
failover-strategy: region
taskmanager:
bind-host: 0.0.0.0
host: 192.168.1.134
numberOfTaskSlots: 2
memory:
process:
size: 1728m
parallelism:
default: 1
high-availability:
type: zookeeper
storageDir: s3://flink-ha-20/ha-storage/
cluster-id: cluster_two
zookeeper:
quorum: 192.168.1.134:2182,192.168.1.135:2182,192.168.1.113:2182
path.root: /flink20
execution:
checkpointing:
interval: 30s
externalized-checkpoint-retention: RETAIN_ON_CANCELLATION
max-concurrent-checkpoints: 1
mode: EXACTLY_ONCE
timeout: 10min
state:
backend:
type: rocksdb
incremental: true
checkpoints:
dir: s3://flink-ha-20/checkpoints/
savepoints:
dir: s3://flink-ha-20/savepoints/
rest:
address: 0.0.0.0
port: 8081
红色部分在另外2个节点根据相应的ip进行修改
6.修改配置文件masters
vi /opt/flink-1.20.5/conf/masters
192.168.1.134:8081
192.168.1.135:8081
这里写2个ip,组成一主一备,也可以写3个ip,组成1主2从
192.168.1.134:8081
192.168.1.135:8081
192.168.1.113:8081
7.修改配置文件workers
vi /opt/flink-1.20.5/conf/workers
192.168.1.134
192.168.1.135
192.168.1.113
5.将整个目录拷贝到另外两个节点
cd /opt/
scp -r flink-1.20.5 root@192.168.1.135:/opt/
scp -r flink-1.20.5 root@192.168.1.113:/opt/
6.修改另外2个节点的配置文件
vi /opt/flink-1.20.5/conf/config.yaml
将相应的ip地址修改为本机的IP地址
6.启动集群
启动方式1:start-cluster.sh
启动，这里启动命令只在一个master节点节点上执行即可,他会自动去启动其他节点的JM和TM服务的
/opt/flink-1.20.5/bin/start-cluster.sh
[root@flink-134 log]# /opt/flink-1.20.5/bin/start-cluster.sh
Starting HA cluster with 2 masters.
Starting standalonesession daemon on host flink-134.
Starting standalonesession daemon on host flink-135.
^CKilled by signal 2.
Starting taskexecutor daemon on host flink-134.
Starting taskexecutor daemon on host flink-135.
^CKilled by signal 2.
Starting taskexecutor daemon on host flink-113.
这里需要输入2次 Ctrl+c 才能全部启动完成
停止命令
[root@flink-134 log]# /opt/flink-1.20.5/bin/stop-cluster.sh
Stopping taskexecutor daemon (pid: 24161) on host flink-134.
Stopping taskexecutor daemon (pid: 15046) on host flink-135.
Stopping taskexecutor daemon (pid: 11524) on host flink-113.
Stopping standalonesession daemon (pid: 22773) on host flink-134.
Stopping standalonesession daemon (pid: 7052) on host flink-135.
启动方式2:jobmanager.sh
192.168.1.134启动JM
/opt/flink-1.20.5/bin/jobmanager.sh start 192.168.1.134 8081
192.168.1.135启动JM
/opt/flink-1.20.5/bin/jobmanager.sh start 192.168.1.135 8081
停止JM:
192.168.1.134
/opt/flink-1.20.5/bin/jobmanager.sh stop 192.168.1.134 8081
192.168.1.135
/opt/flink-1.20.5/bin/jobmanager.sh stop 192.168.1.135 8081
在192.168.1.134,192.168.1.135,192.168.1.113 三台分别启动 TM
启动TM
/opt/flink-1.20.5/bin/taskmanager.sh start
停止TM
/opt/flink-1.20.5/bin/taskmanager.sh stop
7.验证集群状态
192.168.1.134
[root@localhost opt]# jps
10816 TaskManagerRunner
10206 StandaloneSessionClusterEntrypoint
192.168.1.135
[root@host135 conf]# jps
25701 StandaloneSessionClusterEntrypoint
29341 TaskManagerRunner
192.168.1.113
[root@host113 opt]# jps
18239 TaskManagerRunner
8.WebUI 访问
任意一个 JM 地址：
http://192.168.1.134:8081
http://192.168.1.135:8081
浙公网安备 33010602011771号