ubuntu 24.04.4，使用kubeadm + (HaProxy + Keepalived) 在线部署三个master节点的高可用k8s v1.34.10集群

系统，ubuntu 24.04.4
集群规划（3 台服务器）：

主机名：k8s-master-01，内网ip：172.29.128.1

主机名：k8s-worker-01，内网ip：172.29.132.68

主机名：k8s-worker-02，内网ip：172.29.132.69
由于服务器有限，把这三台服务器既当master节点，又当工作节点，部署三个master节点的高可用k8s集群。

节点清单：

节点 内网 IP 角色 
k8s-master-01 172.29.128.1 Master01+Worker + HaProxy + keepalived
k8s-worker-01 172.29.132.68 Master02+Worker + HaProxy + keepalived
k8s-worker-02 172.29.132.69 Master03+Worker + HaProxy + keepalived

 

 

 

 

 

一、所有 3 台节点统一前置初始化（全部执行）

1.1 主机名 & hosts 解析

分别设置主机名

# 节点1执行
hostnamectl set-hostname k8s-master-01
# 节点2执行
hostnamectl set-hostname k8s-worker-01
# 节点3执行
hostnamectl set-hostname k8s-worker-02

所有节点写入 /etc/hosts

cat >> /etc/hosts <<EOF
172.29.128.1    k8s-master-01
172.29.132.68   k8s-worker-01
172.29.132.69   k8s-worker-02
172.29.128.100  k8s-apiserver-vip
EOF

1.2 系统内核配置、关闭 swap、防火墙

 

1.3. 时间同步

apt install chrony -y
systemctl enable --now chronyd
timedatectl set-timezone Asia/Shanghai

 

二，三台节点部署 HAProxy+Keepalived（VIP 负载均衡层）

2.1 所有节点安装组件

 

apt install -y haproxy keepalived

 

 2.2 Keepalived 配置（区分 MASTER/BACKUP）

 节点 k8s-master-01（主 VIP 节点，优先级 100）

cat > /etc/keepalived/keepalived.conf <<'EOF'
# Keepalived 多master apiserver VIP漂移配置
global_defs {
   router_id LVS-k8s-master-01  # 节点唯一标识，三台不能重复
}
vrrp_instance VI_1 {
    state MASTER                # 主节点；备机改为BACKUP
    interface eth0              # 内网业务网卡
    virtual_router_id 51        # 三台集群必须统一
    priority 100                # 优先级，数字越大为主，备机90/80
    advert_int 1                # VRRP报文发送间隔1秒
    unicast_peer {
        172.29.132.68
        172.29.132.69
    }
    authentication {
        auth_type PASS
        auth_pass k8svip99      # 密码1-8字符，三台完全一致
    }
    virtual_ipaddress {
        # 业务VIP1 apiserver访问地址
        172.29.128.200
        # 备用VIP2
        172.29.128.201
    }
}
EOF

 

节点 k8s-worker-01（备节点，优先级 90）

cat > /etc/keepalived/keepalived.conf <<'EOF'
global_defs {
   router_id LVS-k8s-worker-01
}
vrrp_instance VI_1 {
    state BACKUP      #备节点用BACKUP
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    nopreempt         #主节点恢复后不抢占vip，避免断流
    unicast_peer {
        172.29.128.1
        172.29.132.69
    }
    authentication {
        auth_type PASS
        auth_pass k8svip99
    }
    virtual_ipaddress {
        172.29.128.200
        172.29.128.201
    }
}
EOF

 

节点 k8s-worker-02（备节点，优先级 80）

cat > /etc/keepalived/keepalived.conf <<'EOF'
global_defs {
   router_id LVS-k8s-worker-02
}
vrrp_instance VI_1 {
    state BACKUP     #备节点用BACKUP
    interface eth0    #你真实的物理网卡（ip addr查看）
    virtual_router_id 51
    priority 80
    advert_int 1
    nopreempt
    unicast_peer {
        172.29.128.1
        172.29.132.68
    }
    authentication {
        auth_type PASS
        auth_pass k8svip99  #密码长度1到8位
    }
    virtual_ipaddress {
        172.29.128.200
        172.29.128.201
    }
}
EOF

 

参数说明：

router_id可以同名，但尽量不要同名，看日志不好排查问题。

auth_pass密码长度不能超过8位，超过会截断，会出现诡异的问题。

virtual_ipaddress里的ip地址规则：

  • 必须落在 eth0 的大网段 172.29.128.0/20 内 
    允许区间：172.29.128.1 ~ 172.29.143.254
  • IP 必须空闲未占用：不能是 master 节点 IP、网关、其他业务服务器 IP
  • 禁止使用：网段首地址 .0、广播地址 .255
  • 禁止跨网段：不能写 172.29.127.x/ 172.29.144.x

unicast_peer ：里面的ip不能添加本机的ip，否则会自环发包，产生冗余告警，无效心跳、轻微性能损耗，判定为异常冲突，持续打印警告日志刷屏。

分段拆解含义：
(1)  dev eth0
含义：把这个 VIP 绑定在 eth0 这块物理网卡上。

(2) 一块网卡可以写多行，绑定多个 VIP。

virtual_ipaddress {
    172.29.128.200
    172.29.128.201
}

 

校验 + 重启keepalived

# 校验配置语法是否正确
keepalived -t -c /etc/keepalived/keepalived.conf
# 重启服务生效
systemctl restart keepalived
# 设置开机自启
systemctl enable keepalived
#看keepalived实时日志
journalctl -u keepalived -f

#停止keepalived

systemctl stop keepalived

 

3, 查看vip在哪个节点

分别在三台keepalived节点执行命令：

ip a s

会出现的三种情况，在真实网卡（eth0 / ens33）下面的信息：

（1）如果三台全部没有输出带 172.29.128.200的ip地址，表示keepalived启动失败。

（2）如果只有一台输出带 172.29.128.200的ip地址，表示keepalived工作正常。

（3）如果有两台或多台输出带172.29.128.200的 ip地址，说明脑裂，表示节点相互不通，各自都启动一台vip。

输出示例：

root@k8s-master-01:~# ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:16:3e:3c:a2:4a brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname ens5
    inet 172.29.128.1/20 metric 100 brd 172.29.143.255 scope global dynamic eth0
       valid_lft 1891909023sec preferred_lft 1891909023sec
    inet 172.29.128.200/32 scope global eth0:vip
       valid_lft forever preferred_lft forever
    inet 172.29.128.201/32 scope global eth0:vip2
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fe3c:a24a/64 scope link
       valid_lft forever preferred_lft forever

 

2.3 全部节点 HAProxy 配置 /etc/haproxy/haproxy.cfg

追加到文件末尾，负载均衡用6444端口（三台master节点全部执行）
#================ K8s APISERVER TCP负载均衡 6444 =================
frontend k8s-api
    #不要用6443端口，因为k8s apiserver使用6443
    bind 0.0.0.0:6444
    mode tcp
    option tcplog
    timeout client 600000
    default_backend k8s-master-api

backend k8s-master-api
    mode tcp
    balance roundrobin
    timeout server 600000
    #转发k8s apiserver的6443端口 
    server k8s-master-01 172.29.128.1:6443 check inter 3000 rise 2 fall 3
    server k8s-worker-01 172.29.132.68:6443 check inter 3000 rise 2 fall 3
    server k8s-worker-02 172.29.132.69:6443 check inter 3000 rise 2 fall 3

listen stats
    bind 0.0.0.0:8404
    stats enable
    stats auth admin:admin2026
    stats hide-version
    stats refresh 5s

 

校验语法 + 启动自启haproxy

# 校验配置合法性
haproxy -c -f /etc/haproxy/haproxy.cfg
# 重启并开机自启
systemctl restart haproxy
systemctl enable --now haproxy
# 验证6444端口监听
ss -ltnp | grep 6444
ss -ltnp | grep haproxy
# 查看haproxy实时日志
journalctl -u keepalived -f

 

#停止keepalived

systemctl stop haproxy

 

三，安装 containerd + kubeadm(所有节点执行)

1, 安装containerd

# 1. 全局删除所有 download.docker.com 配置
sed -i '/download.docker.com/d' /etc/apt/sources.list
rm -rf /etc/apt/sources.list.d/docker*
apt clean
rm -rf /var/lib/apt/lists/*download.docker.com*

# 2. 重新生成密钥+阿里源
curl -fsSL https://mirrors.aliyun.com/docker-ce/linux/ubuntu/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
chmod 644 /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://mirrors.aliyun.com/docker-ce/linux/ubuntu $(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null

# 3. 校验无官方docker地址
grep -r "download.docker.com" /etc/apt/

# 4. 更新源（不会再出现download.docker.com报错）
apt update
#安装containerd
apt install -y containerd.io
# 生成默认配置
containerd config default | tee /etc/containerd/config.toml

# 修改cgroup驱动为systemd + 配置国内pause镜像
sed -i 's/SystemdCgroup \= false/SystemdCgroup \= true/g' /etc/containerd/config.toml
sed -i 's#sandbox_image = "registry.k8s.io/pause:3.10"#sandbox_image = "registry.ustc.edu.cn/k8s/pause:3.10"#' /etc/containerd/config.toml

# 重启生效
systemctl daemon-reload
systemctl restart containerd
systemctl enable --now containerd

 

查看可用containerd版本（可以跳过）

apt-cache madison containerd.io

 

输出：

apt-cache madison containerd.io
containerd.io | 2.2.6-1~ubuntu.24.04~noble | https://download.docker.com/linux/ubuntu noble/stable amd64 Packages
containerd.io | 2.2.6-1~ubuntu.24.04~noble | https://mirrors.aliyun.com/docker-ce/linux/ubuntu noble/stable amd64 Packages
containerd.io | 2.2.5-1~ubuntu.24.04~noble | https://download.docker.com/linux/ubuntu noble/stable amd64 Packages
containerd.io | 2.2.5-1~ubuntu.24.04~noble | https://mirrors.aliyun.com/docker-ce/linux/ubuntu noble/stable amd64 Packages
containerd.io | 2.2.4-1~ubuntu.24.04~noble | https://download.docker.com/linux/ubuntu noble/stable amd64 Packages
containerd.io | 2.2.4-1~ubuntu.24.04~noble | https://mirrors.aliyun.com/docker-ce/linux/ubuntu noble/stable amd64 Packages
containerd.io | 2.2.3-1~ubuntu.24.04~noble | https://download.docker.com/linux/ubuntu noble/stable amd64 Packages

 

2，安装 kubeadm/kubelet/kubectl v1.34.10

# 导入k8s密钥
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.34/deb/Release.key | gpg --dearmor -o /usr/share/keyrings/kubernetes-apt-keyring.gpg
# 写入apt源
echo 'deb [signed-by=/usr/share/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.34/deb/ /' | tee /etc/apt/sources.list.d/kubernetes.list
apt update
# 锁定指定版本v1.34.10
K8S_VERSION="1.34.10-1.1"
apt install -y kubelet=${K8S_VERSION} kubeadm=${K8S_VERSION} kubectl=${K8S_VERSION}
# 锁定版本防止自动升级
apt-mark hold kubelet kubeadm kubectl
# 校验版本
kubeadm version
kubectl version --client

 

如果不知道源里有哪些版本，先查看可用版本

# 查询kubeadm所有可用版本
apt-cache madison kubeadm
# kubectl
apt-cache madison kubectl
# kubelet
apt-cache madison kubelet

输出：

 apt-cache madison kubeadm
   kubeadm | 1.34.10-1.1 | https://pkgs.k8s.io/core:/stable:/v1.34/deb  Packages
   kubeadm | 1.34.9-1.1 | https://pkgs.k8s.io/core:/stable:/v1.34/deb  Packages
   kubeadm | 1.34.8-1.1 | https://pkgs.k8s.io/core:/stable:/v1.34/deb  Packages
   kubeadm | 1.34.7-1.1 | https://pkgs.k8s.io/core:/stable:/v1.34/deb  Packages
   kubeadm | 1.34.6-1.1 | https://pkgs.k8s.io/core:/stable:/v1.34/deb  Packages
   kubeadm | 1.34.5-1.1 | https://pkgs.k8s.io/core:/stable:/v1.34/deb  Packages
   kubeadm | 1.34.4-1.1 | https://pkgs.k8s.io/core:/stable:/v1.34/deb  Packages
   kubeadm | 1.34.3-1.1 | https://pkgs.k8s.io/core:/stable:/v1.34/deb  Packages
   kubeadm | 1.34.2-1.1 | https://pkgs.k8s.io/core:/stable:/v1.34/deb  Packages
   kubeadm | 1.34.1-1.1 | https://pkgs.k8s.io/core:/stable:/v1.34/deb  Packages
   kubeadm | 1.34.0-1.1 | https://pkgs.k8s.io/core:/stable:/v1.34/deb  Packages

 

四，初始化第一个master节点（仅 k8s-master-01 执行）

1, 执行镜像预拉取（所有节点执行）：

# 源镜像仓库（可替换阿里云/华为/本地私有仓）
SOURCE_REPO="registry.aliyuncs.com/google_containers"
# K8s标准目标仓库固定值
K8S_REPO="registry.k8s.io"
# 版本变量统一管理
K8S_VER="v1.34.10"
COREDNS_VER="v1.12.1"
ETCD_VER="3.6.5-0"
PAUSE_VER="3.10.1"

# 预先拉取镜像（单行命令）
kubeadm config images pull --kubernetes-version ${K8S_VER} --image-repository ${SOURCE_REPO}

# 逐条打tag 4个控制平面镜像
nerdctl tag ${SOURCE_REPO}/kube-apiserver:${K8S_VER}        ${K8S_REPO}/kube-apiserver:${K8S_VER}
nerdctl tag ${SOURCE_REPO}/kube-controller-manager:${K8S_VER} ${K8S_REPO}/kube-controller-manager:${K8S_VER}
nerdctl tag ${SOURCE_REPO}/kube-scheduler:${K8S_VER}        ${K8S_REPO}/kube-scheduler:${K8S_VER}
nerdctl tag ${SOURCE_REPO}/kube-proxy:${K8S_VER}            ${K8S_REPO}/kube-proxy:${K8S_VER}

# 特殊路径镜像单独处理
nerdctl tag ${SOURCE_REPO}/coredns:${COREDNS_VER}  ${K8S_REPO}/coredns/coredns:${COREDNS_VER}
nerdctl tag ${SOURCE_REPO}/pause:${PAUSE_VER}      ${K8S_REPO}/pause:${PAUSE_VER}
nerdctl tag ${SOURCE_REPO}/etcd:${ETCD_VER}        ${K8S_REPO}/etcd:${ETCD_VER}

 

--image-repository 备选仓库：

阿里云：registry.aliyuncs.com/google_containers
腾讯云：mirror.ccs.tencentyun.com/k8s
华为云：repo.huaweicloud.com/kubernetes
道客云：k8s.m.daocloud.io 

 

2, 初始化第一个master节点

kubeadm init \
  --kubernetes-version v1.34.10 \
  --control-plane-endpoint "172.29.128.200:6444" \
  --apiserver-advertise-address=172.29.128.1 \
  --service-cidr=10.96.0.0/12 \
  --pod-network-cidr=192.168.0.0/16 \
  --upload-certs

 

参数说明

  1. --control-plane-endpoint：VIP 地址，所有节点统一接入点
  2. --control-plane-endpoint：第一个master节点的地址
  3. --pod-network-cidr=192.168.0.0/16：适配 Calico 网络插件，用10.244.0.0/16也行
  4. --image-repository：中科大镜像仓库，解决 registry.k8s.io 拉取慢
  5. --upload-certs：将控制平面证书上传集群 secret，其他 master 加入无需手动拷贝证书

初始化成功后输出：

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of control-plane nodes running the following command on each as root:

  kubeadm join 172.29.128.200:6444 --token rjaf1k.vg2n5uygawc8g0rr \
        --discovery-token-ca-cert-hash sha256:5c5242f461e93a87de6f8bf59fa8d9a9107baad6f0cd2a494aa143333d40bafc \
        --control-plane --certificate-key 22b273e5cdf206af3800682e4a38141c7d68d10234134ac71b87de3993c18765

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 172.29.128.200:6444 --token rjaf1k.vg2n5uygawc8g0rr \
        --discovery-token-ca-cert-hash sha256:5c5242f461e93a87de6f8bf59fa8d9a9107baad6f0cd2a494aa143333d40bafc

把上面两条kubeadm join命令保存下来，后加入控制面板与工作面板需需要使用。

3, 配置 kubectl 普通用户权限

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

查看集节点状态

kubectl get nodes
kubectl get pod -A

 

4, 把其它两个节点加入到 控制平面（带 cert-key，有效期 2 小时）

（1）其它master节点加入控制面板，要先停止节点本机的haproxy，加入成功后再启动haproxy

kubeadm join 172.29.128.200:6444 --token abcdef.0123456789abcdef \
  --discovery-token-ca-cert-hash sha256:xxxxxxxxxxxxxxxxxxxxxx \
  --control-plane --certificate-key xxxxxxxxxxxxxxxxxxxxxxx \
-v 6

加 -v 6 输出完整请求、镜像拉取、证书通信日志

 

（2）其它master节点也要配置 kubectl 普通用户权限（各自执行）

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

 

5, 移除 Master 节点污点（关键！让 master 节点可调度 Pod）

#移除某一台节点的污点
kubectl taint nodes --all node-role.kubernetes.io/control-plane- 
kubectl taint nodes --all node-role.kubernetes.io/master-
#一次性清除全部控制平面污点（无需写节点名，适配任意多 master）
kubectl taint nodes --all node-role.kubernetes.io/control-plane-

 

6, 加入 Worker 节点（在任意一台master节点执行）

kubeadm join 172.29.128.200:6444 --token abcdef.0123456789abcdef \
  --discovery-token-ca-cert-hash sha256:xxxxxxxxxxxxxxxxxxxxxx \
-v 6

 

7, 集群验证（在任意一台master节点执行）

kubectl get nodes
kubectl get pod -A
kubectl get cs

 

8, token过期重置 

token 过期重置：

kubeadm token create --print-join-command

证书 key 过期重置：

kubeadm init phase upload-certs --upload-certs

 

9,  查看 etcd 集群成员（3 节点堆叠 etcd）

kubectl exec -n kube-system etcd-k8s-master-01 -- etcdctl \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  member list
# 输出3个etcd节点ID，代表etcd三副本集群正常

 

五，安装 calico网络插件

#下载calico官方配置文件（适合k8s v1.34.10的版本）
curl -O calico-v3.32.1.yaml https://raw.githubusercontent.com/projectcalico/calico/v3.32.1/manifests/calico.yaml
#替换为国内daocloud镜像源
sed -i 's|quay.io/calico|m.daocloud.io/quay.io/calico|g' calico-v3.32.1.yaml
#安装
kubectl apply -f calico-v3.32.1.yaml
# 等待calico全部pod就绪
kubectl get pods -n kube-system -w
# 全部ready后节点状态变为Ready
kubectl get nodes

查看calico健康状态

kubectl get pod -A -n kube-system -w

 

集群健康检查

kubectl get cs
kubectl get nodes
# 三台control-plane+worker全部Ready

 

六 测试 VIP 高可用故障转移

1, 查看当前 VIP 在哪台节点 ip a
2, 停止对应节点 keepalived： 

systemctl stop keepalived

3, 验证VIP 是否有漂移至剩余优先级最高节点。

4，再验证执行 kubectl 命令是否正常。 

# 任意节点通过VIP访问集群
kubectl --server=https://172.29.128.100:6443 get nodes

 

posted @ 2026-08-03 16:24  民工黑猫  阅读(4)  评论(0)    收藏  举报