ubuntu 24.04.4，使用kubeadm + kube-vip 在线部署三个master节点的高可用k8s v1.34.10集群

系统，ubuntu 24.04.4
集群规划（3 台服务器）：

主机名：k8s-master-01，内网ip：172.29.128.1

主机名：k8s-worker-01，内网ip：172.29.132.68

主机名：k8s-worker-02，内网ip：172.29.132.69
由于服务器有限，把这三台服务器既当master节点，又当工作节点，部署三个master节点的高可用k8s集群。

节点清单：

节点	内网 IP	角色
k8s-master-01	172.29.128.1	Master01+Worker
k8s-worker-01	172.29.132.68	Master02+Worker
k8s-worker-02	172.29.132.69	Master03+Worker

 

 

 

 

 

一、所有 3 台节点统一前置初始化（全部执行）

1.1 主机名 & hosts 解析

分别设置主机名

# 节点1执行
hostnamectl set-hostname k8s-master-01
# 节点2执行
hostnamectl set-hostname k8s-worker-01
# 节点3执行
hostnamectl set-hostname k8s-worker-02

所有节点写入 /etc/hosts

cat >> /etc/hosts <<EOF
172.29.128.1    k8s-master-01
172.29.132.68   k8s-worker-01
172.29.132.69   k8s-worker-02
172.29.128.100  k8s-apiserver-vip
EOF

1.2 系统内核配置、关闭 swap、防火墙

# 关闭永久swap
swapoff -a
sed -i '/swap/s/^/#/' /etc/fstab

#关闭 ufw防火墙
systemctl stop ufw
systemctl disable ufw

# 内核转发、iptables桥接
cat >> /etc/sysctl.conf <<EOF
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
#配置ARP广播必需参数
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.all.rp_filter = 2
net.ipv4.ip_nonlocal_bind = 1
# 替换你的网卡名
net.ipv4.conf.ens3.arp_ignore = 1
net.ipv4.conf.ens3.arp_announce = 2
net.ipv4.conf.ens3.rp_filter = 2

EOF
sysctl -p

# 加载overlay/iptables内核模块
modprobe overlay
modprobe br_netfilter

 

1.3. 时间同步

apt install chrony -y
systemctl enable --now chronyd
timedatectl set-timezone Asia/Shanghai

 

二，安装 containerd + kubeadm(所有节点执行)

1, 安装containerd

# 1. 重新生成密钥+阿里源
curl -fsSL https://mirrors.aliyun.com/docker-ce/linux/ubuntu/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
chmod 644 /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://mirrors.aliyun.com/docker-ce/linux/ubuntu $(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null

# 2. 更新源（不会再出现download.docker.com报错）
apt update
#安装containerd
apt install -y containerd.io
# 生成默认配置
containerd config default | tee /etc/containerd/config.toml

# 修改cgroup驱动为systemd + 配置国内pause镜像
sed -i 's/SystemdCgroup \= false/SystemdCgroup \= true/g' /etc/containerd/config.toml
sed -i 's#sandbox_image = "registry.k8s.io/pause:3.10"#sandbox_image = "registry.ustc.edu.cn/k8s/pause:3.10"#' /etc/containerd/config.toml

# 重启生效
systemctl daemon-reload
systemctl restart containerd
systemctl enable --now containerd

 

2, 添加 k8s -v1.34.10官方源

# 导入k8s密钥
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.34/deb/Release.key | gpg --dearmor -o /usr/share/keyrings/kubernetes-apt-keyring.gpg
# 写入apt源
echo 'deb [signed-by=/usr/share/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.34/deb/ /' | tee /etc/apt/sources.list.d/kubernetes.list
apt update

查看k8s可用版本

# 查询kubeadm所有可用版本
apt-cache madison kubeadm
# kubectl
apt-cache madison kubectl
# kubelet
apt-cache madison kubelet

输出：

 apt-cache madison kubeadm
   kubeadm | 1.34.10-1.1 | https://pkgs.k8s.io/core:/stable:/v1.34/deb  Packages
   kubeadm | 1.34.9-1.1 | https://pkgs.k8s.io/core:/stable:/v1.34/deb  Packages
   kubeadm | 1.34.8-1.1 | https://pkgs.k8s.io/core:/stable:/v1.34/deb  Packages
   kubeadm | 1.34.7-1.1 | https://pkgs.k8s.io/core:/stable:/v1.34/deb  Packages
   kubeadm | 1.34.6-1.1 | https://pkgs.k8s.io/core:/stable:/v1.34/deb  Packages
   kubeadm | 1.34.5-1.1 | https://pkgs.k8s.io/core:/stable:/v1.34/deb  Packages
   kubeadm | 1.34.4-1.1 | https://pkgs.k8s.io/core:/stable:/v1.34/deb  Packages
   kubeadm | 1.34.3-1.1 | https://pkgs.k8s.io/core:/stable:/v1.34/deb  Packages
   kubeadm | 1.34.2-1.1 | https://pkgs.k8s.io/core:/stable:/v1.34/deb  Packages
   kubeadm | 1.34.1-1.1 | https://pkgs.k8s.io/core:/stable:/v1.34/deb  Packages
   kubeadm | 1.34.0-1.1 | https://pkgs.k8s.io/core:/stable:/v1.34/deb  Packages

 

3, 安装kubeadm、kubelet、kubectl

# 锁定指定版本v1.34.10
K8S_VERSION="1.34.10-1.1"
apt install -y kubelet=${K8S_VERSION} kubeadm=${K8S_VERSION} kubectl=${K8S_VERSION}
# 锁定版本防止自动升级
apt-mark hold kubelet kubeadm kubectl
# 校验版本
kubeadm version
kubectl version --client

 

三，部署 kube-vip 静态 Pod（所有 3 台 master 节点执行）

kube-vip 以静态 Pod方式运行在每台控制平面节点，接管 VIP 172.29.128.100，实现 APIServer 高可用

1, 预拉取 kube-vip 镜像（三台都执行）

# kube-vip最新稳定版适配k8s 1.34
ctr images pull ghcr.io/kube-vip/kube-vip:v1.2.2

 校验镜像是否存在

ctr images list
ctr images list | grep kube-vip

 

2, 生成 kube-vip配置文件（仅 master-01 执行）mkdir -p /etc/kubernetes/manifests

 

export VIP=172.29.128.100 #vip地址
export INTERFACE=eth0   #填你的真实网卡名
ctr run --rm --net-host ghcr.io/kube-vip/kube-vip:v1.2.2 kv-gen /kube-vip manifest pod \
--interface $INTERFACE \
--address $VIP \
--controlplane \
--services \
--arp \
--enableLoadBalancer \
--leaderElection | sudo tee /etc/kubernetes/manifests/kube-vip.yaml

  

生成的文件内容如下：

apiVersion: v1
kind: Pod
metadata:
  name: kube-vip
  namespace: kube-system
spec:
  containers:
  - args:
    - manager
    env:
    - name: vip_arp
      value: "true"
    - name: port
      value: "6443"
    - name: vip_nodename
      valueFrom:
        fieldRef:
          fieldPath: spec.nodeName
    - name: vip_interface
      value: eth0
    - name: vip_subnet
      value: "32"
    - name: dns_mode
      value: first
    - name: dhcp_mode
      value: ipv4
    - name: cp_enable
      value: "true"
    - name: cp_namespace
      value: kube-system
    - name: svc_enable
      value: "true"
    - name: svc_leasename
      value: plndr-svcs-lock
    - name: vip_leaderelection
      value: "true"
    - name: vip_leasename
      value: plndr-cp-lock
    - name: vip_leaseduration
      value: "15"
    - name: vip_renewdeadline
      value: "10"
    - name: vip_retryperiod
      value: "2"
    - name: lb_enable
      value: "true"
    - name: lb_port
      value: "6443"
    - name: lb_fwdmethod
      value: local
    - name: address
      value: 172.29.128.100
    - name: prometheus_server
      value: :2112
    image: ghcr.io/kube-vip/kube-vip:v1.2.2
    imagePullPolicy: IfNotPresent
    name: kube-vip
    resources: {}
    securityContext:
      capabilities:
        add:
        - NET_ADMIN
        - NET_RAW
        drop:
        - ALL
    volumeMounts:
    - mountPath: /etc/kubernetes/admin.conf
      name: kubeconfig
  hostAliases:
  - hostnames:
    - kubernetes
    ip: 127.0.0.1
  hostNetwork: true
  volumes:
  - hostPath:
      path: /etc/kubernetes/admin.conf
    name: kubeconfig
status: {}

 

3, 修改 /etc/kubernetes/manifests/kube-vip.yaml 文件的配置：

把 /etc/kubernetes/admin.conf 修改 为 /etc/kubernetes/super-admin.conf

 

4, 分发 kube-vip.yaml 到另外几台 master 节点

# master-01执行，替换对应IP
scp /etc/kubernetes/manifests/kube-vip.yaml root@172.29.132.68:/etc/kubernetes/manifests/
scp /etc/kubernetes/manifests/kube-vip.yaml root@172.29.132.69:/etc/kubernetes/manifests/

注意：每台机器的网卡名不一致时，分别修改三台 yaml 内vip_interface字段。

spec:
  containers:
  - args:
    - manager
    env:
    - name: vip_arp
      value: "true"
    - name: port
      value: "6443"
    - name: vip_nodename
      valueFrom:
        fieldRef:
          fieldPath: spec.nodeName
    - name: vip_interface
      value: eth0  #修改为三台机器真实的网卡名称

 

四，初始化master节点

1, 执行镜像预拉取（所有节点执行）：

# 源镜像仓库（可替换阿里云/华为/本地私有仓）
SOURCE_REPO="registry.aliyuncs.com/google_containers"
# K8s标准目标仓库固定值
K8S_REPO="registry.k8s.io"
# 版本变量统一管理
K8S_VER="v1.34.10"
COREDNS_VER="v1.12.1"
ETCD_VER="3.6.5-0"
PAUSE_VER="3.10.1"

# 预先拉取镜像（单行命令）
kubeadm config images pull --kubernetes-version ${K8S_VER} --image-repository ${SOURCE_REPO}

# 逐条打tag 4个控制平面镜像
nerdctl tag ${SOURCE_REPO}/kube-apiserver:${K8S_VER}        ${K8S_REPO}/kube-apiserver:${K8S_VER}
nerdctl tag ${SOURCE_REPO}/kube-controller-manager:${K8S_VER} ${K8S_REPO}/kube-controller-manager:${K8S_VER}
nerdctl tag ${SOURCE_REPO}/kube-scheduler:${K8S_VER}        ${K8S_REPO}/kube-scheduler:${K8S_VER}
nerdctl tag ${SOURCE_REPO}/kube-proxy:${K8S_VER}            ${K8S_REPO}/kube-proxy:${K8S_VER}

# 特殊路径镜像单独处理
nerdctl tag ${SOURCE_REPO}/coredns:${COREDNS_VER}  ${K8S_REPO}/coredns/coredns:${COREDNS_VER}
nerdctl tag ${SOURCE_REPO}/pause:${PAUSE_VER}      ${K8S_REPO}/pause:${PAUSE_VER}
nerdctl tag ${SOURCE_REPO}/etcd:${ETCD_VER}        ${K8S_REPO}/etcd:${ETCD_VER}

 

--image-repository 备选仓库：

阿里云：registry.aliyuncs.com/google_containers
腾讯云：mirror.ccs.tencentyun.com/k8s
华为云：repo.huaweicloud.com/kubernetes
道客云：k8s.m.daocloud.io 

 

2, 集群初始化文件准备

查看kind默认配置

kubeadm config print init-defaults --component-configs KubeProxyConfiguration

生成配置文件样例 kubeadm-config.yaml

kubeadm config print init-defaults --component-configs KubeProxyConfiguration > kubeadm-init.yaml

修改kubeadm-init.yaml的几个配置（必须） 

#1. advertiseAddress: 1.2.3.4  修改成你自己master节点ip
advertiseAddress: 172.29.128.1
#2. nodeRegistration下面的name: node 修改成master节点名称
nodeRegistration:
  name: k8s-master-01
#3. apiServer:{}  ，先删掉左右大括号{}，再添加所有master节点名称及ip、vip的ip或域名。（工作节点不需要添加）
apiServer:
  certSANs:
  - 172.29.128.100
  - k8s-master-01
  - k8s-worker-01
  - k8s-worder-02
  - 172.29.128.1
  - 172.29.132.68
  - 172.29.132.69
#4. 在apiVersion的前面增加controlPlaneEndpoint配置，填vip的ip地址（或域名）及端口
apiServer:
  certSANs:
  - 172.29.128.100
  - k8s-master-01
  - k8s-worker-01
  - k8s-worder-02
  - 172.29.128.1
  - 172.29.132.68
  - 172.29.132.69
controlPlaneEndpoint: 172.29.128.100:6443
apiVersion: kubeadm.k8s.io/v1beta4

#5. 在networking下增加子节点 podSubnet: 10.244.0.0/16或者podSubnet: 192.168.0.0/16
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
  podSubnet: 192.168.0.0/16
#6. ipvs下面的strictARP: false  改成 strictARP: true
ipvs:
  excludeCIDRs: null
  minSyncPeriod: 0s
  scheduler: ""
  strictARP: true
  syncPeriod: 0s
  tcpFinTimeout: 0s
  tcpTimeout: 0s
  udpTimeout: 0s

#7. nftables: 节点上面的mode: ""  改成 mode: "ipvs"
metricsBindAddress: ""
mode: "ipvs"
nftables:
  masqueradeAll: false
  masqueradeBit: null
  minSyncPeriod: 0s
  syncPeriod: 0s
#8. kubernetesVersion: 1.xx.x 版本号改成你要安装的版本
kubernetesVersion: 1.34.10

 

2，初始化master节点

用kubeadm-init.yaml 文件初始化集群

kubeadm init --config kubeadm-init.yaml --upload-certs --v=9

 

用纯命令初始化主master节点

kubeadm init \
  --kubernetes-version=v1.34.10 \
  --control-plane-endpoint=172.29.128.100:6443 \
  --apiserver-advertise-address=172.29.128.1 \
  --pod-network-cidr=192.168.0.0/16 \
  --service-cidr=10.96.0.0/12 \
  --upload-certs \
  --cri-socket=/run/containerd/containerd.sock

参数说明：

• --control-plane-endpoint：kube-vip VIP，集群统一接入地址
• --upload-certs：将控制平面证书上传到 Secret，后续节点 join control-plane 免拷贝证书
• `pod 网段 Calico 标准 192.168.0.0/16

master节点初始化成功后，输出：

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of control-plane nodes running the following command on each as root:

  kubeadm join 172.29.128.100:6443 --token abcdef.0123456789abcdef \
        --discovery-token-ca-cert-hash sha256:2ffb0cb00bae4888b350db6e2ad2438ccec120b5c9955d450c6d1c74424f1fa2 \
        --control-plane --certificate-key 8154e310eafe6f2531a681bf0d8c2e6f586def63dd6b1d6116268840d17002ff

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 172.29.128.100:6443 --token abcdef.0123456789abcdef \
        --discovery-token-ca-cert-hash sha256:2ffb0cb00bae4888b350db6e2ad2438ccec120b5c9955d450c6d1c74424f1fa2

记录输出的 2 条 join 命令（保存备用）

 

3, 拷贝 kubeconfig，添加普通用户执行权限

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

 

4,查看三台master节点有没有VIP

查看pod

kubectl get pod -A

会看到有kube-vip-开头的三条pod记录，状态全部为Running，并且Ready全部为1/1，就表示kube-vip静态pod启动成功。

kube-system   kube-vip-k8s-master-01                     1/1     Running   0 (66s ago)     4m38s
kube-system   kube-vip-k8s-worker-01                     1/1     Running   0 (2m54s ago)   18m
kube-system   kube-vip-k8s-worker-02                     1/1     Running   0 (2m50s ago)   18m

逐个在三台master节点校验eth0网卡有没有VIP

ip a s
ip a s | grep 172.29.128.100

如果vip创建成功，在其中一台的ip a s  命令会输出含有 172.29.128.100/32 的信息：

eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:16:3e:3c:a2:4a brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname ens5
    inet 172.29.128.1/20 metric 100 brd 172.29.143.255 scope global dynamic eth0
       valid_lft 1891831497sec preferred_lft 1891831497sec
    inet 172.29.128.100/32 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fe3c:a24a/64 scope link
       valid_lft forever preferred_lft forever

如果没有 172.29.128.100的输出信息，说明VIP创建失败，集群不可用。只有VIP创建成功后，才可以继续下一步加入其它控制面板。

 

5, 加入其它master节点与工作节点
(1) 另外两台加入控制平面（master 节点）：

kubeadm join 172.29.128.100:6443 \
--token xxx \
--discovery-token-ca-cert-hash sha256:xxxx \
--control-plane \
--certificate-key xxxxx \
--apiserver-advertise-address=172.29.132.68  # 本机内网IP，第二台替换172.29.132.69

 

（2）新 master 节点拷贝 kubeconfig（每台新增 master 执行）

mkdir -p $HOME/.kube
sudo scp root@172.29.128.1:/etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

 

(3) 加入 worker 节点：

kubeadm join 172.29.128.100:6443 --token xxx --discovery-token-ca-cert-hash sha256:xxx -v 9

 

(4) 验证节点数量

kubectl get nodes
# 三台全部为control-plane:True，master污点默认存在
kubectl describe node k8s-master-01 | grep Taints

(5) ：移除 master 节点污点，允许调度 Pod（ALL-IN-ONE 混合节点关键）

# 三台节点分别执行，或在master-01批量执行
kubectl taint nodes k8s-master-01 node-role.kubernetes.io/control-plane:NoSchedule-
kubectl taint nodes k8s-worker-01 node-role.kubernetes.io/control-plane:NoSchedule-
kubectl taint nodes k8s-worker-02 node-role.kubernetes.io/control-plane:NoSchedule-

或者一次性清除所有master节点污点

kubectl taint nodes --all node-role.kubernetes.io/control-plane- 

 

6, token过期重置 

token 过期重置：

kubeadm token create --print-join-command

证书 key 过期重置：

kubeadm init phase upload-certs --upload-certs

 

五，安装 calico网络插件

#下载calico官方配置文件（适合k8s v1.34.10的版本）
curl -O calico-v3.32.1.yaml https://raw.githubusercontent.com/projectcalico/calico/v3.32.1/manifests/calico.yaml
#替换为国内daocloud镜像源
sed -i 's|quay.io/calico|m.daocloud.io/quay.io/calico|g' calico-v3.32.1.yaml
#安装
kubectl apply -f calico-v3.32.1.yaml
# 等待calico全部pod就绪
kubectl get pods -n kube-system -w
# 全部ready后节点状态变为Ready
kubectl get nodes

查看calico健康状态

kubectl get pod -A -n kube-system -w

 

集群健康检查

kubectl get cs
kubectl get nodes
# 三台control-plane+worker全部Ready

 

六，kube-vip VIP高可用校验

1，验证VIP是否有漂移

查看 VIP 在哪台节点：ip addr | grep 172.29.128.100
关闭当前 leader 节点 kube-vip 容器，观察 VIP 是否有自动漂移至另外一台

# 关闭kube-vip静态pod（删除yaml，kubelet自动销毁）
mv /etc/kubernetes/manifests/kube-vip.yaml /tmp/
# 等待10s，再移回，模拟故障
mv /tmp/kube-vip.yaml /etc/kubernetes/manifests/

 

2，API Server 连通性

# 任意节点通过VIP访问集群
kubectl --server=https://172.29.128.100:6443 get nodes

 

七，卸载kube-vip（每台master节点执行）

1，如果要删除kube-vip，就删掉 /etc/kubernetes/manifests/kube-vip.yaml ，然后重启kubelet

# 删除静态清单文件
rm -f /etc/kubernetes/manifests/kube-vip.yaml
# 重启kubelet，立刻销毁kube-vip容器
systemctl restart kubelet

 2. 手动清除宿主机残留 VIP 地址

kube-vip 退出后 VIP 可能残留在网卡，手动清理：

VIP="172.29.128.100"
INTERFACE="eth0"
ip addr del ${VIP}/32 dev ${INTERFACE}
# 确认VIP已移除
ip a | grep ${VIP}

3. 清理本地镜像（可选，保留镜像可跳过）

ctr images rm ghcr.io/kube-vip/kube-vip:v1.2.2