高可用 Kubernetes 集群部署：使用 kubeadm 安装 3 Master 2 Worker 的 Kubernetes 集群（LVS+Keepalived实现高可用）

碰到的难点：代理配置，Linux内核模块和参数，sed命令修改配置, systemctl daemon-reload的作用，网络拓扑

关键词： #Kubernetes 、 #部署手册

实验目标
实验环境
网络规划
安装步骤
验证

实验目标

使用 kubeadm 部署一个
3 Master + 2 Worker 的高可用 Kubernetes 集群（加上相关的两个LB调度器）。

实验环境

项目	配置
操作系统	Rocky Linux release 9.4 (Blue Onyx)
内核版本	5.14.0-427.13.1.el9_4.x86_64
CPU 架构	x86_64
Kubernetes	v1.32.2
Containerd	1.7.29
网络插件	Calico v3.29.2
容器网络 CIDR	10.244.0.0/16
Master 数量	1
Worker 数量	2

网络规划

192.168.250.0/24 是 nat 模式的虚拟网络 VMnet8

角色	主机名	IP 地址	配置
Master1	master-1	192.168.250.151	2C4G
Master2	master-2	192.168.250.152	2C4G
Master3	master-3	192.168.250.153	2C4G
Worker1	worker-1	192.168.250.154	2C4G
Worker2	worker-2	192.168.250.155	2C4G
LB1	lb-1	192.168.250.156	2C4G
LB2	lb-2	192.168.250.157	2C4G
VIP (api-server)		192.168.250.158

注：网关是 192.168.250.2

安装步骤

一、环境准备 ( Master , Worker 和 LB 都执行 )

1.1 配置静态 IP

如下是 master-1 节点的配置静态 IP 地址的流程，其他节点类似

cd /etc/NetworkManager/system-connections
sed -ri "1,/^method=/s/^(method=).*/\1manual/" ens160.nmconnection
sed -ri '0,/^method=/s|^method=.*|&\naddress=192.168.250.151/24,192.168.250.2\ndns=192.168.250.2|' ens160.nmconnection
nmcli c r
nmcli c u ens160

期望的配置 /etc/NetworkManager/system-connections/ens160.nmconnection

  ...
  [ipv4]
+ method=manual
- method=auto
+ address=192.168.250.151/24,192.168.250.2
+ dns=192.168.250.2
  ...

1.2 关闭防火墙和selinux

systemctl disable --now firewalld
setenforce 0
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config

1.3 分别配置主机名（每个节点执行的命令不同）

master-1

hostnamectl set-hostname master-1

master-2

hostnamectl set-hostname master-2

master-3

hostnamectl set-hostname master-3

worker-1

hostnamectl set-hostname worker-1

worker-2

hostnamectl set-hostname worker-2

lb-1

hostnamectl set-hostname lb-1

lb-2

hostnamectl set-hostname lb-2

1.4 配置hosts文件本地解析

cat >> /etc/hosts << EOF
192.168.250.151 master-1
192.168.250.152 master-2
192.168.250.153 master-3
192.168.250.154 worker-1
192.168.250.155 worker-2
192.168.250.156 lb-1
192.168.250.157 lb-2
192.168.250.158 api
EOF

二、环境准备 ( Master 和 Worker 都执行 )

2.1 关闭交换分区

swapoff -a
sed -i '/swap/d' /etc/fstab
free -h

2.2 修改内核参数

cat >> /etc/modules-load.d/k8s.conf <<EOF
br_netfilter
EOF
modprobe br_netfilter

cat >> /etc/sysctl.d/kubernetes.conf<<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF

cat >> /etc/sysctl.conf <<EOF
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
EOF

sysctl --system

2.3 配置时钟同步

dnf -y install chrony 
cat >> /etc/chrony.conf << EOF
server ntp.aliyun.com iburst
server ntp1.aliyun.com iburst
server ntp2.aliyun.com iburst
EOF
systemctl enable --now chronyd

2.4 安装容器运行时 Containerd

安装 containerd

dnf install -y yum-utils 
dnf config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
dnf -y install containerd.io-1.7.29

修改 containerd 配置文件
创建默认配置文件

cd /etc/containerd
mv config.toml config.toml.orig
containerd config default > /etc/containerd/config.toml

修改配置

SystemdCgroup=true
sandbox_image=registry.aliyuncs.com/google_containers/pause:3.8

sed -i "s#SystemdCgroup\ \=\ false#SystemdCgroup\ \=\ $SystemdCgroup#g" /etc/containerd/config.toml
sed -i "s#registry.k8s.io/pause:3.8#$sandbox_image#g" /etc/containerd/config.toml

注：期望的配置状态： /etc/containerd/config.toml

  ...
    [plugins."io.containerd.grpc.v1.cri"]
      ...
+     sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.8"
-     sandbox_image = "registry.k8s.io/pause:3.8"
  ...
            [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
              ...
+             SystemdCgroup = true
-             SystemdCgroup = false
 ...

启动 containerd 服务

systemctl enable --now containerd
systemctl status containerd --no-pager
ctr version

2.5 安装 kubeadm 和 k8s 组件

cat >> /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.32/rpm/
enabled=1
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.32/rpm/repodata/repomd.xml.key
EOF
dnf install -y kubeadm-1.32.2 kubelet-1.32.2 kubectl-1.32.2
systemctl enable --now kubelet

三、环境准备 ( LB 都执行 )

3.1 安装 keepalived 和 ipvsadm

dnf -y install keepalived ipvsadm

3.2 修改内核参数

cat >> /etc/sysctl.conf <<EOF
net.ipv4.conf.all.send_redirects = 0
EOF

sysctl -p

3.3 指定 keepalived 配置

在 LB1 节点运行如下命令

cat > /etc/keepalived/keepalived.conf << EOF
global_defs {
   router_id LVS_DEVEL
}

vrrp_instance VI_1 {
    state MASTER
    interface ens160
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 123456
    }
    virtual_ipaddress {
        192.168.250.158/24
    }
}

virtual_server 192.168.250.158 6443 {
    delay_loop 6
    lb_algo rr                  
    lb_kind DR            
    protocol TCP

    real_server 192.168.250.151 6443 {
        TCP_CHECK {
            connect_timeout 3
        }
    }

    real_server 192.168.250.152 6443 {
        TCP_CHECK {
            connect_timeout 3
        }
    }

    real_server 192.168.250.153 6443 {
        TCP_CHECK {
            connect_timeout 3
        }
    }
}
EOF

在 LB2 节点运行如下命令

cat > /etc/keepalived/keepalived.conf << EOF
global_defs {
   router_id LVS_DEVEL
}

vrrp_instance VI_1 {
    state BACKUP
    interface ens160
    virtual_router_id 51
    priority 90
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 123456
    }
    virtual_ipaddress {
        192.168.250.158/24
    }
}

virtual_server 192.168.250.158 6443 {
    delay_loop 6
    lb_algo rr                  
    lb_kind DR            
    protocol TCP

    real_server 192.168.250.151 6443 {
        TCP_CHECK {
            connect_timeout 3
        }
    }

    real_server 192.168.250.152 6443 {
        TCP_CHECK {
            connect_timeout 3
        }
    }

    real_server 192.168.250.153 6443 {
        TCP_CHECK {
            connect_timeout 3
        }
    }
}
EOF

3.4 启动 keepalived 服务

systemctl enable --now keepalived

四、Master1节点初始化

4.1 添加 api-server 的 vip

nmcli connection add \
  con-name lo-vip \
  type loopback \
  connection.interface-name lo \
  connection.autoconnect yes
nmcli connection modify lo-vip \
  +ipv4.addresses 192.168.250.158/32
nmcli connection up lo-vip

4.2 生成 kubeadm 初始化的配置文件

master1=192.168.250.151

# .kubernetesVersion
kubernetesVersion=1.32.2
# .controlPlaneEndpoint
controlPlaneEndpoint=api:6443
# .imageRepository
imageRepository=registry.aliyuncs.com/google_containers
# .localAPIEndpoint.advertiseAddress
advertiseAddress=$master1
# .nodeRegistration.name
name=master-1
# .networking.podSubnet
podSubnet=10.244.0.0/16

cd
kubeadm config print init-defaults > kubeadm-config.yaml
sed -ri "s|(advertiseAddress:[[:space:]]*).*|\1$advertiseAddress|" kubeadm-config.yaml
sed -ri "s|(name:[[:space:]]*).*|\1$name|" kubeadm-config.yaml
sed -ri "s|(imageRepository:[[:space:]]*).*|\1$imageRepository|" kubeadm-config.yaml
sed -ri "s|(kubernetesVersion:[[:space:]]*).*|\1$kubernetesVersion|" kubeadm-config.yaml
sed -ri "/(serviceSubnet:[[:space:]]*).*/a\ \ podSubnet: $podSubnet" kubeadm-config.yaml
sed -ri "/(kubernetesVersion:[[:space:]]*).*/a controlPlaneEndpoint: $controlPlaneEndpoint" kubeadm-config.yaml

期望的配置

  apiVersion: kubeadm.k8s.io/v1beta4
  bootstrapTokens:
  - groups:
    - system:bootstrappers:kubeadm:default-node-token
    token: abcdef.0123456789abcdef
    ttl: 24h0m0s
    usages:
    - signing
    - authentication
  kind: InitConfiguration
  localAPIEndpoint:
+   advertiseAddress: 192.168.250.151
-   advertiseAddress: 1.2.3.4
    bindPort: 6443
  nodeRegistration:
    criSocket: unix:///var/run/containerd/containerd.sock
    imagePullPolicy: IfNotPresent
    imagePullSerial: true
+   name: master-1
-   name: node
    taints: null
  timeouts:
    controlPlaneComponentHealthCheck: 4m0s
    discovery: 5m0s
	    etcdAPICall: 2m0s
    kubeletHealthCheck: 4m0s
    kubernetesAPICall: 1m0s
    tlsBootstrap: 5m0s
    upgradeManifests: 5m0s
  ---
  apiServer: {}
  apiVersion: kubeadm.k8s.io/v1beta4
  caCertificateValidityPeriod: 87600h0m0s
  certificateValidityPeriod: 8760h0m0s
  certificatesDir: /etc/kubernetes/pki
  clusterName: kubernetes
  controllerManager: {}
  dns: {}
  encryptionAlgorithm: RSA-2048
  etcd:
    local:
      dataDir: /var/lib/etcd
+ imageRepository: registry.aliyuncs.com/google_containers
- imageRepository: registry.k8s.io
  kind: ClusterConfiguration
+ kubernetesVersion: 1.32.2
- kubernetesVersion: 1.32.0
+ controlPlaneEndpoint: api:6443
  networking:
    dnsDomain: cluster.local
    serviceSubnet: 10.96.0.0/12
+   podSubnet: 10.244.0.0/16
  proxy: {}
  scheduler: {}

4.3 下载集群初始化所需镜像

kubeadm config --config=kubeadm-config.yaml images list
kubeadm config --config=kubeadm-config.yaml images pull
ctr -n k8s.io image list

4.4 集群初始化

kubeadm init --config=kubeadm-config.yaml

4.5 配置 kubectl 的认证文件

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

五、Worker 节点加入集群

在 Master1 节点运行如下命令，生成Worker节点加入集群命令的命令，然后将生成的命令在各个 Worker 节点上运行

kubeadm token create --print-join-command

注：示例命令

kubeadm join api:6443 --token wg0ev4.cz{{placehold}}oyq --discovery-token-ca-cert-hash sha256:ba{{placehold}}07e

六、其余 Master 节点加入集群

6.1 Master 节点加入集群

如果要让其余 Master 节点加入集群，需要在生成的 join-command 中增加两个参数 --control-plane 和 --certificate-key 2d{placehold}b

注：certificate-key 的值需要通过如下命令生成

kubeadm init phase upload-certs --upload-certs

注：示例命令

kubeadm join api:6443 --token wg0ev4.cz{{placehold}}oyq --discovery-token-ca-cert-hash sha256:ba{{placehold}}07e --control-plane --certificate-key 2d{placehold}b

6.2 Master 节点添加 api-server 的 vip 到 lo 网卡

nmcli connection add \
  con-name lo-vip \
  type loopback \
  connection.interface-name lo \
  connection.autoconnect yes
nmcli connection modify lo-vip \
  +ipv4.addresses 192.168.250.158/32
nmcli connection up lo-vip

七、安装网络插件 Calico

7.1 配置代理（在所有节点配置代理）

注：接下来，为了避免安装网络插件 Calico 因网络不好而需要导入离线镜像的麻烦，我在这次实验中在这里配置代理，方便拉取相关镜像

全局代理

cat > /etc/profile.d/proxy.sh << 'EOF'
#!/bin/bash

# Configuration
PROXY_URL=http://192.168.250.1:10090
NO_PROXY_URL=localhost,127.0.0.1,::1,192.168.0.0/16,172.16.0.0/12,10.0.0.0/8,.svc,.cluster.local,.localdomain,.example.com,192.168.250.158,api

# Enviroment Variables
export HTTP_PROXY=$PROXY_URL HTTPS_PROXY=$PROXY_URL ALL_PROXY=$PROXY_URL
export http_proxy=$PROXY_URL https_proxy=$PROXY_URL all_proxy=$PROXY_URL
export NO_PROXY=$NO_PROXY_URL no_proxy=$NO_PROXY_URL
EOF
. /etc/profile.d/proxy.sh

containerd代理

mkdir /usr/lib/systemd/system/containerd.service.d
PROXY_URL=http://192.168.250.1:10090
NO_PROXY_URL=localhost,127.0.0.1,::1,192.168.0.0/16,172.16.0.0/12,10.0.0.0/8,.svc,.cluster.local,.localdomain,.example.com,192.168.250.158,api
cat > /usr/lib/systemd/system/containerd.service.d/http-proxy.conf << EOF
[Service]
Environment="NO_PROXY=$NO_PROXY_URL"
Environment="HTTPS_PROXY=$PROXY_URL"
Environment="HTTP_PROXY=$PROXY_URL"
Environment="ALL_PROXY=$PROXY_URL"
Environment="no_proxy=$NO_PROXY_URL"
Environment="https_proxy=$PROXY_URL"
Environment="http_proxy=$PROXY_URL"
Environment="all_proxy=$PROXY_URL"
EOF

systemctl daemon-reload
systemctl restart containerd

注：如果需要在网络受限环境中安装网络插件 Calico，执行如下操作即可

上传离线镜像 calico-3.29.2.image
运行命令 ctr -n k8s.io image import calico-3.29.2.image

7.2 安装网络插件 Calico （仅在一个 master 节点执行）

curl https://raw.githubusercontent.com/projectcalico/calico/v3.29.2/manifests/calico.yaml -O
kubectl create -f calico.yaml

验证

kubectl get po -n kube-system -o wide
kubectl get no
kubectl get cs

posted on 2026-10-26 17:23 笔记和文档阅读(9) 评论(0) 收藏举报

刷新页面返回顶部

iN2gN3jT

导航

公告