ubuntu 26.04，使用kubespray-v2.31.0离线安装5节点高可用k8s-v1.35.4集群

ubuntu 26.04。

集群规划（5 节点）

master节点01，主机名：k8s-master-01，ip：172.21.64.36

master节点02，主机名：k8s-master-02，ip：172.21.64.37

master节点03，主机名：k8s-master-03，ip：172.21.64.38

工作节点01：主机名：k8s-worker-01，ip：172.21.64.39

工作节点02：主机名：k8s-worker-02，ip：172.21.64.40

机器名	内网 ip	角色
k8s-master-01	172.21.64.36	master
k8s-master-02	172.21.64.37	master
k8s-master-03	172.21.64.38	master
k8s-worker-01	172.21.64.39	worker
k8s-worker-02	172.21.64.40	worker

一，运维目录结构规划

/data/kubespray/
├── clusters/
│   ├── prod/
│   │   └── inventory.ini          # 生产集群清单：节点IP、主机名、角色分组（kube_control_plane/etcd/kube_node）
│   ├── test/
│   │   └── inventory.ini          # 测试集群清单
│   └── dev/
│       └── inventory.ini          # 开发集群清单
│
└── kubespray-toolset/
    ├── v2.31.0/
    │   ├── kubespray-src/         # kubespray v2.31.0 原始源码
    │   ├── venv/                  # v2.31.0 独立Python虚拟环境（内网现场构建，不跨机器拷贝）
    │   ├── offline-resources/     # v2.31.0 全套离线资源（源码外部存放，不污染源码）
    │   │   ├── pip-offline/       # python依赖离线whl包
    │   │   ├── deb_packages/      # Ubuntu系统依赖deb包
    │   │   ├── binaries/          # wget下载的kubeadm/kubectl/helm等二进制文件
    │   │   └── images_tar/        # skopeo导出的容器镜像tar包
    │   └── version-vars/
    │       └── group_vars/all/
    │           ├── china_mirror.yml   # 【通用】国内镜像地址配置（在线、离线全部加载）
    │           ├── containerd.yml     # 【通用】容器运行时配置（在线、离线全部加载）
    │           └── offline.yml        # 【离线专属】离线模式开关（download_container: false等，在线禁止加载）
    │
    └── v2.32.0/                    # 后续升级新版本，目录结构完全镜像v2.31.0
        ├── kubespray-src/
        ├── venv/
        ├── offline-resources/
        └── version-vars/
            └── group_vars/all/
                ├── china_mirror.yml
                ├── containerd.yml
                └── offline.yml

二，联网机下载离线包

1. 安装下载环境

apt update
apt install -y git wget skopeo python3 python3-pip python3-venv

2, 配置离线下载脚本

vi /data/kubespray/download_kubespray_offline_v2.31.0.sh

#!/bin/bash
set -euo pipefail

########################### 配置区 ###########################
ROOT=/data/kubespray
KS_VERSION="v2.31.0"

TOOLSET_ROOT=${ROOT}/kubespray-toolset/${KS_VERSION}
SRC_DIR=${TOOLSET_ROOT}/kubespray-src
OUT_RES=${TOOLSET_ROOT}/offline-resources

# 离线资源子目录
DIR_PIP=${OUT_RES}/pip-offline
DIR_DEB=${OUT_RES}/deb_packages
DIR_BIN=${OUT_RES}/binaries
DIR_IMG=${OUT_RES}/images_tar

########################### 创建目录 ###########################
mkdir -p ${SRC_DIR}
mkdir -p ${DIR_PIP} ${DIR_DEB} ${DIR_BIN} ${DIR_IMG}

########################### 1. 克隆指定版本kubespray源码 ###########################
if [ ! -d "${SRC_DIR}/.git" ];then
  git clone https://github.com/kubernetes-sigs/kubespray.git ${SRC_DIR}
fi
cd ${SRC_DIR}
git fetch --all
git checkout ${KS_VERSION}

########################### 2. 下载python离线依赖（pip whl）###########################
pip download -d ${DIR_PIP} \
  --timeout 120 \
  --retries 5 \
  -i https://pypi.tuna.tsinghua.edu.cn/simple \
  -r requirements.txt

########################### 3. 生成二进制清单、镜像清单（官方脚本）###########################
cd ${SRC_DIR}/contrib/offline
rm -rf temp
bash generate_list.sh

FILES_LIST="./temp/files.list"
IMAGES_LIST="./temp/images.list"

# 校验清单是否生成成功
if [ ! -s "${FILES_LIST}" ]; then
  echo "ERROR: files.list 为空！generate_list.sh 执行异常"
  exit 1
fi
if [ ! -s "${IMAGES_LIST}" ]; then
  echo "ERROR: images.list 为空！generate_list.sh 执行异常"
  exit 1
fi
echo "二进制清单数量：$(wc -l ${FILES_LIST})"
echo "镜像清单数量：$(wc -l ${IMAGES_LIST})"

########################### 4. 替换下载地址 → DaoCloud 加速 ###########################
# 二进制下载域名替换
sed -i.bak \
-e 's|https://dl.k8s.io|https://dl.k8s.m.daocloud.io|g' \
-e 's|https://github.com|https://github.m.daocloud.io|g' \
${FILES_LIST}
rm -f ${FILES_LIST}.bak

# 镜像地址替换（供skopeo拉取）
sed -i.bak \
-e 's|registry.k8s.io|k8s.m.daocloud.io|g' \
-e 's|quay.io|quay.m.daocloud.io|g' \
-e 's|ghcr.io|ghcr.m.daocloud.io|g' \
-e 's|docker.io|docker.m.daocloud.io|g' \
${IMAGES_LIST}
rm -f ${IMAGES_LIST}.bak

########################### 5. wget下载二进制文件 ###########################
echo "===== 开始下载二进制文件 ====="
wget -x -P ${DIR_BIN} -i ${FILES_LIST} || echo "警告：存在部分二进制下载失败，后续人工核对"

########################### 6. skopeo 导出镜像 tar包【已移除command-timeout兼容低版本】 ###########################
echo "===== 开始导出镜像tar包 ====="
while read -r IMAGE_NAME; do
  [ -z "${IMAGE_NAME}" ] && continue
  TAR_NAME=$(echo "${IMAGE_NAME}" | tr '/' '_' | tr ':' '_').tar
  OUT_TAR=${DIR_IMG}/${TAR_NAME}
  if [ -f "${OUT_TAR}" ];then
    echo "跳过已存在镜像: ${IMAGE_NAME}"
    continue
  fi
  echo "拉取镜像: ${IMAGE_NAME}"
  if ! skopeo copy --override-arch amd64 docker://${IMAGE_NAME} docker-archive:${OUT_TAR};then
    echo "【失败】镜像拉取异常：${IMAGE_NAME}"
  fi
done < ${IMAGES_LIST}

echo "==================== Download Task Completed ===================="
echo "源码目录:     ${SRC_DIR}"
echo "离线资源根目录:${OUT_RES}"
echo "二进制目录:    ${DIR_BIN}"
echo "镜像Tar目录:   ${DIR_IMG}"

三，前置环境配置，所有节点统一前置初始化脚本（所有机器执行）

1. 卸载、清理历史k8s残留

#!/usr/bin/env bash
set -euo pipefail

echo -e "===== 开始执行【全场景全覆盖】K8s 终极节点重置（适配kubeadm/sealos/kubekey/kubesray/kubeasz/二进制/deb混合残留）====="

# ======================
# 1. 通用集群重置（kubeadm体系）
# ======================
if command -v kubeadm >/dev/null; then
    kubeadm reset -f || true
fi

# ======================
# 2. 停止所有可能存在的服务（全覆盖）
# ======================
systemctl stop \
containerd kubelet \
kube-apiserver kube-controller-manager kube-scheduler kube-lb \
sealos kube-proxy etcd \
2>/dev/null || true

systemctl disable \
containerd kubelet \
kube-apiserver kube-controller-manager kube-scheduler kube-lb \
sealos kube-proxy etcd \
2>/dev/null || true

# ======================
# 3. 清理【二进制部署残留】控制面服务（你之前无限复活的元凶）
# ======================
rm -f /etc/systemd/system/kube-apiserver.service
rm -f /etc/systemd/system/kube-controller-manager.service
rm -f /etc/systemd/system/kube-scheduler.service
rm -f /etc/systemd/system/kube-lb.service
rm -f /etc/systemd/system/etcd.service
rm -f /etc/systemd/system/kube-proxy.service

# ======================
# 4. 清理 Sealos / Kubekey / KubeSray / Kubeasz 专属服务 & 目录
# ======================
# sealos 残留
rm -rf /var/lib/sealos /etc/sealos /root/.sealos
rm -f /etc/systemd/system/sealos.service

# kubekey/kubesray 残留
rm -rf /var/lib/kubekey /etc/kubekey /opt/kubekey
rm -rf /var/lib/kubesphere /etc/kubesphere

# kubeasz 残留
rm -rf /etc/kubeasz /var/lib/kubeasz

# ======================
# 5. 彻底purge卸载deb包（containerd.io、kubeadm、cri-tools全部purge，删除包+配置）
# ======================
apt purge -y --allow-change-held-packages \
    containerd.io containerd \
    kubeadm kubectl kubelet \
    cri-tools kubernetes-cni \
2>/dev/null || true

apt autoremove -y || true
apt autoclean

# 解除版本锁定，保证下次重装无冲突
apt-mark unhold \
    kubeadm kubelet kubectl \
    containerd.io cri-tools kubernetes-cni \
2>/dev/null || true

# ======================
# 6. 彻底清理所有路径 systemd 单元（解决二进制+deb双路径冲突）
# ======================
rm -f /lib/systemd/system/kubelet.service
rm -f /etc/systemd/system/kubelet.service
rm -rf /etc/systemd/system/kubelet.service.d

rm -f /lib/systemd/system/containerd.service
rm -f /etc/systemd/system/containerd.service
rm -rf /etc/systemd/system/containerd.service.d
rm -f /usr/local/lib/systemd/system/containerd.service

# ======================
# 7. 彻底清理所有路径 k8s/容器二进制
# 覆盖：apt/deb / 二进制解压 / kubekey / sealos
# 新增 cni、crictl 二进制清理
# ======================
rm -f /usr/bin/containerd /usr/bin/ctr /usr/bin/crictl /usr/bin/kube*
rm -f /usr/local/bin/containerd /usr/local/bin/ctr /usr/local/bin/crictl /usr/local/bin/kube*
rm -f /opt/kube/bin/* 2>/dev/null || true

# 清理二进制解压的cni插件二进制
rm -rf /opt/cni/bin/* 2>/dev/null || true

# ======================
# 8. 重载 systemd、清除失效状态
# ======================
systemctl daemon-reload
systemctl reset-failed

# ======================
# 9. 绝杀所有残留进程（杜绝复活、端口占用）
# ======================
pkill -9 -f "kube-apiserver|kube-scheduler|kube-controller-manager|kube-lb|kubelet|containerd|sealos|etcd|crictl" 2>/dev/null || true

# ======================
# 10. 清空所有体系数据目录
# ======================

# 先卸载 Calico cgroup 挂载点（否则 rm 报 Operation not permitted）
umount -R /var/run/calico/cgroup 2>/dev/null || true
umount -R /run/calico/cgroup 2>/dev/null || true

# /var/run 和 /run 是 tmpfs，重启自动清空，无需 rm
rm -rf \
/var/lib/containerd \
/var/lib/cri \
/etc/cni \
/opt/cni \
/root/.kube \
/etc/containerd \
/etc/kubernetes \
/var/lib/kubelet \
/var/lib/etcd \
/var/lib/calico \
/var/lib/sealos \
/var/lib/nerdctl \
/var/lib/kubekey \
/var/lib/kubesphere \
/etc/kubeasz \
/etc/cni/net.d/*

# ======================
# 11. 重置网络栈（解决多工具残留网卡、iptables错乱）
# ======================
iptables -F && iptables -t nat -F && iptables -X && ipvsadm -C
ip link del tunl0 2>/dev/null || true
ip link del cni0 2>/dev/null || true
ip link del flannel.1 2>/dev/null || true
ip link del vxlan.calico 2>/dev/null || true

echo -e "✅ 【k8s残留】清理完成！已purge卸载 containerd.io、cri-tools、kubernetes-cni"

2. 修改主机名

# master-01节点执行
hostnamectl set-hostname k8s-master-01
# master-02节点执行
hostnamectl set-hostname k8s-master-02
# master-03节点执行
hostnamectl set-hostname k8s-master-03
# worker-01节点执行
hostnamectl set-hostname k8s-worker-01
# worker-02节点执行
hostnamectl set-hostname k8s-worker-02

3. 配置 hosts 解析（所有节点统一写入 /etc/hosts）

cat >> /etc/hosts << EOF
172.21.64.36   k8s-master-01
172.21.64.37   k8s-master-02
172.21.64.38   k8s-master-03
172.21.64.39   k8s-worker-01
172.21.64.40   k8s-worker-02
EOF

4. 系统内核 & 资源优化

# 关闭永久swap
swapoff -a
sed -i '/swap/s/^/#/' /etc/fstab

#关闭 ufw防火墙
systemctl stop ufw
systemctl disable ufw

# 内核转发、iptables桥接
cat >> /etc/sysctl.conf <<EOF
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1

# 允许绑定本机不存在的VIP（HA/kube-vip核心）
net.ipv4.ip_nonlocal_bind = 1

# 反向路由过滤 全部宽松模式
net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.eth0.rp_filter = 2  #eth0替换为真实的网卡名称（非必填）

# ARP应答策略：仅匹配网卡IP才回复
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.default.arp_ignore = 1
net.ipv4.conf.eth0.arp_ignore = 1  #eth0替换为真实的网卡名称(非必填)

# ARP宣告策略：严格匹配网卡网段
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.eth0.arp_announce = 2  #eth0替换为真实的网卡名称（非必填）

EOF
sysctl -p

# 加载overlay/iptables内核模块
modprobe overlay
modprobe br_netfilter


#系统开机自动加载里面写的内核模块。
echo "overlay" >> /etc/modules-load.d/k8s.conf
echo "br_netfilter" >> /etc/modules-load.d/k8s.conf
# 开启root密码登录
sed -i 's/#PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config
sed -i 's/#PasswordAuthentication no/PasswordAuthentication yes/' /etc/ssh/sshd_config
sed -i 's/#PubkeyAuthentication no/PubkeyAuthentication yes/' /etc/ssh/sshd_config
systemctl restart ssh systemctl enable ssh

#时间同步
apt install chrony -y
systemctl enable --now chronyd
timedatectl set-timezone Asia/Shanghai

5. Master 节点单独执行 SSH 免密分发

# master节点生成密钥
ssh-keygen -t ed25519 -N "" -f /root/.ssh/id_ed25519
# 分发公钥
sshpass -p "root密码" ssh-copy-id root@172.21.64.36
sshpass -p "root密码" ssh-copy-id root@172.21.64.37
sshpass -p "root密码" ssh-copy-id root@172.21.64.38
sshpass -p "root密码" ssh-copy-id root@172.21.64.39
sshpass -p "root密码" ssh-copy-id root@172.21.64.40

# 验证免密连通
ssh root@172.21.64.36 hostname
ssh root@172.21.64.37 hostname
ssh root@172.21.64.38 hostname
ssh root@172.21.64.39 hostname
ssh root@172.21.64.40 hostname

6. 批量校验所有节点 SSH 连通性（最关键前置检查）

nodes=(172.21.64.36 172.21.64.37 172.21.64.38 172.21.64.39 172.21.64.40)
for ip in ${nodes[@]};
do
  echo "=== 测试 $ip SSH ==="
  sshpass -p "你的root密码" ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no root@$ip "hostname; echo 连接正常"
done

四，目录结构规划及节点清单配置(在管控节点执行)

1. 目录结构

/data/kubespray/
├── clusters/
│   └── prod/
│       └── inventory.ini       # 本次生产集群清单【重点下文模板】
└── kubespray-toolset/
    └── v2.31.0/
        ├── kubespray-src/
        ├── venv/
        ├── offline-resources/  # 已经下载完成离线包
        └── version-vars/
            └── group_vars/all/
                ├── china_mirror.yml
                ├── containerd.yml
                └── offline.yml # 离线核心配置

2. 集群节点清单

创建目录

#!/bin/bash
rootdir="v2.31.0"
base_path="/data/kubespray"

mkdir -p ${base_path}/clusters
mkdir -p ${base_path}/kubespray-toolset/${rootdir}/version-vars/group_vars/all

vi /data/kubespray/clusters/prod/inventory.ini

[all]
k8s-master-01 ansible_host=172.21.64.36 ip=172.21.64.36
k8s-master-02 ansible_host=172.21.64.37 ip=172.21.64.37
k8s-master-03 ansible_host=172.21.64.38 ip=172.21.64.38
k8s-worker-01 ansible_host=172.21.64.39 ip=172.21.64.39
k8s-worker-02 ansible_host=172.21.64.40 ip=172.21.64.40

[kube_control_plane]
k8s-master-01
k8s-master-02
k8s-master-03

[etcd]
k8s-master-01
k8s-master-02
k8s-master-03

[kube_node]
k8s-worker-01
k8s-worker-02

[calico_rr]

[k8s_cluster:children]
kube_control_plane
kube_node

3. 离线关键 yaml 配置文件

1) 1. offline.yml（离线模式核心，必须启用）

vi /data/kubespray/kubespray-toolset/v2.31.0/version-vars/group_vars/all/offline.yml

## 开启离线模式
local_release_dir: "/opt/kubespray/releases"
files_repo: "file://{{ local_release_dir }}"

# 禁止网络拉取镜像、禁止下载二进制
download_container: false
download_ceph_image: false
download_binaries: false

# 关闭外部github/网络下载
kubeadm_download_url: ""
kubectl_download_url: ""
kubelet_download_url: ""
etcd_download_url: ""
cni_download_url: ""
crictl_download_url: ""
runc_download_url: ""
containerd_download_url: ""
nerdctl_download_url: ""
helm_download_url: ""

2) china_mirror.yml（在线镜像地址，离线模式不生效保留即可）

vi /data/kubespray/kubespray-toolset/v2.31.0/version-vars/group_vars/all/china_mirror.yml

registry_k8s_mirror: k8s.m.daocloud.io
quay_mirror: quay.m.daocloud.io
ghcr_mirror: ghcr.m.daocloud.io
docker_mirror: docker.m.daocloud.io

3) containerd.yml（运行时配置）

vi /data/kubespray/kubespray-toolset/v2.31.0/version-vars/group_vars/all/containerd.yml

containerd_runtime: containerd
containerd_insecure_registries:
  - "127.0.0.0/8"

4. 拷贝离线包到所有节点

所有节点 /opt/kubespray/releases/ 存放二进制目录结构

# 离线二进制目录
BIN_SRC=/data/kubespray/kubespray-toolset/v2.31.0/offline-resources/binaries
# 目标目录
TARGET=/opt/kubespray/releases

# 循环推送到5台机器
NODE_LIST=(
172.21.64.36
172.21.64.37
172.21.64.38
172.21.64.39
172.21.64.40
)

for node in ${NODE_LIST[@]}; do
  echo "====同步二进制至 $node ===="
  ssh root@$node "mkdir -p ${TARGET}"
  rsync -av ${BIN_SRC}/ root@$node:${TARGET}/
done

所有节点导入镜像 tar 包（containerd k8s.io 命名空间）

TAR_SRC=/data/kubespray/kubespray-toolset/v2.31.0/offline-resources/images_tar
NODE_LIST=(
172.21.64.36
172.21.64.37
172.21.64.38
172.21.64.39
172.21.64.40
)

for node in ${NODE_LIST[@]}; do
echo "====传输镜像tar包至 $node===="
ssh root@$node "mkdir -p /opt/k8s-images"
rsync -av ${TAR_SRC}/*.tar root@$node:/opt/k8s-images/
done

五，部署 k8s

1) 安装虚拟环境

#!/bin/bash
# Kubespray v2.31.0 虚拟环境搭建

# 根路径变量
BASE=/data/kubespray/kubespray-toolset/v2.31.0

# 1. 创建虚拟环境目录
mkdir -p ${BASE}/venv

# 2. 创建虚拟环境
python3 -m venv ${BASE}/venv

# 3. 激活虚拟环境
source ${BASE}/venv/bin/activate

# 4. 进入源码目录安装依赖
cd ${BASE}/kubespray-src
pip install -U pip
pip install -r requirements.txt

# 校验ansible版本
ansible --version

2）进入虚拟环境

source /data/kubespray/kubespray-toolset/v2.31.0/venv/bin/activate

退出虚拟环境

deactivate

3）执行部署

预检

#!/bin/bash
BASE=/data/kubespray/kubespray-toolset/v2.31.0
ansible-playbook \
-i /data/kubespray/clusters/prod/inventory.ini \
-e @${BASE}/version-vars/group_vars/all/china_mirror.yml \
-e @${BASE}/version-vars/group_vars/all/containerd.yml \
-e @${BASE}/version-vars/group_vars/all/offline.yml \
--check cluster.yml

4) 部署

#!/bin/bash
BASE=/data/kubespray/kubespray-toolset/v2.31.0

cd ${BASE}/kubespray-src

ansible-playbook \
-i /data/kubespray/clusters/prod/inventory.ini \
-e @${BASE}/version-vars/group_vars/all/china_mirror.yml \
-e @${BASE}/version-vars/group_vars/all/containerd.yml \
-e @${BASE}/version-vars/group_vars/all/offline.yml \
cluster.yml

部署过程中，在导出镜像阻塞的时候，另外开一个管控机的终端导入镜像（导入镜像需要人工卡点）

NODE_LIST=(
172.21.64.36
172.21.64.37
172.21.64.38
172.21.64.39
172.21.64.40
)

for node in ${NODE_LIST[@]}; do
echo "====节点 $node 开始导入镜像===="
ssh root@$node "cd /opt/k8s-images && for f in *.tar; do ctr -n k8s.io images import \"\$f\"; done"
echo "====节点 $node 镜像导入完成===="
done

5) 授予master本地访问权限(在每个master节点执行)

mkdir -p ~/.kube
cp /etc/kubernetes/admin.conf ~/.kube/config
chmod 600 ~/.kube/config

或者在管控机批量复制到所有master节点

ansible k8s-master-01 -i ../inv/dev-k8s/inventory.ini -m fetch -a "src=/etc/kubernetes/admin.conf dest=./ flat=yes"
cp admin.conf ~/.kube/config
chmod 600 ~/.kube/config

6) 校验集群

kubectl get nodes
kubectl get cs
kubectl get pods -n kube-system

posted @ 2026-08-14 10:00 民工黑猫阅读(7) 评论(0) 收藏举报

刷新页面返回顶部

民工黑猫

ubuntu 26.04，使用kubespray-v2.31.0离线安装5节点高可用k8s-v1.35.4集群

公告