K8s1.28集群基于kubeadm部署&containerd&calico
1. 部署之前环境准备
1.1 主机准备
系统版本ubuntu22.04
10.0.0.115 master115
10.0.0.116 worker116
10.0.0.117 worker117
1.2 关闭swap分区
使用 systemctl --type swap 列出当前的交换单元
systemctl mask #dev-sda3(磁盘位置)#.swap #关闭swap自动激活功能
swapoff -a && sysctl -w vm.swappiness=0 # 临时关闭
sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab # 基于配置文件关闭
1.3 确保各个节点MAC地址、product_uuid唯一
apt update && apt install -y net-tools
ifconfig eth0 | grep ether | awk '{print $2}'
cat /sys/class/dmi/id/product_uuid
温馨提示: 硬件设备会拥有唯一的地址，但是有些虚拟机的地址可能会重复如克隆的虚拟机会导致mac地址一样需重新生成MAC地址。 Kubernetes使用这些值来唯一确定集群中的节点。 如果这些值在每个节点上不唯一，可能会导致安装失败
1.4 检查网络节点是否互通
简而言之，就是检查你的k8s集群各节点网络是否通，可以使用ping命令来测试。
ping jd.com -c 10
1.5 所有节点修改时间
[root@master115 ~]# date -R
Mon, 09 Sep 2024 14:58:34 +0800
[root@master115 ~]# ll /etc/localtime
lrwxrwxrwx 1 root root 33 Aug 30 15:27 /etc/localtime -> /usr/share/zoneinfo/Asia/Shanghai
[root@worker115 ~]# date -R
Mon, 09 Sep 2024 14:59:22 +0800
[root@worker232 ~]# ll /etc/localtime
lrwxrwxrwx 1 root root 33 Aug 30 15:27 /etc/localtime -> /usr/share/zoneinfo/Asia/Shanghai
[root@worker116 ~]# date -R
Mon, 09 Sep 2024 14:59:35 +0800
[root@worker116 ~]# ll /etc/localtime
lrwxrwxrwx 1 root root 33 Aug 30 15:27 /etc/localtime -> /usr/share/zoneinfo/Asia/Shanghai
*****************************************************************************************
如果没有修改使用以下命令进行修改
[root@worker115 ~]# timedatectl set-timezone Asia/Shanghai
1.6 所有节点修改cgroup的管理进程为systemd
(ubuntu 默认cgroup的管理进程为systemd）
1.7 允许iptable检查桥接流量
1) 创建配置文件
cat <<EOF | tee /etc/modules-load.d/k8s.conf
br_netfilter
bridge
EOF
cat <<EOF | tee /etc/sysctl.d/k8s.conf 【以下3个参数是containerd所依赖的内核参数】
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
sysctl --system
2) 临时生效，需要加载模块，所有节点均要操作
[root@master115 ~]# modprobe br_netfilter bridge
[root@master115 ~]# lsmod | grep bridge
bridge 311296 1 br_netfilter
stp 16384 1 bridge
llc 16384 2 bridge,stp
[root@master115 ~]# sysctl -f /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
[root@master115 ~]#
1.8 禁用ufw&apparmor
systemctl disable --now ufw
#ubuntu的ufw类似于Centos的firewalld
systemctl disable --now apparmor
#ubuntu的apparmor类似于Centos的selinux
2. 所有节点安装containerd.1.6.36作为容器运行时
参考地址：https://github.com/containerd/containerd/releases
2.1 安装containerd
1. tar xf cri-containerd-1.6.36-linux-amd64.tar.gz #解压软件包
2. 查看目录是否存在
[root@master115 ~]# ls etc opt usr
etc: #存放的是containerd服务管理配置文件及cni虚拟网卡配置文件。
cni crictl.yaml systemd
opt: #存放的是GCE环境中使用containerd配置文件及CNI插件。
cni containerd
usr: #存放的是containerd运行时文件，包含runC。
local
3. 拷贝containerd运行时文件到宿主机指定路径
[root@master115 ~]# cp usr/local/bin/containerd /usr/local/bin/
[root@master115 ~]#
[root@master115 ~]# ll /usr/local/bin/
total 38136
drwxr-xr-x 2 root root 4096 Sep 26 09:55 ./
drwxr-xr-x 10 root root 4096 Feb 17 2024 ../
-rwxr-xr-x 1 root root 39039560 Sep 26 09:55 containerd*
[root@k8s66 ~]#
4. 拷贝containerd的systemd的配置文件
[root@master115 ~]# cp etc/systemd/system/containerd.service /usr/lib/systemd/system/
[root@master115 ~]#
[root@master115 ~]# ll /usr/lib/systemd/system/containerd.service
-rw-r--r-- 1 root root 1414 Sep 26 09:56 /usr/lib/systemd/system/containerd.service
[root@master115 ~]#
5. 查看containerd帮助信息
[root@master115 ~]# containerd --help
NAME:
containerd -
__ _ __
_________ ____ / /_____ _(_)___ ___ _________/ /
/ ___/ __ \/ __ \/ __/ __ `/ / __ \/ _ \/ ___/ __ /
/ /__/ /_/ / / / / /_/ /_/ / / / / / __/ / / /_/ /
\___/\____/_/ /_/\__/\__,_/_/_/ /_/\___/_/ \__,_/
high performance container runtime
6. 生成配置文件
[root@master115 ~]# mkdir /etc/containerd #创建配置文件目录
[root@master115 ~]# containerd config default > /etc/containerd/config.toml #生成默认的配置文件
7. 修改Cgroup的管理者为systemd组件
[root@master115 ~]# grep SystemdCgroup /etc/containerd/config.toml
SystemdCgroup = false
[root@master115 ~]# sed -ri 's#(SystemdCgroup = )false#\1true#' /etc/containerd/config.toml
[root@master115 ~]# grep SystemdCgroup /etc/containerd/config.toml
SystemdCgroup = true
8. 修改pause的基础镜像名称
[root@master115 ~]# grep sandbox_image /etc/containerd/config.toml
sandbox_image = "registry.k8s.io/pause:3.6"
[root@master115 ~]# sed -i 's#registry.k8s.io/pause:3.6#registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.9#' /etc/containerd/config.toml
[root@master115 ~]# grep sandbox_image /etc/containerd/config.toml
sandbox_image = "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.9"
9. 启动containerd服务
[root@master115 ~]# systemctl enable --now containerd
[root@master115 ~]# systemctl status containerd
● containerd.service - containerd container runtime
Loaded: loaded (/lib/systemd/system/containerd.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2024-09-28 14:43:58 CST; 3h 2min ago
Docs: https://containerd.io
Main PID: 4067 (containerd)
Tasks: 10
Memory: 17.2M
CPU: 22.654s
CGroup: /system.slice/containerd.service
└─4067 /usr/local/bin/containerd
10. 拷贝ctr客户端工具及runc容器运行时
[root@master115 ~]# cp usr/local/bin/ctr /usr/bin/ #拷贝ctr客户端工具到PATH环境变量
[root@master115 ~]# ctr version #查看contrainerd服务版本信息
Client:
Version: v1.6.36
Revision: 88c3d9bc5b5a193f40b7c14fa996d23532d6f956
Go version: go1.22.7
Server:
Version: v1.6.36
Revision: 88c3d9bc5b5a193f40b7c14fa996d23532d6f956
UUID: 15b38c60-a9e6-4994-aad7-236e3c7189b3
11. 拷贝容器的运行时，用于后期启动容器
[root@worker116 ~]# cp usr/local/bin/containerd-shim-runc-v2 /usr/bin/
[root@worker116 ~]# ll /usr/bin/containerd-shim-runc-v2
-rwxr-xr-x 1 root root 8605848 Sep 28 17:47 /usr/bin/containerd-shim-runc-v2*
2.2 单独安装runC v1.1.12
由于二进制包中提供的runC默认需要安装seccomp支持，需要单独安装，且不同runC对seccomp版本要求不尽相同。这会导致安装后无法使用runC的情况。
参考链接：https://github.com/opencontainers/runc/releases
1. 拷贝runC到环境变量
[root@master115 ~]# mv runc.amd64 /usr/sbin/runc
2. 为runc程序添加执行权限
[root@master115 ~]# chmod +x /usr/sbin/runc
[root@master115 ~]# ll /usr/sbin/runc
-rwxr-xr-x 1 root root 10709696 Mar 8 2024 /usr/sbin/runc*
3. 查看runC的版本信息
[root@master115 ~]# runc -v
runc version 1.1.12
commit: v1.1.12-0-g51d5e946
spec: 1.0.2-dev
go: go1.20.13
libseccomp: 2.5.4
3. K8S集群部署所有节点安装kubeadm，kubelet，kubectl
3.1 配置软件源
1. 更新"apt"包索引并安装使用Kubernetes apt仓库所需要的包 【此过程可能中断，需要执行两遍】
apt-get update && apt-get install -y apt-transport-https
2. 下载用于 Kubernetes 软件包仓库的公共签名密钥。所有仓库都使用相同的签名密钥，因此你可以忽略URL中的版本
curl -fsSL https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.28/deb/Release.key |
gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
3. 添加Kubernetes apt仓库
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.28/deb/ /" |
tee /etc/apt/sources.list.d/kubernetes.list
4.查看软件源的k8s版本
apt-cache madison kubeadm
5. 安装kubeadm，kubelet，kubectl
apt-get update
apt-get install -y kubelet kubeadm kubectl
6. 锁定版本 【防止误进行升级操作】
apt-mark hold kubelet kubeadm kubectl
3.2 K8S集群部署之初始化master节点
1. kubeadm初始化生成token 默认保存24小时，24小时后token过期
[root@master115 ~]kubeadm init --kubernetes-version=v1.28.14 --image-repository registry.aliyuncs.com/google_containers --pod-network-cidr=10.100.0.0/16 --service-cidr=10.200.0.0/16 --service-dns-domain=nolenlinux.cn
......
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 10.0.0.115:6443 --token ogo9nn.wlfk3e9p0d9maqb2 \
--discovery-token-ca-cert-hash sha256:4f22369982e1a7df623997109abeb5b6681604dfe67c75c2a5731d95fe95dfba
2. 拷贝认证信息
[root@master115 ~]# mkdir -p $HOME/.kube
[root@masker115 ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@master115 ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config
[root@master115 ~]#
[root@master115 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master115 NotReady control-plane 50s v1.28.14
3.3 worker节点加入K8S集群
1. 复制master节点生成的token加入到K8S节点
[root@worker116 ~]# kubeadm join 10.0.0.115:6443 --token ogo9nn.wlfk3e9p0d9maqb2 \
> --discovery-token-ca-cert-hash sha256:4f22369982e1a7df623997109abeb5b6681604dfe67c75c2a5731d95fe95dfba
[root@worker117 ~]# kubeadm join 10.0.0.115:6443 --token ogo9nn.wlfk3e9p0d9maqb2 \
> --discovery-token-ca-cert-hash sha256:4f22369982e1a7df623997109abeb5b6681604dfe67c75c2a5731d95fe95dfba
2. 查看K8S集群状态
[root@master115 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master115 NotReady control-plane 55m v1.28.14
worker116 NotReady <none> 8m14s v1.28.14
worker117 NotReady <none> 6s v1.28.14
4. 安装calico网络插件 【虚拟机需要FQ拉取资源清单和镜像或者有容器所需要的镜像导入到本地】
4.1 下载calico的资源清单
[root@master115 ~]# wget https://raw.githubusercontent.com/projectcalico/calico/v3.25.2/manifests/tigera-operator.yaml
4.2 下载calico的自定义配置Pod网络
[root@master115 ~]# wget https://raw.githubusercontent.com/projectcalico/calico/v3.25.2/manifests/custom-resources.yaml
4.3 安装部署组件
[root@master231 ~]# kubectl create -f tigera-operator.yaml
4.4 应用资源池定义Pod网段并创建
[root@master115 ~]# grep ipPools: custom-resources.yaml -A 2
ipPools:
- blockSize: 26
cidr: 192.168.0.0/16
[root@master115 ~]#
[root@master115 ~]# sed -i '/cidr/s#192.168#10.100#' custom-resources.yaml
[root@master115 ~]#
[root@master115 ~]# grep ipPools: custom-resources.yaml -A 2
ipPools:
- blockSize: 26
cidr: 10.100.0.0/16
[root@master115 ~]#
[root@master115 ~]# kubectl create -f custom-resources.yaml
installation.operator.tigera.io/default created
apiserver.operator.tigera.io/default created
[root@master115 ~]#
4.5 确认所有的Pod均能正常运行
[root@master115 ~]# kubectl get pods -o wide -n calico-system
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
calico-kube-controllers-6dbf55d9cd-cvd4v 1/1 Running 0 5m5s 10.100.235.196 master115 <none> <none>
calico-node-gl4l4 1/1 Running 1 (2m47s ago) 5m5s 10.0.0.117 worker117 <none> <none>
calico-node-gqzjt 1/1 Running 0 5m5s 10.0.0.116 worker116 <none> <none>
calico-node-sw7r9 1/1 Running 0 5m5s 10.0.0.115 master115 <none> <none>
calico-typha-59d84499db-5rx44 1/1 Running 1 (4m52s ago) 5m5s 10.0.0.116 worker116 <none> <none>
calico-typha-59d84499db-zfjjp 1/1 Running 1 (4m42s ago) 5m1s 10.0.0.117 worker117 <none> <none>
csi-node-driver-dfvdg 2/2 Running 2 (2m47s ago) 5m5s 10.100.101.67 worker117 <none> <none>
csi-node-driver-hfr74 2/2 Running 0 5m5s 10.100.235.194 master115 <none> <none>
csi-node-driver-z6bjn 2/2 Running 2 (3m53s ago) 5m5s 10.100.214.2 worker116 <none> <none>
4.6 查看集群是否就绪
[root@master115 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master115 Ready control-plane 80m v1.28.14
worker116 Ready <none> 32m v1.28.14
worker117 Ready <none> 24m v1.28.14
浙公网安备 33010602011771号