Kubernetes 集群运维管理 Master和worker节点添加和删除

 

 

Kubernetes 集群中添加和删除节点

从Kubernetes 集群删除 Master 节点

当一个Master出现故障时,需要添加或删除Master节点操作,从而修复节点后, 再重新加入集群,避免ETCD错误
前提条件

1. 集群中有 至少一个其他健康的 Master 节点（推荐 3 个或更多保证高可用）
2. 确认要删除的 Master 节点 不再承担调度任务
3. 备份重要配置和 ETCD 数据 

# 备份 ETCD（可选，但强烈建议）
ETCDCTL_API=3 etcdctl snapshot save /tmp/etcd-snapshot.db \
  --endpoints=<MASTER_NODE_IP>:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/peer.crt \
  --key=/etc/kubernetes/pki/etcd/peer.key

 

kubernetes 节点信息

#建议先备份ETCD数据库，再执行删除等操作

kubectl get nodes

[root@master1 ~]# kubectl get node
NAME          STATUS   ROLES           AGE   VERSION
master1.org   Ready    control-plane   29d   v1.35.6
master2.org   Ready    control-plane   37m   v1.35.6
master3.org   Ready    control-plane   24m   v1.35.6
node1.org     Ready    <none>          29d   v1.35.6
node2.org     Ready    <none>          29d   v1.35.6
node3.org     Ready    <none>          29d   v1.35.6

 

 

删除其中一个 Master 节点
注意：删除Master节点后，要保留至少半数以上个Master节点可用的，否则集群失败
当对删除其中的一个master节点时，可以执行下面操作

#在保留的其中一个节点上master1执行下面操作，指定删除master3.org 节点
#1)标记节点不可调度,并驱逐上面的Pod

kubectl drain master3.org --ignore-daemonsets

 #2)从集群中删除节点

kubectl delete node master3.org 

 

#3)在被删除的master3执行删除本机上面的信息
#作用：清理 kubelet、kube-proxy、证书,删除 /etc/kubernetes 配置,注意：这会删除节点上的所有 Kubernetes 配置和 ETCD 数据
kubeadm reset -f --cri-socket=unix:///run/cri-dockerd.sock

rm -rf /etc/cni/net.d/ ~/.kube /etc/kubernetes /opt/cni

dnf -y remove kubeadm kubelet kubectl

 

#4)建议重启清理环境

reboot

 

删除此 Master (ETCD节点) 在 ETCD 中信息
注意：如果此Master节点还充当ETCD成员, 此步必须执行，否则重新添加节点到kubernetes集群时可能会失败

在其他Master的任意一个节点上更新本地ETCD数据库,从中删除此ETCD节点信息

范例: 方法1：自行下载etcdctl进行如下操作

ETCD_VERSION=v3.6.12

wget https://github.com/etcdio/etcd/releases/download/${ETCD_VERSION}/etcd-v3.6.12-linux-amd64.tar.gz

tar xf etcd-${ETCD_VERSION}-linux-amd64.tar.gz

cd etcd-v3.6.12-linux-amd64

mv etcdctl /usr/local/bin/etcdctl

etcdctl --endpoints 127.0.0.1:2379 --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member list

[root@master1 etcd-v3.6.12-linux-amd64]# etcdctl --endpoints 127.0.0.1:2379 --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member list
30035185a83c6bab, started, master1.org, https://192.168.3.60:2380, https://192.168.3.60:2379, false
52740139c69d8255, started, master3.org, https://192.168.3.62:2380, https://192.168.3.62:2379, false
b034099815698f01, started, master2.org, https://192.168.3.61:2380, https://192.168.3.61:2379, false

 

#根据上面命令列出的Master3节点的ID,在容器内删除相应节点,只需要在Master1执行，会自动同步到其它ETCD节点

etcdctl --endpoints 127.0.0.1:2379 --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member remove 52740139c69d8255


 

添加 Master 节点到 Kubernetes 集群

获取添加 Master 的命令

范例：使用原有的添加Master节点的命令因为--certificate-key过期会出现下面错误

  kubeadm join kubeapi.org:6443 --token pfrv1q.qk7x7aloakfmk8fb \
        --discovery-token-ca-cert-hash sha256:f8bf9f366cde46b1bb0c3a609f3d57245e83175e536cb299ef28250b039b87d9  \
        --control-plane --certificate-key d52f8f1ec718c34ee50c8ea048b4885f06ef86249e6be826753167dd5bc4cc97 \
        --cri-socket=unix:///run/cri-dockerd.sock

 

[root@master3 ~]#  kubeadm join kubeapi.org:6443 --token pfrv1q.qk7x7aloakfmk8fb \
>         --discovery-token-ca-cert-hash sha256:f8bf9f366cde46b1bb0c3a609f3d57245e83175e536cb299ef28250b039b87d9  \
>         --control-plane --certificate-key d52f8f1ec718c34ee50c8ea048b4885f06ef86249e6be826753167dd5bc4cc97 \
>         --cri-socket=unix:///run/cri-dockerd.sock
[preflight] Running pre-flight checks
        [WARNING SystemVerification]: kernel release 5.14.0-611.5.1.el9_7.x86_64 is unsupported. Supported LTS versions from the 5.x series are 5.4, 5.10 and 5.15. Any 6.x version is also supported. For cgroups v2 support, the recommended version is 5.10 or newer
[preflight] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[preflight] Use 'kubeadm init phase upload-config kubeadm --config your-config-file' to re-upload it.
W0718 15:33:13.406076    1431 utils.go:69] The recommended value for "bindAddress" in "KubeProxyConfiguration" is: ::; the provided value is: 0.0.0.0
[preflight] Running pre-flight checks before initializing the new control plane instance
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action beforehand using 'kubeadm config images pull'
[download-certs] Downloading the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
error: error execution phase control-plane-prepare/download-certs: error downloading certs: error downloading the secret: Secret "kubeadm-certs" was not found in the "kube-system" Namespace. This Secret might have expired. Please, run `kubeadm init phase upload-certs --upload-certs` on a control plane to generate a new one
To see the stack trace of this error execute with --v=5 or higher
[root@master3 ~]#

 

 

#在master1节点执行上传CA证书，并生成加入集群所需的certificate-key，此key的有效期默认只有2小
时，只在添加master节点才需要

kubeadm init phase upload-certs --upload-certs

......
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kubesystem" Namespace
[upload-certs] Using certificate key:
75d7b4c68674cff2f1bda7c69f668f2a850649031fb413e75096ee814307f9fe #记录此certificate-key

 

kubeadm init phase upload-certs --upload-certs
I0718 15:37:36.535799   11735 version.go:260] remote version is much newer: v1.36.2; falling back to: stable-1.35
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
c5cf1bba9aeb42ffe9869c0e95ba5bbaf6d188ff2ea200d02b77d6e47f65ea12

 

 

#利用上面certificate-key,重新生成添加控制平面节点的命令

kubeadm token create --print-join-command --certificate-key <上面生成的certificate-key>

kubeadm token create --print-join-command --certificate-key c5cf1bba9aeb42ffe9869c0e95ba5bbaf6d188ff2ea200d02b77d6e47f65ea12


#结果如下

kubeadm join kubeapi.wang.org:6443 --token re9e0s.t9fuaqipopck9hzs  --discoverytoken-ca-cert-hash
sha256:cf6d6b66275bd9917563004bc6f71eb3391f083dcd434d109361ea0a7e5db2a8 --
control-plane --control-plane --certificate-key
75d7b4c68674cff2f1bda7c69f668f2a850649031fb413e75096ee814307f9fe

 

kubeadm join kubeapi.org:6443 --token n4uymy.iy2x00x1k5c0p5ge --discovery-token-ca-cert-hash sha256:f8bf9f366cde46b1bb0c3a609f3d57245e83175e536cb299ef28250b039b87d9 --control-plane --certificate-key c5cf1bba9aeb42ffe9869c0e95ba5bbaf6d188ff2ea200d02b77d6e47f65ea12 

kubeadm join kubeapi.org:6443 --token n4uymy.iy2x00x1k5c0p5ge --discovery-token-ca-cert-hash sha256:f8bf9f366cde46b1bb0c3a609f3d57245e83175e536cb299ef28250b039b87d9 --control-plane --certificate-key c5cf1bba9aeb42ffe9869c0e95ba5bbaf6d188ff2ea200d02b77d6e47f65ea12

 

kubeadm join kubeapi.org:6443 --token n4uymy.iy2x00x1k5c0p5ge --discovery-token-ca-cert-hash sha256:f8bf9f366cde46b1bb0c3a609f3d57245e83175e536cb299ef28250b039b87d9 --control-plane --certificate-key c5cf1bba9aeb42ffe9869c0e95ba5bbaf6d188ff2ea200d02b77d6e47f65ea12 --cri-socket=unix:///run/cri-dockerd.sock

 

#上面的信息中
--token re9e0s.t9fuaqipopck9hzs  #有效期默认24小时
--discovery-token-ca-cert-hash
sha256:cf6d6b66275bd9917563004bc6f71eb3391f083dcd434d109361ea0a7e5db2a8 #永久
--certificate-key
75d7b4c68674cff2f1bda7c69f668f2a850649031fb413e75096ee814307f9fe ##有效期默认2小时,
专用于添加master节点，添加worker节点不需要此项

将 Master 重新加入集群

#在新的节点先做环境初始化，主机名，域名解析，内核参数，swap禁用等

过程略

#在需要加入的master节点上执行上面生成的命令，如下面操作加入集群

kubeadm join kubeapi.org:6443 --token n4uymy.iy2x00x1k5c0p5ge --discovery-token-ca-cert-hash sha256:f8bf9f366cde46b1bb0c3a609f3d57245e83175e536cb299ef28250b039b87d9 --control-plane --certificate-key c5cf1bba9aeb42ffe9869c0e95ba5bbaf6d188ff2ea200d02b77d6e47f65ea12 --cri-socket=unix:///run/cri-dockerd.sock

 

授权 Master 管理功能

mkdir -p $HOME/.kube

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

#验证节点加入

[root@master1 ~]# kubectl get nodes
NAME          STATUS   ROLES           AGE     VERSION
master1.org   Ready    control-plane   30d     v1.35.6
master2.org   Ready    control-plane   5h28m   v1.35.6
master3.org   Ready    control-plane   45s     v1.35.6
node1.org     Ready    <none>          30d     v1.35.6
node2.org     Ready    <none>          30d     v1.35.6
node3.org     Ready    <none>          30d     v1.35.6

 

添加和删除 Worker 节点

添加新worker节点和删除worker节点说明

生产中,有时需要对 Worker 节点进行添加和删除操作
缩容：

排空节点上所有Pod

  • delete node 删除节点，过一会儿上面的Pod自动迁移到其它节点，重启kubelet.service服务后会
  • 自动再加该节点回到集群中
  • 或者用下面方式立即驱离节点上面的Pod
  • cordon：标记为不可被作为调度目标, uncordon：标记为可被作业调度目标
  • drain：排空

下线节点
删除节点资源


节点移除过程

#在master节点上删除worker节点信息

#添加污点node.kubernetes.io/unschedulable:NoSchedule,只影响新Pod

kubectl cordon node3.org 

驱离要删除worker节点上的所有pod

驱离已有的pod

kubectl drain node3.org --delete-emptydir-data --force --ignore-daemonsets

 #从集群中删除节点

kubectl delete node node3.org

 

[root@master1 ~]# kubectl get node
NAME          STATUS   ROLES           AGE     VERSION
master1.org   Ready    control-plane   30d     v1.35.6
master2.org   Ready    control-plane   5h42m   v1.35.6
master3.org   Ready    control-plane   14m     v1.35.6
node1.org     Ready    <none>          30d     v1.35.6
node2.org     Ready    <none>          30d     v1.35.6

 

 

#在node节点上移除数据和历史记录文件

清除worker节点的k8s信息

kubeadm reset --cri-socket=unix:///run/cri-dockerd.sock

systemctl stop kubelet docker

rm -rf /etc/kubernetes/ /etc/cni/net.d
dnf -y remove kubeadm kubelet kubectl

#建议重启清理环境
reboot

 

范例: 加入worker节点到kubernetes 集群

#重新加集群
#1)生成将worker节点加集群的命令

kubeadm token create --print-join-command

[root@master1 ~]# kubeadm token create --print-join-command
kubeadm join kubeapi.org:6443 --token imodin.5dem4n5qjvvt0cal --discovery-token-ca-cert-hash sha256:f8bf9f366cde46b1bb0c3a609f3d57245e83175e536cb299ef28250b039b87d9

 

2)在worker节点执行命令,重新加入集群

kubeadm join kubeapi.org:6443 --token imodin.5dem4n5qjvvt0cal --discovery-token-ca-cert-hash sha256:f8bf9f366cde46b1bb0c3a609f3d57245e83175e536cb299ef28250b039b87d9 --cri-socket=unix:///run/cri-dockerd.sock

 

#验证加入成功

[root@master1 ~]# kubectl get nodes 
NAME          STATUS   ROLES           AGE     VERSION
master1.org   Ready    control-plane   30d     v1.35.6
master2.org   Ready    control-plane   5h52m   v1.35.6
master3.org   Ready    control-plane   24m     v1.35.6
node1.org     Ready    <none>          30d     v1.35.6
node2.org     Ready    <none>          30d     v1.35.6
node3.org     Ready    <none>          20s     v1.35.6

 

 

 

Token 管理
默认情况下，kubeadm在创建集群的时候，会使用tls的方式传输信息，而node在加入集群的时候，也会应用该即token信息 ，如果忘了保存,也可以通过如下方式来获取，然后将kubeadm join的命令更改一下即可。
--token 对应的有效期默认为24小时
可以用kubeadm init --token-ttl=0 指定永久有效

 

查看当前 token

#生成加入节点的命令方法1：
#查看各个token的值和时效
kubeadm token list

 

#查看discovery-token-ca-cert-hash值，此hash值是永久不变的

openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

#基于上面信息，生成添加Worker节点命令

kubeadm join kubeapi.wang.org:6443 --token <token值> --discovery-token-ca-certhash sha256:<hash值>

 

#生成加入节点的命令方法2：直接生成加入集群的命令
kubeadm token create --print-join-command
#直接生成加入节点的命令如下形式
kubeadm join kubeapi.wang.org:6443 --token rv0cgm.h9e1xclmgq8nu4an --discoverytoken-ca-cert-hash sha256:2ac34514b2ea12b7e20b62785b9fea6ce52a6925aa81c2bad789473049f68e0f

 

范例:

[root@master1 ~]# kubeadm token list
TOKEN                     TTL         EXPIRES                USAGES                   DESCRIPTION                                                EXTRA GROUPS
imodin.5dem4n5qjvvt0cal   23h         2026-07-19T08:10:47Z   authentication,signing   <none>                                                     system:bootstrappers:kubeadm:default-node-token
kdsu2n.vjgz3mfs5rlh9bey   1h          2026-07-18T09:37:36Z   <none>                   Proxy for managing TTL for the kubeadm-certs secret        <none>
n4uymy.iy2x00x1k5c0p5ge   23h         2026-07-19T07:38:10Z   authentication,signing   <none>                                                     system:bootstrappers:kubeadm:default-node-token
pfrv1q.qk7x7aloakfmk8fb   <forever>   <never>   authentication,signing   The default bootstrap token generated by 'kubeadm init'.   system:bootstrappers:kubeadm:default-node-token
vq7i91.fqugak7alckehm7a   17h         2026-07-19T02:18:39Z   authentication,signing   <none>                                                     system:bootstrappers:kubeadm:default-node-token
[root@master1 ~]#

 

#查询discovery-token-ca-cert-hash的值

openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

[root@master1 ~]# openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
f8bf9f366cde46b1bb0c3a609f3d57245e83175e536cb299ef28250b039b87d9
[root@master1 ~]#

 

[root@master1 ~]# openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
f8bf9f366cde46b1bb0c3a609f3d57245e83175e536cb299ef28250b039b87d9

 

#利用上面信息生成加入Worker节点的命令

kubeadm join kubeapi.wang.org:6443 --token pfrv1q.qk7x7aloakfmk8fb --discovery-token-ca-cert-hash sha256:f8bf9f366cde46b1bb0c3a609f3d57245e83175e536cb299ef28250b039b87d9

 

重新生成 token
默认Token有效期为24小时，如果找不到原有的token或者已经失效，也可以重新生成 Token 信息，但原有未过期的token还可以使用

kubeadm token create --print-join-command

范例

[root@master1 ~]#kubeadm token create --print-join-command
#生成以下信息
kubeadm join kubeapi.wang.org:6443 --token n5upn2.e8qsbm4bimge7exe --discoverytoken-ca-cert-hash 
sha256:cf6d6b66275bd9917563004bc6f71eb3391f083dcd434d109361ea0a7e5db2a8

 

posted @ 2026-07-19 12:16  minger_lcm  阅读(11)  评论(0)    收藏  举报