36.B站薪享宏福笔记——第十三章（1）Cordon & unCordon & Drain
13 B站薪享宏福笔记——第十三章
—— Kubernetes 中必备的工具组件
13.1 Cordon & unCordon & Drain
—— 安全的清空节点
13.1.1 Cordon & unCordon
（1）Cordon 禁止参加调度
注意：设置节点不可调度，仅仅是节点不再参与调度，新的 pod 不会被调度到该节点上，但已存在的 pod 不会被驱逐（与 taint 污点 noschedule 原理相同）
# 语法
kubectl cordon Node [options]
[root@k8s-master01 13.1]# kubectl get node NAME STATUS ROLES AGE VERSION k8s-master01 Ready control-plane 90d v1.29.2 k8s-node01 Ready <none> 90d v1.29.2 k8s-node02 Ready <none> 90d v1.29.2 # 主节点因为有 NoSchedule 不进行 Pod 的调度
[root@k8s-master01 13.1]# kubectl describe node k8s-master01|grep Taints Taints: node-role.kubernetes.io/control-plane:NoSchedule [root@k8s-master01 13.1]# kubectl describe node k8s-node01|grep Taints Taints: <none> [root@k8s-master01 13.1]# kubectl cordon k8s-node01 node/k8s-node01 cordoned [root@k8s-master01 13.1]# kubectl describe node k8s-node01|grep Taints Taints: node.kubernetes.io/unschedulable:NoSchedule [root@k8s-master01 13.1]# kubectl get node NAME STATUS ROLES AGE VERSION k8s-master01 Ready control-plane 90d v1.29.2 k8s-node01 Ready,SchedulingDisabled <none> 90d v1.29.2 k8s-node02 Ready <none> 90d v1.29.2 [root@k8s-master01 13.1]# kubectl create deployment myapp --image=myapp:v1.0 --replicas=10 deployment.apps/myapp created
# 全部 master01、node01 都有 NoSchedule，只能调度到 node02 节点上 [root@k8s-master01 13.1]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES myapp-77d465c645-2jrpg 1/1 Running 0 59s 10.244.58.232 k8s-node02 <none> <none> myapp-77d465c645-4c5zh 1/1 Running 0 59s 10.244.58.234 k8s-node02 <none> <none> myapp-77d465c645-4dgj6 1/1 Running 0 59s 10.244.58.243 k8s-node02 <none> <none> myapp-77d465c645-6pgjs 1/1 Running 0 59s 10.244.58.223 k8s-node02 <none> <none> myapp-77d465c645-7tmzv 1/1 Running 0 59s 10.244.58.244 k8s-node02 <none> <none> myapp-77d465c645-fvqbl 1/1 Running 0 59s 10.244.58.239 k8s-node02 <none> <none> myapp-77d465c645-llvbj 1/1 Running 0 59s 10.244.58.229 k8s-node02 <none> <none> myapp-77d465c645-swplb 1/1 Running 0 59s 10.244.58.235 k8s-node02 <none> <none> myapp-77d465c645-t89hp 1/1 Running 0 59s 10.244.58.231 k8s-node02 <none> <none> myapp-77d465c645-vrkd5 1/1 Running 0 59s 10.244.58.240 k8s-node02 <none> <none>
（2）unCordon 恢复调度
# 语法
kubectl uncordon Node [options]
# 取消污点
[root@k8s-master01 13.1]# kubectl uncordon k8s-node01 node/k8s-node01 uncordoned # 污点的 NoSchedule 已经消失，变成可调度状态 [root@k8s-master01 13.1]# kubectl describe node k8s-node01|grep Taints Taints: <none> [root@k8s-master01 13.1]# kubectl get node NAME STATUS ROLES AGE VERSION k8s-master01 Ready control-plane 91d v1.29.2 k8s-node01 Ready <none> 91d v1.29.2 k8s-node02 Ready <none> 91d v1.29.2
13.1.2 Drain 排空
（1）实验
注意：设置排空，不仅节点不可调度，原节点上 Pod 也将被排除，可以看到驱离的过程，阻塞型命令
# 语法 kubectl drain Node [options] # 常用命令 kubectl drain Node --ignore-daemonsets=true --grace-period=900 --ignore-daemonsets=true 跳过 daemonset 控制器（ daemonset 通常部署 calico、ingress-nginx、node-exporter 等非业务 Pod 运行） --grace-period=900 指定宽限期
# 正常节点状态，node01、node02 正常调度 [root@k8s-master01 13.1]# kubectl get node NAME STATUS ROLES AGE VERSION k8s-master01 Ready control-plane 91d v1.29.2 k8s-node01 Ready <none> 91d v1.29.2 k8s-node02 Ready <none> 91d v1.29.2 [root@k8s-master01 13.1]# kubectl describe node k8s-node02|grep Taints Taints: <none> # node01、node02 负载调度正常 每个节点 5 个 Pod [root@k8s-master01 13.1]# kubectl create deployment myapp --image=myapp:v1.0 --replicas=10 deployment.apps/myapp created [root@k8s-master01 13.1]# kubectl get pod -o wide --show-labels NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES LABELS myapp-77d465c645-29vzj 1/1 Running 0 21m 10.244.58.192 k8s-node02 <none> <none> app=myapp,pod-template-hash=77d465c645 myapp-77d465c645-5vt5j 1/1 Running 0 21m 10.244.58.228 k8s-node02 <none> <none> app=myapp,pod-template-hash=77d465c645 myapp-77d465c645-7gtdj 1/1 Running 0 21m 10.244.58.247 k8s-node02 <none> <none> app=myapp,pod-template-hash=77d465c645 myapp-77d465c645-ckdjk 1/1 Running 0 21m 10.244.85.199 k8s-node01 <none> <none> app=myapp,pod-template-hash=77d465c645 myapp-77d465c645-fgrbd 1/1 Running 0 21m 10.244.85.221 k8s-node01 <none> <none> app=myapp,pod-template-hash=77d465c645 myapp-77d465c645-gcbtx 1/1 Running 0 21m 10.244.85.250 k8s-node01 <none> <none> app=myapp,pod-template-hash=77d465c645 myapp-77d465c645-q9gqh 1/1 Running 0 21m 10.244.58.242 k8s-node02 <none> <none> app=myapp,pod-template-hash=77d465c645 myapp-77d465c645-shn8q 1/1 Running 0 21m 10.244.85.200 k8s-node01 <none> <none> app=myapp,pod-template-hash=77d465c645 myapp-77d465c645-sl5pk 1/1 Running 0 21m 10.244.85.207 k8s-node01 <none> <none> app=myapp,pod-template-hash=77d465c645 myapp-77d465c645-wxztr 1/1 Running 0 21m 10.244.58.199 k8s-node02 <none> <none> app=myapp,pod-template-hash=77d465c645 # 这里使用了 --pod-selector=app=myapp 标签选择器，因为 之前部署的 loki 等有数据，drain 时提示是否需要强制删除，这里为了演示，就只匹配 pod 就可以 # 有驱逐的过程 [root@k8s-master01 13.1]# kubectl drain k8s-node02 --ignore-daemonsets=true --grace-period=900 --pod-selector=app=myapp node/k8s-node02 cordoned evicting pod default/myapp-77d465c645-wxztr evicting pod default/myapp-77d465c645-29vzj evicting pod default/myapp-77d465c645-5vt5j evicting pod default/myapp-77d465c645-7gtdj evicting pod default/myapp-77d465c645-q9gqh pod/myapp-77d465c645-q9gqh evicted pod/myapp-77d465c645-29vzj evicted pod/myapp-77d465c645-5vt5j evicted pod/myapp-77d465c645-7gtdj evicted pod/myapp-77d465c645-wxztr evicted node/k8s-node02 drained # 在驱逐的同时，为节点打上了不可调度的标签 [root@k8s-master01 13.1]# kubectl get node NAME STATUS ROLES AGE VERSION k8s-master01 Ready control-plane 91d v1.29.2 k8s-node01 Ready <none> 91d v1.29.2 k8s-node02 Ready,SchedulingDisabled <none> 91d v1.29.2 [root@k8s-master01 13.1]# kubectl describe node k8s-node02|grep Taints Taints: node.kubernetes.io/unschedulable:NoSchedule # 因为 myapp 是基于 deployment 控制器部署，所以驱逐后，都部署到了 k8s-node01 节点上 [root@k8s-master01 13.1]# kubectl get pod -o wide --show-labels NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES LABELS myapp-77d465c645-4ssf2 1/1 Running 0 4m10s 10.244.85.213 k8s-node01 <none> <none> app=myapp,pod-template-hash=77d465c645 myapp-77d465c645-ckdjk 1/1 Running 0 27m 10.244.85.199 k8s-node01 <none> <none> app=myapp,pod-template-hash=77d465c645 myapp-77d465c645-fgrbd 1/1 Running 0 27m 10.244.85.221 k8s-node01 <none> <none> app=myapp,pod-template-hash=77d465c645 myapp-77d465c645-gcbtx 1/1 Running 0 27m 10.244.85.250 k8s-node01 <none> <none> app=myapp,pod-template-hash=77d465c645 myapp-77d465c645-jmmqk 1/1 Running 0 4m10s 10.244.85.206 k8s-node01 <none> <none> app=myapp,pod-template-hash=77d465c645 myapp-77d465c645-jwhvl 1/1 Running 0 4m10s 10.244.85.254 k8s-node01 <none> <none> app=myapp,pod-template-hash=77d465c645 myapp-77d465c645-lknhq 1/1 Running 0 4m10s 10.244.85.217 k8s-node01 <none> <none> app=myapp,pod-template-hash=77d465c645 myapp-77d465c645-rp2bf 1/1 Running 0 4m10s 10.244.85.203 k8s-node01 <none> <none> app=myapp,pod-template-hash=77d465c645 myapp-77d465c645-shn8q 1/1 Running 0 27m 10.244.85.200 k8s-node01 <none> <none> app=myapp,pod-template-hash=77d465c645 myapp-77d465c645-sl5pk 1/1 Running 0 27m 10.244.85.207 k8s-node01 <none> <none> app=myapp,pod-template-hash=77d465c645
# 恢复使用 uncordon，本质上 drain 是在节点上打一个 cordon，然后删除节点上所有 pod，所以恢复只需要 对节点 uncordon 既可 [root@k8s-master01 13.1]# kubectl uncordon k8s-node02 node/k8s-node02 uncordoned [root@k8s-master01 13.1]# kubectl delete pod --all -n default pod "myapp-77d465c645-4ssf2" deleted pod "myapp-77d465c645-ckdjk" deleted pod "myapp-77d465c645-fgrbd" deleted pod "myapp-77d465c645-gcbtx" deleted pod "myapp-77d465c645-jmmqk" deleted pod "myapp-77d465c645-jwhvl" deleted pod "myapp-77d465c645-lknhq" deleted pod "myapp-77d465c645-rp2bf" deleted pod "myapp-77d465c645-shn8q" deleted pod "myapp-77d465c645-sl5pk" deleted [root@k8s-master01 13.1]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES myapp-77d465c645-2cmhh 1/1 Running 0 11s 10.244.85.195 k8s-node01 <none> <none> myapp-77d465c645-45wlq 1/1 Running 0 12s 10.244.58.251 k8s-node02 <none> <none> myapp-77d465c645-598r6 1/1 Running 0 12s 10.244.85.228 k8s-node01 <none> <none> myapp-77d465c645-97b56 1/1 Running 0 12s 10.244.85.222 k8s-node01 <none> <none> myapp-77d465c645-bmgr5 1/1 Running 0 11s 10.244.58.200 k8s-node02 <none> <none> myapp-77d465c645-kqxqr 1/1 Running 0 12s 10.244.58.253 k8s-node02 <none> <none> myapp-77d465c645-lnkv2 1/1 Running 0 11s 10.244.85.216 k8s-node01 <none> <none> myapp-77d465c645-lw85v 1/1 Running 0 12s 10.244.58.194 k8s-node02 <none> <none> myapp-77d465c645-qq854 1/1 Running 0 12s 10.244.85.236 k8s-node01 <none> <none> myapp-77d465c645-s4gwl 1/1 Running 0 12s 10.244.58.197 k8s-node02 <none> <none>
（2）概念
一旦执行了 kubectl drain <node-name> , Kubernetes 将开始迁移节点上的工作负载，直到所有的 Pod 都已经被成功调度到其他节点为止，在此过程中，新的 Pod 将不会被调度到正在排空的节点上，从而确保在维护期间不会有新的负载进入
排空过程中，Kubernetes 会等待节点上的 Pod 自然地完成它们的任务，并将其调度到其他节点上
对于无法重新调度的 Pod （比如有 nodename 关键字 ），可以使用 --force 标志来强制删除它们，完成排空后，节点将处于无工作负载的状态，可以安全地进行维护操作，完成维护后，可以使用 kubectl uncordon <node-name> 命令来重新启用节点，使其恢复正常的调度功能
（3）特别说明
kubectl drain 操作会将相应节点上的旧 Pod 删除，并在可调度节点上面起一个对应的 Pod，当旧 Pod 没有被正常删除的情况下，新 Pod 不会起来，例如：旧 Pod 一直处于 Terminating 状态
对应的解决方式是通过重启相应节点的 kubelet 或者强制删除该 Pod
# kubeadm 安装方式，重启发生了 Terminating 节点的 kubelet systemctl restart kubelet # 强制删除 Terminating 状态的 Pod kubectl delete pod <PodName> --namespace=<NameSpace> --force --grace-period=0
--force 强制删除
--grace-period=0 容忍时间为 0，从 etcd 中将 pod 元数据删除
13.1.3 总结
Taint（污点）更适合在维护前准备节点时，阻止新的 Pod 被调度到节点上（相同：Taint:NoSchedule、cordon、uncordon）（不允许新的 Pod 进入）
Drain 更适合在维护期间确保节点上的所有工作负载被安全地迁移（相同：Taint:NoExecute、Drain）（在节点已有的 Pod 全部删除，Drain 更理想会打印删除过程）
———————————————————————————————————————————————————————————————————————————
无敌小马爱学习
浙公网安备 33010602011771号