k8s kubectl命令报错 提示证书到期。
线上MongoDB环境运行358day天，2月20号提示就过期了
网上找文档进行更新延期一年
1、查看证书到期时间： kubeadm certs check-expiration 2、备份k8s配置 cp -rp /etc/kubernetes /etc/kubernetes.bak 3、删除旧的证书 rm -rf /etc/kubernetes/pki/apiserver.key 4、重新生成全部证书 kubeadm certs renew all 5、备份旧配置文件 mv /etc/kubernetes/*.conf /tmp/ 6、生成所有配置文件 kubeadm init phase kubeconfig all 7、重启kubelet systemctl restart kubelet 8、替换kubeconfig cp /etc/kubernetes/admin.conf ~/.kube/config 9、查看证书时间（续一年） kubeadm alpha certs check-expiration
替换后问题复现:
kubectl查看pod正常，线上环境没有数据，kubectl delete pod mongo重新apply出一个pod，volumes,configMap,service都没动，
root@xxx k8s]# cat mongodb.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: mongodb
spec:
replicas: 1
serviceName: mongodb
selector:
matchLabels:
name: mongodb
template:
metadata:
labels:
name: mongodb
apply之后get找不到，必须kubectl get statefulset mongodb才能出来状态是pending状态，
kubectl describe pod statefulset mongodb 查看没有任务事件：Events: <none>
排查pending原因： 资源不足，调度限制，节点污点，网络问题都不是；
之后重新run了一个特别小的busybox的pod尝试一下
kubectl run busybox --image busybox:1.28 --restart=Never --rm -it busybox -- sh
还是pending状态，describe没有任何事件，
排查k8s当中的组件：(加蓝是错误日志，其它是重启组件之后日志)
kube-apiserver
ERROR $root.definitions.org.projectcalico.crd.v1.NetworkPolicy.properties.spec.properties.ingress.items.<array>.properties.protocol has invalid property: anyOf
ERROR $root.definitions.org.projectcalico.crd.v1.NetworkPolicy.properties.spec.properties.ingress.items.<array>.properties.source.properties.notPorts.items.<array> has invalid property: anyOf
ERROR $root.definitions.org.projectcalico.crd.v1.NetworkPolicy.properties.spec.properties.ingress.items.<array>.properties.source.properties.ports.items.<array> has invalid property: anyOf
I0223 07:04:55.917852 1 client.go:360] parsed scheme: "endpoint"
I0223 07:04:55.917882 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 <nil> 0 <nil>}]
I0223 07:04:56.216268 1 controller.go:132] OpenAPI AggregationController: action for item : Nothing (removed from the queue).
I0223 07:04:56.216303 1 controller.go:132] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue).
I0223 07:04:56.222289 1 storage_scheduling.go:148] all system priority classes are created successfully or already exist.
I0223 07:05:26.959321 1 controller.go:609] quota admission added evaluator for: controllerrevisions.apps
I0223 07:05:26.992086 1 controller.go:609] quota admission added evaluator for: endpointslices.discovery.k8s.io
I0223 07:05:27.000263 1 controller.go:609] quota admission added evaluator for: events.events.k8s.io
I0223 07:05:27.215105 1 controller.go:609] quota admission added evaluator for: endpoints
I0223 07:05:27.403804 1 client.go:360] parsed scheme: "endpoint"
I0223 07:05:27.403837 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 <nil> 0 <nil>}]
日志提示: 解析名为
anyOf 的属性时出现了错误。根据错误消息，问题似乎与 org.projectcalico.crd.v1.NetworkPolicy 自定义资源定义（CRD）中的某个属性spec.ingress[].source.ports[].<array>.anyOf 这个属性被认为是无效的。网络组件node节点calico无法与apiserver通信
kube-controller-manage
E0223 07:04:21.193538 1 leaderelection.go:325] error retrieving resource lock kube-system/kube-controller-manager: Unauthorized
E0223 07:04:23.333589 1 leaderelection.go:325] error retrieving resource lock kube-system/kube-controller-manager: Unauthorized
E0223 07:04:26.740817 1 leaderelection.go:325] error retrieving resource lock kube-system/kube-controller-manager: Unauthorized
E0223 07:04:30.059519 1 leaderelection.go:325] error retrieving resource lock kube-system/kube-controller-manager: Unauthorized
E0223 07:04:32.357466 1 leaderelection.go:325] error retrieving resource lock kube-system/kube-controller-manager: Unauthorized
E0223 07:04:34.528863 1 leaderelection.go:325] error retrieving resource lock kube-system/kube-controller-manager: Unauthorized
E0223 07:04:38.329459 1 leaderelection.go:325] error retrieving resource lock kube-system/kube-controller-manager: Unauthorized
E0223 07:04:42.637725 1 leaderelection.go:325] error retrieving resource lock kube-system/kube-controller-manager: Unauthorized
E0223 07:04:46.430677 1 leaderelection.go:325] error retrieving resource lock kube-system/kube-controller-manager: Unauthorized
E0223 07:04:49.123341 1 leaderelection.go:325] error retrieving resource lock kube-system/kube-controller-manager: Unauthorized
E0223 07:04:51.741695 1 leaderelection.go:325] error retrieving resource lock kube-system/kube-controller-manager: Get "https://192.168.84.82:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager?timeout=10s": dial tcp 192.168.84.82:6443: connect: connection refused
E0223 07:04:55.348674 1 leaderelection.go:325] error retrieving resource lock kube-system/kube-controller-manager: Unauthorized
Flag --port has been deprecated, see --secure-port instead.
I0223 07:04:58.166706 1 serving.go:331] Generated self-signed cert in-memory
I0223 07:04:58.544156 1 controllermanager.go:176] Version: v1.20.6
I0223 07:04:58.544943 1 dynamic_cafile_content.go:167] Starting request-header::/etc/kubernetes/pki/front-proxy-ca.crt
I0223 07:04:58.544947 1 dynamic_cafile_content.go:167] Starting client-ca-bundle::/etc/kubernetes/pki/ca.crt
I0223 07:04:58.545367 1 secure_serving.go:197] Serving securely on 127.0.0.1:10257
I0223 07:04:58.545410 1 leaderelection.go:243] attempting to acquire leader lease kube-system/kube-controller-manager...
I0223 07:04:58.545443 1 tlsconfig.go:240] Starting DynamicServingCertificateController
I0223 07:05:15.757267 1 leaderelection.go:253] successfully acquired lease kube-system/kube-controller-manager
I0223 07:05:15.757476 1 event.go:291] "Event occurred" object="kube-system/kube-controller-manager" kind="Lease" apiVersion="coordination.k8s.io/v1" type="Normal" reason="LeaderElection" message="bja-public-n9e00.bj_b79dc398-8623-4b9b-b1a0-aefa32f8d188 became leader"
I0223 07:05:16.266609 1 shared_informer.go:240] Waiting for caches to sync for tokens
I0223 07:05:16.269190 1 controllermanager.go:554] Started "bootstrapsigner"
日志提示：错误消息 "Unauthorized" 表示你没有足够的权限来检索资源锁定（resource lock）。这可能是因为你使用的身份验证凭据没有足够的权限执行此操作
kube-scheduler
E0223 07:04:55.265533 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Pod: unknown (get pods)
E0223 07:04:55.265557 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.ReplicationController: unknown (get replicationcontrollers)
E0223 07:04:55.266562 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.StatefulSet: unknown (get statefulsets.apps)
E0223 07:04:55.269981 1 leaderelection.go:325] error retrieving resource lock kube-system/kube-scheduler: leases.coordination.k8s.io "kube-scheduler" is forbidden: User "system:kube-scheduler" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "kube-system"
I0223 07:05:04.469328 1 serving.go:331] Generated self-signed cert in-memory
I0223 07:05:05.052634 1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0223 07:05:05.052664 1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
I0223 07:05:05.052677 1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0223 07:05:05.052685 1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-filec
日志提示: kube-scheduler 组件无法监听
v1.Pod 资源的变化。原因是 kube-scheduler 没有足够的权限来获取 Pod 资源。
解决方法:
重启控制节点kube-apiserver、kube-controller-manager、kube-scheduler组件容器
docker ps |grep kube-apiserver|grep -v pause|awk '{print $1}'|xargs -i docker restart {}
docker ps |grep kube-controller-manage|grep -v pause|awk '{print $1}'|xargs -i docker restart {}
docker ps |grep kube-scheduler|grep -v pause|awk '{print $1}'|xargs -i docker restart {}
查看kube-apiserver、kube-controller-manager、kube-scheduler日志不在打印错误日志就ok了
kubectl get pods 查看pod状态
参考链接:https://blog.csdn.net/caryeko/article/details/134077153
浙公网安备 33010602011771号