k8s更新证书
1查看crt的证书有效期，有二种方式
第一种是利用openssl查看crt证书
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text |grep ' Not '
第二种方式使用certs命令查看
[root@master tls-setup]# kubeadm alpha certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Dec 02, 2022 06:43 UTC 133d no
apiserver Dec 02, 2022 06:43 UTC 133d ca no
apiserver-etcd-client Dec 02, 2022 06:43 UTC 133d etcd-ca no
apiserver-kubelet-client Dec 02, 2022 06:43 UTC 133d ca no
controller-manager.conf Dec 02, 2022 06:43 UTC 133d no
etcd-healthcheck-client Dec 02, 2022 06:43 UTC 133d etcd-ca no
etcd-peer Dec 02, 2022 06:43 UTC 133d etcd-ca no
etcd-server Dec 02, 2022 06:43 UTC 133d etcd-ca no
front-proxy-client Dec 02, 2022 06:43 UTC 133d front-proxy-ca no
scheduler.conf Dec 02, 2022 06:43 UTC 133d no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Nov 30, 2031 06:43 UTC 9y no
etcd-ca Nov 30, 2031 06:43 UTC 9y no
front-proxy-ca Nov 30, 2031 06:43 UTC 9y no
这里我所有证书是到2022-12-02
其中，ca.crt这些根证书是十年，一般也够用了
2生成配置文件和备份
kubeadm config view > /root/kubeadm.yaml
mkdir /etc/kubernetes.bak
cp -r /etc/kubernetes/* /etc/kubernetes.bak
cp -r /var/lib/etcd /var/lib/etcd.bak
mv /etc/kubernetes/admin.conf /etc/kubernetes/admin.conf.bak
mv /etc/kubernetes/kubelet.conf /etc/kubernetes/kubelet.conf.bak
mv /etc/kubernetes/controller-manager.conf /etc/kubernetes/controller-manager.conf.bak
mv /etc/kubernetes/scheduler.conf /etc/kubernetes/scheduler.conf.bak
3更新证书
更新所有证书
[root@master tls-setup]# kubeadm alpha certs renew all --config=/root/kubeadm.yaml
W0721 16:23:09.411913 30234 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
MISSING! certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
MISSING! certificate embedded in the kubeconfig file for the controller manager to use
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
MISSING! certificate embedded in the kubeconfig file for the scheduler manager to use
[root@master tls-setup]# kubeadm alpha certs check-expiration
W0721 16:24:32.730372 30474 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
!MISSING! admin.conf
apiserver Jul 21, 2023 08:23 UTC 364d ca no
apiserver-etcd-client Jul 21, 2023 08:23 UTC 364d etcd-ca no
apiserver-kubelet-client Jul 21, 2023 08:23 UTC 364d ca no
!MISSING! controller-manager.conf
etcd-healthcheck-client Jul 21, 2023 08:23 UTC 364d etcd-ca no
etcd-peer Jul 21, 2023 08:23 UTC 364d etcd-ca no
etcd-server Jul 21, 2023 08:23 UTC 364d etcd-ca no
front-proxy-client Jul 21, 2023 08:23 UTC 364d front-proxy-ca no
!MISSING! scheduler.conf
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Nov 30, 2031 06:43 UTC 9y no
etcd-ca Nov 30, 2031 06:43 UTC 9y no
front-proxy-ca Nov 30, 2031 06:43 UTC 9y no
更新kubeconfig配置文件
root@master tls-setup]# kubeadm init phase kubeconfig all --config /root/kubeadm.yaml
W0721 16:54:19.723619 5064 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[root@master tls-setup]# kubeadm alpha certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Jul 21, 2023 08:54 UTC 364d no
apiserver Jul 21, 2023 08:23 UTC 364d ca no
apiserver-etcd-client Jul 21, 2023 08:23 UTC 364d etcd-ca no
apiserver-kubelet-client Jul 21, 2023 08:23 UTC 364d ca no
controller-manager.conf Jul 21, 2023 08:54 UTC 364d no
etcd-healthcheck-client Jul 21, 2023 08:23 UTC 364d etcd-ca no
etcd-peer Jul 21, 2023 08:23 UTC 364d etcd-ca no
etcd-server Jul 21, 2023 08:23 UTC 364d etcd-ca no
front-proxy-client Jul 21, 2023 08:23 UTC 364d front-proxy-ca no
scheduler.conf Jul 21, 2023 08:54 UTC 364d no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Nov 30, 2031 06:43 UTC 9y no
etcd-ca Nov 30, 2031 06:43 UTC 9y no
front-proxy-ca Nov 30, 2031 06:43 UTC 9y no
更新成功
更新kubelet证书
cp -r /var/lib/kubelet/pki /var/lib/kubelet/pki_old
rm -rf /var/lib/kubelet/pki4重启pod生效证书
# docker ps | grep -E 'k8s_kube-apiserver|k8s_kube-controller-manager|k8s_kube-scheduler|k8s_etcd_etcd' | awk -F ' ' '{print $1}' | xargs docker restart
5验证
[root@master tls-setup]# openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text |grep ' Not '
Not Before: Dec 2 06:43:52 2021 GMT
Not After : Jul 21 08:23:10 2023 GMT
[root@master tls-setup]# openssl x509 -in /etc/kubernetes/pki/apiserver-etcd-client.crt -noout -text |grep ' Not '
Not Before: Dec 2 06:43:55 2021 GMT
Not After : Jul 21 08:23:10 2023 GMT
[root@master tls-setup]# openssl x509 -in /etc/kubernetes/pki/ca.crt -noout -text |grep ' Not '
Not Before: Dec 2 06:43:52 2021 GMT
Not After : Nov 30 06:43:52 2031 GMT
[root@master tls-setup]# openssl x509 -in /etc/kubernetes/pki/front-proxy-ca.crt -noout -text |grep ' Not '
Not Before: Dec 2 06:43:54 2021 GMT
Not After : Nov 30 06:43:54 2031 GMT
[root@master tls-setup]# kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
default demo-7d4bb77dc8-dfzjj 1/1 Running 0 8d
ingress-nginx ingress-nginx-admission-create-qdnzw 0/1 Completed 0 8d
ingress-nginx ingress-nginx-admission-patch-wk9zf 0/1 Completed 1 8d
ingress-nginx ingress-nginx-controller-57fdb6d45b-pj7t4 1/1 Running 0 8d
kube-system coredns-7ff77c879f-kf2c2 1/1 Running 0 50d
kube-system coredns-7ff77c879f-vq4bg 1/1 Running 0 217d
kube-system etcd-master 1/1 Running 0 231d
kube-system kube-apiserver-master 1/1 Running 0 231d
kube-system kube-controller-manager-master 1/1 Running 10 231d
kube-system kube-proxy-5wgv7 1/1 Running 4 231d
kube-system kube-proxy-mfjvb 1/1 Running 0 231d
kube-system kube-scheduler-master 1/1 Running 10 231d
kube-system metrics-server-74b9445bdf-f9mr4 1/1 Running 0 71d
kube-system weave-net-75frc 2/2 Running 7 231d
kube-system weave-net-wfb6l 2/2 Running 0 231d
monitoring alertmanager-my-release-prometheus-oper-alertmanager-0 2/2 Running 0 224d
monitoring my-adapter-prometheus-adapter-7c68497875-7lxzk 1/1 Running 11 84d
monitoring my-release-grafana-675c9d959f-p426b 2/2 Running 0 217d
monitoring my-release-kube-state-metrics-844f86b568-9rjlm 1/1 Running 0 217d
monitoring my-release-prometheus-node-exporter-46gdg 1/1 Running 1 224d
monitoring my-release-prometheus-node-exporter-tt7z4 1/1 Running 0 224d
monitoring my-release-prometheus-oper-operator-65565f586-82t4g 2/2 Running 0 224d
monitoring prometheus-my-release-prometheus-oper-prometheus-0 3/3 Running 1 70d
[root@master tls-setup]# kubectl get node
NAME STATUS ROLES AGE VERSION
master Ready master 231d v1.18.0
node2 Ready <none> 231d v1.18.0
master2
master3
由于到期时间还是一年，这里还是添加一下监控告警
部署（所有节点）
# replace with latest version released
VERSION=3.X.X
TMP=$(mktemp)
curl -L -o ${TMP} https://github.com/enix/x509-certificate-exporter/releases/download/v${VERSION}/x509-certificate-exporter-linux-amd64.tar.gz
sudo tar xvfz ${TMP} -C /usr/local/bin/ x509-certificate-exporter
sudo chown root:root /usr/local/bin/x509-certificate-exporter
sudo chmod 755 /usr/local/bin/x509-certificate-exporter
curl -L -o ${TMP} https://raw.githubusercontent.com/enix/x509-certificate-exporter/master/deploy/linux/x509-certificate-exporter.service
sudo install -o root -g root -m 644 ${TMP} /etc/systemd/system/x509-certificate-exporter.service
rm -f ${TMP}
# edit exporter arguments in /etc/systemd/system/x509-certificate-exporter.service
# no configuration is supported
sudo systemctl daemon-reload
sudo systemctl enable --now x509-certificate-exporter
sudo systemctl status x509-certificate-exporter编辑配置文件/etc/systemd/system/x509-certificate-exporter.service
[root@hybxvuca01 ~]# cat /etc/systemd/system/x509-certificate-exporter.service
[Unit]
Description=Prometheus exporter for X.509 certificates
Documentation=https://github.com/enix/x509-certificate-exporter
After=network.target
[Service]
Type=exec
Restart=on-failure
ExecStart=/usr/local/bin/x509-certificate-exporter --watch-kubeconf=/etc/kubernetes/admin.conf --watch-kubeconf=/etc/kubernetes/controller-manager.conf --watch-kubeconf=/etc/kubernetes/scheduler.conf --watch-dir=/etc/kubernetes/pki --watch-file=/var/lib/kubelet/pki/kubelet-client-current.pem
MemoryAccounting=true
MemoryMax=32M
[Install]
WantedBy=multi-user.target接入prom
[root@hybxvuca01 ~]# cd /app/stage/monitoring/
#添加自定义监控
[root@hybxvuca01 monitoring]# cat prometheus-additional.yaml
- job_name: 'rym-user-center'
metrics_path: '/actuator/prometheus'
scrape_interval: 2s
static_configs:
- targets: ['10.153.184.46:17030']
labels:
application: 'rym-user-center'
- job_name: 'x509-certificate-exporter'
metrics_path: '/metrics'
scrape_interval: 2s
static_configs:
- targets: ['10.152.194.151:9793','10.152.194.152:9793','10.152.194.153:9793','10.152.194.154:9793','10.152.194.155:9793','10.152.194.156:9793','10.152.194.6:9793']
labels:
application: 'x509-certificate-exporter'
# 更新secret
kubectl create secret generic additional-scrape-configs --from-file=prometheus-additional.yaml -n kubesphere-monitoring-system --dry-run -o yaml > additional-scrape-configs.yaml
kubectl apply -f additional-scrape-configs.yaml 验证
接入grafana
验证
浙公网安备 33010602011771号