Kubernetes（k8s）中Pod资源的健康检查
Kubernetes（k8s）中Pod资源的健康检查
1、Pod的健康检查，也叫做探针，探针的种类有两种。
答：1）、livenessProbe，健康状态检查，周期性检查服务是否存活，检查结果失败，将重启容器。
2）、readinessProbe，可用性检查，周期性检查服务是否可用，不可用将从service的endpoints中移除。
2、探针的检测方法。
答：1）、exec，执行一段命令。
2）、httpGet，检测某个http请求的返回状态码。
3）、tcpSocket，测试某个端口是否能够连接。
3、创建exec的探针检测方法。
1 [root@k8s-master health]# vim pod_nginx_exec.yaml
1 apiVersion: v1 2 kind: Pod 3 metadata: 4 name: exec 5 spec: 6 containers: 7 - name: nginx 8 image: 192.168.110.133:5000/nginx:1.13 9 ports: 10 - containerPort: 80 11 args: 12 - /bin/sh 13 - -c 14 - touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600 15 livenessProbe: 16 exec: 17 command: 18 - cat 19 - /tmp/healthy 20 initialDelaySeconds: 5 21 periodSeconds: 5 22
开始创建这个Pod，操作如下所示：
1 [root@k8s-master health]# kubectl create -f pod_nginx_exec.yaml 2 pod "exec" created 3 [root@k8s-master health]# kubectl get all -o wide 4 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR 5 rc/mysql 1 1 1 1h mysql 192.168.110.133:5000/mysql:5.7.30 app=mysql 6 rc/myweb 1 1 1 1h myweb 192.168.110.133:5000/tomcat:latest app=myweb 7 8 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR 9 svc/kubernetes 10.254.0.1 <none> 443/TCP 13d <none> 10 svc/mysql 10.254.207.238 <none> 3306/TCP 1h app=mysql 11 svc/myweb 10.254.29.22 <nodes> 8080:30008/TCP 1h app=myweb 12 13 NAME READY STATUS RESTARTS AGE IP NODE 14 po/busybox2 1/1 Running 1 35m 172.16.16.3 k8s-master 15 po/exec 1/1 Running 0 15s 172.16.59.4 k8s-node2 16 po/mysql-lmx4s 1/1 Running 0 1h 172.16.59.2 k8s-node2 17 po/myweb-hsdwn 1/1 Running 0 1h 172.16.32.3 k8s-node3 18 [root@k8s-master health]#
可用通过详细查看Pod的启动状况。
1 [root@k8s-master health]# kubectl describe pod exec
2 Name: exec
3 Namespace: default
4 Node: k8s-node2/192.168.110.134
5 Start Time: Thu, 18 Jun 2020 16:36:16 +0800
6 Labels: <none>
7 Status: Running
8 IP: 172.16.59.4
9 Controllers: <none>
10 Containers:
11 nginx:
12 Container ID: docker://54e08c980bd1ad05c756de4de50b0f61c448a01b7ffcda2b97dc8b5c3a6e2749
13 Image: 192.168.110.133:5000/nginx:1.13
14 Image ID: docker-pullable://192.168.110.133:5000/nginx@sha256:e4f0474a75c510f40b37b6b7dc2516241ffa8bde5a442bde3d372c9519c84d90
15 Port: 80/TCP
16 Args:
17 /bin/sh
18 -c
19 touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600
20 State: Running
21 Started: Thu, 18 Jun 2020 16:37:32 +0800
22 Last State: Terminated
23 Reason: Error
24 Exit Code: 137
25 Started: Thu, 18 Jun 2020 16:36:18 +0800
26 Finished: Thu, 18 Jun 2020 16:37:32 +0800
27 Ready: True
28 Restart Count: 1
29 Liveness: exec [cat /tmp/healthy] delay=5s timeout=1s period=5s #success=1 #failure=3
30 Volume Mounts: <none>
31 Environment Variables: <none>
32 Conditions:
33 Type Status
34 Initialized True
35 Ready True
36 PodScheduled True
37 No volumes.
38 QoS Class: BestEffort
39 Tolerations: <none>
40 Events:
41 FirstSeen LastSeen Count From SubObjectPath Type Reason Message
42 --------- -------- ----- ---- ------------- -------- ------ -------
43 1m 1m 1 {default-scheduler } Normal Scheduled Successfully assigned exec to k8s-node2
44 1m 1m 1 {kubelet k8s-node2} spec.containers{nginx} Normal Created Created container with docker id ff079949b2cb; Security:[seccomp=unconfined]
45 1m 1m 1 {kubelet k8s-node2} spec.containers{nginx} Normal Started Started container with docker id ff079949b2cb
46 1m 1m 3 {kubelet k8s-node2} spec.containers{nginx} Warning Unhealthy Liveness probe failed: cat: /tmp/healthy: No such file or directory
47
48 1m 33s 2 {kubelet k8s-node2} spec.containers{nginx} Normal Pulled Container image "192.168.110.133:5000/nginx:1.13" already present on machine
49 33s 33s 1 {kubelet k8s-node2} spec.containers{nginx} Normal Killing Killing container with docker id ff079949b2cb: pod "exec_default(c69fbd14-b13e-11ea-80b4-000c2919d52d)" container "nginx" is unhealthy, it will be killed and re-created.
50 33s 33s 1 {kubelet k8s-node2} spec.containers{nginx} Normal Created Created container with docker id 54e08c980bd1; Security:[seccomp=unconfined]
51 33s 33s 1 {kubelet k8s-node2} spec.containers{nginx} Normal Started Started container with docker id 54e08c980bd1
52 [root@k8s-master health]# kubectl get all -o wide
53 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR
54 rc/mysql 1 1 1 1h mysql 192.168.110.133:5000/mysql:5.7.30 app=mysql
55 rc/myweb 1 1 1 1h myweb 192.168.110.133:5000/tomcat:latest app=myweb
56
57 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
58 svc/kubernetes 10.254.0.1 <none> 443/TCP 13d <none>
59 svc/mysql 10.254.207.238 <none> 3306/TCP 1h app=mysql
60 svc/myweb 10.254.29.22 <nodes> 8080:30008/TCP 1h app=myweb
61
62 NAME READY STATUS RESTARTS AGE IP NODE
63 po/busybox2 1/1 Running 1 37m 172.16.16.3 k8s-master
64 po/exec 1/1 Running 1 2m 172.16.59.4 k8s-node2
65 po/mysql-lmx4s 1/1 Running 0 1h 172.16.59.2 k8s-node2
66 po/myweb-hsdwn 1/1 Running 0 1h 172.16.32.3 k8s-node3
67 [root@k8s-master health]#
pod重启的次数，可以看到这个Pod的重启了多少次了。
4、创建httpGet的探针检测方法，liveness探针的httpGet使用。
1 [root@k8s-master health]# vim nginx_pod_httpGet.yaml
具体内容，如下所示：
1 apiVersion: v1 2 kind: Pod 3 metadata: 4 name: httpget 5 spec: 6 containers: 7 - name: nginx 8 image: 192.168.110.133:5000/nginx:1.13 9 ports: 10 - containerPort: 80 11 livenessProbe: 12 httpGet: 13 path: /index.html 14 port: 80 15 initialDelaySeconds: 3 16 periodSeconds: 3 17
创建httpGet的探针检测方法，liveness探针的httpGet使用。
1 [root@k8s-master health]# kubectl get all -o wide
2 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR
3 rc/mysql 1 1 1 1h mysql 192.168.110.133:5000/mysql:5.7.30 app=mysql
4 rc/myweb 1 1 1 1h myweb 192.168.110.133:5000/tomcat:latest app=myweb
5
6 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
7 svc/kubernetes 10.254.0.1 <none> 443/TCP 13d <none>
8 svc/mysql 10.254.207.238 <none> 3306/TCP 1h app=mysql
9 svc/myweb 10.254.29.22 <nodes> 8080:30008/TCP 1h app=myweb
10
11 NAME READY STATUS RESTARTS AGE IP NODE
12 po/busybox2 1/1 Running 1 46m 172.16.16.3 k8s-master
13 po/exec 0/1 CrashLoopBackOff 6 11m 172.16.59.4 k8s-node2
14 po/httpget 1/1 Running 0 22s 172.16.16.4 k8s-master
15 po/mysql-lmx4s 1/1 Running 0 1h 172.16.59.2 k8s-node2
16 po/myweb-hsdwn 1/1 Running 0 1h 172.16.32.3 k8s-node3
17 [root@k8s-master health]# kubectl describe pod httpget
18 Name: httpget
19 Namespace: default
20 Node: k8s-master/192.168.110.133
21 Start Time: Thu, 18 Jun 2020 16:47:03 +0800
22 Labels: <none>
23 Status: Running
24 IP: 172.16.16.4
25 Controllers: <none>
26 Containers:
27 nginx:
28 Container ID: docker://f9e0ea6a16a26d3fca57b1e1b521f642f89eb05e6951a05830933e68dce1253a
29 Image: 192.168.110.133:5000/nginx:1.13
30 Image ID: docker-pullable://docker.io/nginx@sha256:b1d09e9718890e6ebbbd2bc319ef1611559e30ce1b6f56b2e3b479d9da51dc35
31 Port: 80/TCP
32 State: Running
33 Started: Thu, 18 Jun 2020 16:47:04 +0800
34 Ready: True
35 Restart Count: 0
36 Liveness: http-get http://:80/index.html delay=3s timeout=1s period=3s #success=1 #failure=3
37 Volume Mounts: <none>
38 Environment Variables: <none>
39 Conditions:
40 Type Status
41 Initialized True
42 Ready True
43 PodScheduled True
44 No volumes.
45 QoS Class: BestEffort
46 Tolerations: <none>
47 Events:
48 FirstSeen LastSeen Count From SubObjectPath Type Reason Message
49 --------- -------- ----- ---- ------------- -------- ------ -------
50 1m 1m 1 {default-scheduler } Normal Scheduled Successfully assigned httpget to k8s-master
51 1m 1m 1 {kubelet k8s-master} spec.containers{nginx} Normal Pulled Container image "192.168.110.133:5000/nginx:1.13" already present on machine
52 1m 1m 1 {kubelet k8s-master} spec.containers{nginx} Normal Created Created container with docker id f9e0ea6a16a2; Security:[seccomp=unconfined]
53 1m 1m 1 {kubelet k8s-master} spec.containers{nginx} Normal Started Started container with docker id f9e0ea6a16a2
54 [root@k8s-master health]#
默认，这个首页index.html是可以访问到的，这里也可以让它访问不到的。
1 [root@k8s-master health]# kubectl exec -it httpget bash
2 root@httpget:/# cd /usr/share/html
3 bash: cd: /usr/share/html: No such file or directory
4 root@httpget:/# cd /usr/share/nginx/html
5 root@httpget:/usr/share/nginx/html# ls
6 50x.html index.html
7 root@httpget:/usr/share/nginx/html# mv index.html /tmp/
8 root@httpget:/usr/share/nginx/html# exit
9 exit
10 [root@k8s-master health]# kubectl get pod httpget
11 NAME READY STATUS RESTARTS AGE
12 httpget 1/1 Running 1 3m
13 [root@k8s-master health]# kubectl describe pod httpget
14 Name: httpget
15 Namespace: default
16 Node: k8s-master/192.168.110.133
17 Start Time: Thu, 18 Jun 2020 16:47:03 +0800
18 Labels: <none>
19 Status: Running
20 IP: 172.16.16.4
21 Controllers: <none>
22 Containers:
23 nginx:
24 Container ID: docker://7bb1ef7b33b24e16aad18053b3238077baac8e4b507ba85a6943dc9e9ae3ea29
25 Image: 192.168.110.133:5000/nginx:1.13
26 Image ID: docker-pullable://docker.io/nginx@sha256:b1d09e9718890e6ebbbd2bc319ef1611559e30ce1b6f56b2e3b479d9da51dc35
27 Port: 80/TCP
28 State: Running
29 Started: Thu, 18 Jun 2020 16:50:07 +0800
30 Last State: Terminated
31 Reason: Completed
32 Exit Code: 0
33 Started: Thu, 18 Jun 2020 16:47:04 +0800
34 Finished: Thu, 18 Jun 2020 16:50:06 +0800
35 Ready: True
36 Restart Count: 1
37 Liveness: http-get http://:80/index.html delay=3s timeout=1s period=3s #success=1 #failure=3
38 Volume Mounts: <none>
39 Environment Variables: <none>
40 Conditions:
41 Type Status
42 Initialized True
43 Ready True
44 PodScheduled True
45 No volumes.
46 QoS Class: BestEffort
47 Tolerations: <none>
48 Events:
49 FirstSeen LastSeen Count From SubObjectPath Type Reason Message
50 --------- -------- ----- ---- ------------- -------- ------ -------
51 3m 3m 1 {default-scheduler } Normal Scheduled Successfully assigned httpget to k8s-master
52 3m 3m 1 {kubelet k8s-master} spec.containers{nginx} Normal Created Created container with docker id f9e0ea6a16a2; Security:[seccomp=unconfined]
53 3m 3m 1 {kubelet k8s-master} spec.containers{nginx} Normal Started Started container with docker id f9e0ea6a16a2
54 3m 13s 2 {kubelet k8s-master} spec.containers{nginx} Normal Pulled Container image "192.168.110.133:5000/nginx:1.13" already present on machine
55 19s 13s 3 {kubelet k8s-master} spec.containers{nginx} Warning Unhealthy Liveness probe failed: HTTP probe failed with statuscode: 404
56 13s 13s 1 {kubelet k8s-master} spec.containers{nginx} Normal Killing Killing container with docker id f9e0ea6a16a2: pod "httpget_default(4857d09b-b140-11ea-80b4-000c2919d52d)" container "nginx" is unhealthy, it will be killed and re-created.
57 13s 13s 1 {kubelet k8s-master} spec.containers{nginx} Normal Created Created container with docker id 7bb1ef7b33b2; Security:[seccomp=unconfined]
58 12s 12s 1 {kubelet k8s-master} spec.containers{nginx} Normal Started Started container with docker id 7bb1ef7b33b2
59 [root@k8s-master health]# kubectl describe pod httpget
60 Name: httpget
61 Namespace: default
62 Node: k8s-master/192.168.110.133
63 Start Time: Thu, 18 Jun 2020 16:47:03 +0800
64 Labels: <none>
65 Status: Running
66 IP: 172.16.16.4
67 Controllers: <none>
68 Containers:
69 nginx:
70 Container ID: docker://7bb1ef7b33b24e16aad18053b3238077baac8e4b507ba85a6943dc9e9ae3ea29
71 Image: 192.168.110.133:5000/nginx:1.13
72 Image ID: docker-pullable://docker.io/nginx@sha256:b1d09e9718890e6ebbbd2bc319ef1611559e30ce1b6f56b2e3b479d9da51dc35
73 Port: 80/TCP
74 State: Running
75 Started: Thu, 18 Jun 2020 16:50:07 +0800
76 Last State: Terminated
77 Reason: Completed
78 Exit Code: 0
79 Started: Thu, 18 Jun 2020 16:47:04 +0800
80 Finished: Thu, 18 Jun 2020 16:50:06 +0800
81 Ready: True
82 Restart Count: 1
83 Liveness: http-get http://:80/index.html delay=3s timeout=1s period=3s #success=1 #failure=3
84 Volume Mounts: <none>
85 Environment Variables: <none>
86 Conditions:
87 Type Status
88 Initialized True
89 Ready True
90 PodScheduled True
91 No volumes.
92 QoS Class: BestEffort
93 Tolerations: <none>
94 Events:
95 FirstSeen LastSeen Count From SubObjectPath Type Reason Message
96 --------- -------- ----- ---- ------------- -------- ------ -------
97 3m 3m 1 {default-scheduler } Normal Scheduled Successfully assigned httpget to k8s-master
98 3m 3m 1 {kubelet k8s-master} spec.containers{nginx} Normal Created Created container with docker id f9e0ea6a16a2; Security:[seccomp=unconfined]
99 3m 3m 1 {kubelet k8s-master} spec.containers{nginx} Normal Started Started container with docker id f9e0ea6a16a2
100 3m 18s 2 {kubelet k8s-master} spec.containers{nginx} Normal Pulled Container image "192.168.110.133:5000/nginx:1.13" already present on machine
101 24s 18s 3 {kubelet k8s-master} spec.containers{nginx} Warning Unhealthy Liveness probe failed: HTTP probe failed with statuscode: 404
102 18s 18s 1 {kubelet k8s-master} spec.containers{nginx} Normal Killing Killing container with docker id f9e0ea6a16a2: pod "httpget_default(4857d09b-b140-11ea-80b4-000c2919d52d)" container "nginx" is unhealthy, it will be killed and re-created.
103 18s 18s 1 {kubelet k8s-master} spec.containers{nginx} Normal Created Created container with docker id 7bb1ef7b33b2; Security:[seccomp=unconfined]
104 17s 17s 1 {kubelet k8s-master} spec.containers{nginx} Normal Started Started container with docker id 7bb1ef7b33b2
105 [root@k8s-master health]#
5、liveness探针的tcpSocket使用。
1 [root@k8s-master health]# vim nginx_pod_tcpSocket.yaml
使用tcpSocket监控的是80的端口，配置文件的内容如下所示：
1 apiVersion: v1 2 kind: Pod 3 metadata: 4 name: tcpsocket 5 spec: 6 containers: 7 - name: nginx 8 image: 192.168.110.133:5000/nginx:1.13 9 ports: 10 - containerPort: 80 11 livenessProbe: 12 tcpSocket: 13 port: 80 14 initialDelaySeconds: 3 15 periodSeconds: 3 16
创建，如下所示：
1 [root@k8s-master health]# kubectl create -f nginx_pod_tcpSocket.yaml 2 pod "tcpsocket" created 3 [root@k8s-master health]# kubectl get all -o wide 4 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR 5 svc/kubernetes 10.254.0.1 <none> 443/TCP 13d <none> 6 7 NAME READY STATUS RESTARTS AGE IP NODE 8 po/busybox2 1/1 Running 1 54m 172.16.16.3 k8s-master 9 po/exec 0/1 CrashLoopBackOff 7 18m 172.16.59.4 k8s-node2 10 po/httpget 1/1 Running 1 8m 172.16.16.4 k8s-master 11 po/tcpsocket 1/1 Running 0 6s 172.16.59.2 k8s-node2
6、readiness探针的httpGet使用。
1 [root@k8s-master health]# vim nginx_rc_readiness.yaml
具体内容，如下所示：
需要访问qiangge.html，如果不存在就一直加入不到svc中。
1 apiVersion: v1 2 kind: ReplicationController 3 metadata: 4 name: readiness 5 spec: 6 replicas: 2 7 selector: 8 app: readiness 9 template: 10 metadata: 11 labels: 12 app: readiness 13 spec: 14 containers: 15 - name: readiness 16 image: 192.168.110.133:5000/nginx:1.13 17 ports: 18 - containerPort: 80 19 readinessProbe: 20 httpGet: 21 path: /qiangge.html 22 port: 80 23 initialDelaySeconds: 3 24 periodSeconds: 3
创建完rc之后，需要创建一个svc的，这里使用命令创建，不再使用配置文件进行创建了。
1 [root@k8s-master health]# kubectl expose rc readiness --port=80 2 service "readiness" exposed 3 [root@k8s-master health]# kubectl describe svc readiness 4 Name: readiness 5 Namespace: default 6 Labels: app=readiness 7 Selector: app=readiness 8 Type: ClusterIP 9 IP: 10.254.218.107 10 Port: <unset> 80/TCP 11 Endpoints: 12 Session Affinity: None 13 No events. 14 [root@k8s-master health]# kubectl get all -o wide 15 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR 16 rc/readiness 2 2 0 2m readiness 192.168.110.133:5000/nginx:1.13 app=readiness 17 18 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR 19 svc/kubernetes 10.254.0.1 <none> 443/TCP 13d <none> 20 svc/readiness 10.254.218.107 <none> 80/TCP 28s app=readiness 21 22 NAME READY STATUS RESTARTS AGE IP NODE 23 po/busybox2 1/1 Running 1 1h 172.16.16.3 k8s-master 24 po/exec 1/1 Running 11 28m 172.16.59.4 k8s-node2 25 po/httpget 1/1 Running 1 18m 172.16.16.4 k8s-master 26 po/readiness-66j6c 0/1 Running 0 2m 172.16.59.5 k8s-node2 27 po/readiness-l2v6z 0/1 Running 0 2m 172.16.32.3 k8s-node3 28 po/tcpsocket 1/1 Running 0 10m 172.16.59.2 k8s-node2 29 [root@k8s-master health]#
可以看到readiness已经启动了两个Pod了，但是后端节点里面是空的。此时可以创建一个指定的html文件，就可以了。
此时发现后端节点也不为空了，Pod也正常启动了，另外一个可以类似出来，就可以将两个Pod正常启动起来了。
1 [root@k8s-master health]# kubectl exec -it readiness-66j6c bash 2 root@readiness-66j6c:/# cd /usr/share/nginx/html/ 3 root@readiness-66j6c:/usr/share/nginx/html# echo hello nginx > qiangge.html 4 root@readiness-66j6c:/usr/share/nginx/html# cat qiangge.html 5 hello nginx 6 root@readiness-66j6c:/usr/share/nginx/html# exit 7 exit 8 [root@k8s-master health]# kubectl describe svc readiness 9 Name: readiness 10 Namespace: default 11 Labels: app=readiness 12 Selector: app=readiness 13 Type: ClusterIP 14 IP: 10.254.218.107 15 Port: <unset> 80/TCP 16 Endpoints: 172.16.59.5:80 17 Session Affinity: None 18 No events. 19 [root@k8s-master health]# kubectl get all -o wide 20 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR 21 rc/readiness 2 2 1 6m readiness 192.168.110.133:5000/nginx:1.13 app=readiness 22 23 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR 24 svc/kubernetes 10.254.0.1 <none> 443/TCP 13d <none> 25 svc/readiness 10.254.218.107 <none> 80/TCP 4m app=readiness 26 27 NAME READY STATUS RESTARTS AGE IP NODE 28 po/busybox2 1/1 Running 1 1h 172.16.16.3 k8s-master 29 po/exec 0/1 CrashLoopBackOff 11 32m 172.16.59.4 k8s-node2 30 po/httpget 1/1 Running 1 21m 172.16.16.4 k8s-master 31 po/readiness-66j6c 1/1 Running 0 6m 172.16.59.5 k8s-node2 32 po/readiness-l2v6z 0/1 Running 0 6m 172.16.32.3 k8s-node3 33 po/tcpsocket 1/1 Running 0 13m 172.16.59.2 k8s-node2 34 [root@k8s-master health]#
浙公网安备 33010602011771号