容器的资源需求，资源限制
requests: 需求，最低保障；
limits: 限制，硬限制；
CPU:
1颗逻辑CPU
1=1000, millicores
500m=0.5CPU
内存:
E、P、T、G、M、K
Ei、Pi
Qos:
Guranteed: 确保保证，每个容器
同时设置CPU和内存的requests和limits.
cpu.limits=cpu.requests
memory.limits=memory.request
Burstable:
至少又一个容器设置CPU或内存资源的requests属性
BestEffor: 没有任何一个容器设置了requests或limits属性；最低优先级别
kubectl top
HeapSter
kubelet
cAdvisor
InfluxDB
HeapSter
Grafana
https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
master:
[root@master ~]# kubectl explain pods.spec.containers
[root@master ~]# kubectl explain pods.spec.containers.resources
[root@master ~]# cd manifests/
[root@master manifests]# mkdir metrics
[root@master manifests]# cd metrics/
[root@master metrics]# cp ../schedule/pod-demo.yaml ./
[root@master metrics]# vim pod-demo.yaml
apiVersion: v1
kind: Pod
metadata:
name: pod-demo
namespace: default
labels:
app: myapp
tier: frontend
spec:
containers:
- name: myapp
image: ikubernetes/stress-ng
command: ["/usr/bin/stress-ng", "-m 1","-c 1","--metrics-brief"]
resources:
requests:
cpu: "200m"
memory: "128Mi"
limits:
cpu: "500m"
memory: "200Mi"
[root@master metrics]# kubectl apply -f pod-demo.yaml
[root@master metrics]# kubectl get pods
NAME READY STATUS RESTARTS AGE
myapp-deploy-5d9c6985f5-7sbdk 1/1 Running 0 23h
myapp-deploy-5d9c6985f5-rcxvj 1/1 Running 0 23h
myapp-deploy-5d9c6985f5-ssdf6 1/1 Running 0 23h
pod-demo 0/1 Pending 0 1m
[root@master metrics]# kubectl taint node node01.smoke.com node-type-
[root@master metrics]# kubectl taint node node02.smoke.com node-type-
[root@master metrics]# kubectl get pods
NAME READY STATUS RESTARTS AGE
myapp-deploy-5d9c6985f5-7sbdk 1/1 Running 0 23h
myapp-deploy-5d9c6985f5-rcxvj 1/1 Running 0 23h
myapp-deploy-5d9c6985f5-ssdf6 1/1 Running 0 23h
pod-demo 0/1 ContainerCreating 0 4m
[root@master metrics]# kubectl describe pods pod-demo
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 3m (x25 over 4m) default-scheduler 0/3 nodes are available: 3 node(s) had taints that the pod didn't tolerate.
Normal Pulling 2m kubelet, node01.smoke.com pulling image "ikubernetes/stress-ng"
[root@master metrics]# kubectl get pods
NAME READY STATUS RESTARTS AGE
myapp-deploy-5d9c6985f5-7sbdk 1/1 Running 0 23h
myapp-deploy-5d9c6985f5-rcxvj 1/1 Running 0 23h
myapp-deploy-5d9c6985f5-ssdf6 1/1 Running 0 23h
pod-demo 1/1 Running 0 19m
[root@master metrics]# kubectl exec -it pod-demo -- /bin/sh
/ # top
Mem: 1648528K used, 216780K free, 35060K shrd, 1128K buff, 750504K cached
CPU: 21% usr 7% sys 0% nic 69% idle 0% io 0% irq 0% sirq
Load average: 1.32 1.01 0.64 6/548 672
PID PPID USER STAT VSZ %VSZ CPU %CPU COMMAND
7 1 root R 6884 0% 1 13% {stress-ng-cpu} /usr/bin/stress-ng -m 1 -c 1 --metrics-brief
672 8 root R 262m 14% 0 0% {stress-ng-vm} /usr/bin/stress-ng -m 1 -c 1 --metrics-brief
8 1 root S 6244 0% 0 0% {stress-ng-vm} /usr/bin/stress-ng -m 1 -c 1 --metrics-brief
1 0 root S 6244 0% 0 0% /usr/bin/stress-ng -m 1 -c 1 --metrics-brief
653 0 root S 1508 0% 1 0% /bin/sh
667 653 root R 1500 0% 1 0% top
/ # free -m
total used free shared buffers cached
Mem: 1821 1561 260 0 1 733
-/+ buffers/cache: 827 994
Swap: 2047 3 2044
/ # exit
[root@master metrics]# kubectl delete -f pod-demo.yaml
[root@master metrics]# vim pod-demo.yaml
apiVersion: v1
kind: Pod
metadata:
name: pod-demo
namespace: default
labels:
app: myapp
tier: frontend
spec:
containers:
- name: myapp
image: ikubernetes/stress-ng
command: ["/usr/bin/stress-ng", "-c 1","--metrics-brief"]
resources:
requests:
cpu: "200m"
memory: "128Mi"
limits:
cpu: "500m"
memory: "512Mi"
[root@master metrics]# kubectl apply -f pod-demo.yaml
[root@master metrics]# kubectl get pods
NAME READY STATUS RESTARTS AGE
myapp-deploy-5d9c6985f5-7sbdk 1/1 Running 0 1d
myapp-deploy-5d9c6985f5-rcxvj 1/1 Running 0 1d
myapp-deploy-5d9c6985f5-ssdf6 1/1 Running 0 1d
pod-demo 1/1 Running 0 23s
[root@master metrics]# kubectl exec pod-demo -- top
Mem: 1718820K used, 146488K free, 28768K shrd, 2108K buff, 1102016K cached
CPU: 26% usr 1% sys 0% nic 71% idle 0% io 0% irq 0% sirq
Load average: 0.97 0.55 0.34 6/396 11
PID PPID USER STAT VSZ %VSZ CPU %CPU COMMAND
6 1 root R 6892 0% 1 25% {stress-ng-cpu} /usr/bin/stress-ng
1 0 root S 6244 0% 1 0% /usr/bin/stress-ng -c 1 --metrics-
7 0 root R 1504 0% 0 0% top
[root@master metrics]# kubectl describe pods pod-demo
QoS Class: Burstable
[root@master metrics]# kubectl delete -f pod-demo.yaml
[root@master metrics]# vim pod-demo.yaml
apiVersion: v1
kind: Pod
metadata:
name: pod-demo
namespace: default
labels:
app: myapp
tier: frontend
spec:
containers:
- name: myapp
image: ikubernetes/myapp:v1
resources:
requests:
cpu: "200m"
memory: "512Mi"
limits:
cpu: "200m"
memory: "512Mi"
[root@master metrics]# kubectl apply -f pod-demo.yaml
[root@master metrics]# kubectl describe pods pod-demo
QoS Class: Guaranteed
[root@master metrics]# kubectl get pods
NAME READY STATUS RESTARTS AGE
myapp-deploy-5d9c6985f5-7sbdk 1/1 Running 0 1d
myapp-deploy-5d9c6985f5-rcxvj 1/1 Running 0 1d
myapp-deploy-5d9c6985f5-ssdf6 1/1 Running 0 1d
pod-demo 1/1 Running 0 4m
[root@master metrics]# kubectl describe pods myapp-deploy-5d9c6985f5-7sbdk
QoS Class: BestEffort
[root@master metrics]# kubectl top pod
Error from server (NotFound): the server could not find the requested resource (get services http:heapster:)
[root@master metrics]# ss -tnl
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 127.0.0.1:38405 *:*
LISTEN 0 128 127.0.0.1:10248 *:*
LISTEN 0 128 127.0.0.1:10249 *:*
LISTEN 0 128 127.0.0.1:10251 *:*
LISTEN 0 128 127.0.0.1:2379 *:*
LISTEN 0 128 127.0.0.1:10252 *:*
LISTEN 0 128 127.0.0.1:2380 *:*
LISTEN 0 128 *:22 *:*
LISTEN 0 100 127.0.0.1:25 *:*
LISTEN 0 128 172.20.0.70:10010 *:*
LISTEN 0 128 :::30080 :::*
LISTEN 0 128 :::10250 :::*
LISTEN 0 128 :::9099 :::*
LISTEN 0 128 :::30443 :::*
LISTEN 0 128 :::6443 :::*
LISTEN 0 128 :::10256 :::*
LISTEN 0 128 :::31762 :::*
LISTEN 0 128 :::22 :::*
LISTEN 0 100 ::1:25
安装HeapSter
https://github.com/kubernetes-retired/heapster/tree/master/deploy/kube-config
master:
[root@master metrics]# wget https://raw.githubusercontent.com/kubernetes-retired/heapster/master/deploy/kube-config/influxdb/influxdb.yaml
[root@master metrics]# vim influxdb.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: monitoring-influxdb
namespace: kube-system
spec:
replicas: 1
selector:
matchLabels:
task: monitoring
k8s-app: influxdb
template:
metadata:
labels:
task: monitoring
k8s-app: influxdb
spec:
containers:
- name: influxdb
image: k8s.gcr.io/heapster-influxdb-amd64:v1.5.2
volumeMounts:
- mountPath: /data
name: influxdb-storage
volumes:
- name: influxdb-storage
emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
labels:
task: monitoring
# For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)
# If you are NOT using this as an addon, you should comment out this line.
kubernetes.io/cluster-service: 'true'
kubernetes.io/name: monitoring-influxdb
name: monitoring-influxdb
namespace: kube-system
spec:
ports:
- port: 8086
targetPort: 8086
selector:
k8s-app: influxdb
[root@master metrics]# kubectl apply -f influxdb.yaml
[root@master metrics]# k8s.gcr.io/heapster-influxdb-amd64:v1.5.2^C
[root@master metrics]# kubectl get svc -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP 63d
kubernetes-dashboard NodePort 10.96.159.129 <none> 443:31762/TCP 33d
monitoring-influxdb ClusterIP 10.100.218.196 <none> 8086/TCP 23h
[root@master metrics]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
canal-997tb 3/3 Running 0 24d
canal-j6t4j 3/3 Running 0 24d
canal-jxq25 3/3 Running 0 24d
coredns-78fcdf6894-bt5g6 1/1 Running 1 63d
coredns-78fcdf6894-zzbll 1/1 Running 1 63d
etcd-master.smoke.com 1/1 Running 1 63d
kube-apiserver-master.smoke.com 1/1 Running 1 63d
kube-controller-manager-master.smoke.com 1/1 Running 1 63d
kube-flannel-ds-g69pn 1/1 Running 0 23h
kube-flannel-ds-rkd4c 1/1 Running 0 23h
kube-flannel-ds-stnlp 1/1 Running 0 23h
kube-proxy-5jppm 1/1 Running 1 62d
kube-proxy-7lg96 1/1 Running 1 63d
kube-proxy-qmrq7 1/1 Running 1 62d
kube-scheduler-master.smoke.com 1/1 Running 1 63d
kubernetes-dashboard-6948bdb78-fdpt2 1/1 Running 0 8d
monitoring-influxdb-848b9b66f6-xx7mq 1/1 Running 0 23h
[root@master metrics]# kubectl logs monitoring-influxdb-848b9b66f6-xx7mq -n kube-system
[root@master metrics]# kubectl apply -f https://raw.githubusercontent.com/kubernetes-retired/heapster/master/deploy/kube-config/rbac/heapster-rbac.yaml
[root@master metrics]# wget https://raw.githubusercontent.com/kubernetes-retired/heapster/master/deploy/kube-config/influxdb/heapster.yaml
[root@master metrics]# ll
总用量 12
-rw-r--r--. 1 root root 1100 8月 18 21:56 heapster.yaml
-rw-r--r--. 1 root root 1025 8月 17 22:06 influxdb.yaml
-rw-r--r--. 1 root root 318 8月 11 22:04 pod-demo.yaml
[root@master metrics]# vim heapster.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: heapster
namespace: kube-system
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: heapster
namespace: kube-system
spec:
replicas: 1
selector:
matchLabels:
task: monitoring
k8s-app: heapster
template:
metadata:
labels:
task: monitoring
k8s-app: heapster
spec:
serviceAccountName: heapster
containers:
- name: heapster
image: k8s.gcr.io/heapster-amd64:v1.5.4
imagePullPolicy: IfNotPresent
command:
- /heapster
- --source=kubernetes:https://kubernetes.default
- --sink=influxdb:http://monitoring-influxdb.kube-system.svc:8086
---
apiVersion: v1
kind: Service
metadata:
labels:
task: monitoring
# For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)
# If you are NOT using this as an addon, you should comment out this line.
kubernetes.io/cluster-service: 'true'
kubernetes.io/name: Heapster
name: heapster
namespace: kube-system
spec:
ports:
- port: 80
targetPort: 8082
type: NodePort
selector:
k8s-app: heapster
[root@master metrics]# kubectl apply -f heapster.yaml
[root@master metrics]# kubectl get svc -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
heapster NodePort 10.105.208.122 <none> 80:31474/TCP 23h
kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP 64d
kubernetes-dashboard NodePort 10.96.159.129 <none> 443:31762/TCP 34d
monitoring-influxdb ClusterIP 10.100.218.196 <none> 8086/TCP 1d
[root@master metrics]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
canal-997tb 3/3 Running 0 25d
canal-j6t4j 3/3 Running 0 25d
canal-jxq25 3/3 Running 0 25d
coredns-78fcdf6894-bt5g6 1/1 Running 1 64d
coredns-78fcdf6894-zzbll 1/1 Running 1 64d
etcd-master.smoke.com 1/1 Running 1 64d
heapster-84c9bc48c4-7sbss 1/1 Running 0 23h
kube-apiserver-master.smoke.com 1/1 Running 1 64d
kube-controller-manager-master.smoke.com 1/1 Running 1 64d
kube-flannel-ds-g69pn 1/1 Running 0 1d
kube-flannel-ds-rkd4c 1/1 Running 0 1d
kube-flannel-ds-stnlp 1/1 Running 0 1d
kube-proxy-5jppm 1/1 Running 1 63d
kube-proxy-7lg96 1/1 Running 1 64d
kube-proxy-qmrq7 1/1 Running 1 63d
kube-scheduler-master.smoke.com 1/1 Running 1 64d
kubernetes-dashboard-6948bdb78-fdpt2 1/1 Running 0 9d
monitoring-influxdb-848b9b66f6-xx7mq 1/1 Running 0 1d
通过宿主机的浏览器访问http://172.20.0.66:31474
master:
[root@master metrics]# kubectl logs heapster-84c9bc48c4-7sbss -n kube-system
I0818 15:52:57.954486 1 heapster.go:78] /heapster --source=kubernetes:https://kubernetes.default --sink=influxdb:http://monitoring-influxdb.kube-system.svc:8086
I0818 15:52:57.954561 1 heapster.go:79] Heapster version v1.5.4
I0818 15:52:57.955282 1 configs.go:61] Using Kubernetes client with master "https://kubernetes.default" and version v1
I0818 15:52:57.955310 1 configs.go:62] Using kubelet port 10255
I0818 15:52:58.276440 1 influxdb.go:312] created influxdb sink with options: host:monitoring-influxdb.kube-system.svc:8086 user:root db:k8s
I0818 15:52:58.276478 1 heapster.go:202] Starting with InfluxDB Sink
I0818 15:52:58.276487 1 heapster.go:202] Starting with Metric Sink
I0818 15:52:58.291886 1 heapster.go:112] Starting heapster on port 8082
[root@master metrics]# wget https://raw.githubusercontent.com/kubernetes-retired/heapster/master/deploy/kube-config/influxdb/grafana.yaml
[root@master metrics]# vim grafana.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: monitoring-grafana
namespace: kube-system
spec:
replicas: 1
selector:
matchLabels:
task: monitoring
k8s-app: grafana
template:
metadata:
labels:
task: monitoring
k8s-app: grafana
spec:
containers:
- name: grafana
image: k8s.gcr.io/heapster-grafana-amd64:v5.0.4
ports:
- containerPort: 3000
protocol: TCP
volumeMounts:
- mountPath: /etc/ssl/certs
name: ca-certificates
readOnly: true
- mountPath: /var
name: grafana-storage
env:
- name: INFLUXDB_HOST
value: monitoring-influxdb
- name: GF_SERVER_HTTP_PORT
value: "3000"
# The following env variables are required to make Grafana accessible via
# the kubernetes api-server proxy. On production clusters, we recommend
# removing these env variables, setup auth for grafana, and expose the grafana
# service using a LoadBalancer or a public IP.
- name: GF_AUTH_BASIC_ENABLED
value: "false"
- name: GF_AUTH_ANONYMOUS_ENABLED
value: "true"
- name: GF_AUTH_ANONYMOUS_ORG_ROLE
value: Admin
- name: GF_SERVER_ROOT_URL
# If you're only using the API Server proxy, set this value instead:
# value: /api/v1/namespaces/kube-system/services/monitoring-grafana/proxy
value: /
volumes:
- name: ca-certificates
hostPath:
path: /etc/ssl/certs
- name: grafana-storage
emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
labels:
# For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)
# If you are NOT using this as an addon, you should comment out this line.
kubernetes.io/cluster-service: 'true'
kubernetes.io/name: monitoring-grafana
name: monitoring-grafana
namespace: kube-system
spec:
# In a production setup, we recommend accessing Grafana through an external Loadbalancer
# or through a public IP.
# type: LoadBalancer
# You could also use NodePort to expose the service at a randomly-generated port
# type: NodePort
ports:
- port: 80
targetPort: 3000
selector:
k8s-app: grafana
type: NodePort
[root@master metrics]# kubectl apply -f grafana.yaml
[root@master metrics]# kubectl get svc -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
heapster NodePort 10.105.208.122 <none> 80:31474/TCP 1d
kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP 65d
kubernetes-dashboard NodePort 10.96.159.129 <none> 443:31762/TCP 35d
monitoring-grafana NodePort 10.104.205.6 <none> 80:30175/TCP 23h
monitoring-influxdb ClusterIP 10.100.218.196 <none> 8086/TCP 2d
[root@master metrics]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
canal-997tb 3/3 Running 0 25d
canal-j6t4j 3/3 Running 0 25d
canal-jxq25 3/3 Running 0 25d
coredns-78fcdf6894-bt5g6 1/1 Running 1 65d
coredns-78fcdf6894-zzbll 1/1 Running 1 65d
etcd-master.smoke.com 1/1 Running 1 65d
heapster-84c9bc48c4-7sbss 1/1 Running 0 1d
kube-apiserver-master.smoke.com 1/1 Running 1 65d
kube-controller-manager-master.smoke.com 1/1 Running 1 65d
kube-flannel-ds-g69pn 1/1 Running 0 2d
kube-flannel-ds-rkd4c 1/1 Running 0 2d
kube-flannel-ds-stnlp 1/1 Running 0 2d
kube-proxy-5jppm 1/1 Running 1 64d
kube-proxy-7lg96 1/1 Running 1 65d
kube-proxy-qmrq7 1/1 Running 1 64d
kube-scheduler-master.smoke.com 1/1 Running 1 65d
kubernetes-dashboard-6948bdb78-fdpt2 1/1 Running 0 10d
monitoring-grafana-555545f477-jjwkw 1/1 Running 0 23h
monitoring-influxdb-848b9b66f6-xx7mq 1/1 Running 0 2d
通过宿主机的浏览器访问http://172.20.0.66:30175
点击New dashboard
点击Configuration -- Data Sources
点击Dashboards -- New dashboard -- Graph
master:
[root@master metrics]# kubectl top pod W0820 21:45:27.657772 107517 top_pod.go:263] Metrics not available for pod default/myapp-deploy-5d9c6985f5-7sbdk, age: 239h59m18.657630377s error: Metrics not available for pod default/myapp-deploy-5d9c6985f5-7sbdk, age: 239h59m18.657630377s [root@master metrics]# kubectl top node node01.smoke.com Error from server (NotFound): the server could not find the requested resource (get services http:heapster:) [root@master metrics]# kubectl top nodes error: metrics not available yet [root@master metrics]# kubectl top pods #还是没有数据 W0820 21:46:51.685669 108366 top_pod.go:263] Metrics not available for pod default/myapp-deploy-5d9c6985f5-7sbdk, age: 240h0m42.685644659s error: Metrics not available for pod default/myapp-deploy-5d9c6985f5-7sbdk, age: 240h0m42.685644659s [root@master metrics]# kubectl log heapster-84c9bc48c4-7sbss -n kube-system #无法获取到容器数据 E0820 13:47:05.005711 1 manager.go:101] Error in scraping containers from kubelet:172.20.0.70:10255: failed to get all container stats from Kubelet URL "http://172.20.0.70:10255/stats/container/": Post http://172.20.0.70:10255/stats/container/: dial tcp 172.20.0.70:10255: getsockopt: connection refused E0820 13:47:05.023006 1 manager.go:101] Error in scraping containers from kubelet:172.20.0.66:10255: failed to get all container stats from Kubelet URL "http://172.20.0.66:10255/stats/container/": Post http://172.20.0.66:10255/stats/container/: dial tcp 172.20.0.66:10255: getsockopt: connection refused E0820 13:47:05.023449 1 manager.go:101] Error in scraping containers from kubelet:172.20.0.67:10255: failed to get all container stats from Kubelet URL "http://172.20.0.67:10255/stats/container/": Post http://172.20.0.67:10255/stats/container/: dial tcp 172.20.0.67:10255: getsockopt: connection refused W0820 13:47:25.000584 1 manager.go:152] Failed to get all responses in time (got 0/3)
node02:
[root@node02 ~]# ss -tnl #10255端口没有监听，查看官网在v1.11版本已经将heapster废除，因此不能用 State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 128 *:34630 *:* LISTEN 0 128 127.0.0.1:10248 *:* LISTEN 0 128 127.0.0.1:10249 *:* LISTEN 0 128 127.0.0.1:41769 *:* LISTEN 0 128 *:111 *:* LISTEN 0 128 *:22 *:* LISTEN 0 100 127.0.0.1:25 *:* LISTEN 0 128 172.20.0.67:10010 *:* LISTEN 0 128 :::30175 :::* LISTEN 0 128 :::30080 :::* LISTEN 0 128 :::48325 :::* LISTEN 0 128 :::10250 :::* LISTEN 0 128 :::9099 :::* LISTEN 0 128 :::30443 :::* LISTEN 0 128 :::111 :::* LISTEN 0 128 :::10256 :::* LISTEN 0 128 :::31474 :::* LISTEN 0 128 :::31762 :::* LISTEN 0 128 :::22 :::* LISTEN 0 100 ::1:25 :::*
浙公网安备 33010602011771号