HPA 自动水平伸缩(基于CPU)
自动水平伸缩，是指运行在k8s上的应用负载(POD)，可以根据资源使用率进行自动扩容、缩容，它依赖metrics-server服务pod使用资源指标收集；我们知道应用的资源使用率通常都有高峰和低谷，所以k8s的HPA特性应运而生；它也是最能体现区别于传统运维的优势之一，不仅能够弹性伸缩，而且完全自动化！
我们在生产中通常用得最多的就是基于服务pod的cpu使用率metrics来自动扩容pod数量，下面来以生产的标准来实战测试下（注意：使用HPA前我们要确保K8s集群的dns服务和metrics服务是正常运行的，并且我们所创建的服务需要配置指标分配）
Github地址:
https://github.com/kubernetes/kubernetes/blob/release-1.19/cluster/addons/metrics-server/
解决不能下载google 镜像仓库的问题：
由于无法下载google的k8s.gcr.io镜像，选择国内的镜像代理站点下载同步镜像：
例如：
docker pull k8s.gcr.io/metrics-server-amd64:v0.3.6
替换成阿里云镜像
docker pull registry.aliyuncs.com/google_containers/metrics-server-amd64:v0.3.6
*** 如果生产使用建议放在harbor上.
1 安装前修改kube-apiserver.yaml配置文件:
vim /etc/kubernetes/manifests/kube-apiserver.yaml
。。。。。。省略
- --enable-bootstrap-token-auth=true
- --enable-aggregator-routing=true # 添加此行
- --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
。。。。。。省略
*** 一定要添加上: "- --enable-aggregator-routing=true" 保证有这行.
2 安装metrics-server v0.3.6
mkdir metrics-server
cd metrics-server
#2.1 下载yaml文件：
wget https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.6/components.yaml
#2.2 修改components.yaml文件
vim components.yaml
......其他不做变动省略即可......
apiVersion: apps/v1
kind: Deployment
metadata:
name: metrics-server
namespace: kube-system
labels:
k8s-app: metrics-server
spec:
selector:
matchLabels:
k8s-app: metrics-server
template:
metadata:
name: metrics-server
labels:
k8s-app: metrics-server
spec:
serviceAccountName: metrics-server
volumes:
# mount in tmp so we can safely use from-scratch images and/or read-only containers
- name: tmp-dir
emptyDir: {}
containers:
- name: metrics-server
#image: k8s.gcr.io/metrics-server-amd64:v0.3.6 # 注释原行
image: registry.aliyuncs.com/google_containers/metrics-server-amd64:v0.3.6 # 替换成阿里云的代理镜像
imagePullPolicy: IfNotPresent
args:
- --cert-dir=/tmp
- --secure-port=4443
- /metrics-server #新增
- --kubelet-preferred-address-types=InternalIP #新增
- --kubelet-insecure-tls #新增
ports:
- name: main-port
......其他不做变动省略即可......
## 2.3 安装metrics-server ：
kubectl apply -f components.yaml
## 2.4 查看metrics-server服务状态：
[root@k8s-master1 data]# kubectl get pod -n kube-system | grep metrics-server
metrics-server-6ddbc8ff55-zwqdm 1/1 Running 0 38m
## 2.5 检查接口是否有异常：
[root@k8s-master1 data]# kubectl api-versions
......
events.k8s.io/v1beta1
extensions/v1beta1
metrics.k8s.io/v1beta1 # 需要有此行
networking.k8s.io/v1
networking.k8s.io/v1beta1
......
[root@k8s-master1 data]# kubectl describe apiservice v1beta1.metrics.k8s.io # 看到下面的信息API接口即正常
Name: v1beta1.metrics.k8s.io
Namespace:
Labels: <none>
Annotations: <none>
API Version: apiregistration.k8s.io/v1
Kind: APIService
Metadata:
Creation Timestamp: 2021-11-11T09:22:34Z
Managed Fields:
API Version: apiregistration.k8s.io/v1
......省略部分
3 配置一个deploment，并添加资源限制条件：
## pod内资源分配的配置格式如下：
## 默认可以只配置requests，但根据生产中的经验，建议把limits资源限制也加上，因为对K8s来说，只有这两个都配置了且配置的值都要一样，这个pod资源的优先级才是最高的，在node资源不够的情况下，首先是把没有任何资源分配配置的pod资源给干掉，其次是只配置了requests的，最后才是两个都配置的情况，仔细品品
vim deployment-nginx.yaml
kind: Deployment
metadata:
labels:
app: ng-web
name: ng-web
spec:
replicas: 1
selector:
matchLabels:
app: ng-web
template:
metadata:
creationTimestamp: null
labels:
app: ng-web
spec:
containers:
- image: 192.168.10.19/devops/nginx:1.14.0
name: nginx
resources: # 资源使用
limits: # (最高)限制单个POD 最多能使用1核(1000m 毫核)CPU以及2G内存
cpu: '1'
memory: 20Gi
requests: # (最低)保证此pod 初始获取这么多资源
cpu: '1'
memory: 20Mi
ports:
- containerPort: 80
volumeMounts:
- name: my-nfs
mountPath: /usr/share/nginx/html
- name: k8snfs-db
mountPath: /data/nginx/html
volumes:
- name: my-nfs
nfs:
server: 192.168.10.19
path: /data/k8sdata
- name: k8snfs-db
nfs:
server: 192.168.10.19
path: /data/k8sdb
---
apiVersion: v1
kind: Service
metadata:
labels:
app: ng-web
name: ng-web
spec:
ports:
- port: 80
protocol: TCP
targetPort: 80
selector:
app: ng-web
type: ClusterIP
****** ****** ****** ****** ****** ****** ****** ****** ****** ****** ****** ****** ****** ****** ****** ****** ******
# 安装部署测试的deployment：
kubectl apply -f deployment-nginx.yaml
# 查看pod 的资源使用:
[root@k8s-master1 data]# kubectl top pod
NAME CPU(cores) MEMORY(bytes)
ng-web-59f56fdfb6-58ch6 0m 1Mi
ng-web-59f56fdfb6-9ds9x 0m 1Mi
ng-web-59f56fdfb6-xmnks 0m 2Mi
# 查看Node 使用情况:
[root@k8s-master1 data]# kubectl top node
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
k8s-master1.example.com 242m 12% 1172Mi 61%
node1.example.com 66m 3% 505Mi 56%
node2.example.com 56m 2% 520Mi 58%
node3.example.com 80m 4% 534Mi 60%
4 为测试deployment，创建一个HPA:
# 为deployment资源web创建HPA，pod数量上限3个，最低1个，在pod平均CPU达到50%后开始扩容
kubectl autoscale deployment ng-web --max=3 --min=1 --cpu-percent=50
# 下面提示说到，HPA缺少最小资源分配的request参数:
[root@k8s-master1 linux39_ng]# kubectl describe hpa web1
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedGetScale 3m30s (x3 over 142m) horizontal-pod-autoscaler deployments/scale.apps "ng-web" not found
Warning FailedGetResourceMetric 3m (x2 over 3m15s) horizontal-pod-autoscaler unable to get metrics for resource cpu: no metrics returned from resource metrics API
Warning FailedComputeMetricsReplicas 3m (x2 over 3m15s) horizontal-pod-autoscaler invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
Warning FailedGetResourceMetric 2m15s (x3 over 2m45s) horizontal-pod-autoscaler missing request for cpu
Warning FailedComputeMetricsReplicas 2m15s (x3 over 2m45s) horizontal-pod-autoscaler invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: missing request for cpu
***** 所以在deployment-nginx.yaml里面一定要加入resources limits request 这几个资源限制参数.
5 查看HPA的资源信息：
## 添加资源限制后重新应用：
kubectl apply -f deployment-nginx.yaml
[root@k8s-master1 linux39_ng]# kubectl autoscale deployment web1 --max=3 --min=1 --cpu-percent=1
horizontalpodautoscaler.autoscaling/web1 autoscaled
# --cpu-percent=1 CPU百分比，测试建议放低点
## 等待一会，可以看到相关的hpa信息（K8s上metrics服务收集所有pod资源的时间间隔大概在60s的时间）
[root@k8s-master1 web1]# kubectl get hpa -w
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
web1 Deployment/web1 0%/1% 1 3 1 48m
## 再次查看HPA：
[root@k8s-master1 linux39_ng]# kubectl describe hpa web1
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True ScaleDownStabilized recent recommendations were higher than current one, applying the highest recent recommendation
ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
6 模拟业务流量增长，看看hpa自动伸缩的效果：
# 获取ng-web 的CluseterIP：
[root@k8s-master2 ng]# kubectl get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
web1 ClusterIP 172.16.1.108 <none> 80/TCP 16d
# 其他node模拟流量,随便写了个html页面用于测试:
[root@k8s-master2 ~]# while :;do curl 172.16.1.108 ;done
<h1>this is NFS PV PVC pod show</h1>
<img src='123.png' >
<h1>this is NFS PV PVC pod show</h1>
一台不行就2个节点上量.....
# 过一会在看看HPA 信息，不会上来就自动扩容, 需要一定的时间全达到阈值范围，就开始自动扩容.
[root@k8s-master1 web1]# kubectl get hpa -w
web1 Deployment/web1 10%/1% 1 3 1 48m
web1 Deployment/web1 10%/1% 1 3 3 48m
web1 Deployment/web1 0%/1% 1 3 3 49m
# 停止Node 并发访问，在查看HPA是不是慢慢的进行了缩容：
[root@k8s-master1 web1]# kubectl get hpa -w
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
web1 Deployment/web1 10%/1% 1 3 1 48m
web1 Deployment/web1 0%/1% 1 3 3 54m
web1 Deployment/web1 0%/1% 1 3 1 54m
7 文件方式创建HPA:
vim hpa_web1.yaml
---
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
name: web1
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: web1
minReplicas: 1
maxReplicas: 3
targetCPUUtilizationPercentage: 50
## 执行文件：
kubectl apply -f hpa_web1.yaml
错误：
1. kubectl get hpa -w 不显示资源使用率：
问题 1：
[root@k8s-master1 web1]# kubectl delete ingress web1
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
Error from server (NotFound): ingresses.extensions "web1" not found
[root@k8s-master1 web1]# kubectl describe hpa web1
Name: web1
Namespace: default
Labels: <none>
Annotations: <none>
CreationTimestamp: Tue, 23 Nov 2021 16:13:41 +0800
Reference: Deployment/web1
Metrics: ( current / target )
resource cpu on pods (as a percentage of request): <unknown> / 1%
Min replicas: 1
Max replicas: 3
Deployment pods: 1 current / 0 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale
ScalingActive False FailedGetResourceMetric the HPA was unable to compute the replica count: missing request for cpu
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedComputeMetricsReplicas 10s (x12 over 2m58s) horizontal-pod-autoscaler invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: missing request for cpu
Warning FailedGetResourceMetric <invalid> (x20 over 2m58s) horizontal-pod-autoscaler missing request for cpu
问题 2 ：
[root@k8s-master1 web1]# kubectl get hpa -w
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
web1 Deployment/web1 <unknown>/1% 1 3 0 12s
web1 Deployment/web1 <unknown>/1% 1 3 1 15s
*************************************************************************************************************************************
解决：
以上的两个问题都是因为创建deployment 时的resources 这部分的限制超标或者资源限制的配置格式不对. 仔细检查deployment 清单里的资源限制书写格式部分.
正确的限制资源格式如下：
......
# resources: {} # 这行很重要,命令行导出自带的, 1 删除 2 把资源限制写在里面. 我就直接注释，大家可以选择删除.
resources: # 资源使用
limits: # (最高)限制单个POD 最多能使用1核(1000m 毫核)CPU以及2G内存
cpu: "20m"
memory: "50Mi"
requests: # (最低)保证此pod 初始获取这么多资源
cpu: "10m"
memory: "20Mi"
......
