AIGC标识 k8s~HPA对服务高可用的支撑

Kubernetes HPA（Horizontal Pod Autoscaler）详解

HPA是Kubernetes中自动水平伸缩Pod副本数的核心组件，它可以根据监控指标自动调整应用的副本数量。

1. HPA基本概念

工作原理

监控指标变化 → HPA控制器检测 → 计算所需副本数 → 调整Deployment/StatefulSet副本数

支持的指标类型

# HPA v2支持三种指标类型：
1. Resource Metrics（资源指标）    - CPU/Memory
2. Custom Metrics（自定义指标）    - 应用自定义指标
3. External Metrics（外部指标）    - 集群外部指标

2. HPA配置详解

2.1 基础HPA配置

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: myapp-hpa
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment       # 也可以支持StatefulSet、ReplicaSet
    name: myapp-deployment # 要伸缩的目标资源
  minReplicas: 2           # 最小副本数
  maxReplicas: 10          # 最大副本数
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization  # 或 AverageValue
        averageUtilization: 70  # CPU使用率目标70%
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80  # 内存使用率目标80%

2.2 完整HPA v2示例

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: comprehensive-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp
  minReplicas: 3
  maxReplicas: 20
  # 行为配置（Kubernetes 1.18+）
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300  # 缩容稳定窗口300秒
      policies:
      - type: Percent
        value: 50                      # 一次最多缩容50%的Pod
        periodSeconds: 60
      - type: Pods
        value: 5                       # 一次最多缩容5个Pod
        periodSeconds: 60
      selectPolicy: Max                # 选择限制最严格的策略
    scaleUp:
      stabilizationWindowSeconds: 0    # 扩容立即执行
      policies:
      - type: Percent
        value: 100                     # 一次最多扩容100%的Pod
        periodSeconds: 60
      - type: Pods
        value: 10                      # 一次最多扩容10个Pod
        periodSeconds: 60
      selectPolicy: Max
  metrics:
  # 1. CPU指标
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  # 2. 内存指标
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80
  # 3. 自定义指标（需要安装Custom Metrics API）
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: 1000  # 目标：每个Pod每秒1000个请求
  # 4. 外部指标（需要安装External Metrics API）
  - type: External
    external:
      metric:
        name: queue_messages
        selector:
          matchLabels:
            queue: "myqueue"
      target:
        type: AverageValue
        averageValue: 30  # 目标：每个副本处理30个消息

3. HPA计算算法详解

副本数计算公式

所需副本数 = ceil[当前副本数 × (当前指标值 / 期望指标值)]

示例计算

# 假设：
当前副本数 = 3
当前CPU使用率 = 90%
目标CPU使用率 = 70%

# 计算：
所需副本数 = ceil[3 × (90 / 70)] = ceil[3 × 1.2857] = ceil[3.857] = 4

多指标时的计算逻辑

1. 对每个指标单独计算所需副本数
2. 取所有计算结果中的最大值
3. 确保副本数在minReplicas和maxReplicas之间

4. HPA行为控制

4.1 伸缩策略详解

behavior:
  scaleDown:
    stabilizationWindowSeconds: 300  # 缩容稳定窗口
    policies:
    - type: Percent
      value: 50      # 每60秒最多减少50%的Pod
      periodSeconds: 60
    - type: Pods
      value: 5       # 每60秒最多减少5个Pod
      periodSeconds: 60
    selectPolicy: Max  # 使用限制最严格的策略
  scaleUp:
    stabilizationWindowSeconds: 0    # 立即扩容
    policies:
    - type: Percent
      value: 200     # 每60秒最多增加200%的Pod
      periodSeconds: 60
    - type: Pods
      value: 10      # 每60秒最多增加10个Pod
      periodSeconds: 60
    selectPolicy: Max

4.2 稳定窗口的作用

# 防止频繁伸缩的"抖动"现象
scaleDown:
  stabilizationWindowSeconds: 300  # 5分钟稳定窗口
  
# 工作原理：
# - 在稳定窗口期内，HPA会记录所有建议的副本数
# - 选择窗口期内最大的建议副本数
# - 避免因指标短暂波动导致的频繁伸缩

5. 基于自定义指标的HPA

5.1 准备工作

# 1. 安装Custom Metrics API
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install prometheus-adapter prometheus-community/prometheus-adapter

# 2. 验证安装
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq .

5.2 应用暴露自定义指标

// Spring Boot Actuator示例
// pom.xml添加依赖
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

// application.properties
management.endpoints.web.exposure.include=metrics,prometheus
management.metrics.export.prometheus.enabled=true

5.3 自定义指标HPA配置

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: custom-metric-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: 500  # 每个Pod每秒处理500个请求
  - type: Object
    object:
      metric:
        name: queue_messages
      describedObject:
        apiVersion: v1
        kind: Service
        name: myapp-service
      target:
        type: Value
        value: 1000  # 整个服务队列消息目标1000

6. 基于外部指标的HPA

6.1 Kafka队列示例

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: kafka-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: kafka-consumer
  minReplicas: 2
  maxReplicas: 20
  metrics:
  - type: External
    external:
      metric:
        name: kafka_topic_lag
        selector:
          matchLabels:
            topic: "orders"
            consumer_group: "myapp"
      target:
        type: AverageValue
        averageValue: 100  # 每个副本处理100个消息延迟

6.2 Prometheus外部指标

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: prometheus-external-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp
  minReplicas: 2
  maxReplicas: 15
  metrics:
  - type: External
    external:
      metric:
        name: prometheus_metric
        selector:
          matchLabels:
            query: "rate(http_requests_total[5m])"
      target:
        type: Value
        value: 10000  # 目标值：10000请求/秒

7. 实战示例

7.1 Web应用基于QPS的伸缩

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: webapp-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: webapp
  minReplicas: 3
  maxReplicas: 30
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 600  # 缩容等待10分钟
      policies:
      - type: Percent
        value: 10
        periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 60   # 扩容等待1分钟
      policies:
      - type: Percent
        value: 100
        periodSeconds: 60
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: 1000

7.2 批处理作业基于队列的伸缩

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: batch-job-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: batch-processor
  minReplicas: 1
  maxReplicas: 50
  metrics:
  - type: External
    external:
      metric:
        name: sqs_approximate_number_of_messages_visible
        selector:
          matchLabels:
            queue_name: "process-queue"
      target:
        type: AverageValue
        averageValue: 1000  # 每个Pod处理1000个消息

8. HPA调试与监控

8.1 查看HPA状态

# 查看HPA基本信息
kubectl get hpa

# 查看详细状态
kubectl describe hpa <hpa-name>

# 查看HPA事件
kubectl describe hpa <hpa-name> | grep -A20 Events

# 以YAML格式查看
kubectl get hpa <hpa-name> -o yaml

8.2 HPA状态字段解释

# kubectl describe hpa 输出解析
Status:
  Current Replicas: 5        # 当前副本数
  Desired Replicas: 7        # 期望副本数
  Current Metrics:
  - Type: Resource
    Resource:
      Name: cpu
    Current:
      Average Utilization: 85  # 当前平均使用率
      Average Value: 85m       # 当前平均值
  - Type: Pods
    Pods:
      Metric:
        Name: http_requests_per_second
      Current:
        Average Value: 1200    # 当前平均值
  Conditions:
  - Type: AbleToScale         # 是否可以伸缩
    Status: True
  - Type: ScalingActive       # 伸缩是否活跃
    Status: True
  - Type: ScalingLimited      # 是否被限制
    Status: False

8.3 监控HPA指标

# 查看HPA相关指标
kubectl get --raw /apis/metrics.k8s.io/v1beta1/pods | jq .

# 查看特定Pod的指标
kubectl top pod <pod-name>

# 查看节点的指标
kubectl top node

8.4 调试命令

# 1. 检查Metrics Server是否正常工作
kubectl get apiservices | grep metrics
kubectl get pods -n kube-system | grep metrics-server

# 2. 检查Pod资源请求是否设置
kubectl get deployment <deployment-name> -o yaml | grep -A5 resources

# 3. 模拟负载测试观察HPA
# 使用hey或wrk进行压力测试
kubectl run -i --tty load-generator --rm \
  --image=rakyll/hey \
  --restart=Never \
  -- /bin/sh -c "hey -z 300s -c 50 http://myapp-service"

# 4. 实时观察HPA变化
watch -n 5 'kubectl get hpa && kubectl get pods -l app=myapp'

9. 常见问题与解决方案

问题1：HPA不工作

# 排查步骤：
# 1. 检查Metrics Server
kubectl get pods -n kube-system -l k8s-app=metrics-server

# 2. 检查Pod资源请求
kubectl describe pod <pod-name> | grep -A5 Requests

# 3. 检查HPA配置
kubectl describe hpa <hpa-name>

# 4. 查看HPA控制器日志
kubectl logs -n kube-system deployment/hpa-controller-manager

问题2：频繁伸缩（抖动）

# 解决方案：增加稳定窗口和调整策略
behavior:
  scaleDown:
    stabilizationWindowSeconds: 600  # 10分钟稳定窗口
    policies:
    - type: Percent
      value: 10                      # 每次最多缩容10%
      periodSeconds: 300             # 每5分钟评估一次
  scaleUp:
    stabilizationWindowSeconds: 300   # 5分钟稳定窗口

问题3：伸缩速度太慢

# 解决方案：调整伸缩策略
behavior:
  scaleUp:
    stabilizationWindowSeconds: 0    # 立即扩容
    policies:
    - type: Percent
      value: 300                     # 一次最多扩容300%
      periodSeconds: 60
    - type: Pods
      value: 20                      # 一次最多扩容20个Pod
      periodSeconds: 60

10. 最佳实践

10.1 资源请求配置

# Pod模板中必须设置资源请求
containers:
- name: app
  resources:
    requests:
      cpu: 100m     # HPA基于requests计算使用率
      memory: 128Mi
    limits:
      cpu: 200m
      memory: 256Mi

10.2 HPA配置建议

# 推荐配置参数
spec:
  minReplicas: 2                    # 至少2个副本保证高可用
  maxReplicas: 10                   # 根据实际需求设置上限
  
  # 设置合理的稳定窗口
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300  # 缩容谨慎
    scaleUp:
      stabilizationWindowSeconds: 60   # 扩容相对快速
      
  # 多指标组合使用
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Pods
    pods:
      metric:
        name: qps
      target:
        type: AverageValue
        averageValue: 500

10.3 与PDB配合使用

# HPA缩容时，PDB确保最少可用副本
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: myapp-pdb
spec:
  minAvailable: 2        # 确保至少2个Pod可用
  selector:
    matchLabels:
      app: myapp

10.4 与VPA（垂直伸缩）配合

# HPA负责水平伸缩（副本数），VPA负责垂直伸缩（资源大小）
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: myapp-vpa
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: myapp
  updatePolicy:
    updateMode: "Auto"  # 自动调整Pod资源请求

11. 完整的生产环境示例

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: production-hpa
  namespace: production
  labels:
    app: myapp
    component: autoscaler
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp-deployment
  minReplicas: 3
  maxReplicas: 50
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 900
      policies:
      - type: Pods
        value: 1
        periodSeconds: 600
      selectPolicy: Disabled  # 在非高峰时段禁用缩容
    scaleUp:
      stabilizationWindowSeconds: 60
      policies:
      - type: Percent
        value: 200
        periodSeconds: 60
      - type: Pods
        value: 10
        periodSeconds: 60
      selectPolicy: Max
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 65
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 75
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: 800
---
# 对应的Deployment配置
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-deployment
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
    spec:
      containers:
      - name: app
        image: myapp:latest
        ports:
        - containerPort: 8080
        resources:
          requests:
            cpu: 200m
            memory: 256Mi
          limits:
            cpu: 500m
            memory: 512Mi
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 60
          periodSeconds: 15

12. HPA版本对比

特性 autoscaling/v1 autoscaling/v2beta1 autoscaling/v2
支持CPU ✅ ✅ ✅
支持内存 ❌ ✅ ✅
自定义指标 ❌ ✅ ✅
外部指标 ❌ ✅ ✅
多指标 ❌ ✅ ✅
行为控制 ❌ ❌ ✅
稳定窗口 ❌ ❌ ✅

总结

HPA是Kubernetes自动伸缩的核心组件，通过合理配置可以实现：

  1. 资源优化：根据负载自动调整副本数
  2. 成本节约：在低负载时减少资源使用
  3. 高可用性：在高负载时自动扩容保证服务
  4. 灵活策略：支持多种指标和自定义行为

关键配置要点：

  • 必须设置Pod资源请求
  • 合理设置min/max副本数
  • 使用行为控制避免频繁伸缩
  • 结合多种指标进行综合判断
  • 与PDB、VPA等其他组件配合使用
posted @ 2026-09-24 10:22  张占岭  阅读(17)  评论(0)    收藏  举报