防止kubepods过载

 

 

 

---
- name: Limit kubepods.slice resources by systemd cgroup v1
  hosts: all
  become: yes
  gather_facts: no

  vars:
    kubepods_slice_dir: /etc/systemd/system/kubepods.slice.d
    kubepods_limit_file: /etc/systemd/system/kubepods.slice.d/limit.conf

    cpu_quota: "3800%"
    memory_limit: "56G"
    tasks_max: "30000"

  tasks:

    - name: Check cgroup v1 directories
      stat:
        path: "{{ item }}"
      register: cgroup_check
      loop:
        - /sys/fs/cgroup/cpu/kubepods.slice
        - /sys/fs/cgroup/memory/kubepods.slice
        - /sys/fs/cgroup/pids/kubepods.slice


    - name: Fail if kubepods cgroup not exists
      fail:
        msg: >
          kubepods.slice cgroup not found,
          this node may not use systemd cgroup driver
      when:
        - not item.stat.exists
      loop: "{{ cgroup_check.results }}"


    - name: Create systemd drop-in directory
      file:
        path: "{{ kubepods_slice_dir }}"
        state: directory
        owner: root
        group: root
        mode: "0755"


    - name: Configure kubepods.slice resource limit
      copy:
        dest: "{{ kubepods_limit_file }}"
        owner: root
        group: root
        mode: "0644"
        content: |
          [Slice]
          CPUQuota={{ cpu_quota }}
          MemoryLimit={{ memory_limit }}
          TasksMax={{ tasks_max }}


    - name: Reload systemd daemon
      systemd:
        daemon_reload: yes


    - name: Restart kubelet
      systemd:
        name: kubelet
        state: restarted
        enabled: yes


    - name: Wait kubelet startup
      pause:
        seconds: 10


    - name: Check systemd kubepods.slice config
      shell: |
        systemctl show kubepods.slice \
        | grep -E "CPUQuota|Memory|Tasks"
      register: systemd_limit
      changed_when: false


    - name: Show systemd limit result
      debug:
        var: systemd_limit.stdout_lines


    - name: Check CPU cgroup limit
      shell: |
        cat /sys/fs/cgroup/cpu/kubepods.slice/cpu.cfs_quota_us
      register: cpu_limit
      changed_when: false


    - name: Check memory cgroup limit
      shell: |
        cat /sys/fs/cgroup/memory/kubepods.slice/memory.limit_in_bytes
      register: memory_limit_result
      changed_when: false


    - name: Check pids cgroup limit
      shell: |
        cat /sys/fs/cgroup/pids/kubepods.slice/pids.max
      register: pids_limit
      changed_when: false


    - name: Show cgroup limit result
      debug:
        msg:
          - "CPU quota(us): {{ cpu_limit.stdout }}"
          - "Memory limit(bytes): {{ memory_limit_result.stdout }}"
          - "Tasks max: {{ pids_limit.stdout }}"


    - name: Check kubelet cgroup driver
      shell: |
        grep cgroupDriver /var/lib/kubelet/config.yaml || true
      register: kubelet_driver
      changed_when: false


    - name: Show kubelet driver
      debug:
        var: kubelet_driver.stdout_lines
[es-ops@man workstation]$ 

　　

 

 

 

 

你的环境是 cgroup v1 + systemd 管理的 kubepods.slice。
直接 echo > /sys/fs/cgroup/... 只是临时生效，重启后会丢失。

永久限制建议使用 systemd slice drop-in。

目标：

kubepods.slice

CPU:
32核

Memory:
48G

Tasks:
30000

---

1. 创建 kubepods.slice 配置目录

mkdir -p /etc/systemd/system/kubepods.slice.d

---

2. 创建限制配置

vi /etc/systemd/system/kubepods.slice.d/limit.conf

写入：

[Slice]
CPUQuota=3200%
MemoryLimit=48G
TasksMax=30000

说明：

参数	值	含义
CPUQuota=3200%	32核	100%=1 CPU
MemoryLimit=48G	48GB	cgroup memory限制
TasksMax=30000	30000 task	PID/线程总数

---

3. 重新加载 systemd

systemctl daemon-reload

---

4. 重启 kubelet

因为 kubepods.slice 是 kubelet 创建管理的，需要让 kubelet重新识别：

systemctl restart kubelet

---

5. 验证 systemd 配置

查看：

systemctl show kubepods.slice | grep -E "CPUQuota|Memory|Tasks"

期望：

CPUQuotaPerSecUSec=3200%
MemoryLimit=51539607552
TasksMax=30000

---

6. 验证 cgroup v1 文件

CPU

cat /sys/fs/cgroup/cpu/kubepods.slice/cpu.cfs_quota_us

应该：

3200000

计算：

3200000 / 100000 = 32 CPU

---

Memory

cat /sys/fs/cgroup/memory/kubepods.slice/memory.limit_in_bytes

应该：

51539607552

---

Tasks

cat /sys/fs/cgroup/pids/kubepods.slice/pids.max

应该：

30000

---

重要：你的环境可能需要 kubelet 参数

你当前：

kubepods.slice
├── kubepods-besteffort.slice
└── kubepods-burstable.slice

说明 kubelet 已经使用 systemd cgroup driver。

确认：

cat /var/lib/kubelet/config.yaml | grep cgroupDriver

应该：

cgroupDriver: systemd

如果是：

cgroupDriver: cgroupfs

则需要改：

cgroupDriver: systemd

然后：

systemctl restart kubelet

---

重启验证

重启机器：

reboot

回来后：

systemctl show kubepods.slice | grep -E "CPUQuota|Memory|Tasks"

以及：

cat /sys/fs/cgroup/cpu/kubepods.slice/cpu.cfs_quota_us

cat /sys/fs/cgroup/memory/kubepods.slice/memory.limit_in_bytes

cat /sys/fs/cgroup/pids/kubepods.slice/pids.max

仍然保持：

3200000
51539607552
30000

即永久生效。

---

补充一点：你这台机器 systemd DefaultTasksMax 已经是：

DefaultTasksMax=18446744073709551615

说明 systemd 没有全局 task 限制，所以设置 kubepods.slice 的 TasksMax=30000 不会和 systemd 默认值冲突。你只限制 Kubernetes 容器，不影响系统服务。