k8s集群节点配置GPU

默认已有以下前置条件

k8s 1.10+
操作系统：麒麟v10 SP2（Centos，Ubuntu等见“安装NVIDIA Container Toolkit”中其他源配置）
GPU驱动：已安装

官方文档：https://github.com/NVIDIA/k8s-device-plugin

一、验证GPU挂载以及驱动是否安装

验证是否挂载

$ lspci -vv | grep -i nvidia
00:0d.0 VGA compatible controller: NVIDIA Corporation TU104GL [Quadro RTX 5000] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: NVIDIA Corporation Device 129f
        Kernel driver in use: nvidia
        Kernel modules: nouveau, nvidia_drm, nvidia
00:0e.0 Audio device: NVIDIA Corporation TU104 HD Audio Controller (rev a1)
        Subsystem: NVIDIA Corporation Device 129f
00:0f.0 USB controller: NVIDIA Corporation TU104 USB 3.1 Host Controller (rev a1) (prog-if 30 [XHCI])
        Subsystem: NVIDIA Corporation Device 129f
00:10.0 Serial bus controller [0c80]: NVIDIA Corporation TU104 USB Type-C UCSI Controller (rev a1)
        Subsystem: NVIDIA Corporation Device 129f

验证驱动是否安装

nvidia-smi全称是NVIDIA System Management Interface ，它是一个基于前面介绍过的NVIDIA Management Library(NVML)构建的命令行实用工具，旨在帮助管理和监控NVIDIA GPU设备。

# nvidia-smi
     
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.113.01             Driver Version: 535.113.01   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Quadro RTX 5000                Off | 00000000:00:0D.0 Off |                  Off |
| 29%   23C    P8              12W / 230W |      0MiB / 17236MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

查看动态监控显卡状态

$ watch -t -n 1 nvidia-smi

验证cuda版本

nvcc -V

二、增加GPU节点标签

$ kubectl label nodes 节点名称 node-role.kubernetes.io/gpu=true

三、在容器中启用 GPU 支持

需要安装NVIDIA Container来支持docker调用显卡

如不设置则会报如下错误：

docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].

NVIDIA Container主要组件包括nvidia-container-runtime, nvidia-container-toolkit, libnvidia-container，

nvidia-container-toolkit：CUDA Toolkit是一个用于开发和优化CUDA应用程序的软件包，其中包括CUDA驱动程序和CUDA运行时库。
nvidia-container-runtime：CUDA运行时库是一个用于在GPU上执行CUDA应用程序的软件组件，它提供了一组CUDA API函数，于管理GPU内存和执行CUDA内核。

官方架构地址：https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/arch-overview.html

依赖关系：

├─ nvidia-container-toolkit (version)
│    ├─ libnvidia-container-tools (>= version)
│    └─ nvidia-container-toolkit-base (version)
│
├─ libnvidia-container-tools (version)
│    └─ libnvidia-container1 (>= version)
└─ libnvidia-container1 (version)

由于nvidia-container-toolkit已包含运行时，这里我们选择安装nvidia-container-toolkit，用于后续k8s节点使用GPU

安装NVIDIA Container Toolkit

运行NVIDIA Container Toolkit的条件：

内核版本 > 3.10 的 GNU/Linux x86_64
Docker >= 19.03（推荐，但某些发行版可能包含旧版本的 Docker。支持的最低版本为 1.12）
架构 >= Kepler（或计算能力 3.0）的 NVIDIA GPU
NVIDIA Linux 驱动程序>= 418.81.07（请注意，不支持较旧的驱动程序版本或分支。）

麒麟系统 v10 sp2设置源：

$ curl -s -L https://nvidia.github.io/nvidia-docker/centos8/nvidia-docker.repo | sudo tee /etc/yum.repos.d/nvidia-docker.repo

$ sudo yum install -y nvidia-container-toolkit

CentOS源

$ curl -s -L https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo | \
  sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo

$ sudo yum install -y nvidia-container-toolkit

Ubuntu源

$ curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list \
  && \
    sudo apt-get update

$ sudo apt-get install -y nvidia-container-toolkit

其他系统存储库配置:https://nvidia.github.io/nvidia-docker/

Docker显卡适配

在Docker在19.03版本以后直接安装NVIDIA Container，可以不需要再独立安装nvidia-docker也就是docker2直接设置就可以支持显卡

直接设置daemon.json,增加如下

  "default-runtime": "nvidia",
  "runtimes": {
      "nvidia": {
          "path": "nvidia-container-runtime",
          "runtimeArgs": []
      }
  }

配置完成后进行重启

$ sudo systemctl restart docker

Cotainerd显卡适配

在config.toml，增加如下：

version = 2
[plugins]
  [plugins."io.containerd.grpc.v1.cri"]
    [plugins."io.containerd.grpc.v1.cri".containerd]
      default_runtime_name = "nvidia"

      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia]
          privileged_without_host_devices = false
          runtime_engine = ""
          runtime_root = ""
          runtime_type = "io.containerd.runc.v2"
          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia.options]
            BinaryName = "/usr/bin/nvidia-container-runtime"

配置完成后进行重启

sudo systemctl restart containerd

容器内验证

docker run --rm --gpus all nvidia/cuda:11.0-base nvidia-smi
Tue Nov 14 12:29:39 2023       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.113.01             Driver Version: 535.113.01   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Quadro RTX 5000                Off | 00000000:00:0D.0 Off |                  Off |
| 29%   23C    P8              12W / 230W |      0MiB / 17236MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

如上为成功设置

四、在Kubernetes中启用 GPU 支持

安装k8s插件 nvidia-device-plugin

$ kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.14.2/nvidia-device-plugin.yml

nvidia-device-plugin.yml内容：

# Copyright (c) 2019, NVIDIA CORPORATION.  All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nvidia-device-plugin-daemonset
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: nvidia-device-plugin-ds
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        name: nvidia-device-plugin-ds
    spec:
      tolerations:
        - key: node-usage.project/name
          operator: Equal
          value: gpu
      priorityClassName: system-node-critical
      containers:
      - image: 'nvidia/k8s-device-plugin:1.11'
        name: nvidia-device-plugin-ctr
        env:
          - name: FAIL_ON_INIT_ERROR
            value: "false"
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop: ["ALL"]
        volumeMounts:
        - name: device-plugin
          mountPath: /var/lib/kubelet/device-plugins
      volumes:
      - name: device-plugin
        hostPath:
          path: /var/lib/kubelet/device-plugins

运行后查看打标签的节点gpu是否启用

$ kubectl get nodes "-o=custom columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu"
NAME          GPU
k8s-worker01   <none>
k8s-worker02   1
k8s-worker03   1

k8s部署验证

部署job进行验证

$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod
spec:
  restartPolicy: Never
  containers:
    - name: cuda-container
      image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda10.2
      resources:
        limits:
          nvidia.com/gpu: 1 # requesting 1 GPU
  tolerations:
  - key: nvidia.com/gpu
    operator: Exists
    effect: NoSchedule
EOF

如下输出则k8s启用GPU成功

 $ kubectl logs gpu-pod
Loading NVML
Fetching devices.
Starting FS watcher.
Starting OS watcher.
Starting to serve on /var/lib/kubelet/device-plugins/nvidia.sock
Registered device plugin with Kubelet

posted @ 2023-11-16 13:31 shookm 阅读(156) 评论(0) 编辑收藏举报

会员力量，点亮园子希望

刷新页面返回顶部

shookm