【异常修复】Docker启动容器时无法使用GPU
硬件配置
显卡：Tesla V100
问题呈现
- 某次重启后，发现服务器nvidia-docker无法
start已有的部分容器
$ docker start ******
Error response from daemon: failed to create shim: OCI runtime create failed
: container_linux.go:380: starting container process caused
: process_linux.go:545: container init caused
: Running hook #0:: error running hook: exit status 1, stdout: , stderr
: nvidia-container-cli: initialization error: nvml error: driver not loaded: unknown
Error: failed to start containers: ******
- 为排查问题的可能原因，尝试另行启动容器，发现启动时无法使用GPU
$ docker run -it --gpus all [Image_Name]:[Image_Tag]
docker:
Error response from daemon: failed to create shim: OCI runtime create failed
: container_linux.go:380: starting container process caused
: process_linux.go:545: container init caused
: Running hook #0:: error running hook: exit status 1, stdout: , stderr
: nvidia-container-cli: initialization error: nvml error: driver not loaded: unknown.
- 为了进一步排查问题的可能原因，使用
nvidia-smi命令查看显卡情况，发现异常
$ nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver.
Make sure that the latest NVIDIA driver is installed and running.
问题定位
docker报错中包含nvml error: driver not loaded: unknown，因此问题产生原因可能是显卡驱动被挤掉了
解决方案
- 查验现状
用nvcc -V确认显卡驱动存在
$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Wed_Jul_22_19:09:09_PDT_2020
Cuda compilation tools, release 11.0, V11.0.221
Build cuda_11.0_bu.TC445_37.28845127_0
- 查看已安装驱动的版本信息
$ ls /usr/src | grep nvidia
nvidia-450.51.06
- 安装一个不知道什么东西，再安一个不知道什么东西（注意需要sudo权限，无权限联系管理员处理）
$ sudo apt-get install dkms
$ sudo dkms install -m nvidia -v 450.51.06 # 注意这个版本号和上一步查到的相对应
- 完成操作后，docker和Nvidia-SMI均恢复正常
