NVIDIA NVML Driver/library version mismatch

sudo rmmod nvidia_drm
sudo rmmod nvidia_modeset
sudo rmmod nvidia_uvm
sudo lsof /dev/nvidia*
confirm you successfully unload those kmods

lsmod | grep nvidia

you should get nothing, then confirm you can load the correct driver

nvidia-smi

服务器更新nvidia driver 版本之后，经常会出现执行nvidia-smi出现
Failed to initialize NVML: Driver/library version mismatch的问题
执行如下操作，即可修复

sudo rmmod nvidia
sudo nvidia-smi