可微渲染 SoftRas 实践
SoftRas 是目前主流三角网格可微渲染器之一。
可微渲染通过计算渲染过程的导数，使得从单张图片学习三维结构逐渐成为现实。可微渲染目前被广泛地应用于三维重建，特别是人体重建、人脸重建和三维属性估计等应用中。
安装
conda 安装 PyTorch 环境：
conda create -n torch python=3.8 -y
conda activate torch
conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c nvidia -y
conda activate torch
python - <<-EOF
import platform
import torch
print(f"Python : {platform.python_version()}")
print(f"PyTorch: {torch.__version__}")
print(f" CUDA : {torch.version.cuda}")
EOF
Python : 3.8.10
PyTorch: 1.9.0
CUDA : 11.1
获取代码并安装：
git clone https://github.com/ShichenLiu/SoftRas.git
cd SoftRas
python setup.py install
可设 setup.py 镜像源：
cat <<-EOF > ~/.pydistutils.cfg
[easy_install]
index_url = http://mirrors.aliyun.com/pypi/simple
EOF
应用
安装模型查看工具：
snap install ogre-meshviewer
# 或
snap install meshlab
渲染物体
渲染测试：
CUDA_VISIBLE_DEVICES=0 python examples/demo_render.py
渲染结果：
对比前后模型:
ogre-meshviewer data/obj/spot/spot_triangulated.obj
ogre-meshviewer data/results/output_render/saved_spot.obj
Mesh 重建
下载数据集：
bash examples/recon/download_dataset.sh
训练模型：
$ CUDA_VISIBLE_DEVICES=0 python examples/recon/train.py -eid recon
Loading dataset: 100%|██████████████████████████| 13/13 [00:35<00:00, 2.74s/it]
Iter: [0/250000] Time 1.189 Loss 0.655 lr 0.000100 sv 0.000100
Iter: [100/250000] Time 0.464 Loss 0.405 lr 0.000100 sv 0.000100
...
Iter: [250000/250000] Time 0.450 Loss 0.128 lr 0.000030 sv 0.000030
测试模型：
$ CUDA_VISIBLE_DEVICES=0 python examples/recon/test.py -eid recon \
-d 'data/results/models/recon/checkpoint_0250000.pth.tar'
Loading dataset: 100%|██████████████████████████| 13/13 [00:03<00:00, 3.25it/s]
Iter: [0/97] Time 0.419 IoU 0.697
=================================
Mean IoU: 65.586 for class Airplane
Iter: [0/43] Time 0.095 IoU 0.587
=================================
Mean IoU: 49.798 for class Bench
Iter: [0/37] Time 0.089 IoU 0.621
=================================
Mean IoU: 68.975 for class Cabinet
Iter: [0/179] Time 0.088 IoU 0.741
Iter: [100/179] Time 0.083 IoU 0.772
=================================
Mean IoU: 74.224 for class Car
Iter: [0/162] Time 0.086 IoU 0.565
Iter: [100/162] Time 0.085 IoU 0.522
=================================
Mean IoU: 52.933 for class Chair
Iter: [0/26] Time 0.094 IoU 0.681
=================================
Mean IoU: 60.553 for class Display
Iter: [0/55] Time 0.087 IoU 0.526
=================================
Mean IoU: 45.751 for class Lamp
Iter: [0/38] Time 0.086 IoU 0.580
=================================
Mean IoU: 65.626 for class Loudspeaker
Iter: [0/56] Time 0.090 IoU 0.783
=================================
Mean IoU: 68.683 for class Rifle
Iter: [0/76] Time 0.092 IoU 0.647
=================================
Mean IoU: 68.111 for class Sofa
Iter: [0/204] Time 0.090 IoU 0.405
Iter: [100/204] Time 0.087 IoU 0.435
Iter: [200/204] Time 0.086 IoU 0.567
=================================
Mean IoU: 46.206 for class Table
Iter: [0/25] Time 0.097 IoU 0.901
=================================
Mean IoU: 82.261 for class Telephone
Iter: [0/46] Time 0.087 IoU 0.503
=================================
Mean IoU: 61.019 for class Watercraft
=================================
Mean IoU: 62.287 for all classes
Mesh 重建：
# 获取 `softras_recon.py` 进 `examples/recon/`
# https://github.com/ikuokuo/start-3d-recon/blob/master/samples/softras_recon.py
# 注释 `iou` 直接返回 0，位于 `examples/recon/models.py` `evaluate_iou()`
# 2D 图像重构 3D Mesh
CUDA_VISIBLE_DEVICES=0 python examples/recon/softras_recon.py \
-s '.' \
-d 'data/results/models/recon/checkpoint_0250000.pth.tar' \
-img 'data/car_64x64.png'
ogre-meshviewer data/car_64x64.obj
重建图像：
重建结果：
或重建 ShapeNet 数据集内图像：
# mesh recon images of ShapeNet dataset
CUDA_VISIBLE_DEVICES=0 python examples/recon/softras_recon.py \
-s '.' \
-d 'data/results/models/recon/checkpoint_0250000.pth.tar' \
-imgs 'data/datasets/02958343_test_images.npz'
或使用 SoftRas 训练好的模型：
- SoftRas trained with silhouettes supervision (62+ IoU): google drive
- SoftRas trained with shading supervision (64+ IoU, test with
--shading-modelarg): google drive - SoftRas reconstructed meshes with color (random sampled): google drive
更多
GoCoding 个人实践的经验分享，可关注公众号！
