2025.7.17学习日记

1.YML配置环境

今天学习到了一种新式配置python环境的方法，例如在GitHub上的开源项目SAR3D中，装上这个环境的步骤比较长，我们采用逐步装环境法.将dependencies和pip的部分分离，对于dependencies，测试的效果如下：

  1. PLAN_A
  • defaults:
    首先在默认的阿里源下，缺失的模块包括如下，耗时为29s
ResolvePackageNotFound: 
  - decorator==5.1.1=pyhd8ed1ab_0
  - toml==0.10.2=pyhd8ed1ab_0
  - openexr-python==1.3.9=py39hbd9bb45_1
  - zlib==1.2.13=h166bdaf_4
  - _libgcc_mutex==0.1=conda_forge
  - executing==2.0.1=pyhd8ed1ab_0
  - libstdcxx-ng==12.2.0=h46fd767_19
  - cuda-runtime==11.7.1=0
  - conda-pack==0.8.0=pyhd8ed1ab_0
  - _openmp_mutex==4.5=2_kmp_llvm
  - ptyprocess==0.7.0=pyhd3deb0d_0
  - cuda-libraries==11.7.1=0
  - certifi==2024.7.4=pyhd8ed1ab_0
  - libcublas==11.10.3.66=0
  - libnvjpeg==11.8.0.2=0
  - ilmbase==2.5.5=h780b84a_0
  - libcurand==10.3.2.56=0
  - libcusparse==11.7.4.91=0
  - libgcc-ng==12.2.0=h65d4601_19
  - python_abi==3.9=2_cp39
  - cuda-cupti==11.7.101=0
  - typing_extensions==4.12.2=pyha770c72_0
  - cuda-nvrtc==11.7.99=0
  - llvm-openmp==16.0.1=h417c0b6_0
  - libcufile==1.6.0.25=0
  - libnpp==11.7.4.75=0
  - libcusolver==11.4.0.1=0
  - ca-certificates==2024.7.4=hbcca054_0
  - pure_eval==0.2.2=pyhd8ed1ab_0
  - openexr==3.1.7=h5c7bc04_0
  - libzlib==1.2.13=h166bdaf_4
  - asttokens==2.4.1=pyhd8ed1ab_0
  - openssl==1.1.1w=hd590300_0
  - pickleshare==0.7.5=py_1003
  - imath==3.1.7=h3eb15da_1
  - stack_data==0.6.2=pyhd8ed1ab_0
  - libcufft==10.7.2.124=h4fbf590_0
  - cuda-nvtx==11.7.91=0
  - cuda-cudart==11.7.99=0
  • defaults+pytorch:
    加上pytorch源,缺失的模块如下，耗时为31s
ResolvePackageNotFound: 
  - llvm-openmp==16.0.1=h417c0b6_0
  - libcublas==11.10.3.66=0
  - certifi==2024.7.4=pyhd8ed1ab_0
  - ptyprocess==0.7.0=pyhd3deb0d_0
  - openssl==1.1.1w=hd590300_0
  - pure_eval==0.2.2=pyhd8ed1ab_0
  - libcusolver==11.4.0.1=0
  - executing==2.0.1=pyhd8ed1ab_0
  - cuda-cupti==11.7.101=0
  - python_abi==3.9=2_cp39
  - libzlib==1.2.13=h166bdaf_4
  - cuda-runtime==11.7.1=0
  - typing_extensions==4.12.2=pyha770c72_0
  - cuda-nvrtc==11.7.99=0
  - pickleshare==0.7.5=py_1003
  - libnvjpeg==11.8.0.2=0
  - cuda-nvtx==11.7.91=0
  - openexr-python==1.3.9=py39hbd9bb45_1
  - libcufft==10.7.2.124=h4fbf590_0
  - libnpp==11.7.4.75=0
  - _openmp_mutex==4.5=2_kmp_llvm
  - stack_data==0.6.2=pyhd8ed1ab_0
  - cuda-cudart==11.7.99=0
  - _libgcc_mutex==0.1=conda_forge
  - libcurand==10.3.2.56=0
  - libstdcxx-ng==12.2.0=h46fd767_19
  - cuda-libraries==11.7.1=0
  - libgcc-ng==12.2.0=h65d4601_19
  - conda-pack==0.8.0=pyhd8ed1ab_0
  - ilmbase==2.5.5=h780b84a_0
  - decorator==5.1.1=pyhd8ed1ab_0
  - imath==3.1.7=h3eb15da_1
  - zlib==1.2.13=h166bdaf_4
  - libcusparse==11.7.4.91=0
  - asttokens==2.4.1=pyhd8ed1ab_0
  - libcufile==1.6.0.25=0
  - toml==0.10.2=pyhd8ed1ab_0
  - ca-certificates==2024.7.4=hbcca054_0
  - openexr==3.1.7=h5c7bc04_0
  • defaults+pytorch+nvidia：
    加上nvidia源后，缺失的模块如下，耗时为32s
ResolvePackageNotFound: 
  - libzlib==1.2.13=h166bdaf_4
  - ca-certificates==2024.7.4=hbcca054_0
  - certifi==2024.7.4=pyhd8ed1ab_0
  - asttokens==2.4.1=pyhd8ed1ab_0
  - zlib==1.2.13=h166bdaf_4
  - libstdcxx-ng==12.2.0=h46fd767_19
  - ilmbase==2.5.5=h780b84a_0
  - openssl==1.1.1w=hd590300_0
  - stack_data==0.6.2=pyhd8ed1ab_0
  - openexr-python==1.3.9=py39hbd9bb45_1
  - pure_eval==0.2.2=pyhd8ed1ab_0
  - pickleshare==0.7.5=py_1003
  - toml==0.10.2=pyhd8ed1ab_0
  - ptyprocess==0.7.0=pyhd3deb0d_0
  - _openmp_mutex==4.5=2_kmp_llvm
  - executing==2.0.1=pyhd8ed1ab_0
  - imath==3.1.7=h3eb15da_1
  - conda-pack==0.8.0=pyhd8ed1ab_0
  - python_abi==3.9=2_cp39
  - _libgcc_mutex==0.1=conda_forge
  - libgcc-ng==12.2.0=h65d4601_19
  - openexr==3.1.7=h5c7bc04_0
  - decorator==5.1.1=pyhd8ed1ab_0
  - typing_extensions==4.12.2=pyha770c72_0
  - llvm-openmp==16.0.1=h417c0b6_0
  • defaults+pytorch+nvidia+conda-forge:
    加上conda-forge源后，没有缺失但是被kill掉了，说明内存不够(你的内存必须大于2G)，需要开启显卡模式,开启完显卡模式后，无缺失模块，正常进入安装程序，但是由于耗时太久，煮啵另寻出路去了
    【注】：这边会报两个警告，首先是*匹配所有版本的这个警告，被废弃了，其次是要我们升级conda，均可以忽略
  1. PLAN_B
  • 属于conda-forge源的环境安装
    感觉是由于检索源文件(repodata.json)的难度比较大，所以环境安装起来会比较困难,这里我找到了清华源中有关conda-forge的镜像，成功安装完成，耗时2min左右,编写如下文件
# 文件名为conda-forge-envs.yml
name: sar3d
channels:
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/linux-64
dependencies:
  - libzlib==1.2.13=h166bdaf_4
  - ca-certificates==2024.7.4=hbcca054_0
  - certifi==2024.7.4=pyhd8ed1ab_0
  - asttokens==2.4.1=pyhd8ed1ab_0
  - zlib==1.2.13=h166bdaf_4
  - libstdcxx-ng==12.2.0=h46fd767_19
  - ilmbase==2.5.5=h780b84a_0
  - openssl==1.1.1w=hd590300_0
  - stack_data==0.6.2=pyhd8ed1ab_0
  - openexr-python==1.3.9=py39hbd9bb45_1
  - pure_eval==0.2.2=pyhd8ed1ab_0
  - pickleshare==0.7.5=py_1003
  - toml==0.10.2=pyhd8ed1ab_0
  - ptyprocess==0.7.0=pyhd3deb0d_0
  - _openmp_mutex==4.5=2_kmp_llvm
  - executing==2.0.1=pyhd8ed1ab_0
  - imath==3.1.7=h3eb15da_1
  - conda-pack==0.8.0=pyhd8ed1ab_0
  - python_abi==3.9=2_cp39
  - _libgcc_mutex==0.1=conda_forge
  - libgcc-ng==12.2.0=h65d4601_19
  - openexr==3.1.7=h5c7bc04_0
  - decorator==5.1.1=pyhd8ed1ab_0
  - typing_extensions==4.12.2=pyha770c72_0
  - llvm-openmp==16.0.1=h417c0b6_0

直接运行conda env create -f conda-forge-envs.yml,等待2-3min左右即可安装成功

  • 属于nvidia源的环境安装
    下面是属于nvidia源的包，建议利用.sh脚本进行增量增加，编写如下nvidia-envs.sh脚本
#!/bin/bash
conda install -n sar3d \
  libcublas=11.10.3.66=0 \
  libcusolver=11.4.0.1=0 \
  cuda-cupti=11.7.101=0 \
  cuda-runtime=11.7.1=0 \
  cuda-nvrtc=11.7.99=0 \
  libnvjpeg=11.8.0.2=0 \
  cuda-nvtx=11.7.91=0 \
  libcufft=10.7.2.124=h4fbf590_0 \
  libnpp=11.7.4.75=0 \
  cuda-cudart=11.7.99=0 \
  libcurand=10.3.2.56=0 \
  cuda-libraries=11.7.1=0 \
  libcusparse=11.7.4.91=0 \
  libcufile=1.6.0.25=0 \
  -y -c nvidia
echo "nvidia pkg installed"

直接运行bash nvidia-envs.sh,等待8min即可安装成功,貌似也有清华源，应该可以使用-c参数指定，清华源的地址为：https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch/linux-64/

  • 属于pytorch源的环境安装
    下面是始于pytorch源的包，同理编写如下脚本，对比了一下发现完全一样，执行到上一条目就行了
  • 属于defaults源的环境安装
    这里大概4min
#!/bin/bash
conda install -n sar3d \
  blas=1.0=mkl \
  bzip2=1.0.8=h7b6447c_0 \
  freetype=2.12.1=h4a9f257_0 \
  gmp=6.2.1=h295c915_3 \
  gnutls=3.6.15=he1e5248_0 \
  lcms2=2.12=h3be6417_0 \
  libffi=3.4.2=h6a678d5_6 \
  libiconv=1.16=h7f8727e_2 \
  libtasn1=4.16.0=h27cfd23_0 \
  ncurses=6.4=h6a678d5_0 \
  readline=8.2=h5eee18b_0 \
  tk=8.6.12=h1ccaba5_0 \
  xz=5.2.10=h5eee18b_1 \
  zstd=1.5.2=ha4553b6_0\
  python=3.9.16=h7a1cb2a_2 \
  setuptools=65.6.3=py39h06a4308_0 \
  wheel=0.38.4=py39h06a4308_0 \
  sqlite=3.41.1=h5eee18b_0 \
  pyopenssl=23.0.0=py39h06a4308_0 \
  urllib3=1.26.14=py39h06a4308_0 \
  brotlipy=0.7.0=py39h27cfd23_1003 \
  cryptography=39.0.1=py39h9ce1e76_0 \
  cffi=1.15.1=py39h5eee18b_3 \
  pycparser=2.21=pyhd3eb1b0_0 \
  six=1.16.0=pyhd3eb1b0_1 \
  idna=3.4=py39h06a4308_0 \
  charset-normalizer=2.0.4=pyhd3eb1b0_0 \
  pysocks=1.7.1=py39h06a4308_0\
  mkl=2021.4.0=h06a4308_640 \
  mkl-service=2.4.0=py39h7f8727e_0 \
  mkl_fft=1.3.1=py39hd3c417c_0 \
  mkl_random=1.2.2=py39h51133e4_0 \
  intel-openmp=2021.4.0=h06a4308_3561 \
  gmpy2=2.1.2=py39heeb90bb_0 \
  mpfr=4.0.2=hb69a4c5_1 \
  mpc=1.1.0=h10f8cd9_1 \
  mpmath=1.2.1=py39h06a4308_0 \
  sympy=1.11.1=py39h06a4308_0 \
  networkx=2.8.4=py39h06a4308_1\
  ffmpeg=4.3=hf484d3e_0 \
  openh264=2.1.1=h4ff587b_0 \
  lame=3.100=h7b6447c_0 \
  giflib=5.2.1=h5eee18b_3 \
  jpeg=9e=h5eee18b_1 \
  libpng=1.6.39=h5eee18b_0 \
  libtiff=4.5.0=h6a678d5_2 \
  libwebp=1.2.4=h11a3e52_1 \
  libwebp-base=1.2.4=h5eee18b_1 \
  pillow=9.4.0=py39h6a678d5_0\
  pytorch=2.0.0=py3.9_cuda11.7_cudnn8.5.0_0 \
  torchvision=0.15.0=py39_cu117 \
  torchaudio=2.0.0=py39_cu117 \
  pytorch-cuda=11.7=h778d358_3 \
  pytorch-mutex=1.0=cuda \
  torchtriton=2.0.0=py39\
  filelock=3.9.0=py39h06a4308_0 \
  flit-core=3.8.0=py39h06a4308_0 \
  jinja2=3.1.2=py39h06a4308_0 \
  markupsafe=2.1.1=py39h7f8727e_0 \
  ld_impl_linux-64=2.38=h1181459_1 \
  lerc=3.0=h295c915_0 \
  libdeflate=1.17=h5eee18b_0 \
  libidn2=2.3.2=h7f8727e_0 \
  libunistring=0.9.10=h27cfd23_0 \
  lz4-c=1.9.4=h6a678d5_0 \
  nettle=3.7.3=hbbd107a_1\
  -y
posted @ 2025-07-18 01:26  BingUw  阅读(25)  评论(0)    收藏  举报