大连人工智能计算平台——华为昇腾AI平台——高性能计算HPC——官方的运行配置文件中的错误修正——MPI启动配置
官方的HPC运行配置文件：
#!/usr/bin/env bash echo ----- print env vars ----- if [ "${CCS_ALLOC_FILE}" != "" ]; then echo " " ls -la ${CCS_ALLOC_FILE} echo ------ cat ${CCS_ALLOC_FILE} cat ${CCS_ALLOC_FILE} fi export HOSTFILE=/tmp/hostfile.$$ rm -rf $HOSTFILE touch $HOSTFILE # parse CCS_ALLOC_FILE ## node name, cores, tasks, task_list # hpcbuild002 8 1 container_22_default_00001_e01_000002 # hpctest005 8 1 container_22_default_00000_e01_000001 ntask=`cat ${CCS_ALLOC_FILE} | awk -v fff="$HOSTFILE" '{} { split($0, a, " ") if (length(a[1]) >0 && length(a[3]) >0) { print a[1]" slots="a[2] >> fff total_task+=a[3] } }END{print total_task}'` echo "openmpi hostfile $HOSTFILE generated:" echo "-----------------------" cat $HOSTFILE echo "-----------------------" echo "Total tasks is $ntask" echo "mpirun -hostfile $HOSTFILE -n $ntask <your application>"
相关知识已在前文给出具体介绍，这里不再讨论，有需要可以翻阅之前blog。
===========================================
启动HPC上MPI的命令：
/opt/batch/cli/bin/dsub -n xxxxxxx -A xxxxxxxxxxxx --priority 9999 --job_retry 10 --job_type hmpi -R "cpu=10;mem=128" -N 100 -eo error.txt -oo output.txt xxxxxxxx.sh
可以看到，上面的命令是启动100个任务（-N 100），每个任务需要的资源为10个CPU和128MB的内存（-R "cpu=10;mem=128"）。
用华为官方给出的HPC启动配置文件并将配置输出：
----- print env vars ----- -rw-rw----+ 1 ccs_agent ccs_agent 4777 Aug 25 08:40 /tmp/.ccscheduler/xxxxxx/mpi/allocFile_container_21302_default_00000_e01_000001 ------ cat /tmp/.ccscheduler/xxxxxx/mpi/allocFile_container_21302_default_00000_e01_000001 dlhpcshare-agent-46 40 4 container_21302_default_00000_e01_000001 container_21302_default_00022_e01_000023 container_21302_default_00074_e01_000075 container_21302_default_00082_e01_000083 dlhpcshare-agent-25 10 1 container_21302_default_00039_e01_000040 dlhpcshare-agent-49 30 3 container_21302_default_00060_e01_000061 container_21302_default_00009_e01_000010 container_21302_default_00054_e01_000055 dlhpcshare-agent-28 70 7 container_21302_default_00047_e01_000048 container_21302_default_00023_e01_000024 container_21302_default_00094_e01_000095 container_21302_default_00053_e01_000054 container_21302_default_00018_e01_000019 container_21302_default_00079_e01_000080 container_21302_default_00013_e01_000014 dlhpcshare-agent-21 10 1 container_21302_default_00067_e01_000068 dlhpcshare-agent-44 20 2 container_21302_default_00099_e01_000100 container_21302_default_00077_e01_000078 dlhpcshare-agent-61 10 1 container_21302_default_00069_e01_000070 dlhpcshare-agent-40 10 1 container_21302_default_00033_e01_000034 dlhpcshare-agent-41 20 2 container_21302_default_00042_e01_000043 container_21302_default_00003_e01_000004 dlhpcshare-agent-42 30 3 container_21302_default_00034_e01_000035 container_21302_default_00037_e01_000038 container_21302_default_00015_e01_000016 dlhpcshare-agent-20 80 8 container_21302_default_00011_e01_000012 container_21302_default_00056_e01_000057 container_21302_default_00093_e01_000094 container_21302_default_00002_e01_000003 container_21302_default_00076_e01_000077 container_21302_default_00087_e01_000088 container_21302_default_00092_e01_000093 container_21302_default_00031_e01_000032 dlhpcshare-agent-8 60 6 container_21302_default_00050_e01_000051 container_21302_default_00058_e01_000059 container_21302_default_00030_e01_000031 container_21302_default_00055_e01_000056 container_21302_default_00012_e01_000013 container_21302_default_00014_e01_000015 dlhpcshare-agent-6 10 1 container_21302_default_00044_e01_000045 dlhpcshare-agent-2 20 2 container_21302_default_00081_e01_000082 container_21302_default_00064_e01_000065 dlhpcshare-agent-14 60 6 container_21302_default_00084_e01_000085 container_21302_default_00086_e01_000087 container_21302_default_00043_e01_000044 container_21302_default_00071_e01_000072 container_21302_default_00098_e01_000099 container_21302_default_00052_e01_000053 dlhpcshare-agent-36 100 10 container_21302_default_00095_e01_000096 container_21302_default_00038_e01_000039 container_21302_default_00061_e01_000062 container_21302_default_00091_e01_000092 container_21302_default_00090_e01_000091 container_21302_default_00001_e01_000002 container_21302_default_00078_e01_000079 container_21302_default_00085_e01_000086 container_21302_default_00066_e01_000067 container_21302_default_00007_e01_000008 dlhpcshare-agent-59 10 1 container_21302_default_00065_e01_000066 dlhpcshare-agent-15 40 4 container_21302_default_00051_e01_000052 container_21302_default_00072_e01_000073 container_21302_default_00073_e01_000074 container_21302_default_00068_e01_000069 dlhpcshare-agent-39 10 1 container_21302_default_00080_e01_000081 dlhpcshare-agent-17 40 4 container_21302_default_00057_e01_000058 container_21302_default_00070_e01_000071 container_21302_default_00075_e01_000076 container_21302_default_00028_e01_000029 dlhpcshare-agent-54 30 3 container_21302_default_00083_e01_000084 container_21302_default_00010_e01_000011 container_21302_default_00059_e01_000060 dlhpcshare-agent-12 50 5 container_21302_default_00097_e01_000098 container_21302_default_00096_e01_000097 container_21302_default_00048_e01_000049 container_21302_default_00063_e01_000064 container_21302_default_00089_e01_000090 dlhpcshare-agent-34 120 12 container_21302_default_00027_e01_000028 container_21302_default_00016_e01_000017 container_21302_default_00032_e01_000033 container_21302_default_00036_e01_000037 container_21302_default_00020_e01_000021 container_21302_default_00029_e01_000030 container_21302_default_00019_e01_000020 container_21302_default_00040_e01_000041 container_21302_default_00024_e01_000025 container_21302_default_00004_e01_000005 container_21302_default_00017_e01_000018 container_21302_default_00045_e01_000046 dlhpcshare-agent-57 30 3 container_21302_default_00006_e01_000007 container_21302_default_00049_e01_000050 container_21302_default_00088_e01_000089 dlhpcshare-agent-13 30 3 container_21302_default_00005_e01_000006 container_21302_default_00041_e01_000042 container_21302_default_00035_e01_000036 dlhpcshare-agent-53 60 6 container_21302_default_00021_e01_000022 container_21302_default_00062_e01_000063 container_21302_default_00046_e01_000047 container_21302_default_00008_e01_000009 container_21302_default_00025_e01_000026 container_21302_default_00026_e01_000027 openmpi hostfile /tmp/hostfile.1217297 generated: ----------------------- dlhpcshare-agent-46 slots=40 dlhpcshare-agent-25 slots=10 dlhpcshare-agent-49 slots=30 dlhpcshare-agent-28 slots=70 dlhpcshare-agent-21 slots=10 dlhpcshare-agent-44 slots=20 dlhpcshare-agent-61 slots=10 dlhpcshare-agent-40 slots=10 dlhpcshare-agent-41 slots=20 dlhpcshare-agent-42 slots=30 dlhpcshare-agent-20 slots=80 dlhpcshare-agent-8 slots=60 dlhpcshare-agent-6 slots=10 dlhpcshare-agent-2 slots=20 dlhpcshare-agent-14 slots=60 dlhpcshare-agent-36 slots=100 dlhpcshare-agent-59 slots=10 dlhpcshare-agent-15 slots=40 dlhpcshare-agent-39 slots=10 dlhpcshare-agent-17 slots=40 dlhpcshare-agent-54 slots=30 dlhpcshare-agent-12 slots=50 dlhpcshare-agent-34 slots=120 dlhpcshare-agent-57 slots=30 dlhpcshare-agent-13 slots=30 dlhpcshare-agent-53 slots=60 ----------------------- Total tasks is 100 mpirun -hostfile /tmp/hostfile.1217297 -n 100 <your application>
运行HPC上的MPI测试代码：
hello.py
import os import mpi4py.MPI as MPI import sys import numpy as np import time def func1(queue, num): # time.sleep(num) # time.sleep(180) x = np.random.rand(100) for _ in range(2000000): x += np.random.rand(100) num += np.sum(x) queue.put(num) def run_queue(): from multiprocessing import Process, Queue ps = 10 queue = Queue(maxsize=200) # the following attribute can call in anywhere process = [Process(target=func1, args=(queue, num)) for num in range(ps)] [p.start() for p in process] [p.join() for p in process] return sum([queue.get() for p in process]) comm = MPI.COMM_WORLD comm_rank = comm.Get_rank() comm_size = comm.Get_size() node_name = MPI.Get_processor_name() # point to point communication data_send = [comm_rank]*1 comm.send(data_send,dest=(comm_rank+1)%comm_size) res = run_queue() ### data_recv =comm.recv(source=(comm_rank-1)%comm_size)
结果：
my rank is 0/100, and node_name: dlhpcshare-agent-46 Ireceived:[99]1000016705.767213 my rank is 1/100, and node_name: dlhpcshare-agent-46 Ireceived:[0]999965729.3735983 my rank is 2/100, and node_name: dlhpcshare-agent-46 Ireceived:[1]1000055354.1766404 my rank is 3/100, and node_name: dlhpcshare-agent-46 Ireceived:[2]999938702.6090965 my rank is 4/100, and node_name: dlhpcshare-agent-46 Ireceived:[3]999932647.7499591 my rank is 5/100, and node_name: dlhpcshare-agent-46 Ireceived:[4]999970098.8907137 my rank is 6/100, and node_name: dlhpcshare-agent-46 Ireceived:[5]1000035017.3560394 my rank is 7/100, and node_name: dlhpcshare-agent-46 Ireceived:[6]999983266.860093 my rank is 8/100, and node_name: dlhpcshare-agent-46 Ireceived:[7]1000019754.1181364 my rank is 9/100, and node_name: dlhpcshare-agent-46 Ireceived:[8]1000028583.4510313 my rank is 10/100, and node_name: dlhpcshare-agent-46 Ireceived:[9]999948045.2254038 my rank is 11/100, and node_name: dlhpcshare-agent-46 Ireceived:[10]1000000535.3682672 my rank is 12/100, and node_name: dlhpcshare-agent-46 Ireceived:[11]1000060573.6321007 my rank is 13/100, and node_name: dlhpcshare-agent-46 Ireceived:[12]999989400.5199456 my rank is 14/100, and node_name: dlhpcshare-agent-46 Ireceived:[13]1000004276.757213 my rank is 15/100, and node_name: dlhpcshare-agent-46 Ireceived:[14]1000036378.0970109 my rank is 16/100, and node_name: dlhpcshare-agent-46 Ireceived:[15]999941671.5626221 my rank is 17/100, and node_name: dlhpcshare-agent-46 Ireceived:[16]1000016780.6566744 my rank is 18/100, and node_name: dlhpcshare-agent-46 Ireceived:[17]999976054.5543907 my rank is 19/100, and node_name: dlhpcshare-agent-46 Ireceived:[18]999981141.0937772 my rank is 20/100, and node_name: dlhpcshare-agent-46 Ireceived:[19]1000015375.1100855 my rank is 21/100, and node_name: dlhpcshare-agent-46 Ireceived:[20]1000073895.3809344 my rank is 22/100, and node_name: dlhpcshare-agent-46 Ireceived:[21]999949026.6361238 my rank is 23/100, and node_name: dlhpcshare-agent-46 Ireceived:[22]999966888.9603195 my rank is 24/100, and node_name: dlhpcshare-agent-46 Ireceived:[23]999972898.4622048 my rank is 25/100, and node_name: dlhpcshare-agent-46 Ireceived:[24]999949881.6119624 my rank is 26/100, and node_name: dlhpcshare-agent-46 Ireceived:[25]999952754.392655 my rank is 27/100, and node_name: dlhpcshare-agent-46 Ireceived:[26]1000003606.3229789 my rank is 28/100, and node_name: dlhpcshare-agent-46 Ireceived:[27]999978698.6484457 my rank is 29/100, and node_name: dlhpcshare-agent-46 Ireceived:[28]1000023654.3494583 my rank is 30/100, and node_name: dlhpcshare-agent-46 Ireceived:[29]999979493.4737692 my rank is 31/100, and node_name: dlhpcshare-agent-46 Ireceived:[30]999952955.1322489 my rank is 32/100, and node_name: dlhpcshare-agent-46 Ireceived:[31]1000023147.510355 my rank is 33/100, and node_name: dlhpcshare-agent-46 Ireceived:[32]999988621.8661311 my rank is 34/100, and node_name: dlhpcshare-agent-46 Ireceived:[33]1000032886.1024017 my rank is 35/100, and node_name: dlhpcshare-agent-46 Ireceived:[34]1000080786.645305 my rank is 36/100, and node_name: dlhpcshare-agent-46 Ireceived:[35]999985738.2168791 my rank is 37/100, and node_name: dlhpcshare-agent-46 Ireceived:[36]1000035874.901148 my rank is 38/100, and node_name: dlhpcshare-agent-46 Ireceived:[37]999986126.0029303 my rank is 39/100, and node_name: dlhpcshare-agent-46 Ireceived:[38]1000005594.235587 my rank is 40/100, and node_name: dlhpcshare-agent-25 Ireceived:[39]999994753.8205698 my rank is 41/100, and node_name: dlhpcshare-agent-25 Ireceived:[40]999999613.1839917 my rank is 42/100, and node_name: dlhpcshare-agent-25 Ireceived:[41]1000032886.4796938 my rank is 43/100, and node_name: dlhpcshare-agent-25 Ireceived:[42]1000015299.8141469 my rank is 44/100, and node_name: dlhpcshare-agent-25 Ireceived:[43]999978839.5215104 my rank is 45/100, and node_name: dlhpcshare-agent-25 Ireceived:[44]1000056945.34637 my rank is 46/100, and node_name: dlhpcshare-agent-25 Ireceived:[45]999987546.3340306 my rank is 47/100, and node_name: dlhpcshare-agent-25 Ireceived:[46]999977891.1204312 my rank is 48/100, and node_name: dlhpcshare-agent-25 Ireceived:[47]999964171.7260697 my rank is 49/100, and node_name: dlhpcshare-agent-25 Ireceived:[48]1000002815.8499967 my rank is 50/100, and node_name: dlhpcshare-agent-49 Ireceived:[49]1000088232.1541688 my rank is 51/100, and node_name: dlhpcshare-agent-49 Ireceived:[50]999922549.1663526 my rank is 52/100, and node_name: dlhpcshare-agent-49 Ireceived:[51]1000036873.3701472 my rank is 53/100, and node_name: dlhpcshare-agent-49 Ireceived:[52]999946075.0287105 my rank is 54/100, and node_name: dlhpcshare-agent-49 Ireceived:[53]1000071587.8812076 my rank is 55/100, and node_name: dlhpcshare-agent-49 Ireceived:[54]999950657.5166117 my rank is 56/100, and node_name: dlhpcshare-agent-49 Ireceived:[55]999933878.6885785 my rank is 57/100, and node_name: dlhpcshare-agent-49 Ireceived:[56]1000066159.5095633 my rank is 58/100, and node_name: dlhpcshare-agent-49 Ireceived:[57]999963883.8407422 my rank is 59/100, and node_name: dlhpcshare-agent-49 Ireceived:[58]1000010886.356683 my rank is 60/100, and node_name: dlhpcshare-agent-49 Ireceived:[59]999998936.3161815 my rank is 61/100, and node_name: dlhpcshare-agent-49 Ireceived:[60]1000032457.9024019 my rank is 62/100, and node_name: dlhpcshare-agent-49 Ireceived:[61]1000008817.6049147 my rank is 63/100, and node_name: dlhpcshare-agent-49 Ireceived:[62]999916332.1086562 my rank is 64/100, and node_name: dlhpcshare-agent-49 Ireceived:[63]1000028239.993777 my rank is 65/100, and node_name: dlhpcshare-agent-49 Ireceived:[64]1000025534.4065487 my rank is 66/100, and node_name: dlhpcshare-agent-49 Ireceived:[65]999965383.8654947 my rank is 67/100, and node_name: dlhpcshare-agent-49 Ireceived:[66]999901860.5872655 my rank is 68/100, and node_name: dlhpcshare-agent-49 Ireceived:[67]1000061659.5934151 my rank is 69/100, and node_name: dlhpcshare-agent-49 Ireceived:[68]1000018931.4276459 my rank is 70/100, and node_name: dlhpcshare-agent-49 Ireceived:[69]1000066546.8614273 my rank is 71/100, and node_name: dlhpcshare-agent-49 Ireceived:[70]999979458.4319509 my rank is 72/100, and node_name: dlhpcshare-agent-49 Ireceived:[71]1000010398.3486695 my rank is 73/100, and node_name: dlhpcshare-agent-49 Ireceived:[72]999996877.6207495 my rank is 74/100, and node_name: dlhpcshare-agent-49 Ireceived:[73]999963902.0169674 my rank is 75/100, and node_name: dlhpcshare-agent-49 Ireceived:[74]999949408.9980878 my rank is 76/100, and node_name: dlhpcshare-agent-49 Ireceived:[75]1000033206.5989758 my rank is 77/100, and node_name: dlhpcshare-agent-49 Ireceived:[76]999980319.9703398 my rank is 78/100, and node_name: dlhpcshare-agent-49 Ireceived:[77]1000096159.5354401 my rank is 79/100, and node_name: dlhpcshare-agent-49 Ireceived:[78]1000012855.660957 my rank is 80/100, and node_name: dlhpcshare-agent-28 Ireceived:[79]999945256.5891613 my rank is 81/100, and node_name: dlhpcshare-agent-28 Ireceived:[80]999936662.6228058 my rank is 82/100, and node_name: dlhpcshare-agent-28 Ireceived:[81]1000047847.3015394 my rank is 83/100, and node_name: dlhpcshare-agent-28 Ireceived:[82]1000009892.1034278 my rank is 84/100, and node_name: dlhpcshare-agent-28 Ireceived:[83]1000017076.6121353 my rank is 85/100, and node_name: dlhpcshare-agent-28 Ireceived:[84]1000018412.7315669 my rank is 86/100, and node_name: dlhpcshare-agent-28 Ireceived:[85]999982459.9707924 my rank is 87/100, and node_name: dlhpcshare-agent-28 Ireceived:[86]1000033177.8524352 my rank is 88/100, and node_name: dlhpcshare-agent-28 Ireceived:[87]999969004.9046781 my rank is 89/100, and node_name: dlhpcshare-agent-28 Ireceived:[88]999985019.450158 my rank is 90/100, and node_name: dlhpcshare-agent-28 Ireceived:[89]1000057133.9164723 my rank is 91/100, and node_name: dlhpcshare-agent-28 Ireceived:[90]1000076183.055097 my rank is 92/100, and node_name: dlhpcshare-agent-28 Ireceived:[91]1000001585.1896353 my rank is 93/100, and node_name: dlhpcshare-agent-28 Ireceived:[92]999997285.7032727 my rank is 94/100, and node_name: dlhpcshare-agent-28 Ireceived:[93]999958897.370945 my rank is 95/100, and node_name: dlhpcshare-agent-28 Ireceived:[94]999987025.9069097 my rank is 96/100, and node_name: dlhpcshare-agent-28 Ireceived:[95]1000040273.2979946 my rank is 97/100, and node_name: dlhpcshare-agent-28 Ireceived:[96]1000019695.0531096 my rank is 98/100, and node_name: dlhpcshare-agent-28 Ireceived:[97]1000010986.5963597 my rank is 99/100, and node_name: dlhpcshare-agent-28 Ireceived:[98]1000039139.5256413
统计下结果中运行的主机情况：
----------------------------------------------------------
这个结果说明，HPC的调度分配是用容器作为task_list进行分配的，我们运行的目的是启动100个task，每个task使用10个CPU，最终HPC调度器也是如此分配的，但是在HPC的MPI启动配置文件时就出了错误，这时候是启动100个task，每个task使用1个CPU，结果就是调度器分配的26台主机，1000个CPU，而最终运行HPC时却只使用了4个主机和100个CPU，可以知道这个错误造成的性能差距是十分巨大的。
----------------------------------------------------------
更改运行的MPI代码：
加入代码：os.cpu_count()
结果：
my rank is 0/100, and node_name: dlhpcshare-agent-53 Ireceived:[99] 128 1000106515.606065 my rank is 1/100, and node_name: dlhpcshare-agent-53 Ireceived:[0] 128 999939847.1516143 my rank is 2/100, and node_name: dlhpcshare-agent-53 Ireceived:[1] 128 999893013.0117773 my rank is 3/100, and node_name: dlhpcshare-agent-53 Ireceived:[2] 128 1000006650.8617853 my rank is 4/100, and node_name: dlhpcshare-agent-53 Ireceived:[3] 128 1000014617.9322758 my rank is 5/100, and node_name: dlhpcshare-agent-53 Ireceived:[4] 128 1000035850.8999186 my rank is 6/100, and node_name: dlhpcshare-agent-53 Ireceived:[5] 128 1000056091.3767813 my rank is 7/100, and node_name: dlhpcshare-agent-53 Ireceived:[6] 128 1000092650.9862612 my rank is 8/100, and node_name: dlhpcshare-agent-53 Ireceived:[7] 128 999992376.4017116 my rank is 9/100, and node_name: dlhpcshare-agent-53 Ireceived:[8] 128 999990154.7197974 my rank is 10/100, and node_name: dlhpcshare-agent-53 Ireceived:[9] 128 999945251.9710757 my rank is 11/100, and node_name: dlhpcshare-agent-53 Ireceived:[10] 128 1000018748.3208461 my rank is 12/100, and node_name: dlhpcshare-agent-53 Ireceived:[11] 128 1000014784.1006019 my rank is 13/100, and node_name: dlhpcshare-agent-53 Ireceived:[12] 128 1000018181.9926168 my rank is 14/100, and node_name: dlhpcshare-agent-53 Ireceived:[13] 128 999958054.3204675 my rank is 15/100, and node_name: dlhpcshare-agent-53 Ireceived:[14] 128 999924067.4712228 my rank is 16/100, and node_name: dlhpcshare-agent-53 Ireceived:[15] 128 999988603.3355628 my rank is 17/100, and node_name: dlhpcshare-agent-53 Ireceived:[16] 128 1000012689.8649071 my rank is 18/100, and node_name: dlhpcshare-agent-53 Ireceived:[17] 128 999984526.9629173 my rank is 19/100, and node_name: dlhpcshare-agent-53 Ireceived:[18] 128 1000021401.943769 my rank is 20/100, and node_name: dlhpcshare-agent-53 Ireceived:[19] 128 999929078.5520971 my rank is 21/100, and node_name: dlhpcshare-agent-53 Ireceived:[20] 128 999952623.3748196 my rank is 22/100, and node_name: dlhpcshare-agent-53 Ireceived:[21] 128 999959034.3172395 my rank is 23/100, and node_name: dlhpcshare-agent-53 Ireceived:[22] 128 999987730.0943791 my rank is 24/100, and node_name: dlhpcshare-agent-53 Ireceived:[23] 128 999965389.0376471 my rank is 25/100, and node_name: dlhpcshare-agent-53 Ireceived:[24] 128 1000037427.562461 my rank is 26/100, and node_name: dlhpcshare-agent-53 Ireceived:[25] 128 1000018241.0460006 my rank is 27/100, and node_name: dlhpcshare-agent-53 Ireceived:[26] 128 999971382.2166605 my rank is 28/100, and node_name: dlhpcshare-agent-53 Ireceived:[27] 128 999972307.7894782 my rank is 29/100, and node_name: dlhpcshare-agent-53 Ireceived:[28] 128 999946836.4048017 my rank is 30/100, and node_name: dlhpcshare-agent-53 Ireceived:[29] 128 999964303.4481132 my rank is 31/100, and node_name: dlhpcshare-agent-53 Ireceived:[30] 128 999984879.3717456 my rank is 32/100, and node_name: dlhpcshare-agent-53 Ireceived:[31] 128 999965798.9835973 my rank is 33/100, and node_name: dlhpcshare-agent-53 Ireceived:[32] 128 1000106757.4002326 my rank is 34/100, and node_name: dlhpcshare-agent-53 Ireceived:[33] 128 999995905.8208525 my rank is 35/100, and node_name: dlhpcshare-agent-53 Ireceived:[34] 128 999939982.0805119 my rank is 36/100, and node_name: dlhpcshare-agent-53 Ireceived:[35] 128 1000040325.6754566 my rank is 37/100, and node_name: dlhpcshare-agent-53 Ireceived:[36] 128 999974755.9749557 my rank is 38/100, and node_name: dlhpcshare-agent-53 Ireceived:[37] 128 999990948.2711018 my rank is 39/100, and node_name: dlhpcshare-agent-53 Ireceived:[38] 128 1000005991.5601768 my rank is 40/100, and node_name: dlhpcshare-agent-53 Ireceived:[39] 128 1000026937.8002824 my rank is 41/100, and node_name: dlhpcshare-agent-53 Ireceived:[40] 128 1000013302.4165831 my rank is 42/100, and node_name: dlhpcshare-agent-53 Ireceived:[41] 128 1000048050.6615318 my rank is 43/100, and node_name: dlhpcshare-agent-53 Ireceived:[42] 128 1000050569.9395372 my rank is 44/100, and node_name: dlhpcshare-agent-53 Ireceived:[43] 128 999994676.4451874 my rank is 45/100, and node_name: dlhpcshare-agent-53 Ireceived:[44] 128 999965814.8078717 my rank is 46/100, and node_name: dlhpcshare-agent-53 Ireceived:[45] 128 1000063779.5651985 my rank is 47/100, and node_name: dlhpcshare-agent-53 Ireceived:[46] 128 999912822.4157392 my rank is 48/100, and node_name: dlhpcshare-agent-53 Ireceived:[47] 128 999928593.3258204 my rank is 49/100, and node_name: dlhpcshare-agent-53 Ireceived:[48] 128 1000079311.6429831 my rank is 50/100, and node_name: dlhpcshare-agent-53 Ireceived:[49] 128 999969785.781623 my rank is 51/100, and node_name: dlhpcshare-agent-53 Ireceived:[50] 128 999976691.4177382 my rank is 52/100, and node_name: dlhpcshare-agent-53 Ireceived:[51] 128 999981874.8187928 my rank is 53/100, and node_name: dlhpcshare-agent-53 Ireceived:[52] 128 999978211.3062496 my rank is 54/100, and node_name: dlhpcshare-agent-53 Ireceived:[53] 128 1000030697.1055744 my rank is 55/100, and node_name: dlhpcshare-agent-53 Ireceived:[54] 128 999949893.7772596 my rank is 56/100, and node_name: dlhpcshare-agent-53 Ireceived:[55] 128 1000048383.3265806 my rank is 57/100, and node_name: dlhpcshare-agent-53 Ireceived:[56] 128 999981985.1777877 my rank is 58/100, and node_name: dlhpcshare-agent-53 Ireceived:[57] 128 1000038613.5794754 my rank is 59/100, and node_name: dlhpcshare-agent-53 Ireceived:[58] 128 1000025518.5350524 my rank is 60/100, and node_name: dlhpcshare-agent-25 Ireceived:[59] 128 999977504.5852437 my rank is 61/100, and node_name: dlhpcshare-agent-25 Ireceived:[60] 128 1000064099.5901086 my rank is 62/100, and node_name: dlhpcshare-agent-25 Ireceived:[61] 128 999958375.7325293 my rank is 63/100, and node_name: dlhpcshare-agent-25 Ireceived:[62] 128 1000032327.017775 my rank is 64/100, and node_name: dlhpcshare-agent-25 Ireceived:[63] 128 999973698.7042892 my rank is 65/100, and node_name: dlhpcshare-agent-25 Ireceived:[64] 128 1000005174.9065778 my rank is 66/100, and node_name: dlhpcshare-agent-25 Ireceived:[65] 128 1000014848.157102 my rank is 67/100, and node_name: dlhpcshare-agent-25 Ireceived:[66] 128 1000007693.8751913 my rank is 68/100, and node_name: dlhpcshare-agent-25 Ireceived:[67] 128 1000007725.2601458 my rank is 69/100, and node_name: dlhpcshare-agent-25 Ireceived:[68] 128 999985807.2807001 my rank is 70/100, and node_name: dlhpcshare-agent-49 Ireceived:[69] 128 999996348.5611703 my rank is 71/100, and node_name: dlhpcshare-agent-49 Ireceived:[70] 128 1000019451.4848017 my rank is 72/100, and node_name: dlhpcshare-agent-49 Ireceived:[71] 128 999930652.0451655 my rank is 73/100, and node_name: dlhpcshare-agent-49 Ireceived:[72] 128 1000056083.7573292 my rank is 74/100, and node_name: dlhpcshare-agent-49 Ireceived:[73] 128 999974813.2241176 my rank is 75/100, and node_name: dlhpcshare-agent-49 Ireceived:[74] 128 1000010925.1539313 my rank is 76/100, and node_name: dlhpcshare-agent-49 Ireceived:[75] 128 1000053204.7815492 my rank is 77/100, and node_name: dlhpcshare-agent-49 Ireceived:[76] 128 1000002623.7976538 my rank is 78/100, and node_name: dlhpcshare-agent-49 Ireceived:[77] 128 999995721.4458407 my rank is 79/100, and node_name: dlhpcshare-agent-49 Ireceived:[78] 128 1000016805.8214403 my rank is 80/100, and node_name: dlhpcshare-agent-28 Ireceived:[79] 128 999993741.0873865 my rank is 81/100, and node_name: dlhpcshare-agent-28 Ireceived:[80] 128 999985355.8043406 my rank is 82/100, and node_name: dlhpcshare-agent-28 Ireceived:[81] 128 1000020033.2797922 my rank is 83/100, and node_name: dlhpcshare-agent-28 Ireceived:[82] 128 1000025818.03303 my rank is 84/100, and node_name: dlhpcshare-agent-28 Ireceived:[83] 128 999994803.3162259 my rank is 85/100, and node_name: dlhpcshare-agent-28 Ireceived:[84] 128 999949729.7594 my rank is 86/100, and node_name: dlhpcshare-agent-28 Ireceived:[85] 128 1000026843.4581102 my rank is 87/100, and node_name: dlhpcshare-agent-28 Ireceived:[86] 128 1000021547.2364391 my rank is 88/100, and node_name: dlhpcshare-agent-28 Ireceived:[87] 128 999972651.1080276 my rank is 89/100, and node_name: dlhpcshare-agent-28 Ireceived:[88] 128 999978283.9195337 my rank is 90/100, and node_name: dlhpcshare-agent-28 Ireceived:[89] 128 1000060266.1070893 my rank is 91/100, and node_name: dlhpcshare-agent-28 Ireceived:[90] 128 1000058939.073361 my rank is 92/100, and node_name: dlhpcshare-agent-28 Ireceived:[91] 128 1000003214.8985409 my rank is 93/100, and node_name: dlhpcshare-agent-28 Ireceived:[92] 128 999972223.4514489 my rank is 94/100, and node_name: dlhpcshare-agent-28 Ireceived:[93] 128 999999667.1805131 my rank is 95/100, and node_name: dlhpcshare-agent-28 Ireceived:[94] 128 999999557.5432729 my rank is 96/100, and node_name: dlhpcshare-agent-28 Ireceived:[95] 128 1000041692.9375429 my rank is 97/100, and node_name: dlhpcshare-agent-28 Ireceived:[96] 128 1000001575.5039369 my rank is 98/100, and node_name: dlhpcshare-agent-28 Ireceived:[97] 128 999997981.0942448 my rank is 99/100, and node_name: dlhpcshare-agent-28 Ireceived:[98] 128 1000024796.3548242
可以看到每个进程都可以识别到当前所在主机的128个CPU，这也就更加印证了之前blog中的分析，华为的HPC调度中的容器这个划分其实并没有对CPU进行分割，每个在主机上运行的进程都是可以识别并使用当前主机上的所有CPU，即128个CPU的，而在HPC调度器上申请的数量只是用于计费和监督的，我们完全可以对其进行绕过的。
有了这个分析后我们可以对华为官方给出的启动配置进行修改：
将函数：
ntask=`cat ${CCS_ALLOC_FILE} | awk -v fff="$HOSTFILE" '{}
{
split($0, a, " ")
if (length(a[1]) >0 && length(a[3]) >0) {
print a[1]" slots="a[2] >> fff
total_task+=a[3]
}
}END{print total_task}'`
修改为：
ntask=`cat ${CCS_ALLOC_FILE} | awk -v fff="$HOSTFILE" '{}
{
split($0, a, " ")
if (length(a[1]) >0 && length(a[3]) >0) {
print a[1]" slots="a[3] >> fff
total_task+=a[3]
}
}END{print total_task}'`
============================================
修改完成后，进行验证：
代码：
import os import mpi4py.MPI as MPI import sys import numpy as np import time def func1(queue, num): # time.sleep(num) # time.sleep(180) x = np.random.rand(100) for _ in range(2000000): x += np.random.rand(100) num += np.sum(x) queue.put(num) def run_queue(): from multiprocessing import Process, Queue ps = 10 queue = Queue(maxsize=200) # the following attribute can call in anywhere process = [Process(target=func1, args=(queue, num)) for num in range(ps)] [p.start() for p in process] [p.join() for p in process] return sum([queue.get() for p in process]) comm = MPI.COMM_WORLD comm_rank = comm.Get_rank() comm_size = comm.Get_size() node_name = MPI.Get_processor_name() # point to point communication data_send = [comm_rank]*1 comm.send(data_send,dest=(comm_rank+1)%comm_size) res = run_queue() ### data_recv =comm.recv(source=(comm_rank-1)%comm_size) # print("my rank is %d, and Ireceived:" % comm_rank, data_recv, file=sys.stdout, flush=True) # print(data_recv) with open("/home/share/xxxxxxxxxx/home/xxxxxx/xxxxxx/results/{}.txt".format(comm_rank, ), "w") as f: f.write("my rank is %d/%d, and node_name: %s Ireceived:" % (comm_rank, comm_size, node_name) + str(data_recv)+" "+str(os.cpu_count()) +" "+ str(res) + "\n" )
运行的配置log：
----- print env vars ----- -rw-rw----+ 1 ccs_agent ccs_agent 4777 Aug 25 10:49 /tmp/.ccscheduler/xxxxxx/mpi/allocFile_container_21305_default_00000_e01_000001 ------ cat /tmp/.ccscheduler/xxxxxx/mpi/allocFile_container_21305_default_00000_e01_000001 dlhpcshare-agent-28 40 4 container_21305_default_00000_e01_000001 container_21305_default_00046_e01_000047 container_21305_default_00092_e01_000093 container_21305_default_00033_e01_000034 dlhpcshare-agent-25 40 4 container_21305_default_00086_e01_000087 container_21305_default_00009_e01_000010 container_21305_default_00002_e01_000003 container_21305_default_00070_e01_000071 dlhpcshare-agent-49 20 2 container_21305_default_00020_e01_000021 container_21305_default_00061_e01_000062 dlhpcshare-agent-21 10 1 container_21305_default_00051_e01_000052 dlhpcshare-agent-44 30 3 container_21305_default_00066_e01_000067 container_21305_default_00076_e01_000077 container_21305_default_00099_e01_000100 dlhpcshare-agent-46 110 11 container_21305_default_00098_e01_000099 container_21305_default_00029_e01_000030 container_21305_default_00055_e01_000056 container_21305_default_00064_e01_000065 container_21305_default_00079_e01_000080 container_21305_default_00072_e01_000073 container_21305_default_00083_e01_000084 container_21305_default_00041_e01_000042 container_21305_default_00022_e01_000023 container_21305_default_00023_e01_000024 container_21305_default_00090_e01_000091 dlhpcshare-agent-61 10 1 container_21305_default_00097_e01_000098 dlhpcshare-agent-40 10 1 container_21305_default_00087_e01_000088 dlhpcshare-agent-42 10 1 container_21305_default_00084_e01_000085 dlhpcshare-agent-20 30 3 container_21305_default_00013_e01_000014 container_21305_default_00030_e01_000031 container_21305_default_00062_e01_000063 dlhpcshare-agent-60 80 8 container_21305_default_00089_e01_000090 container_21305_default_00004_e01_000005 container_21305_default_00063_e01_000064 container_21305_default_00065_e01_000066 container_21305_default_00080_e01_000081 container_21305_default_00056_e01_000057 container_21305_default_00060_e01_000061 container_21305_default_00054_e01_000055 dlhpcshare-agent-8 60 6 container_21305_default_00091_e01_000092 container_21305_default_00024_e01_000025 container_21305_default_00040_e01_000041 container_21305_default_00038_e01_000039 container_21305_default_00047_e01_000048 container_21305_default_00081_e01_000082 dlhpcshare-agent-6 10 1 container_21305_default_00019_e01_000020 dlhpcshare-agent-2 10 1 container_21305_default_00015_e01_000016 dlhpcshare-agent-14 50 5 container_21305_default_00031_e01_000032 container_21305_default_00053_e01_000054 container_21305_default_00093_e01_000094 container_21305_default_00059_e01_000060 container_21305_default_00069_e01_000070 dlhpcshare-agent-36 50 5 container_21305_default_00006_e01_000007 container_21305_default_00085_e01_000086 container_21305_default_00039_e01_000040 container_21305_default_00045_e01_000046 container_21305_default_00021_e01_000022 dlhpcshare-agent-59 30 3 container_21305_default_00036_e01_000037 container_21305_default_00073_e01_000074 container_21305_default_00074_e01_000075 dlhpcshare-agent-37 50 5 container_21305_default_00011_e01_000012 container_21305_default_00018_e01_000019 container_21305_default_00025_e01_000026 container_21305_default_00050_e01_000051 container_21305_default_00005_e01_000006 dlhpcshare-agent-15 60 6 container_21305_default_00037_e01_000038 container_21305_default_00095_e01_000096 container_21305_default_00096_e01_000097 container_21305_default_00048_e01_000049 container_21305_default_00016_e01_000017 container_21305_default_00003_e01_000004 dlhpcshare-agent-17 20 2 container_21305_default_00077_e01_000078 container_21305_default_00094_e01_000095 dlhpcshare-agent-54 50 5 container_21305_default_00068_e01_000069 container_21305_default_00043_e01_000044 container_21305_default_00034_e01_000035 container_21305_default_00026_e01_000027 container_21305_default_00049_e01_000050 dlhpcshare-agent-12 60 6 container_21305_default_00082_e01_000083 container_21305_default_00057_e01_000058 container_21305_default_00058_e01_000059 container_21305_default_00017_e01_000018 container_21305_default_00071_e01_000072 container_21305_default_00008_e01_000009 dlhpcshare-agent-34 100 10 container_21305_default_00012_e01_000013 container_21305_default_00007_e01_000008 container_21305_default_00088_e01_000089 container_21305_default_00035_e01_000036 container_21305_default_00075_e01_000076 container_21305_default_00027_e01_000028 container_21305_default_00001_e01_000002 container_21305_default_00042_e01_000043 container_21305_default_00044_e01_000045 container_21305_default_00032_e01_000033 dlhpcshare-agent-57 10 1 container_21305_default_00028_e01_000029 dlhpcshare-agent-13 20 2 container_21305_default_00052_e01_000053 container_21305_default_00010_e01_000011 dlhpcshare-agent-53 30 3 container_21305_default_00067_e01_000068 container_21305_default_00014_e01_000015 container_21305_default_00078_e01_000079 openmpi hostfile /tmp/hostfile.1610738 generated: ----------------------- dlhpcshare-agent-28 slots=4 dlhpcshare-agent-25 slots=4 dlhpcshare-agent-49 slots=2 dlhpcshare-agent-21 slots=1 dlhpcshare-agent-44 slots=3 dlhpcshare-agent-46 slots=11 dlhpcshare-agent-61 slots=1 dlhpcshare-agent-40 slots=1 dlhpcshare-agent-42 slots=1 dlhpcshare-agent-20 slots=3 dlhpcshare-agent-60 slots=8 dlhpcshare-agent-8 slots=6 dlhpcshare-agent-6 slots=1 dlhpcshare-agent-2 slots=1 dlhpcshare-agent-14 slots=5 dlhpcshare-agent-36 slots=5 dlhpcshare-agent-59 slots=3 dlhpcshare-agent-37 slots=5 dlhpcshare-agent-15 slots=6 dlhpcshare-agent-17 slots=2 dlhpcshare-agent-54 slots=5 dlhpcshare-agent-12 slots=6 dlhpcshare-agent-34 slots=10 dlhpcshare-agent-57 slots=1 dlhpcshare-agent-13 slots=2 dlhpcshare-agent-53 slots=3 ----------------------- Total tasks is 100 mpirun -hostfile /tmp/hostfile.1610738 -n 100 <your application>
运行结果：
my rank is 0/100, and node_name: dlhpcshare-agent-28 Ireceived:[99] 128 1000001548.0369567 my rank is 1/100, and node_name: dlhpcshare-agent-28 Ireceived:[0] 128 1000044246.7773606 my rank is 2/100, and node_name: dlhpcshare-agent-28 Ireceived:[1] 128 999962097.0663284 my rank is 3/100, and node_name: dlhpcshare-agent-28 Ireceived:[2] 128 1000016400.6064295 my rank is 4/100, and node_name: dlhpcshare-agent-25 Ireceived:[3] 128 1000008126.1840497 my rank is 5/100, and node_name: dlhpcshare-agent-25 Ireceived:[4] 128 999961675.040424 my rank is 6/100, and node_name: dlhpcshare-agent-25 Ireceived:[5] 128 1000021516.2687825 my rank is 7/100, and node_name: dlhpcshare-agent-25 Ireceived:[6] 128 1000044540.8409615 my rank is 8/100, and node_name: dlhpcshare-agent-49 Ireceived:[7] 128 999952778.3406386 my rank is 9/100, and node_name: dlhpcshare-agent-49 Ireceived:[8] 128 1000005490.1392791 my rank is 10/100, and node_name: dlhpcshare-agent-21 Ireceived:[9] 128 999986083.2240833 my rank is 11/100, and node_name: dlhpcshare-agent-44 Ireceived:[10] 128 999979176.7722001 my rank is 12/100, and node_name: dlhpcshare-agent-44 Ireceived:[11] 128 1000024127.996607 my rank is 13/100, and node_name: dlhpcshare-agent-44 Ireceived:[12] 128 999986125.1458977 my rank is 14/100, and node_name: dlhpcshare-agent-46 Ireceived:[13] 128 999968627.061783 my rank is 15/100, and node_name: dlhpcshare-agent-46 Ireceived:[14] 128 999972671.8341721 my rank is 16/100, and node_name: dlhpcshare-agent-46 Ireceived:[15] 128 999956437.583436 my rank is 17/100, and node_name: dlhpcshare-agent-46 Ireceived:[16] 128 999998544.5230424 my rank is 18/100, and node_name: dlhpcshare-agent-46 Ireceived:[17] 128 1000062838.9180896 my rank is 19/100, and node_name: dlhpcshare-agent-46 Ireceived:[18] 128 999977571.1755748 my rank is 20/100, and node_name: dlhpcshare-agent-46 Ireceived:[19] 128 1000026783.9051983 my rank is 21/100, and node_name: dlhpcshare-agent-46 Ireceived:[20] 128 999944040.7103922 my rank is 22/100, and node_name: dlhpcshare-agent-46 Ireceived:[21] 128 1000032367.5440634 my rank is 23/100, and node_name: dlhpcshare-agent-46 Ireceived:[22] 128 1000000648.6932456 my rank is 24/100, and node_name: dlhpcshare-agent-46 Ireceived:[23] 128 1000000311.0008336 my rank is 25/100, and node_name: dlhpcshare-agent-61 Ireceived:[24] 128 1000081391.8491123 my rank is 26/100, and node_name: dlhpcshare-agent-40 Ireceived:[25] 128 999958520.3115258 my rank is 27/100, and node_name: dlhpcshare-agent-42 Ireceived:[26] 128 1000029731.7126155 my rank is 28/100, and node_name: dlhpcshare-agent-20 Ireceived:[27] 128 999989692.1419857 my rank is 29/100, and node_name: dlhpcshare-agent-20 Ireceived:[28] 128 1000007198.1436479 my rank is 30/100, and node_name: dlhpcshare-agent-20 Ireceived:[29] 128 1000033574.3678787 my rank is 31/100, and node_name: dlhpcshare-agent-60 Ireceived:[30] 128 999947556.4409302 my rank is 32/100, and node_name: dlhpcshare-agent-60 Ireceived:[31] 128 1000064412.7062727 my rank is 33/100, and node_name: dlhpcshare-agent-60 Ireceived:[32] 128 1000007761.7041193 my rank is 34/100, and node_name: dlhpcshare-agent-60 Ireceived:[33] 128 999925453.4237553 my rank is 35/100, and node_name: dlhpcshare-agent-60 Ireceived:[34] 128 1000017841.1207856 my rank is 36/100, and node_name: dlhpcshare-agent-60 Ireceived:[35] 128 999973881.0131118 my rank is 37/100, and node_name: dlhpcshare-agent-60 Ireceived:[36] 128 999970098.9094045 my rank is 38/100, and node_name: dlhpcshare-agent-60 Ireceived:[37] 128 999952826.1036501 my rank is 39/100, and node_name: dlhpcshare-agent-8 Ireceived:[38] 128 1000046255.7144414 my rank is 40/100, and node_name: dlhpcshare-agent-8 Ireceived:[39] 128 1000000229.5421265 my rank is 41/100, and node_name: dlhpcshare-agent-8 Ireceived:[40] 128 1000022290.0252079 my rank is 42/100, and node_name: dlhpcshare-agent-8 Ireceived:[41] 128 1000011328.9696794 my rank is 43/100, and node_name: dlhpcshare-agent-8 Ireceived:[42] 128 999943992.9039626 my rank is 44/100, and node_name: dlhpcshare-agent-8 Ireceived:[43] 128 1000026325.9412429 my rank is 45/100, and node_name: dlhpcshare-agent-6 Ireceived:[44] 128 999996594.9427018 my rank is 46/100, and node_name: dlhpcshare-agent-2 Ireceived:[45] 128 1000041001.7400087 my rank is 47/100, and node_name: dlhpcshare-agent-14 Ireceived:[46] 128 1000019117.6120685 my rank is 48/100, and node_name: dlhpcshare-agent-14 Ireceived:[47] 128 1000004723.1141306 my rank is 49/100, and node_name: dlhpcshare-agent-14 Ireceived:[48] 128 1000000249.2617867 my rank is 50/100, and node_name: dlhpcshare-agent-14 Ireceived:[49] 128 999937020.4400109 my rank is 51/100, and node_name: dlhpcshare-agent-14 Ireceived:[50] 128 999991492.5603293 my rank is 52/100, and node_name: dlhpcshare-agent-36 Ireceived:[51] 128 999966666.7490039 my rank is 53/100, and node_name: dlhpcshare-agent-36 Ireceived:[52] 128 999923990.1804005 my rank is 54/100, and node_name: dlhpcshare-agent-36 Ireceived:[53] 128 999961214.0865831 my rank is 55/100, and node_name: dlhpcshare-agent-36 Ireceived:[54] 128 999962973.8227273 my rank is 56/100, and node_name: dlhpcshare-agent-36 Ireceived:[55] 128 1000033223.1139216 my rank is 57/100, and node_name: dlhpcshare-agent-59 Ireceived:[56] 128 999994139.1621871 my rank is 58/100, and node_name: dlhpcshare-agent-59 Ireceived:[57] 128 999996580.0694407 my rank is 59/100, and node_name: dlhpcshare-agent-59 Ireceived:[58] 128 999932381.9617747 my rank is 60/100, and node_name: dlhpcshare-agent-37 Ireceived:[59] 128 999952736.7008783 my rank is 61/100, and node_name: dlhpcshare-agent-37 Ireceived:[60] 128 999961251.6714846 my rank is 62/100, and node_name: dlhpcshare-agent-37 Ireceived:[61] 128 999999478.042908 my rank is 63/100, and node_name: dlhpcshare-agent-37 Ireceived:[62] 128 999979645.2011644 my rank is 64/100, and node_name: dlhpcshare-agent-37 Ireceived:[63] 128 999990214.933797 my rank is 65/100, and node_name: dlhpcshare-agent-15 Ireceived:[64] 128 999921783.2922219 my rank is 66/100, and node_name: dlhpcshare-agent-15 Ireceived:[65] 128 999930730.5282868 my rank is 67/100, and node_name: dlhpcshare-agent-15 Ireceived:[66] 128 999998188.9918685 my rank is 68/100, and node_name: dlhpcshare-agent-15 Ireceived:[67] 128 999967715.6977485 my rank is 69/100, and node_name: dlhpcshare-agent-15 Ireceived:[68] 128 1000079199.5909462 my rank is 70/100, and node_name: dlhpcshare-agent-15 Ireceived:[69] 128 999994935.3312064 my rank is 71/100, and node_name: dlhpcshare-agent-17 Ireceived:[70] 128 999971604.2596891 my rank is 72/100, and node_name: dlhpcshare-agent-17 Ireceived:[71] 128 1000072224.268106 my rank is 73/100, and node_name: dlhpcshare-agent-54 Ireceived:[72] 128 999990133.4711875 my rank is 74/100, and node_name: dlhpcshare-agent-54 Ireceived:[73] 128 999978027.9042766 my rank is 75/100, and node_name: dlhpcshare-agent-54 Ireceived:[74] 128 999990773.9722083 my rank is 76/100, and node_name: dlhpcshare-agent-54 Ireceived:[75] 128 999985109.0805329 my rank is 77/100, and node_name: dlhpcshare-agent-54 Ireceived:[76] 128 1000040580.0881176 my rank is 78/100, and node_name: dlhpcshare-agent-12 Ireceived:[77] 128 999952277.021426 my rank is 79/100, and node_name: dlhpcshare-agent-12 Ireceived:[78] 128 1000006113.7835764 my rank is 80/100, and node_name: dlhpcshare-agent-12 Ireceived:[79] 128 1000016257.5420581 my rank is 81/100, and node_name: dlhpcshare-agent-12 Ireceived:[80] 128 1000009820.5689441 my rank is 82/100, and node_name: dlhpcshare-agent-12 Ireceived:[81] 128 999948478.4103069 my rank is 83/100, and node_name: dlhpcshare-agent-12 Ireceived:[82] 128 999974697.9367532 my rank is 84/100, and node_name: dlhpcshare-agent-34 Ireceived:[83] 128 999979055.5054404 my rank is 85/100, and node_name: dlhpcshare-agent-34 Ireceived:[84] 128 1000041499.0674162 my rank is 86/100, and node_name: dlhpcshare-agent-34 Ireceived:[85] 128 1000032224.6438636 my rank is 87/100, and node_name: dlhpcshare-agent-34 Ireceived:[86] 128 999951417.6075771 my rank is 88/100, and node_name: dlhpcshare-agent-34 Ireceived:[87] 128 1000032051.0978342 my rank is 89/100, and node_name: dlhpcshare-agent-34 Ireceived:[88] 128 1000000387.5620804 my rank is 90/100, and node_name: dlhpcshare-agent-34 Ireceived:[89] 128 999916308.7998385 my rank is 91/100, and node_name: dlhpcshare-agent-34 Ireceived:[90] 128 999932974.9200188 my rank is 92/100, and node_name: dlhpcshare-agent-34 Ireceived:[91] 128 1000075592.2132264 my rank is 93/100, and node_name: dlhpcshare-agent-34 Ireceived:[92] 128 1000026815.8564372 my rank is 94/100, and node_name: dlhpcshare-agent-57 Ireceived:[93] 128 1000013768.6777565 my rank is 95/100, and node_name: dlhpcshare-agent-13 Ireceived:[94] 128 1000045400.8612764 my rank is 96/100, and node_name: dlhpcshare-agent-13 Ireceived:[95] 128 999995092.0445201 my rank is 97/100, and node_name: dlhpcshare-agent-53 Ireceived:[96] 128 999993829.6496035 my rank is 98/100, and node_name: dlhpcshare-agent-53 Ireceived:[97] 128 1000046106.0476944 my rank is 99/100, and node_name: dlhpcshare-agent-53 Ireceived:[98] 128 999911786.4949732
可以看到，修改后的HPC上MPI代码可以成功利用所有的调度器分配的主机与CPU核心。
PS:
需要注意，本文的修改主要目的是解决官方的配置文件的错误问题，也就是解决单job中的每个task不能使用多个CPU核心的问题，当然传统的MPI设计中单task往往都是使用单个CPU的，但是由于现在的计算Job往往很复杂，如AI方向的，我们已经难以满足单Job下的各子task只使用单个CPU的设计，使用本文给出的更正后的配置可以实现单task下使用多CPU核心的需求，然后我们可以在单task中通过使用from multiprocessing import Process, Queue的方式来实现多个CPU核心的利用，以此实现更高性能的需求。
============================================
本博客是博主个人学习时的一些记录，不保证是为原创，个别文章加入了转载的源地址，还有个别文章是汇总网上多份资料所成，在这之中也必有疏漏未加标注处，如有侵权请与博主联系。
如果未特殊标注则为原创，遵循 CC 4.0 BY-SA 版权协议。
posted on 2023-08-25 09:25 Angry_Panda 阅读(122) 评论(0) 收藏 举报
浙公网安备 33010602011771号