【CaRT】Collective and RPC Transport

什么是CaRT

简单的说，CaRT就是支持点对点或向集群发RPC调用的通信库。

官方说明：

Collective and RPC Transport (CaRT)：集体和 RPC 传输 (CaRT)

⚠️ 警告：CaRT 正在大力开发中。 使用风险自负。

CaRT 是用于大数据和 Exascale HPC 的开源 RPC 传输层。 它支持传统的 P2P RPC和集体 RPC，后者通过可扩展的基于树的消息传播在一组目标服务器上调用 RPC。

Gurt Useful Routines and Types (GURT)

GURT 是一个辅助函数和数据类型的开源库。 该库使得操作列表、哈希表、堆和日志记录变得容易。

所有 Gurt 有用的例程和类型都以“d”为前缀，这是字母表中的第 4 个字母，因为 gurt 有用的单词有 4 个字母。

License

CaRT is open source software distributed under a BSD license. Please see the LICENSE & NOTICE files for more information.

​​​
CaRT 是在 BSD 许可下分发的开源软件。 请参阅

许可证（https://github.com/liangchaoxi/daos/blob/master/src/cart/LICENSE）和

通知文件（https://github.com/liangchaoxi/daos/blob/master/src/cart/NOTICE）

了解更多信息。

​

Build

CaRT 需要支持 C99 的编译器和 scons 构建工具来构建。

CaRT的是基于点对点（P2P) RPC的库Mercury：

CaRT 依赖于一些第三方库：

      【RPC】水星Mercury RPC_progress loop-CSDN博客

      ompi 需要用外部 PMIx/hwloc 编译(一个配置例子 "./configure --with-pmix=/your_pmix_install_path / --with-hwloc=/your_hwloc_install_path --with-libevent=external").

安装完所有依赖模块后，可以在最上层目录中执行“scons”来构建它。

故障注入

daos/src/cart/README.swifi

软件实现的故障注入 (SWIFI)

注入故障：
1）定义属性结构：例如 my_attr
2) 根据 fault-inject-cart.yaml 的初始化属性结构
读取配置文件，根据 id 选择正确的行。
3）在您要注入故障的地方注释您的代码。
例如，你想让一个 malloc 失败，你会这样做：

/*
* d_inject_fault 是一个全局变量，用于打开/关闭故障注入。 它
* 在我看来，使用虚拟函数表之类的方法不提供
*任何优势。 if 测试更快，而且我们不需要拆分代码
* 出一个新功能。
*/

      if (SHOULD_FAIL(fault_id)) {
        return NULL;
      } else {
        return malloc(32);
      }

4) 使用开关变量 d_inject_fault 全局开启/关闭故障注入。

环境变量 

daos/src/cart/README.env

This file lists the environment variables used in CaRT.

 . CRT_PHY_ADDR_STR
   It determines which mercury NA plugin to be used:
   1: set it as "ofi+verbs;ofi_rxm" to use OFI verbs;ofi_rxm provider
   2: set it as "ofi+psm2" to use OFI psm2 provider
   3: set it as "ofi+gni" to use OFI gni provider
   4: set it as "sm" to use SM plugin which only works within single node
   5: set it as "ofi+tcp;ofi_rxm" to use OFI tcp;ofi_rxm provider.
   6: set it as "ofi+sockets" to use OFI sockets provider
      NOTE: This provider is deprecated in favor of "ofi+tcp;ofi_rxm"
   7: by default (not set or set as any other value) it will use ofi tcp
      provider.

 . D_LOG_FILE
   Set it as a file path (for example "/tmp/crt.xxx.log") to make GURT log debug
   messages to that file. Or will log to stdout.

 . D_LOG_STDERR_IN_LOG
   For debugging purpose, it will allow to redirect stderr messages in log file.
   Thus, messages from external libs, like libfabric, will be merged in log
   in correct sequence.

 . D_LOG_FILE_APPEND_PID
   Set this to a non-zero value to create separate log files based upon the
   process id (pid). The pid number will be appended to the log file specified
   in D_LOG_FILE. If not defined or setting to zero value will result in this
   feature being disabled.

 . D_LOG_SIZE
   DAOS debug logs (both server and client) have a 1GB file size limit by
   default. When this limit is reached, the current log file is closed and
   renamed with a .old suffix, and a new one is opened. This mechanism will
   repeat each time the limit is reached, meaning that available saved log
   records could be found in both ${D_LOG_FILE} and last generation of
   ${D_LOG_FILE}.old files, to a maximum of the most recent 2*D_LOG_SIZE
   records.  This can be modified by setting this environment variable
   ("D_LOG_SIZE=536870912"). Sizes can also be specified in human-readable form
   using `k`, `m`, `g`, `K`, `M`, and `G`. The lower-case specifiers are
   base-10 multipliers and the upper case specifiers are base-2 multipliers.

 . D_LOG_TRUNCATE
   Set this to truncate log file to size=0 before writing to it. Otherwise log
   file will be appended. Log file is specified by D_LOG_FILE envariable.

 . D_LOG_MASK
   Can set different log levels for different subsystem.
   GURT defined several debug subsystem(log facility): MISC, MEM, RPC, CORPC,
   BULK, GRP, LM, HG, etc.
   Can set different log level for different subsystems, for example:
   "export D_LOG_MASK=DEBUG" will set all subsystems' loglevel as DEBUG
   "export D_LOG_MASK=ERR,RPC=DEBUG,BULK=INFO" will set RPC's loglevel as
   DEBUG, BULK's log level as INFO, all other subsystems' loglevel as ERR.
   If neither above 2 or 3 ENVs set, the default loglevel is WARN and log device
   is stdout.

 . DD_MASK
   Can enable different debug streams for finer-grained debugging per subsystem,
   for example: "DD_MASK="trace"" will log only debug messages with the bit mask
   DB_TRACE associated with them. Multiple streams can be set as well, for
   example: "DD_MASK="io,mem"" will allow logging of a subset of debug messages
   related to io path (bit mask=DB_IO) and memory operations (bit mask=DB_MEM).
   Since these are debug masks, the subsystem/facility must have the debug log
   set ("D_LOG_MASK=DEBUG"), otherwise other priorities take precedence. The bit
   streams are passed as a parameter to D_DEBUG(mask, fmt, ...).

 . D_LOG_FLUSH
   Specifies priority level for flushing of messages into the log file.
   Can be set to one of the following "DBUG,INFO,NOTE,WARN,ERR,CRIT,ALRT,EMRG,EMIT"
   If not set will default to "WARN".

 . DD_STDERR
   User can specify the priority level to output to stderr, for example:
   "DD_STDERR="info"" will log all DLOG_INFO priority messages to stderr.
   The default for cart is DLOG_EMERG ("fatal"), and for daos is DLOG_CRIT
   ("critical"). Options include: "info/note/warn(ing)/err(or)/crit(ical)/fatal"

 . DD_SUBSYS
   User can specify which subsystems to enable. Gurt default subsystems which
   are always enabled are MEM, MISC, and CLOG. All CaRT facilities are enabled
   by default, and can be specified using DD_SUBSYS, for example:
   "DD_SUBSYS=RPC,BULK,CORPC,GRP,LM,HG,PMIX,ST,IV" or also "DD_SUBSYS=all".

 . CRT_TIMEOUT
   Set it as integer in the range of (0, 3600] to set the global timeout value
   of all RPCs (second). Without setting it or set it as any other value will
   take the default timeout value of 60 second.

.  CRT_ATTACH_INFO_PATH
   Set this environment variable in order to specify a custom prefix path for
   '.attach_info_tmp' file generated.
   When set to the "path" the file generated will have a form of:
       <path>/<cart group name>.attach_info_tmp
   When not set the file generated will have a form of:
      /tmp/<cart group name>.attach_info_tmp
   Calling crt_group_config_path_set() will overwrite the prefix path setting by
   CRT_ATTACH_INFO_PATH, however this ENV setting provides a method to control
   the prefix path without calling API which needs to change code.

 . OFI_PORT
   Set it as a port number, when creating NA classes it will use the consecutive
   port numbers start from it.
   When it is not set, then will use random port number.

 . OFI_INTERFACE
   Set it as the network device name to be used for OFI communication, for
   example "eth0", "ib0" or "ens33" etc.

 . OFI_DOMAIN
   Set it as the network domain name to be used for OFI communications. Domain
   name can be determined by running 'fi_info' app. If not set, OFI_DOMAIN is
   assumed to be the same as OFI_INTERFACE.

 . CRT_CREDIT_EP_CTX
   Set it as the max number of inflight RPCs to a target endpoint context, the
   valid range is [0, 256].
   If it is not set then will use the default value of 32.
   It its value exceed 256, then will use 256 for flow control.
   Set it to zero means disable the flow control in cart.

 . CRT_CTX_SHARE_ADDR
   Set it to non-zero to make all the contexts share one network address, in
   this case CaRT will create one SEP and each context maps to one tx/rx
   context pair.
   When the ENV not set or set to 0 each context will create one separate SEP.

 . CRT_CTX_NUM
   Set it to the number of contexts user wants to create. It is meaningful only
   when CRT_CTX_SHARE_ADDR is set to non-zero. If CRT_CTX_NUM exceeds the OFI
   provider capability, NA layer will fail to initialize. If user creates more
   contexts than CRT_CTX_NUM, context creation will fail.

 . D_FI_CONFIG
   Specifies the fault injection configuration file. If this variable is not set
   or set to empty, fault injection is disabled.

 . CRT_ENABLE_MEM_PIN
   Enables memory pinning for CART-890 workaround

 . CRT_MRC_ENABLE
   When not set automatically disables MR caching via FI_MR_CACHE_MAX_COUNT=0
   envariable setting. Set to non 0 to re-enable MR caching in the provider.

 . CRT_TEST_CONT
   When set to 1, orterun does not automatically shut down other servers when
   one server is shutdown. Used in cart internal testing.

posted on 2022-09-12 16:23  bdy  阅读(4)  评论(0)    收藏  举报  来源

导航