Parallel NetCDF 简介
Parallel NetCDF API
- 所有C接口前加
ncmpi前缀，Fortran接口前加nfmpi前缀 - 函数返回整数 NetCDF 状态变量
1. Variable and Parameter Types
函数采用MPI_Offset类型来表示大小参数，与size_t相比（32-bit）MPI_Offset为64位变量，表示数据几乎不受限制。
有关变量起始下标编号start，各个维度长度count，及间隔大小stride等标量或向量都需定义为MPI_Offset类型。
2. Dataset Functions
ncmpi_create与ncmpi_open函数多了一个附加参数MPI_Info，这个参数主要用于传递提示变量。调用时传递MPI_INFO_NULL则可以忽略此功能。
int ncmpi_create(MPI_Comm comm,
const char *path,
int cmode,
MPI_Info info,
int *ncidp)
int ncmpi_open(MPI_Comm comm,
const char *path,
int omode,
MPI_Info info,
int *ncidp)
3. Define Mode Functions
所有进程必须采用相同值调用这类函数，在定义结束后，所有进程定义内容会进行检查与比较。若其不相同，函数ncmpi_enddef会返回错误代码。
4. Inquiry Functions
Inquiry函数可以在定义模式（define mode）或数据模式（data mode）下被调用。
5. Attribute Functions
Attributes（属性）主要在NetCDF中储存标量或是向量来描述变量。
在原始接口中，attribute函数可以在定义模式或数据模式下调用；然而，在数据模式状态下修改attributes的值有可能会失败。主要由于文件所需空间可能会改变。
6. Data Mode Functions
数据模式（data mode）可分为两个状态：总体模式（collective mode）与独立模式（independent mode）。当用户调用ncmpi_enddef或ncmpi_open后，文件自动进入总体模式。
在总体模式内，所有进程必须在代码相同位置调用相同的函数。调用参数如 start，count，stride 等则可以不同；在独立模式内，进程不必共同调用API。
在定义状态（define mode）下不能进入独立模式，需要首先调用ncmpi_enddef来离开定义状态随后进入数据模式。
数据模式函数分为两类。第一类模仿传统的NetCDF函数并且将其简单的又传统NetCDF接口迁移成为并行NetCDF函数接口。我们称这类数据接口为高级数据模式接口(high level data mode interface)。
第二个类函数使用更多的MPI功能来提供更好的处理内部数据，并且更充分地展示MPI-IO处理应用程序的能力。所有的第一类函数将按照这类函数实现。我们这类称为灵活数据模式接口（flexible data mode interface）。
在两类函数中，都提供了包括独立模式与总体模式操作。总体模式函数名后以_all结尾。所有这些进程必须同时调用该函数。
6.1. High Level Data Mode Interface
每个独立函数都类似于NetCDF数据模式接口。主要变化就是使用MPI_Offset代替size_t类型数据。
ncmpi_put_var_<type>将变量所有值写入Netcdf文件；ncmpi_put_vara_<type>写入数据部分由start向量指定起始位置，count指定各维度长度；ncmpi_put_vars_<type>写入数据部分由start向量指定起始位置，count指定各维度长度，stride指定各维度间隔；ncmpi_put_varm_<type>
6.2. Flexible Data Mode Interface
6.3. Mapping Between NetCDF and MPI Types
7. Q & A
For more details, please refer to Parallel netCDF Q&A
Q: How do I use the buffered nonblocking write APIs?
A: Buffered nonblocking write APIs copy the contents of user buffers into an internally allocated buffer, so the user buffers can be reused immediately after the calls return. A typical way to use these APIs is described below.
- First, tell PnetCDF how much space can be allocated to be used by the APIs.
- Make calls to the buffered put APIs.
- Make calls to the (collective) wait APIs.
- Free the space allocated by the internal buffer.
For further information about the buffered nonblocking APIs, readers are referred to this page.
Q: What is the difference between collective and independent APIs?
A: Collective APIs requires all MPI processes to participate the call. This requirement allows MPI-IO and PnetCDF to coordinate the I/O requesting processes to rearrange requests into a form that can achieve the best performance from the underlying file system. On the contrary, independent APIs (also referred as non-collective) has no such requirement. All PnetCDF collective APIs (except create, open, and close) have a suffix of _all, corresponding to their independent counterparts. To switch from collective data mode to independent mode, users must call ncmpi_begin_indep_data. API ncmpi_begin_indep_data is to exit the independent mode.
Q: Should I use collective APIs or independent APIs?
A: Users are encouraged to use collective APIs whenever possible. Collective API calls require the participation of all MPI processes that open the shared file. This requirement allows MPI-IO and PnetCDF to coordinate the I/O requesting processes to rearrange requests into a form that can achieve the best performance from the underlying file system. If the nature of user's I/O does not permit to call collective APIs (such as the number of requests are not equal among processes, or is determined at the run time), then we recommend the followings.
- Force all the processes participate the collective calls. When a process has nothing to request, users can still call a collective API with zero-length request. This is achieved by set the contents of argument count to zero.
- Use nonblocking APIs. Individual processes can make any number of calls to nonblocking APIs independently from other processes. At the end, a collective wait API,
ncmpi_wait_all, is recommended to used to allow all nonblocking requests to commit to the file system.
总结：推荐使用集合接口（collective APIs），不适用也尽量使。
8. Example
/*********************************************************************
*
* Copyright (C) 2012, Northwestern University and Argonne National Laboratory
* See COPYRIGHT notice in top-level directory.
*
*********************************************************************/
/* $Id$ */
/* simple demonstration of pnetcdf
* text attribute on dataset
* write out rank into 1-d array collectively.
* The most basic way to do parallel i/o with pnetcdf */
/* This program creates a file, say named output.nc, with the following
contents, shown by running ncmpidump command .
% mpiexec -n 4 pnetcdf-write-standard /orangefs/wkliao/output.nc
% ncmpidump /orangefs/wkliao/output.nc
netcdf output {
// file format: CDF-2 (large file)
dimensions:
d1 = 4 ;
time = UNLIMITED ; // (2 currently)
variables:
int v1(time, d1) ;
int v2(d1) ;
// global attributes:
:string = "Hello World\n",
"" ;
data:
v1 =
0, 1, 2, 3,
1, 2, 3, 4 ;
v2 = 0, 1, 2, 3 ;
}
*/
#include <stdlib.h>
#include <mpi.h>
#include <pnetcdf.h>
#include <stdio.h>
static void handle_error(int status, int lineno)
{
fprintf(stderr, "Error at line %d: %s\n", lineno, ncmpi_strerror(status));
MPI_Abort(MPI_COMM_WORLD, 1);
}
int main(int argc, char **argv) {
int ret, ncfile, nprocs, rank, dimid1, dimid2, varid1, varid2, ndims;
MPI_Offset start, count=1;
int t, i;
int v1_dimid[2];
MPI_Offset v1_start[2], v1_count[2];
int v1_data[4];
char buf[13] = "Hello World\n";
int data;
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &nprocs);
if (argc != 2) {
if (rank == 0) printf("Usage: %s filename\n", argv[0]);
MPI_Finalize();
exit(-1);
}
ret = ncmpi_create(MPI_COMM_WORLD, argv[1],
NC_CLOBBER, MPI_INFO_NULL, &ncfile);
if (ret != NC_NOERR) handle_error(ret, __LINE__);
ret = ncmpi_def_dim(ncfile, "d1", nprocs, &dimid1);
if (ret != NC_NOERR) handle_error(ret, __LINE__);
ret = ncmpi_def_dim(ncfile, "time", NC_UNLIMITED, &dimid2);
if (ret != NC_NOERR) handle_error(ret, __LINE__);
v1_dimid[0] = dimid2;
v1_dimid[1] = dimid1;
ndims = 2;
ret = ncmpi_def_var(ncfile, "v1", NC_INT, ndims, v1_dimid, &varid1);
if (ret != NC_NOERR) handle_error(ret, __LINE__);
ndims = 1;
ret = ncmpi_def_var(ncfile, "v2", NC_INT, ndims, &dimid1, &varid2);
if (ret != NC_NOERR) handle_error(ret, __LINE__);
ret = ncmpi_put_att_text(ncfile, NC_GLOBAL, "string", 13, buf);
if (ret != NC_NOERR) handle_error(ret, __LINE__);
/* all processors defined the dimensions, attributes, and variables,
* but here in ncmpi_enddef is the one place where metadata I/O
* happens. Behind the scenes, rank 0 takes the information and writes
* the netcdf header. All processes communicate to ensure they have
* the same (cached) view of the dataset */
ret = ncmpi_enddef(ncfile);
if (ret != NC_NOERR) handle_error(ret, __LINE__);
start=rank, count=1, data=rank;
ret = ncmpi_put_vara_int_all(ncfile, varid2, &start, &count, &data);
if (ret != NC_NOERR) handle_error(ret, __LINE__);
for (t = 0; t<2; t++){
v1_start[0] = t, v1_start[1] = rank;
v1_count[0] = 1, v1_count[1] = 1;
for (i = 0; i<4; i++){
v1_data[i] = rank+t;
}
/* in this simple example every process writes its rank to two 1d variables */
ret = ncmpi_put_vara_int_all(ncfile, varid1, v1_start, v1_count, v1_data);
if (ret != NC_NOERR) handle_error(ret, __LINE__);
}
ret = ncmpi_close(ncfile);
if (ret != NC_NOERR) handle_error(ret, __LINE__);
MPI_Finalize();
return 0;
}
