China AZURE Infniband Powered GPU VM

        最近 China Azure 上刚刚发布了 NCv3 系列的 GPU 虚拟主机,细心的小伙伴可能注意到,NCv3 中有一款型号为 NC24rs 的虚拟主机,这个字母 r 来的些许蹊跷,代表什么意思呢?通过查阅官网对于该主机的介绍说明可见,原来 r 代表 RDMA,表示该虚拟主机支持 infiniband 来加持的 RDMA 网络。infiniband 可以帮计算节点获得高带宽低延迟的加持,在集群计算场景下可以大大提成效率。今天我们一起来看一下 infiniband 加持后节点通信性能几何。

        我们准备两台 NC24rsv3 的虚拟主机,为保证两台主机可以通过 infiniband 进行互通,我们需要讲两台虚拟主机部署在同一可用性集中(availability set)。部署完成后,登陆虚拟主机安装 RDMA 相关组件,本文以 ubuntu 16.04 LTS 为例,操作步骤如下:

1. 安装 RDMA 相关组件

sudo apt-get update
sudo apt-get install libdapl2 libmlx4-1

2. 使能 RDMA

通过 root 权限修改 /etc/waagent.conf  文件

OS.EnableRDMA=y

OS.UpdateRdmaDriver=y

3. 打开内存限制

通过 root 权限修改 /etc/security/limits.conf

* hard memlock unlimited
* soft memlock unlimited

4. 重启主机生效配置

sudo reboot

5. 安装 Intel MPI 驱动

下载驱动

wget http://registrationcenter-download.intel.com/akdlm/irc_nas/tec/9278/l_mpi_p_5.1.3.223.tgz

解压缩驱动,进入驱动压缩包所在目录

tar -xzf l_mpi_p_5.1.3.223.tgz

进入解压的安装目录,进行安装

cd l_mpi_p_5.1.3.223/
./install.sh

选择 1, 并按照提示输入 root 密码

Please make your selection by entering an option.
Root access is recommended for evaluation.

1. Run as a root for system wide access for all users [default]
2. Run using sudo privileges and password for system wide access for all users
3. Run as current user to limit access to user level

h. Help
q. Quit

确认,并接受 EULA

Step 1 of 6 | Welcome
--------------------------------------------------------------------------------
Welcome to the Intel(R) MPI Library 5.1 Update 3 for Linux* setup program.


--------------------------------------------------------------------------------
You will complete the steps below during setup process:
Step 1 : Welcome
Step 2 : License agreement
Step 3 : Activation
Step 4 : Options
Step 5 : Installation
Step 6 : Complete

--------------------------------------------------------------------------------
Press "Enter" key to continue or "q" to quit:
Step 2 of 6 | License agreement
To continue with the installation of this product you are required to accept
the terms and conditions of the End User License Agreement (EULA). The EULA
is displayed using the 'more' utility. Press the spacebar to advance to the
next page or enter 'q' to skip to the end. After reading the EULA, you must
enter 'accept' to continue the installation or 'decline' to return to the
previous menu.
--------------------------------------------------------------------------------
IMPORTANT INFORMATION ABOUT YOUR RIGHTS, OBLIGATIONS AND THE USE OF YOUR DATA -
READ AND AGREE BEFORE COPYING, INSTALLING OR USING

This Agreement forms a legally binding contract between you, or the company or
other legal entity ("Legal Entity") for which you represent and warrant that you
have the legal authority to bind that Legal Entity, are agreeing to this
Agreement (each, "You" or "Your") and Intel Corporation and its subsidiaries
(collectively "Intel") regarding Your use of the Materials.  By copying,
installing, distributing, publicly displaying, or otherwise using the Materials,
You agree to be bound by the terms of this Agreement.  If You do not agree to
the terms of this Agreement, do not copy, install, distribute, publicly display,
or use the Materials. You affirm that You are 18 years old or older or, if not,
Your parent, legal guardian or Legal Entity must agree and enter into this
Agreement.

DATA COLLECTION.  The Materials may contain certain features that generate,
collect, and transmit data to Intel about the installation, setup, and use of
the Materials.  The purposes of data collection are: 1) to verify compliance
with the terms of this Agreement; and 2) to enable Intel to develop, improve,
and support Intel's products and services.  When data is collected to verify
compliance with the terms of this Agreement, this collection may be mandatory
and a condition of using the Materials.  This data includes the Material's
unique serial number combined with other information about the Materials and
Your computer.

When Materials are made available for use free of charge, the collection of
usage data (such as randomly generated unique identifier and
component/feature/function usage) may also be mandatory and a condition of using
the Materials.  Data collected about the installation, setup, and use of the
Materials may be collated with other available data only if: 1) the purpose is
to develop, improve, and support Intel's products and services, and 2) the data
will not be used to identify or contact You or other individuals.

To learn more about Intel's data collection for these Materials, please visit:
https://software.intel.com/en-us/articles/data-collection.  To learn more about
Intel's privacy practices, please visit http://www.intel.com/privacy.

Third Party Programs (as defined below), even if included with the distribution
of the Materials, are governed by separate license terms, including without
limitation, third party license terms, other Intel software license terms, and
open source software license terms.  Such separate license terms (and not this
Agreement) solely govern Your use of the Third Party Programs.

1.  LICENSE DEFINITIONS:

   A. "Confidential Information" means all Materials (as defined below),
   including any portions thereof, that are identified (in the product release
   notes, on Intel's download website for the Materials or elsewhere) or labeled
   as Intel confidential information or a similar legend.

   B. "Excluded License" means a license that requires, as a condition of use,
   modification, or distribution, that the licensed software or other software
   incorporated into, derived from or distributed with such software (a) be
   disclosed or distributed in Source Code form; (b) be licensed by the user to
   third parties for the purpose of making and/or distributing derivative works;
   or (c) be redistributable at no charge.  Excluded Licenses include, without
   limitation, licenses that license or distribute software under any of the
   following licenses or distribution models, or licenses or distribution models
   substantially similar to any of the following: (a) GNU's General Public
   License (GPL) or Lesser/Library GPL (LGPL), (b) the Artistic License (e.g.,
   PERL), (c) the Mozilla Public License, (d) the Netscape Public License, (e)
   the Sun Community Source License (SCSL), (f) the Sun Industry Source License
   (SISL), and (g) the Common Public License (CPL).

   C. "Licensed Patent Claims" means the claims of Intel's patents that are
   necessarily and directly infringed by the reproduction and distribution of
   the Materials that is authorized in Section 2 below, when the Materials is in
   its unmodified form as delivered by Intel to You and not modified or combined
   with anything else.  Licensed Patent Claims are only those claims that Intel
   can license without paying, or getting the consent of, a third party.

   D. "Materials" are defined as the software, documentation, the software
   product serial number and license key codes (if applicable), and other
   materials, including any modifications, updates and upgrades thereto, that
   are provided to You under this Agreement.  Materials also include any
   Redistributables, Source Code, and Pre-Release Materials, as defined below
   but do not include Third Party Programs.

   E. "Microsoft Platforms" means any current and future Microsoft operating
   system products, Microsoft run-time technologies (such as the .NET
   Framework), and Microsoft application platforms (such as Microsoft Office or
   Microsoft Dynamics) that Microsoft offers.

   F. "Pre-Release Materials" means the Materials, or portions thereof, that are
   identified (in the product release notes, on Intel's download website for the
   Materials or elsewhere) or labeled as pre-release, and as such the
   Pre-Release Materials are deemed to be pre-release code (e.g., alpha or beta
   release, etc.), which may not be fully functional and which Intel may
   substantially modify in development of a  commercial version, and for which
   Intel makes no assurances that it will ever develop or make generally
   available a commercial version.

   G. "Redistributables" (if any) are the files listed in the following text
   files that may be included in the Materials for the applicable Intel Software
   Development Product: clredist.txt, credist.txt, fredist.txt, redist.txt, and
   redist-rt.txt.

   H. "Sample Source Code" is those portions of the Materials that are Source
   Code files and are identified as sample source code, including without
   limitation, the IPP Sample Source.

   I. "Source Code" is defined as the software (and not documentation or text)
   portion of the Materials provided in human readable format, and includes
   modifications to the Source Code that You make or are made on Your behalf as
   expressly permitted under the terms of this Agreement.

   J. "Third Party Programs" (if any) are the files listed in the
   "third-party-programs.txt" text file that may be included in the Materials
   for the applicable software.

   K. "Your Product" means one or more applications or products developed by or
   for You using the Materials.

2. LICENSE GRANT:

2.1     Subject to the terms and conditions of this Agreement, and timely
payment of any fees (if applicable), Intel grants You a non-exclusive,
worldwide, perpetual (subject to Section 11 below), non-assignable (except as
expressly permitted hereunder), limited right and license:

A. under its copyrights, to:

   (1) reproduce copies of the Materials for Your internal business use in
   accordance with the documentation included as part of the Materials, and
   subject to the applicable license rights and restrictions specified in
   Section 3 below; provided, however, that this license does not include the
   right to sublicense and may only be exercised by You or Your employees;

   (2) use the Materials solely for Your internal business use to develop
   Your Product, in accordance with the applicable license rights and
   restrictions specified in Section 3 below and the documentation or text files
   included as part of the Materials; provided, however, that this license does
   not include the right to sublicense and may only be exercised by You or Your
   employees;

   (3) modify or create derivative works of the Materials, or any portions
   thereof, that are provided in Source Code form, provided, however, that this
   license does not include the right to sublicense and may be exercised only by
   You or Your employees;

   (4) publicly perform, display, and distribute (directly and through Your
   distributors, resellers and other channel partners) or otherwise make
   publicly available the Redistributables, including any modifications to or
   derivative works of the Redistributables made pursuant to Section 2.1.A(3),
   or any portions thereof, subject to the following restrictions:

      (i) any distribution of the Redistributables must only be as part of
      Your Product which must add significantly more functionality than the
      Redistributables themselves;

      (ii) any additional restrictions which may appear in the
      Redistributables text files specified in Section 1.G above and in Section
      3 below; and

      (iii) the license under Section 2.1.A(4) includes the right to
      sublicense the Redistributables, but the sublicense rights are
      limited to sublicensing of any Intel copyrights in the
      Redistributables and only to the extent necessary to perform,
      display, and distribute the Redistributables (including Your
      modifications and derivative works thereto) solely as incorporated
      in Your Product. IF YOU RECEIVED THE MATERIALS FOR EVALUATION, YOU
      HAVE NO RIGHTS TO DISTRIBUTE THE REDISTRIBUTABLES, INCLUDING
      WITHOUT LIMITATION, ANY PORTIONS, MODIFICATIONS OR DERIVATIVE
      WORKS.

      (iv) Distribution of the Redistributables is also subject to the
      following limitations:  You (a) will be solely responsible to Your
      customers for any update, support obligation or other liability
      which may arise from the distribution, (b) will not make any
      statement that Your Product is "certified" or that its performance
      is guaranteed by Intel, (c) will not use Intel's name or
      trademarks to market Your Product without written permission from
      Intel, (d) will provide the Redistributables subject to a license
      agreement that prohibits disassembly and reverse engineering of
      the Redistributables except in cases when you provide Your Product
      subject to an open source license that is not an Excluded License,
      for example, the BSD license, or the MIT license, (e) will
      indemnify, hold harmless, and defend Intel and its suppliers from
      and against any claims or lawsuits, including attorney's fees,
      that arise or result from Your modifications, derivative works or
      Your distribution of Your Product.

and

B. under Intel's Licensed Patent Claims, to:
   (1) make copies of the Materials only as specified in Section 2.1.A(1);

   (2) use the Materials only as specified in Section 2.1.A(2); and

   (3) offer to distribute, and distribute, but not sell, the Redistributables
   only as part of Your Product, under Intel's copyright license granted in
   Section 2.1(A), but only under the terms of that copyright license and not as
   a sale (but this right does not include the right to sub-license);

   (4) provided, further, that the license under the Licensed Patent Claims does
   not and will not apply to any modifications to, or derivative works of, the
   Materials, whether made by You, Your customer (which, for all purposes under
   this Agreement, will mean either a customer, reseller, distributor or other
   channel partner), or any third party even if the modification and derivative
   works are permitted under 2.1(A)(3).

2.2 If the Materials You receive are packaged, as a single orderable item (i.e.,
as a single SKU), with hardware that includes one or more Intel manufactured
microprocessors ("Intel Target Hardware"), then the licenses granted in Section
2.1 above are restricted to the sole purpose of producing and releasing Your
Product to execute on computer systems that include the same or new versions of
the Intel manufactured microprocessor included in the Intel Target Hardware.

Intel expressly does not grant You a patent license in this Agreement to any
modifications or derivative works of the Materials, whether made by You, Your
contractor, Your customer, or any other third party in creating the derivative
works even to the extent creation of derivative works is permitted under Section
2.1(A)(3) above.

3. LICENSE CONDITIONS:

   A. If You are an entity, each of Your employees and Your contractors may use
   the Materials as specified in Section 2 above, provided: (i) their use of the
   Materials is solely on behalf of and in support of Your business, (ii) they
   agree to the terms and conditions of this Agreement, and (iii) You are solely
   responsible for their use of the Materials.

   B. If Your Product is a software development library, then attribution (if
   any), as specified in the product release notes of the corresponding
   Materials shall be displayed prominently in Your Product's associated
   documentation and on the web site (if any) for Your Product.

   C. If You receive Your first copy of the Materials electronically, and a
   second copy on media, then you may use the second copy only in accordance
   with Your applicable license stated in this Agreement, or for backup or
   archival purposes.  You may not provide the second copy to another user.

   D. If the Materials You received are identified as Pre-Release Materials, (i)
   You have the right to use the Pre-Release Materials only for the duration of
   the pre-release term, which is specified in the product release notes, on
   Intel's download website for the Materials or elsewhere, or until the
   commercial release, if any, of the Pre-Release Materials, whichever is
   shorter, and (ii) You may not disclose to any third party any benchmarks,
   performance results, or other information relating to the Pre-Release
   Materials.

   E. Notwithstanding anything to the contrary in this Agreement, if the
   Materials include the text file named "site_license_materials.txt" the files
   specified in that text file may be installed on computer systems located only
   at a single site (unless multiple sites are specified in the purchase order
   accepted by Intel or its resellers), and those files may be accessed or used
   by unlimited and simultaneous users, subject to their compliance with all of
   the terms and conditions of this Agreement.

   F. Except as expressly provided in this Agreement, You may NOT: (i) use,
   copy, distribute, or publicly display the Materials; (ii) rent or lease the
   Materials to any third party; (iii) assign this Agreement or transfer the
   Materials; (iv) modify, adapt, or translate the Materials in whole or in
   part; (v) reverse engineer, decompile, or disassemble the Materials; (vi)
   attempt to modify or tamper with the normal function of any license manager
   that may regulate usage of the Materials; (vii) distribute, sublicense or
   transfer the Source Code form of any components of the Materials or
   derivatives thereof to any third party; (viii) distribute Redistributables
   except as part of a larger program that adds significant primary
   functionality different from that of the Redistributables; (ix) distribute
   the Redistributables to run on a platform other than a Microsoft Platform if
   according to the accompanying user documentation the Materials are meant to
   execute only on a Microsoft Platform; (x) include the Redistributables in
   malicious, deceptive, or unlawful programs or products; or (xi) modify,
   create a derivative work, link, or distribute the Materials so that any part
   of it becomes subject to an Excluded License.

   G. The scope and term of Your license depends on the type of license You are
   provided by Intel.  The variety of license types are set forth below, which
   may not be available for all "Intel(R) Software Development Products" and
   therefore may not apply to the particular Materials You are licensing.  For
   more information on the types of licenses, please contact Intel or Your sales
   representative.

   i. EVALUATION LICENSE: If You obtained the Materials pursuant to an
   evaluation license, You may use the Materials only for internal evaluation
   purposes and only for the term of the evaluation period, as specified on
   Intel's download website or which may be controlled by the license key for
   the Materials.  NOTWITHSTANDING ANYTHING TO THE CONTRARY ELSEWHERE IN THIS
   AGREEMENT, YOU MAY USE THE MATERIALS ONLY FOR EVALUATION PURPOSES AND ONLY
   FOR THE TERM OF THE EVALUATION, YOU MAY NOT DISTRIBUTE ANY PORTION OF THE
   MATERIALS, AND THE APPLICATION AND/OR PRODUCT DEVELOPED BY YOU MAY ONLY BE
   USED FOR EVALUATION PURPOSES AND ONLY FOR THE TERM OF THE EVALUATION.  You
   may install copies of the Materials on a reasonable number of computers to
   conduct Your evaluation provided that You are the only individual using the
   Materials and only one copy of the Materials is in use at any one time.  A
   separate license key is required for each additional use and/or individual
   user in all other cases, including without limitation, use by persons,
   computer systems, and other use methods known now and in the future.  Intel
   may provide You with a license key that enables the Materials for an
   evaluation license.  If You are an entity, Intel grants You the right to
   designate one individual within Your organization to have the sole right to
   use the Materials in the manner provided above.

   ii. NONCOMMERCIAL USE LICENSE:  If You obtained the Materials under a
   noncommercial use license, You may use the Materials only for non-commercial
   use where You receive no fee, salary or any other form of compensation.  The
   Materials may not be used for any other purpose, whether "for profit" or "not
   for profit."  Any work performed or produced as a result of use of the
   Materials cannot be performed or produced for the benefit of other parties
   for a fee, compensation or any other reimbursement or remuneration.  You may
   install copies of the Materials on an unlimited number of computers provided
   that You are the only individual using the Materials and only one copy of the
   Materials is in use at any one time.  A separate license is required for each
   additional use and/or individual user in all other cases, including without
   limitation, use by persons, computer systems, and other methods of use known
   now and in the future.  Intel will provide You with a license key that
   enables the Materials for a noncommercial-use license.  If You obtained a
   time-limited noncommercial-use license, the duration (time period) of Your
   license and Your ability to use the Materials is limited to the time period
   of the obtained license, which is specified on Intel's download website,
   specified in the applicable documentation or controlled by the license key
   for the Materials.

   iii. NAMED-USER LICENSE: If You obtained the Materials under a named-user
   license, You may allow only one (1) individual to install and use the
   Materials on no more than three (3) computers provided that same individual
   is using the Materials only on one (1) computer at a time.  If You obtained a
   time-limited named-user license, the term of Your license and your ability to
   use the Materials is limited to the time period of the obtained license,
   which is specified on Intel's download website, specified in the applicable
   documentation or controlled by the license key for the Materials.

   iv. NODE-LOCKED LICENSE: If You obtained the Materials under a node-locked
   license, You may use the Materials only on a single designated computer by no
   more than the authorized number of concurrent users. If You obtained a
   time-limited node-locked license, the term of Your license and Your ability
   to use the Materials is limited to the time period of the obtained license,
   which is specified on Intel's download website, specified in the applicable
   documentation or controlled by the license key for the Materials.

   v. FLOATING LICENSE: If You obtained the Materials under a floating license,
   you may (a) install the Materials on an unlimited number of computers that
   are connected to the designated network and (b) use the Material by no more
   than the authorized number of concurrent individual users.  If You obtained a
   time-limited Floating license key, the term of Your license and Your ability
   to use the Materials is limited to the time period of the obtained license,
   which is specified on Intel's download website, specified in the applicable
   documentation or controlled by the license key for the Materials.

   H. MEDIA FORMAT CODECS AND DIGITAL RIGHTS MANAGEMENT.  You acknowledge and
   agree that your use of the Materials or distribution of the Materials with
   Your Product as permitted by this license may require you to procure
   license(s) from one or more third parties that may hold intellectual property
   rights applicable to any media decoding, encoding or transcoding technology
   (such as, for example, through use of an audio or video codec) and/or digital
   rights management capabilities of the Materials, if any.  Should any such
   additional licenses be required, You are solely responsible for obtaining any
   such licenses and agree to obtain any such licenses at Your own expense.

   I. MATERIALS TRANSFER:  Except for the Pre-Release Licenses or Evaluation
   Licenses or Non-Commercial Licenses, as specified above, You may permanently
   transfer the Materials you received pursuant to a license type listed in
   Section 4(G) above, and all of Your rights under this Agreement, to another
   party ("Recipient") solely in conjunction with a change of ownership, merger,
   acquisition, sale or transfer of all or substantially all of Your business or
   assets, either voluntarily, by operation of law or otherwise subject to the
   following: You must notify Intel of the transfer by sending a letter to Intel
   (i) identifying the legal entities of Recipient and You, (ii) identifying the
   Materials (i.e., the specific Intel software and version) and the associated
   serial numbers to be transferred, (iii) certifying that You retain no copies
   of the Materials or portions thereof, (iv) certifying that the Recipient has
   agreed in writing to be bound by all of the terms and conditions of this
   Agreement, (v) certifying that the Recipient has been notified that in order
   to receive support from Intel for the Materials they must notify Intel in
   writing of the transfer and provide Intel with the information specified in
   subsection (ii) above along with the name and email address of the individual
   assigned to use the Materials, and (vi) providing Your email address so that
   Intel may confirm receipt of Your letter.  Please send such letter to:

Intel Corporation
2111 NE 25th Avenue
Hillsboro, OR 97124
Attn: DPD Contracts Management, JF1-15

4. PRIVACY:

   A. Data Collection:  Based on the personal information You provided to Intel
   when You registered the license to the Materials with Intel, Intel has
   collected or will collect certain personal information from You in order to
   contact You regarding updates to the Materials, and regarding Your experience
   with obtaining, installing and otherwise using Materials, including sending
   You surveys to obtain the aforementioned information.

   B. Revoking Consent to Data Collection:  You can revoke Your consent to this
   collection of personal information at any time by clicking on the link to
   "unsubscribe" at the bottom of any communication from Intel related to the
   Materials which will allow You to opt-out of receiving future messages
   related to the Materials.

   C. Intel's Privacy Notice:  Intel is committed to respecting Your privacy. To
   learn more about Intel's privacy practices, please visit
   http://www.intel.com/privacy.

5. OWNERSHIP: Title to the Materials and all copies thereof remain with Intel or
its suppliers.  The Materials are protected by intellectual property rights,
including without limitation, United States copyright laws and international
treaty provisions.  You will not remove any copyright or other proprietary
notice from the Materials.  You agree to prevent any unauthorized copying of the
Materials.  Except as expressly provided herein, no license or right is granted
to You directly or by implication, inducement, estoppel or otherwise;
specifically Intel does not grant any express or implied right to You under
Intel patents, copyrights, trademarks, or trade secrets.

6. NO WARRANTY AND NO SUPPORT:  Disclaimer.  Intel disclaims all warranties of
any kind and the terms and remedies provided in this Agreement are instead of
any other warranty or condition, express, implied or statutory, including those
regarding merchantability, fitness for any particular purpose, non-infringement
or any warranty arising out of any course of dealing, usage of trade, proposal,
specification or sample.  Intel does not assume (and does not authorize any
person to assume on its behalf) any other liability.

Intel may make changes to the Materials, or to items referenced therein, at any
time without notice, but is not obligated to support, update or provide training
for the Materials. Intel may in its sole discretion offer such support, update
or training services under separate terms at Intel's then-current rates. You may
request additional information on Intel's service offerings from an Intel sales
representative.

7. LIMITATION OF LIABILITY:  Neither Intel nor its suppliers shall be liable for
any damages whatsoever (including, without limitation, damages for loss of
business profits, business interruption, loss of business information, or other
loss) arising out of the use of or inability to use the Materials, even if Intel
has been advised of the possibility of such damages.  Because some jurisdictions
prohibit the exclusion or limitation of liability for consequential or
incidental damages, the above limitation may not apply to you.

8. UNAUTHORIZED USE:  The Materials are not designed, intended, or authorized
for use in any type of a system or application in which the failure of the
Materials could create a situation where personal injury or death may occur
(e.g.,  medical systems, life sustaining or lifesaving systems).  Should You use
the Materials for any such unintended or unauthorized use, You hereby indemnify,
defend, and hold Intel and its officers, subsidiaries and affiliates harmless
against all claims, costs, damages, expenses, and reasonable attorney fees
arising out of, directly or indirectly, such use and any claim of product
liability, personal injury or death associated with such unintended or
unauthorized use, even if such claim alleges that Intel was negligent regarding
the design or manufacture of the Materials.

9. USER SUBMISSIONS:  This Agreement does not obligate You to provide Intel with
materials, information, comments, suggestions or other communication regarding
the Materials.  However, You agree that any material, information, comments,
suggestions or other communication You transmit or post to an Intel website
(including but not limited to, submissions to the Intel Premier Support and/or
other customer support websites or online portals) or provide to Intel under
this Agreement are not controlled by the International Traffic in Arms
Regulations (ITAR) or the Export Administration Regulation (EAR), and if related
to the features, functions, performance or use of the Materials are deemed
non-confidential and non-proprietary ("Communications").  Intel will have no
obligations with respect to the Communications.  You hereby grant to Intel a
non-exclusive, perpetual, irrevocable, royalty-free, copyright license to copy,
modify, create derivative works, publicly display, disclose, distribute, license
and sublicense through multiple tiers of distribution and licensees, incorporate
and otherwise use the Communications and all data, images, sounds, text, and
other things embodied therein, including derivative works thereto, for any and
all commercial or non-commercial purposes. You are prohibited from posting or
transmitting to or from an Intel website or provide to Intel any unlawful,
threatening, libelous, defamatory, obscene, pornographic, or other material that
would violate any law.  If You wish to provide Intel with information that You
intend to be treated as confidential information, Intel requires that such
confidential information be provided pursuant to a non-disclosure agreement
("NDA"), so please contact Your Intel representative to ensure the proper NDA is
in place.

Nothing in this Agreement will be construed as preventing Intel from reviewing
Your Communications and errors or defects in Intel products discovered while
reviewing Your Communications. Furthermore, nothing in this Agreement will be
construed as preventing Intel from implementing independently-developed
enhancements to Intel's own error diagnosis methodology to detect errors or
defects in Intel products discovered while reviewing Your Communications or to
implement bug fixes or enhancements in Intel products. The foregoing may include
the right to include Your Communications in regression test suites.

10.  NON-DISCLOSURE: The following provisions will apply if there is no existing
non-disclosure agreement between You and Intel. You will maintain the
confidentiality of the Confidential Information (if any) with at least the same
degree of care that You use to protect Your own confidential and proprietary
information, but no less than a reasonable degree of care under the
circumstances.  You will not disclose the Confidential Information to any
employees or to any third parties except to Your employees who have a need to
know and who agree to abide by nondisclosure terms at least as comprehensive as
those set forth herein; provided that You will be liable for breach by any such
entity.  For the purposes of this Agreement, the term "employee" will include
Your independent contractors, who have signed confidentiality agreements with
You.  You will not make any copies of the Confidential Information except as
necessary for Your employees with a need to know.  Any copies which are made
will be identified as belonging to Intel and marked "confidential",
"proprietary" or with similar legend.  You will not be liable for the disclosure
of any Confidential Information which is (a) generally made available publicly
or to third parties by Intel without restriction on disclosure; (b) rightfully
received from a third party without obligation of confidentiality; (c)
rightfully known to You without any limitation on disclosure prior to Your
receipt from Intel; (d) independently developed by Your employees; or (e)
required to be disclosed in accordance with applicable laws, regulations, court,
judicial or other government order, provided that You will give Intel reasonable
notice prior to such disclosure and will comply with any applicable protective
order.

11. TERMINATION OF THIS LICENSE: This Agreement becomes effective on the date
You accept this Agreement and will continue until terminated as provided for in
this Agreement.  If You are using the Materials under a time-limited license,
for example an Evaluation License, this Agreement terminates without notice on
the last day of the time period, which is specified in the Materials or on
Intel's website, and/or controlled by the license key code for the Materials.
Intel may terminate this license immediately if You are in breach of any of its
terms and conditions and such breach is not cured within thirty (30) days of
written notice from Intel.  Upon termination, You will immediately return to
Intel or destroy the Materials and all copies thereof.  In the event of
termination of this Agreement, the license grant to any Materials or
Redistributables distributed by You in accordance with the terms and conditions
of this Agreement, prior to the effective date of such termination, will survive
any such termination of this Agreement. Sections 1, 4, 5, 6, 7, 8, 9, 10, 11,
12, and 13 will survive expiration or termination of this Agreement.

12. U.S. GOVERNMENT RESTRICTED RIGHTS: The technical data and computer software
covered by this license is a "Commercial Item," as such term is defined by the
FAR 2.101 (48 C.F.R. 2.101) and is "commercial computer software" and
"commercial computer software documentation" as specified under FAR 12.212 (48
C.F.R. 12.212) or DFARS 227.7202 (48 C.F.R. 227.7202), as applicable. This
commercial computer software and related documentation is provided to end users
for use by and on behalf of the U.S. Government, with only those rights as are
granted to all other end users pursuant to the terms and conditions herein. Use
for or on behalf of the U.S. Government is permitted only if the party acquiring
or using this software is properly authorized by an appropriate U.S. Government
official. This use by or for the U.S. Government clause is in lieu of, and
supersedes, any other FAR, DFARS, or other provision that addresses Government
rights in the computer software or documentation covered by this license.  All
copyright licenses granted to the U.S. Government are coextensive with the
technical data and computer software licenses granted herein. The U.S.
Government will only have the right to reproduce, distribute, perform, display,
and prepare derivative works as needed to implement those rights.

13. GENERAL PROVISIONS

   A. ENTIRE AGREEMENT: This Agreement contains the complete and exclusive
   agreement and understanding between the parties concerning the subject matter
   of this Agreement, and supersedes all prior and contemporaneous proposals,
   agreements, understanding, negotiations, representations, warranties,
   conditions, and communications, oral or written, between the parties relating
   to the same subject matter. This Agreement, including without limitation its
   termination, has no effect on any signed non-disclosure agreements between
   the parties, which remain in full force and effect as separate agreements to
   their terms. Each party acknowledges and agrees that in entering into this
   Agreement it has not relied on, and will not be entitled to rely on, any oral
   or written representations, warranties, conditions, understanding, or
   communications between the parties that are not expressly set forth in this
   Agreement. The express provisions of this Agreement control over any course
   of performance, course of dealing, or usage of the trade inconsistent with
   any of the provisions of this Agreement. The provisions of this Agreement
   will prevail notwithstanding any different, conflicting, or additional
   provisions that may appear on any purchase order, acknowledgement, invoice,
   or other writing issued by either party in connection with this Agreement. No
   modification or amendment to this Agreement will be effective unless in
   writing and signed by authorized representatives of each party, and must
   specifically identify this Agreement by its title and version (e.g., "End
   User License Agreement for the Intel(R) Software Development Products
   (Version March 2016)).  If You received a copy of this Agreement translated
   into another language, the English language version of this Agreement will
   prevail in the event of any conflict between versions.  Intel may make
   changes to the Agreement as it distributes new versions of the Materials.
   When these changes are made, Intel will make a new version of the Agreement
   available on its website:
   https://software.intel.com/en-us/articles/end-user-license-agreement

   B. EXPORT. You acknowledge that the Materials and all related technical
   information are subject to export controls under the laws and regulations of
   the United States and any other applicable governments. You agree to comply
   with these laws and regulations governing export, re-export, import,
   transfer, distribution, and use of the Materials. In particular, but without
   limitation, the Materials may not be exported or re-exported (a) into any
   U.S. embargoed countries or (b) to any person or entity listed on a denial
   order published by the U.S. government or any other applicable governments.
   By using the Materials, you represent and warrant that you are not located in
   any such country or on any such list. You also agree that you will not use
   the Materials for any purposes prohibited by the U.S. government or other
   applicable governments, including, without limitation, the development,
   design, manufacture or production of nuclear, missile, chemical or biological
   weapons. You confirm that the Materials will not be re-exported or sold to a
   third party who is known or suspected to be involved in activities including,
   without limitation, the development, design, manufacture, or production of
   nuclear, missile, chemical or biological weapons.

   C. GOVERNING LAW, JURISDICTION, AND VENUE: All disputes arising out of or
   related to this Agreement, whether based on contract, tort, or any other
   legal or equitable theory, will in all respects be governed by, and construed
   and interpreted under, the laws of the United States of America and the State
   of Delaware, without reference to conflict of laws principles. The parties
   agree that the United Nations Convention on Contracts for the International
   Sale of Goods (1980) is specifically excluded from and will not apply to this
   Agreement. All disputes arising out of or related to this Agreement, whether
   based on contract, tort, or any other legal or equitable theory, will be
   subject to the exclusive jurisdiction of the courts of the State of Delaware
   or of the Federal courts sitting in that State. Each party submits to the
   personal jurisdiction of those courts and waives all objections to that
   jurisdiction and venue for those disputes.

   D. SEVERABILITY: The parties intend that if a court holds that any provision
   or part of this Agreement is invalid or unenforceable under applicable law,
   the court will modify the provision to the minimum extent necessary to make
   it valid and enforceable, or if it cannot be made valid and enforceable, the
   parties intend that the court will sever and delete the provision or part
   from this Agreement. Any change to or deletion of a provision or part of this
   Agreement under this Section will not affect the validity or enforceability
   of the remainder of this Agreement, which will continue in full force and
   effect.

Document Title and Version: End User License Agreement for the Intel(R) Software
Development Products (Version March 2016)

* Other names and brands may be claimed as the property of others



--------------------------------------------------------------------------------
Do you agree to be bound by the terms and conditions of this license agreement?
Type 'accept' to continue or 'decline' to go back to the previous menu: accept

选择使用 evaluation 授权

Step 3 of 6 | Activation
--------------------------------------------------------------------------------
If you have purchased this product and have the serial number and a connection
to the internet you can choose to activate the product at this time.
Alternatively, you can choose to evaluate the product or defer activation by
choosing the evaluate option. Evaluation software will time out in about one
month. You can also use license file or Intel(R) Software License Manager.
--------------------------------------------------------------------------------
1. Use existing trial license (31 day(s) left) [default]
2. I want to activate my product using a serial number
3. I want to activate by using a license file, or by using Intel(R) Software
License Manager

h. Help
b. Back to the previous menu
q. Quit
--------------------------------------------------------------------------------
Please type a selection or press "Enter" to accept default choice [1]:

选择完成配置安装

Step 4 of 6 | Options > Configure Cluster Installation
--------------------------------------------------------------------------------
This product can be installed on cluster nodes.
--------------------------------------------------------------------------------
1. Finish configuring installation target [default]

2. Installation target                           [ Current system only ]

h. Help
b. Back to the previous menu
q. Quit
--------------------------------------------------------------------------------
Please type a selection or press "Enter" to accept default choice [1]:

选择 start installation, 记录安装路径 /opt/intel

Step 4 of 6 | Options > Pre-install Summary
--------------------------------------------------------------------------------
Install location:
    /opt/intel

Component(s) selected:
    Intel(R) MPI Library 5.1 Update 3                                      763MB
        Intel MPI Benchmarks
        Intel MPI Library, Runtime Environment for applications running on
Intel(R) 64 Architecture
        Intel MPI Library for applications running on Intel(R) 64 Architecture
        Intel MPI Library for applications running on Intel(R) Many Integrated
Core Architecture

Install Space Required:  840MB

Installation target:
    Install on the current system only


1. Start installation Now [default]
2. Customize installation

h. Help
b. Back to the previous menu
q. Quit
--------------------------------------------------------------------------------
Please type a selection or press "Enter" to accept default choice [1]:

安装完毕确认

Step 5 of 6 | Installation
--------------------------------------------------------------------------------
Each component will be installed individually. If you cancel the installation,
some components might remain on your system. This installation may take several
minutes, depending on your system and the options you selected.
--------------------------------------------------------------------------------
Installing Intel MPI Benchmarks component... done
--------------------------------------------------------------------------------
Installing Intel MPI Library, Runtime Environment for applications running on
Intel(R) 64 Architecture component... done
--------------------------------------------------------------------------------
Installing Intel MPI Library for applications running on Intel(R) 64
Architecture component... done
--------------------------------------------------------------------------------
Installing Intel MPI Library for applications running on Intel(R) Many
Integrated Core Architecture component... done
--------------------------------------------------------------------------------
Finalizing product configuration...
--------------------------------------------------------------------------------
Press "Enter" key to continue

完成安装

Step 6 of 6 | Complete
--------------------------------------------------------------------------------
Thank you for installing and for using Intel(R) MPI Library 5.1 Update 3
for Linux*.

Support services start from the time you install or activate your product. If
you have not already done so, please create your support account now to take
full advantage of your product purchase.

Your support account gives you access to free product updates and upgrades as
well as interactive technical support at Intel(R) Premier Support.

--------------------------------------------------------------------------------
Press "Enter" key to quit:

6. 允许节点自动 ssh 登陆

通过此步设置以满足后续 benchmark 测试程序可以在主节点上跨节点操作

sudo apt-get nmap sshpass

创建下述脚本 ./user_authentication.sh,并替换下述脚本中 chown azureuser:azureuser /home/azureuser/.ssh/config 的用户名和用户组为当前环境的用户名和用户组 

#!/bin/bash
# For CentOS user must first install epel-release, sshpass, and nmap (sshpass and nmap are available from epel-release for CentOS)

# usage ./user_authentication.sh [username] [password] [internalIP prefix]
# ./user_authentication.sh azureuser Azure@123 10.32.0
USER=$1
PASS=$2
IPPRE=$3
HEADNODE=`hostname`

mkdir -p .ssh
echo -e  'y\n' | ssh-keygen -f .ssh/id_rsa -t rsa -N ''

echo 'Host *' >> .ssh/config
echo 'StrictHostKeyChecking no' >> .ssh/config
chmod 400 .ssh/config
chown azureuser:azureuser /home/azureuser/.ssh/config

nmap -sn $IPPRE.* | grep $IPPRE. | awk '{print $5}' > nodeips.txt
for NAME in `cat nodeips.txt`; do sshpass -p $PASS ssh -o ConnectTimeout=2 $USER@$NAME 'hostname' >> nodenames.txt;done

NAMES=`cat nodenames.txt` #names from names.txt file
for NAME in $NAMES; do
    sshpass -p $PASS scp -o "StrictHostKeyChecking no" -o ConnectTimeout=2 /home/$USER/nodenames.txt $USER@$NAME:/home/$USER/
    sshpass -p $PASS ssh -o ConnectTimeout=2 $USER@$NAME "mkdir .ssh && chmod 700 .ssh"
    sshpass -p $PASS ssh -o ConnectTimeout=2 $USER@$NAME "echo -e  'y\n' | ssh-keygen -f .ssh/id_rsa -t rsa -N ''"
    sshpass -p $PASS ssh -o ConnectTimeout=2 $USER@$NAME 'touch /home/'$USER'/.ssh/config'
    sshpass -p $PASS ssh -o ConnectTimeout=2 $USER@$NAME 'echo "Host *" >  /home/'$USER'/.ssh/config'
    sshpass -p $PASS ssh -o ConnectTimeout=2 $USER@$NAME 'echo StrictHostKeyChecking no >> /home/'$USER'/.ssh/config'
    sshpass -p $PASS ssh -o ConnectTimeout=2 $USER@$NAME 'chmod 400 /home/'$USER'/.ssh/config'
    cat .ssh/id_rsa.pub | sshpass -p $PASS ssh -o ConnectTimeout=2 $USER@$NAME 'cat >> .ssh/authorized_keys'
    sshpass -p $PASS scp -o "StrictHostKeyChecking no" -o ConnectTimeout=2 $USER@$NAME:/home/$USER/.ssh/id_rsa.pub .ssh/sub_node.pub

    for SUBNODE in `cat nodeips.txt`; do
         sshpass -p $PASS ssh -o ConnectTimeout=2 $USER@$SUBNODE 'mkdir -p .ssh'
         cat .ssh/sub_node.pub | sshpass -p $PASS ssh -o ConnectTimeout=2 $USER@$SUBNODE 'cat >> .ssh/authorized_keys'
    done
    sshpass -p $PASS ssh -o ConnectTimeout=2 $USER@$NAME 'chmod 700 .ssh/'
    sshpass -p $PASS ssh -o ConnectTimeout=2 $USER@$NAME 'chmod 640 .ssh/authorized_keys'
done

查找当前虚拟主机所在子网,将主机位删除,保留子网字段

ifconfig eth0 | grep -w inet | awk '{print $2}'

执行 ./user_authentication.sh <myusername> <mypassword> 10.1.3, myusername 填入测试集群中 ssh 登陆的用户名,mypassword 填入测试集群中 ssh 登陆的密码, 后面填入测试集群所在子网

7. 验证配置完成

关联 mpi 程序

source /opt/intel/impi/5.1.3.223/bin64/mpivars.sh

将 vm1ipaddress 和 vm2ipaddress 地址替换为测试集群主机的 vnet 内的地址

mpirun -ppn 1 -n 2 -hosts <vm1ipaddress>,<vm2ipaddress> -env I_MPI_FABRICS=shm:dapl -env I_MPI_DAPL_PROVIDER=ofa-v2-ib0 -env I_MPI_DYNAMIC_CONNECTION=0 hostname

如果运行正常会分别在本机和远端主机上执行 hostname 命令,所以即打印出测试集群中主机的主机名称

8. 进行 Benchmark 测试

Intel 的 MPI 程序包中已经携带安装了 benchmark 工具,可以通过如下指令执行

 mpirun -hosts <vm1ipaddress>,<vm2ipaddress> -ppn 1 -n 2 -env I_MPI_FABRICS=shm:dapl -env I_MPI_DAPL_PROVIDER=ofa-v2-ib0 -env I_MPI_DYN
AMIC_CONNECTION=0 IMB-MPI1 pingpong

上述命令只是执行了 benchmark 中的 pingpong 测试,该测试可以给出延迟和带宽在 infiniband 网络上的表现,以下是测试环境中的实测结果。 结果中的延迟时间是一个 oneway 的测试延迟,从结果可见 oneway 延迟在 0 字节负载时 1.82 us,带宽最大可到 44Gbps。

MPI benchmark 包含很多其它的测试项目,关于 MPI benchmark 的更多详情可以参阅如下文档: https://software.intel.com/en-us/imb-user-guide-mpi-1-benchmarks

# PingPong

#---------------------------------------------------
# Benchmarking PingPong
# #processes = 2
#---------------------------------------------------
       #bytes #repetitions      t[usec]   Mbytes/sec
            0         1000         1.82         0.00
            1         1000         1.85         0.52
            2         1000         1.98         0.96
            4         1000         1.84         2.08
            8         1000         1.84         4.14
           16         1000         1.84         8.29
           32         1000         2.76        11.05
           64         1000         2.77        22.05
          128         1000         2.83        43.11
          256         1000         2.99        81.60
          512         1000         3.13       155.95
         1024         1000         3.46       282.40
         2048         1000         4.01       487.49
         4096         1000         5.29       738.15
         8192         1000         6.67      1171.02
        16384         1000         8.97      1742.70
        32768         1000        14.70      2125.27
        65536          640        18.43      3391.43
       131072          320        31.41      3980.12
       262144          160        54.78      4563.61
       524288           80        98.57      5072.55
      1048576           40       185.34      5395.56
      2097152           20       359.73      5559.79
      4194304           10       715.24      5592.50


# All processes entering MPI_Finalize

这个延迟大概是什么量级呢,我们跟某厂家的 40G/100G 限速的以太网交换机比较一下

 

 

        

posted @ 2018-08-30 00:30 wekang 阅读(...) 评论(...) 编辑 收藏