China AZURE Infniband Powered GPU VM
最近 China Azure 上刚刚发布了 NCv3 系列的 GPU 虚拟主机，细心的小伙伴可能注意到，NCv3 中有一款型号为 NC24rs 的虚拟主机，这个字母 r 来的些许蹊跷，代表什么意思呢？通过查阅官网对于该主机的介绍说明可见，原来 r 代表 RDMA，表示该虚拟主机支持 infiniband 来加持的 RDMA 网络。infiniband 可以帮计算节点获得高带宽低延迟的加持，在集群计算场景下可以大大提成效率。今天我们一起来看一下 infiniband 加持后节点通信性能几何。
我们准备两台 NC24rsv3 的虚拟主机，为保证两台主机可以通过 infiniband 进行互通，我们需要讲两台虚拟主机部署在同一可用性集中（availability set）。部署完成后，登陆虚拟主机安装 RDMA 相关组件，本文以 ubuntu 16.04 LTS 为例，操作步骤如下：
1. 安装 RDMA 相关组件
sudo apt-get update sudo apt-get install libdapl2 libmlx4-1
2. 使能 RDMA
通过 root 权限修改 /etc/waagent.conf 文件
OS.EnableRDMA=y
OS.UpdateRdmaDriver=y
3. 打开内存限制
通过 root 权限修改 /etc/security/limits.conf
* hard memlock unlimited
* soft memlock unlimited
4. 重启主机生效配置
sudo reboot
5. 安装 Intel MPI 驱动
下载驱动
wget http://registrationcenter-download.intel.com/akdlm/irc_nas/tec/9278/l_mpi_p_5.1.3.223.tgz
解压缩驱动，进入驱动压缩包所在目录
tar -xzf l_mpi_p_5.1.3.223.tgz
进入解压的安装目录，进行安装
cd l_mpi_p_5.1.3.223/ ./install.sh
选择 1， 并按照提示输入 root 密码
Please make your selection by entering an option. Root access is recommended for evaluation. 1. Run as a root for system wide access for all users [default] 2. Run using sudo privileges and password for system wide access for all users 3. Run as current user to limit access to user level h. Help q. Quit
确认，并接受 EULA
Step 1 of 6 | Welcome -------------------------------------------------------------------------------- Welcome to the Intel(R) MPI Library 5.1 Update 3 for Linux* setup program. -------------------------------------------------------------------------------- You will complete the steps below during setup process: Step 1 : Welcome Step 2 : License agreement Step 3 : Activation Step 4 : Options Step 5 : Installation Step 6 : Complete -------------------------------------------------------------------------------- Press "Enter" key to continue or "q" to quit:
Step 2 of 6 | License agreement To continue with the installation of this product you are required to accept the terms and conditions of the End User License Agreement (EULA). The EULA is displayed using the 'more' utility. Press the spacebar to advance to the next page or enter 'q' to skip to the end. After reading the EULA, you must enter 'accept' to continue the installation or 'decline' to return to the previous menu. -------------------------------------------------------------------------------- IMPORTANT INFORMATION ABOUT YOUR RIGHTS, OBLIGATIONS AND THE USE OF YOUR DATA - READ AND AGREE BEFORE COPYING, INSTALLING OR USING This Agreement forms a legally binding contract between you, or the company or other legal entity ("Legal Entity") for which you represent and warrant that you have the legal authority to bind that Legal Entity, are agreeing to this Agreement (each, "You" or "Your") and Intel Corporation and its subsidiaries (collectively "Intel") regarding Your use of the Materials. By copying, installing, distributing, publicly displaying, or otherwise using the Materials, You agree to be bound by the terms of this Agreement. If You do not agree to the terms of this Agreement, do not copy, install, distribute, publicly display, or use the Materials. You affirm that You are 18 years old or older or, if not, Your parent, legal guardian or Legal Entity must agree and enter into this Agreement. DATA COLLECTION. The Materials may contain certain features that generate, collect, and transmit data to Intel about the installation, setup, and use of the Materials. The purposes of data collection are: 1) to verify compliance with the terms of this Agreement; and 2) to enable Intel to develop, improve, and support Intel's products and services. When data is collected to verify compliance with the terms of this Agreement, this collection may be mandatory and a condition of using the Materials. This data includes the Material's unique serial number combined with other information about the Materials and Your computer. When Materials are made available for use free of charge, the collection of usage data (such as randomly generated unique identifier and component/feature/function usage) may also be mandatory and a condition of using the Materials. Data collected about the installation, setup, and use of the Materials may be collated with other available data only if: 1) the purpose is to develop, improve, and support Intel's products and services, and 2) the data will not be used to identify or contact You or other individuals. To learn more about Intel's data collection for these Materials, please visit: https://software.intel.com/en-us/articles/data-collection. To learn more about Intel's privacy practices, please visit http://www.intel.com/privacy. Third Party Programs (as defined below), even if included with the distribution of the Materials, are governed by separate license terms, including without limitation, third party license terms, other Intel software license terms, and open source software license terms. Such separate license terms (and not this Agreement) solely govern Your use of the Third Party Programs. 1. LICENSE DEFINITIONS: A. "Confidential Information" means all Materials (as defined below), including any portions thereof, that are identified (in the product release notes, on Intel's download website for the Materials or elsewhere) or labeled as Intel confidential information or a similar legend. B. "Excluded License" means a license that requires, as a condition of use, modification, or distribution, that the licensed software or other software incorporated into, derived from or distributed with such software (a) be disclosed or distributed in Source Code form; (b) be licensed by the user to third parties for the purpose of making and/or distributing derivative works; or (c) be redistributable at no charge. Excluded Licenses include, without limitation, licenses that license or distribute software under any of the following licenses or distribution models, or licenses or distribution models substantially similar to any of the following: (a) GNU's General Public License (GPL) or Lesser/Library GPL (LGPL), (b) the Artistic License (e.g., PERL), (c) the Mozilla Public License, (d) the Netscape Public License, (e) the Sun Community Source License (SCSL), (f) the Sun Industry Source License (SISL), and (g) the Common Public License (CPL). C. "Licensed Patent Claims" means the claims of Intel's patents that are necessarily and directly infringed by the reproduction and distribution of the Materials that is authorized in Section 2 below, when the Materials is in its unmodified form as delivered by Intel to You and not modified or combined with anything else. Licensed Patent Claims are only those claims that Intel can license without paying, or getting the consent of, a third party. D. "Materials" are defined as the software, documentation, the software product serial number and license key codes (if applicable), and other materials, including any modifications, updates and upgrades thereto, that are provided to You under this Agreement. Materials also include any Redistributables, Source Code, and Pre-Release Materials, as defined below but do not include Third Party Programs. E. "Microsoft Platforms" means any current and future Microsoft operating system products, Microsoft run-time technologies (such as the .NET Framework), and Microsoft application platforms (such as Microsoft Office or Microsoft Dynamics) that Microsoft offers. F. "Pre-Release Materials" means the Materials, or portions thereof, that are identified (in the product release notes, on Intel's download website for the Materials or elsewhere) or labeled as pre-release, and as such the Pre-Release Materials are deemed to be pre-release code (e.g., alpha or beta release, etc.), which may not be fully functional and which Intel may substantially modify in development of a commercial version, and for which Intel makes no assurances that it will ever develop or make generally available a commercial version. G. "Redistributables" (if any) are the files listed in the following text files that may be included in the Materials for the applicable Intel Software Development Product: clredist.txt, credist.txt, fredist.txt, redist.txt, and redist-rt.txt. H. "Sample Source Code" is those portions of the Materials that are Source Code files and are identified as sample source code, including without limitation, the IPP Sample Source. I. "Source Code" is defined as the software (and not documentation or text) portion of the Materials provided in human readable format, and includes modifications to the Source Code that You make or are made on Your behalf as expressly permitted under the terms of this Agreement. J. "Third Party Programs" (if any) are the files listed in the "third-party-programs.txt" text file that may be included in the Materials for the applicable software. K. "Your Product" means one or more applications or products developed by or for You using the Materials. 2. LICENSE GRANT: 2.1 Subject to the terms and conditions of this Agreement, and timely payment of any fees (if applicable), Intel grants You a non-exclusive, worldwide, perpetual (subject to Section 11 below), non-assignable (except as expressly permitted hereunder), limited right and license: A. under its copyrights, to: (1) reproduce copies of the Materials for Your internal business use in accordance with the documentation included as part of the Materials, and subject to the applicable license rights and restrictions specified in Section 3 below; provided, however, that this license does not include the right to sublicense and may only be exercised by You or Your employees; (2) use the Materials solely for Your internal business use to develop Your Product, in accordance with the applicable license rights and restrictions specified in Section 3 below and the documentation or text files included as part of the Materials; provided, however, that this license does not include the right to sublicense and may only be exercised by You or Your employees; (3) modify or create derivative works of the Materials, or any portions thereof, that are provided in Source Code form, provided, however, that this license does not include the right to sublicense and may be exercised only by You or Your employees; (4) publicly perform, display, and distribute (directly and through Your distributors, resellers and other channel partners) or otherwise make publicly available the Redistributables, including any modifications to or derivative works of the Redistributables made pursuant to Section 2.1.A(3), or any portions thereof, subject to the following restrictions: (i) any distribution of the Redistributables must only be as part of Your Product which must add significantly more functionality than the Redistributables themselves; (ii) any additional restrictions which may appear in the Redistributables text files specified in Section 1.G above and in Section 3 below; and (iii) the license under Section 2.1.A(4) includes the right to sublicense the Redistributables, but the sublicense rights are limited to sublicensing of any Intel copyrights in the Redistributables and only to the extent necessary to perform, display, and distribute the Redistributables (including Your modifications and derivative works thereto) solely as incorporated in Your Product. IF YOU RECEIVED THE MATERIALS FOR EVALUATION, YOU HAVE NO RIGHTS TO DISTRIBUTE THE REDISTRIBUTABLES, INCLUDING WITHOUT LIMITATION, ANY PORTIONS, MODIFICATIONS OR DERIVATIVE WORKS. (iv) Distribution of the Redistributables is also subject to the following limitations: You (a) will be solely responsible to Your customers for any update, support obligation or other liability which may arise from the distribution, (b) will not make any statement that Your Product is "certified" or that its performance is guaranteed by Intel, (c) will not use Intel's name or trademarks to market Your Product without written permission from Intel, (d) will provide the Redistributables subject to a license agreement that prohibits disassembly and reverse engineering of the Redistributables except in cases when you provide Your Product subject to an open source license that is not an Excluded License, for example, the BSD license, or the MIT license, (e) will indemnify, hold harmless, and defend Intel and its suppliers from and against any claims or lawsuits, including attorney's fees, that arise or result from Your modifications, derivative works or Your distribution of Your Product. and B. under Intel's Licensed Patent Claims, to: (1) make copies of the Materials only as specified in Section 2.1.A(1); (2) use the Materials only as specified in Section 2.1.A(2); and (3) offer to distribute, and distribute, but not sell, the Redistributables only as part of Your Product, under Intel's copyright license granted in Section 2.1(A), but only under the terms of that copyright license and not as a sale (but this right does not include the right to sub-license); (4) provided, further, that the license under the Licensed Patent Claims does not and will not apply to any modifications to, or derivative works of, the Materials, whether made by You, Your customer (which, for all purposes under this Agreement, will mean either a customer, reseller, distributor or other channel partner), or any third party even if the modification and derivative works are permitted under 2.1(A)(3). 2.2 If the Materials You receive are packaged, as a single orderable item (i.e., as a single SKU), with hardware that includes one or more Intel manufactured microprocessors ("Intel Target Hardware"), then the licenses granted in Section 2.1 above are restricted to the sole purpose of producing and releasing Your Product to execute on computer systems that include the same or new versions of the Intel manufactured microprocessor included in the Intel Target Hardware. Intel expressly does not grant You a patent license in this Agreement to any modifications or derivative works of the Materials, whether made by You, Your contractor, Your customer, or any other third party in creating the derivative works even to the extent creation of derivative works is permitted under Section 2.1(A)(3) above. 3. LICENSE CONDITIONS: A. If You are an entity, each of Your employees and Your contractors may use the Materials as specified in Section 2 above, provided: (i) their use of the Materials is solely on behalf of and in support of Your business, (ii) they agree to the terms and conditions of this Agreement, and (iii) You are solely responsible for their use of the Materials. B. If Your Product is a software development library, then attribution (if any), as specified in the product release notes of the corresponding Materials shall be displayed prominently in Your Product's associated documentation and on the web site (if any) for Your Product. C. If You receive Your first copy of the Materials electronically, and a second copy on media, then you may use the second copy only in accordance with Your applicable license stated in this Agreement, or for backup or archival purposes. You may not provide the second copy to another user. D. If the Materials You received are identified as Pre-Release Materials, (i) You have the right to use the Pre-Release Materials only for the duration of the pre-release term, which is specified in the product release notes, on Intel's download website for the Materials or elsewhere, or until the commercial release, if any, of the Pre-Release Materials, whichever is shorter, and (ii) You may not disclose to any third party any benchmarks, performance results, or other information relating to the Pre-Release Materials. E. Notwithstanding anything to the contrary in this Agreement, if the Materials include the text file named "site_license_materials.txt" the files specified in that text file may be installed on computer systems located only at a single site (unless multiple sites are specified in the purchase order accepted by Intel or its resellers), and those files may be accessed or used by unlimited and simultaneous users, subject to their compliance with all of the terms and conditions of this Agreement. F. Except as expressly provided in this Agreement, You may NOT: (i) use, copy, distribute, or publicly display the Materials; (ii) rent or lease the Materials to any third party; (iii) assign this Agreement or transfer the Materials; (iv) modify, adapt, or translate the Materials in whole or in part; (v) reverse engineer, decompile, or disassemble the Materials; (vi) attempt to modify or tamper with the normal function of any license manager that may regulate usage of the Materials; (vii) distribute, sublicense or transfer the Source Code form of any components of the Materials or derivatives thereof to any third party; (viii) distribute Redistributables except as part of a larger program that adds significant primary functionality different from that of the Redistributables; (ix) distribute the Redistributables to run on a platform other than a Microsoft Platform if according to the accompanying user documentation the Materials are meant to execute only on a Microsoft Platform; (x) include the Redistributables in malicious, deceptive, or unlawful programs or products; or (xi) modify, create a derivative work, link, or distribute the Materials so that any part of it becomes subject to an Excluded License. G. The scope and term of Your license depends on the type of license You are provided by Intel. The variety of license types are set forth below, which may not be available for all "Intel(R) Software Development Products" and therefore may not apply to the particular Materials You are licensing. For more information on the types of licenses, please contact Intel or Your sales representative. i. EVALUATION LICENSE: If You obtained the Materials pursuant to an evaluation license, You may use the Materials only for internal evaluation purposes and only for the term of the evaluation period, as specified on Intel's download website or which may be controlled by the license key for the Materials. NOTWITHSTANDING ANYTHING TO THE CONTRARY ELSEWHERE IN THIS AGREEMENT, YOU MAY USE THE MATERIALS ONLY FOR EVALUATION PURPOSES AND ONLY FOR THE TERM OF THE EVALUATION, YOU MAY NOT DISTRIBUTE ANY PORTION OF THE MATERIALS, AND THE APPLICATION AND/OR PRODUCT DEVELOPED BY YOU MAY ONLY BE USED FOR EVALUATION PURPOSES AND ONLY FOR THE TERM OF THE EVALUATION. You may install copies of the Materials on a reasonable number of computers to conduct Your evaluation provided that You are the only individual using the Materials and only one copy of the Materials is in use at any one time. A separate license key is required for each additional use and/or individual user in all other cases, including without limitation, use by persons, computer systems, and other use methods known now and in the future. Intel may provide You with a license key that enables the Materials for an evaluation license. If You are an entity, Intel grants You the right to designate one individual within Your organization to have the sole right to use the Materials in the manner provided above. ii. NONCOMMERCIAL USE LICENSE: If You obtained the Materials under a noncommercial use license, You may use the Materials only for non-commercial use where You receive no fee, salary or any other form of compensation. The Materials may not be used for any other purpose, whether "for profit" or "not for profit." Any work performed or produced as a result of use of the Materials cannot be performed or produced for the benefit of other parties for a fee, compensation or any other reimbursement or remuneration. You may install copies of the Materials on an unlimited number of computers provided that You are the only individual using the Materials and only one copy of the Materials is in use at any one time. A separate license is required for each additional use and/or individual user in all other cases, including without limitation, use by persons, computer systems, and other methods of use known now and in the future. Intel will provide You with a license key that enables the Materials for a noncommercial-use license. If You obtained a time-limited noncommercial-use license, the duration (time period) of Your license and Your ability to use the Materials is limited to the time period of the obtained license, which is specified on Intel's download website, specified in the applicable documentation or controlled by the license key for the Materials. iii. NAMED-USER LICENSE: If You obtained the Materials under a named-user license, You may allow only one (1) individual to install and use the Materials on no more than three (3) computers provided that same individual is using the Materials only on one (1) computer at a time. If You obtained a time-limited named-user license, the term of Your license and your ability to use the Materials is limited to the time period of the obtained license, which is specified on Intel's download website, specified in the applicable documentation or controlled by the license key for the Materials. iv. NODE-LOCKED LICENSE: If You obtained the Materials under a node-locked license, You may use the Materials only on a single designated computer by no more than the authorized number of concurrent users. If You obtained a time-limited node-locked license, the term of Your license and Your ability to use the Materials is limited to the time period of the obtained license, which is specified on Intel's download website, specified in the applicable documentation or controlled by the license key for the Materials. v. FLOATING LICENSE: If You obtained the Materials under a floating license, you may (a) install the Materials on an unlimited number of computers that are connected to the designated network and (b) use the Material by no more than the authorized number of concurrent individual users. If You obtained a time-limited Floating license key, the term of Your license and Your ability to use the Materials is limited to the time period of the obtained license, which is specified on Intel's download website, specified in the applicable documentation or controlled by the license key for the Materials. H. MEDIA FORMAT CODECS AND DIGITAL RIGHTS MANAGEMENT. You acknowledge and agree that your use of the Materials or distribution of the Materials with Your Product as permitted by this license may require you to procure license(s) from one or more third parties that may hold intellectual property rights applicable to any media decoding, encoding or transcoding technology (such as, for example, through use of an audio or video codec) and/or digital rights management capabilities of the Materials, if any. Should any such additional licenses be required, You are solely responsible for obtaining any such licenses and agree to obtain any such licenses at Your own expense. I. MATERIALS TRANSFER: Except for the Pre-Release Licenses or Evaluation Licenses or Non-Commercial Licenses, as specified above, You may permanently transfer the Materials you received pursuant to a license type listed in Section 4(G) above, and all of Your rights under this Agreement, to another party ("Recipient") solely in conjunction with a change of ownership, merger, acquisition, sale or transfer of all or substantially all of Your business or assets, either voluntarily, by operation of law or otherwise subject to the following: You must notify Intel of the transfer by sending a letter to Intel (i) identifying the legal entities of Recipient and You, (ii) identifying the Materials (i.e., the specific Intel software and version) and the associated serial numbers to be transferred, (iii) certifying that You retain no copies of the Materials or portions thereof, (iv) certifying that the Recipient has agreed in writing to be bound by all of the terms and conditions of this Agreement, (v) certifying that the Recipient has been notified that in order to receive support from Intel for the Materials they must notify Intel in writing of the transfer and provide Intel with the information specified in subsection (ii) above along with the name and email address of the individual assigned to use the Materials, and (vi) providing Your email address so that Intel may confirm receipt of Your letter. Please send such letter to: Intel Corporation 2111 NE 25th Avenue Hillsboro, OR 97124 Attn: DPD Contracts Management, JF1-15 4. PRIVACY: A. Data Collection: Based on the personal information You provided to Intel when You registered the license to the Materials with Intel, Intel has collected or will collect certain personal information from You in order to contact You regarding updates to the Materials, and regarding Your experience with obtaining, installing and otherwise using Materials, including sending You surveys to obtain the aforementioned information. B. Revoking Consent to Data Collection: You can revoke Your consent to this collection of personal information at any time by clicking on the link to "unsubscribe" at the bottom of any communication from Intel related to the Materials which will allow You to opt-out of receiving future messages related to the Materials. C. Intel's Privacy Notice: Intel is committed to respecting Your privacy. To learn more about Intel's privacy practices, please visit http://www.intel.com/privacy. 5. OWNERSHIP: Title to the Materials and all copies thereof remain with Intel or its suppliers. The Materials are protected by intellectual property rights, including without limitation, United States copyright laws and international treaty provisions. You will not remove any copyright or other proprietary notice from the Materials. You agree to prevent any unauthorized copying of the Materials. Except as expressly provided herein, no license or right is granted to You directly or by implication, inducement, estoppel or otherwise; specifically Intel does not grant any express or implied right to You under Intel patents, copyrights, trademarks, or trade secrets. 6. NO WARRANTY AND NO SUPPORT: Disclaimer. Intel disclaims all warranties of any kind and the terms and remedies provided in this Agreement are instead of any other warranty or condition, express, implied or statutory, including those regarding merchantability, fitness for any particular purpose, non-infringement or any warranty arising out of any course of dealing, usage of trade, proposal, specification or sample. Intel does not assume (and does not authorize any person to assume on its behalf) any other liability. Intel may make changes to the Materials, or to items referenced therein, at any time without notice, but is not obligated to support, update or provide training for the Materials. Intel may in its sole discretion offer such support, update or training services under separate terms at Intel's then-current rates. You may request additional information on Intel's service offerings from an Intel sales representative. 7. LIMITATION OF LIABILITY: Neither Intel nor its suppliers shall be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other loss) arising out of the use of or inability to use the Materials, even if Intel has been advised of the possibility of such damages. Because some jurisdictions prohibit the exclusion or limitation of liability for consequential or incidental damages, the above limitation may not apply to you. 8. UNAUTHORIZED USE: The Materials are not designed, intended, or authorized for use in any type of a system or application in which the failure of the Materials could create a situation where personal injury or death may occur (e.g., medical systems, life sustaining or lifesaving systems). Should You use the Materials for any such unintended or unauthorized use, You hereby indemnify, defend, and hold Intel and its officers, subsidiaries and affiliates harmless against all claims, costs, damages, expenses, and reasonable attorney fees arising out of, directly or indirectly, such use and any claim of product liability, personal injury or death associated with such unintended or unauthorized use, even if such claim alleges that Intel was negligent regarding the design or manufacture of the Materials. 9. USER SUBMISSIONS: This Agreement does not obligate You to provide Intel with materials, information, comments, suggestions or other communication regarding the Materials. However, You agree that any material, information, comments, suggestions or other communication You transmit or post to an Intel website (including but not limited to, submissions to the Intel Premier Support and/or other customer support websites or online portals) or provide to Intel under this Agreement are not controlled by the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulation (EAR), and if related to the features, functions, performance or use of the Materials are deemed non-confidential and non-proprietary ("Communications"). Intel will have no obligations with respect to the Communications. You hereby grant to Intel a non-exclusive, perpetual, irrevocable, royalty-free, copyright license to copy, modify, create derivative works, publicly display, disclose, distribute, license and sublicense through multiple tiers of distribution and licensees, incorporate and otherwise use the Communications and all data, images, sounds, text, and other things embodied therein, including derivative works thereto, for any and all commercial or non-commercial purposes. You are prohibited from posting or transmitting to or from an Intel website or provide to Intel any unlawful, threatening, libelous, defamatory, obscene, pornographic, or other material that would violate any law. If You wish to provide Intel with information that You intend to be treated as confidential information, Intel requires that such confidential information be provided pursuant to a non-disclosure agreement ("NDA"), so please contact Your Intel representative to ensure the proper NDA is in place. Nothing in this Agreement will be construed as preventing Intel from reviewing Your Communications and errors or defects in Intel products discovered while reviewing Your Communications. Furthermore, nothing in this Agreement will be construed as preventing Intel from implementing independently-developed enhancements to Intel's own error diagnosis methodology to detect errors or defects in Intel products discovered while reviewing Your Communications or to implement bug fixes or enhancements in Intel products. The foregoing may include the right to include Your Communications in regression test suites. 10. NON-DISCLOSURE: The following provisions will apply if there is no existing non-disclosure agreement between You and Intel. You will maintain the confidentiality of the Confidential Information (if any) with at least the same degree of care that You use to protect Your own confidential and proprietary information, but no less than a reasonable degree of care under the circumstances. You will not disclose the Confidential Information to any employees or to any third parties except to Your employees who have a need to know and who agree to abide by nondisclosure terms at least as comprehensive as those set forth herein; provided that You will be liable for breach by any such entity. For the purposes of this Agreement, the term "employee" will include Your independent contractors, who have signed confidentiality agreements with You. You will not make any copies of the Confidential Information except as necessary for Your employees with a need to know. Any copies which are made will be identified as belonging to Intel and marked "confidential", "proprietary" or with similar legend. You will not be liable for the disclosure of any Confidential Information which is (a) generally made available publicly or to third parties by Intel without restriction on disclosure; (b) rightfully received from a third party without obligation of confidentiality; (c) rightfully known to You without any limitation on disclosure prior to Your receipt from Intel; (d) independently developed by Your employees; or (e) required to be disclosed in accordance with applicable laws, regulations, court, judicial or other government order, provided that You will give Intel reasonable notice prior to such disclosure and will comply with any applicable protective order. 11. TERMINATION OF THIS LICENSE: This Agreement becomes effective on the date You accept this Agreement and will continue until terminated as provided for in this Agreement. If You are using the Materials under a time-limited license, for example an Evaluation License, this Agreement terminates without notice on the last day of the time period, which is specified in the Materials or on Intel's website, and/or controlled by the license key code for the Materials. Intel may terminate this license immediately if You are in breach of any of its terms and conditions and such breach is not cured within thirty (30) days of written notice from Intel. Upon termination, You will immediately return to Intel or destroy the Materials and all copies thereof. In the event of termination of this Agreement, the license grant to any Materials or Redistributables distributed by You in accordance with the terms and conditions of this Agreement, prior to the effective date of such termination, will survive any such termination of this Agreement. Sections 1, 4, 5, 6, 7, 8, 9, 10, 11, 12, and 13 will survive expiration or termination of this Agreement. 12. U.S. GOVERNMENT RESTRICTED RIGHTS: The technical data and computer software covered by this license is a "Commercial Item," as such term is defined by the FAR 2.101 (48 C.F.R. 2.101) and is "commercial computer software" and "commercial computer software documentation" as specified under FAR 12.212 (48 C.F.R. 12.212) or DFARS 227.7202 (48 C.F.R. 227.7202), as applicable. This commercial computer software and related documentation is provided to end users for use by and on behalf of the U.S. Government, with only those rights as are granted to all other end users pursuant to the terms and conditions herein. Use for or on behalf of the U.S. Government is permitted only if the party acquiring or using this software is properly authorized by an appropriate U.S. Government official. This use by or for the U.S. Government clause is in lieu of, and supersedes, any other FAR, DFARS, or other provision that addresses Government rights in the computer software or documentation covered by this license. All copyright licenses granted to the U.S. Government are coextensive with the technical data and computer software licenses granted herein. The U.S. Government will only have the right to reproduce, distribute, perform, display, and prepare derivative works as needed to implement those rights. 13. GENERAL PROVISIONS A. ENTIRE AGREEMENT: This Agreement contains the complete and exclusive agreement and understanding between the parties concerning the subject matter of this Agreement, and supersedes all prior and contemporaneous proposals, agreements, understanding, negotiations, representations, warranties, conditions, and communications, oral or written, between the parties relating to the same subject matter. This Agreement, including without limitation its termination, has no effect on any signed non-disclosure agreements between the parties, which remain in full force and effect as separate agreements to their terms. Each party acknowledges and agrees that in entering into this Agreement it has not relied on, and will not be entitled to rely on, any oral or written representations, warranties, conditions, understanding, or communications between the parties that are not expressly set forth in this Agreement. The express provisions of this Agreement control over any course of performance, course of dealing, or usage of the trade inconsistent with any of the provisions of this Agreement. The provisions of this Agreement will prevail notwithstanding any different, conflicting, or additional provisions that may appear on any purchase order, acknowledgement, invoice, or other writing issued by either party in connection with this Agreement. No modification or amendment to this Agreement will be effective unless in writing and signed by authorized representatives of each party, and must specifically identify this Agreement by its title and version (e.g., "End User License Agreement for the Intel(R) Software Development Products (Version March 2016)). If You received a copy of this Agreement translated into another language, the English language version of this Agreement will prevail in the event of any conflict between versions. Intel may make changes to the Agreement as it distributes new versions of the Materials. When these changes are made, Intel will make a new version of the Agreement available on its website: https://software.intel.com/en-us/articles/end-user-license-agreement B. EXPORT. You acknowledge that the Materials and all related technical information are subject to export controls under the laws and regulations of the United States and any other applicable governments. You agree to comply with these laws and regulations governing export, re-export, import, transfer, distribution, and use of the Materials. In particular, but without limitation, the Materials may not be exported or re-exported (a) into any U.S. embargoed countries or (b) to any person or entity listed on a denial order published by the U.S. government or any other applicable governments. By using the Materials, you represent and warrant that you are not located in any such country or on any such list. You also agree that you will not use the Materials for any purposes prohibited by the U.S. government or other applicable governments, including, without limitation, the development, design, manufacture or production of nuclear, missile, chemical or biological weapons. You confirm that the Materials will not be re-exported or sold to a third party who is known or suspected to be involved in activities including, without limitation, the development, design, manufacture, or production of nuclear, missile, chemical or biological weapons. C. GOVERNING LAW, JURISDICTION, AND VENUE: All disputes arising out of or related to this Agreement, whether based on contract, tort, or any other legal or equitable theory, will in all respects be governed by, and construed and interpreted under, the laws of the United States of America and the State of Delaware, without reference to conflict of laws principles. The parties agree that the United Nations Convention on Contracts for the International Sale of Goods (1980) is specifically excluded from and will not apply to this Agreement. All disputes arising out of or related to this Agreement, whether based on contract, tort, or any other legal or equitable theory, will be subject to the exclusive jurisdiction of the courts of the State of Delaware or of the Federal courts sitting in that State. Each party submits to the personal jurisdiction of those courts and waives all objections to that jurisdiction and venue for those disputes. D. SEVERABILITY: The parties intend that if a court holds that any provision or part of this Agreement is invalid or unenforceable under applicable law, the court will modify the provision to the minimum extent necessary to make it valid and enforceable, or if it cannot be made valid and enforceable, the parties intend that the court will sever and delete the provision or part from this Agreement. Any change to or deletion of a provision or part of this Agreement under this Section will not affect the validity or enforceability of the remainder of this Agreement, which will continue in full force and effect. Document Title and Version: End User License Agreement for the Intel(R) Software Development Products (Version March 2016) * Other names and brands may be claimed as the property of others -------------------------------------------------------------------------------- Do you agree to be bound by the terms and conditions of this license agreement? Type 'accept' to continue or 'decline' to go back to the previous menu: accept
选择使用 evaluation 授权
Step 3 of 6 | Activation -------------------------------------------------------------------------------- If you have purchased this product and have the serial number and a connection to the internet you can choose to activate the product at this time. Alternatively, you can choose to evaluate the product or defer activation by choosing the evaluate option. Evaluation software will time out in about one month. You can also use license file or Intel(R) Software License Manager. -------------------------------------------------------------------------------- 1. Use existing trial license (31 day(s) left) [default] 2. I want to activate my product using a serial number 3. I want to activate by using a license file, or by using Intel(R) Software License Manager h. Help b. Back to the previous menu q. Quit -------------------------------------------------------------------------------- Please type a selection or press "Enter" to accept default choice [1]:
选择完成配置安装
Step 4 of 6 | Options > Configure Cluster Installation -------------------------------------------------------------------------------- This product can be installed on cluster nodes. -------------------------------------------------------------------------------- 1. Finish configuring installation target [default] 2. Installation target [ Current system only ] h. Help b. Back to the previous menu q. Quit -------------------------------------------------------------------------------- Please type a selection or press "Enter" to accept default choice [1]:
选择 start installation， 记录安装路径 /opt/intel
Step 4 of 6 | Options > Pre-install Summary -------------------------------------------------------------------------------- Install location: /opt/intel Component(s) selected: Intel(R) MPI Library 5.1 Update 3 763MB Intel MPI Benchmarks Intel MPI Library, Runtime Environment for applications running on Intel(R) 64 Architecture Intel MPI Library for applications running on Intel(R) 64 Architecture Intel MPI Library for applications running on Intel(R) Many Integrated Core Architecture Install Space Required: 840MB Installation target: Install on the current system only 1. Start installation Now [default] 2. Customize installation h. Help b. Back to the previous menu q. Quit -------------------------------------------------------------------------------- Please type a selection or press "Enter" to accept default choice [1]:
安装完毕确认
Step 5 of 6 | Installation -------------------------------------------------------------------------------- Each component will be installed individually. If you cancel the installation, some components might remain on your system. This installation may take several minutes, depending on your system and the options you selected. -------------------------------------------------------------------------------- Installing Intel MPI Benchmarks component... done -------------------------------------------------------------------------------- Installing Intel MPI Library, Runtime Environment for applications running on Intel(R) 64 Architecture component... done -------------------------------------------------------------------------------- Installing Intel MPI Library for applications running on Intel(R) 64 Architecture component... done -------------------------------------------------------------------------------- Installing Intel MPI Library for applications running on Intel(R) Many Integrated Core Architecture component... done -------------------------------------------------------------------------------- Finalizing product configuration... -------------------------------------------------------------------------------- Press "Enter" key to continue
完成安装
Step 6 of 6 | Complete -------------------------------------------------------------------------------- Thank you for installing and for using Intel(R) MPI Library 5.1 Update 3 for Linux*. Support services start from the time you install or activate your product. If you have not already done so, please create your support account now to take full advantage of your product purchase. Your support account gives you access to free product updates and upgrades as well as interactive technical support at Intel(R) Premier Support. -------------------------------------------------------------------------------- Press "Enter" key to quit:
6. 允许节点自动 ssh 登陆
通过此步设置以满足后续 benchmark 测试程序可以在主节点上跨节点操作
sudo apt-get nmap sshpass
创建下述脚本 ./user_authentication.sh，并替换下述脚本中 chown azureuser:azureuser /home/azureuser/.ssh/config 的用户名和用户组为当前环境的用户名和用户组
#!/bin/bash # For CentOS user must first install epel-release, sshpass, and nmap (sshpass and nmap are available from epel-release for CentOS) # usage ./user_authentication.sh [username] [password] [internalIP prefix] # ./user_authentication.sh azureuser Azure@123 10.32.0 USER=$1 PASS=$2 IPPRE=$3 HEADNODE=`hostname` mkdir -p .ssh echo -e 'y\n' | ssh-keygen -f .ssh/id_rsa -t rsa -N '' echo 'Host *' >> .ssh/config echo 'StrictHostKeyChecking no' >> .ssh/config chmod 400 .ssh/config chown azureuser:azureuser /home/azureuser/.ssh/config nmap -sn $IPPRE.* | grep $IPPRE. | awk '{print $5}' > nodeips.txt for NAME in `cat nodeips.txt`; do sshpass -p $PASS ssh -o ConnectTimeout=2 $USER@$NAME 'hostname' >> nodenames.txt;done NAMES=`cat nodenames.txt` #names from names.txt file for NAME in $NAMES; do sshpass -p $PASS scp -o "StrictHostKeyChecking no" -o ConnectTimeout=2 /home/$USER/nodenames.txt $USER@$NAME:/home/$USER/ sshpass -p $PASS ssh -o ConnectTimeout=2 $USER@$NAME "mkdir .ssh && chmod 700 .ssh" sshpass -p $PASS ssh -o ConnectTimeout=2 $USER@$NAME "echo -e 'y\n' | ssh-keygen -f .ssh/id_rsa -t rsa -N ''" sshpass -p $PASS ssh -o ConnectTimeout=2 $USER@$NAME 'touch /home/'$USER'/.ssh/config' sshpass -p $PASS ssh -o ConnectTimeout=2 $USER@$NAME 'echo "Host *" > /home/'$USER'/.ssh/config' sshpass -p $PASS ssh -o ConnectTimeout=2 $USER@$NAME 'echo StrictHostKeyChecking no >> /home/'$USER'/.ssh/config' sshpass -p $PASS ssh -o ConnectTimeout=2 $USER@$NAME 'chmod 400 /home/'$USER'/.ssh/config' cat .ssh/id_rsa.pub | sshpass -p $PASS ssh -o ConnectTimeout=2 $USER@$NAME 'cat >> .ssh/authorized_keys' sshpass -p $PASS scp -o "StrictHostKeyChecking no" -o ConnectTimeout=2 $USER@$NAME:/home/$USER/.ssh/id_rsa.pub .ssh/sub_node.pub for SUBNODE in `cat nodeips.txt`; do sshpass -p $PASS ssh -o ConnectTimeout=2 $USER@$SUBNODE 'mkdir -p .ssh' cat .ssh/sub_node.pub | sshpass -p $PASS ssh -o ConnectTimeout=2 $USER@$SUBNODE 'cat >> .ssh/authorized_keys' done sshpass -p $PASS ssh -o ConnectTimeout=2 $USER@$NAME 'chmod 700 .ssh/' sshpass -p $PASS ssh -o ConnectTimeout=2 $USER@$NAME 'chmod 640 .ssh/authorized_keys' done
查找当前虚拟主机所在子网，将主机位删除，保留子网字段
ifconfig eth0 | grep -w inet | awk '{print $2}'
执行 ./user_authentication.sh <myusername> <mypassword> 10.1.3， myusername 填入测试集群中 ssh 登陆的用户名，mypassword 填入测试集群中 ssh 登陆的密码， 后面填入测试集群所在子网
7. 验证配置完成
关联 mpi 程序
source /opt/intel/impi/5.1.3.223/bin64/mpivars.sh
将 vm1ipaddress 和 vm2ipaddress 地址替换为测试集群主机的 vnet 内的地址
mpirun -ppn 1 -n 2 -hosts <vm1ipaddress>,<vm2ipaddress> -env I_MPI_FABRICS=shm:dapl -env I_MPI_DAPL_PROVIDER=ofa-v2-ib0 -env I_MPI_DYNAMIC_CONNECTION=0 hostname
如果运行正常会分别在本机和远端主机上执行 hostname 命令，所以即打印出测试集群中主机的主机名称
8. 进行 Benchmark 测试
Intel 的 MPI 程序包中已经携带安装了 benchmark 工具，可以通过如下指令执行
mpirun -hosts <vm1ipaddress>,<vm2ipaddress> -ppn 1 -n 2 -env I_MPI_FABRICS=shm:dapl -env I_MPI_DAPL_PROVIDER=ofa-v2-ib0 -env I_MPI_DYN AMIC_CONNECTION=0 IMB-MPI1 pingpong
上述命令只是执行了 benchmark 中的 pingpong 测试，该测试可以给出延迟和带宽在 infiniband 网络上的表现，以下是测试环境中的实测结果。 结果中的延迟时间是一个 oneway 的测试延迟，从结果可见 oneway 延迟在 0 字节负载时 1.82 us，带宽最大可到 44Gbps。
MPI benchmark 包含很多其它的测试项目，关于 MPI benchmark 的更多详情可以参阅如下文档： https://software.intel.com/en-us/imb-user-guide-mpi-1-benchmarks
# PingPong #--------------------------------------------------- # Benchmarking PingPong # #processes = 2 #--------------------------------------------------- #bytes #repetitions t[usec] Mbytes/sec 0 1000 1.82 0.00 1 1000 1.85 0.52 2 1000 1.98 0.96 4 1000 1.84 2.08 8 1000 1.84 4.14 16 1000 1.84 8.29 32 1000 2.76 11.05 64 1000 2.77 22.05 128 1000 2.83 43.11 256 1000 2.99 81.60 512 1000 3.13 155.95 1024 1000 3.46 282.40 2048 1000 4.01 487.49 4096 1000 5.29 738.15 8192 1000 6.67 1171.02 16384 1000 8.97 1742.70 32768 1000 14.70 2125.27 65536 640 18.43 3391.43 131072 320 31.41 3980.12 262144 160 54.78 4563.61 524288 80 98.57 5072.55 1048576 40 185.34 5395.56 2097152 20 359.73 5559.79 4194304 10 715.24 5592.50 # All processes entering MPI_Finalize
这个延迟大概是什么量级呢，我们跟某厂家的 40G/100G 限速的以太网交换机比较一下
