Roce ofed 环境搭建与测试

Roce ofed环境搭建与测试

一、安装包下载:

mellanox驱动下载地址:

1、进入到网址:https://cn.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed

2、在打开的页面上找到自己平台,如:Linux SW/Drivers,这里以centos 8为例;

 

 

 

3、在页面的下方找到对应的版本进行下载;

 

 

 

这里以tar.gz的格式进行相应的说明使用;

二、安装

1、将下载好的驱动包上传到服务器,上传的步骤这里不叙述;

2、解压上传好的驱动包,等待解压完成;

 [root@localhost ~]# tar xzvf MLNX_OFED_LINUX-5.0-2.1.8.0-rhel8.1-x86_64.tgz

./MLNX_OFED_LINUX-5.0-2.1.8.0-rhel8.1-x86_64/

./MLNX_OFED_LINUX-5.0-2.1.8.0-rhel8.1-x86_64/RPM-GPG-KEY-Mellanox

./MLNX_OFED_LINUX-5.0-2.1.8.0-rhel8.1-x86_64/uninstall.sh

./MLNX_OFED_LINUX-5.0-2.1.8.0-rhel8.1-x86_64/.mlnx

./MLNX_OFED_LINUX-5.0-2.1.8.0-rhel8.1-x86_64/.arch

./MLNX_OFED_LINUX-5.0-2.1.8.0-rhel8.1-x86_64/distro

………………………

3、安装

 [root@localhost ~]# cd MLNX_OFED_LINUX-5.0-2.1.8.0-rhel8.1-x86_64/

 [root@localhost MLNX_OFED_LINUX-5.0-2.1.8.0-rhel8.1-x86_64]# ./mlnxofedinstall

Logs dir: /tmp/MLNX_OFED_LINUX.10300.logs

General log file: /tmp/MLNX_OFED_LINUX.10300.logs/general.log

Verifying KMP rpms compatibility with target kernel...

……………………….

Complete!

等待系统安装完成即可;

 

 

 

三、常用检查配置;

1、InfiniBand 状态:

[root@centos222 ~]# ibstat

CA 'mlx4_0'

        CA type: MT4099

        Number of ports: 1

        Firmware version: 2.42.5000

        Hardware version: 1

        Node GUID: 0xf452140300880760

        System image GUID: 0xf452140300880760

        Port 1:

                State: Active

                Physical state: LinkUp

                Rate: 10

                Base lid: 0

                LMC: 0

                SM lid: 0

                Capability mask: 0x00010000

                Port GUID: 0xf65214fffe880760

                Link layer: Ethernet

2、InfiniBand 状态:

[root@centos222 ~]# ibstatus

Infiniband device 'mlx4_0' port 1 status:

        default gid:     fe80:0000:0000:0000:f652:14ff:fe88:0760

        base lid:        0x0

        sm lid:          0x0

        state:           4: ACTIVE

        phys state:      5: LinkUp

        rate:            10 Gb/sec (1X QDR)

        link_layer:      Ethernet

3、网卡的对应关系:

[root@centos222 ~]# ibdev2netdev

mlx4_0 port 1 ==> enp2s0 (Up)

4、网卡协商相关信息:

[root@centos222 ~]# ethtool enp2s0

Settings for enp2s0:

        Supported ports: [ FIBRE ]

        Supported link modes:   1000baseKX/Full

                                10000baseKR/Full

        Supported pause frame use: Symmetric Receive-only

        Supports auto-negotiation: No

        Supported FEC modes: Not reported

        Advertised link modes:  1000baseKX/Full

                                10000baseKR/Full

        Advertised pause frame use: Symmetric

        Advertised auto-negotiation: No

        Advertised FEC modes: Not reported

        Speed: 10000Mb/s

        Duplex: Full

        Port: Direct Attach Copper

        PHYAD: 0

        Transceiver: internal

        Auto-negotiation: off

        Supports Wake-on: d

        Wake-on: d

        Current message level: 0x00000014 (20)

                               link ifdown

        Link detected: yes

5、网卡支持的gid等相关信息:

[root@centos222 ~]# show_gids

DEV     PORT    INDEX   GID                                     IPv4            VER     DEV

---     ----    -----   ---                                     ------------    ---     ---

mlx4_0  1       0       fe80:0000:0000:0000:f652:14ff:fe88:0760                 v1      enp2s0

n_gids_found=1

6、网卡工作模式:

[root@centos222 ~]# ibstatus

Infiniband device 'mlx4_0' port 1 status:

        default gid:     fe80:0000:0000:0000:f652:14ff:fe88:0760

        base lid:        0x0

        sm lid:          0x0

        state:           4: ACTIVE

        phys state:      5: LinkUp

        rate:            10 Gb/sec (1X QDR)

        link_layer:      Ethernet

查看网卡当前的link 工作模式:

[root@centos7221 ~]# connectx_port_config -s

--------------------------------

Port configuration for PCI device: 0000:86:00.0 is:

eth

--------------------------------

[root@centos7221 ~]# connectx_port_config

 

ConnectX PCI devices :

|----------------------------|

| 1             0000:86:00.0 |

|----------------------------|

 

Before port change:

eth

 

|----------------------------|

| Possible port modes:       |

| 1: Infiniband              |

| 2: Ethernet                |

| 3: AutoSense               |

|----------------------------|

Select mode for port 1 (1,2,3):

按照需要进行选择;

Note:

Connectx-3只支持Ethernet模式

7、解释:

$ ib_send_bw -d mlx5_4 -x 3 //在一边服务器上启动收包测试, 用index3, RoCEv2:这里要注意的是Index 值;

$ sudo ib_send_bw -d mlx5_4 192.168.1.1 --report_gbits -F -x 3 //另外一边发包

Note:

# 记得给你的网卡绑定个IP, 两边能ping通

[root@centos222 ~]# nmcli c modify enp2s0 ipv4.addresses 10.10.10.222/24 autoconnect yes ipv4.method manual

[root@centos222 ~]# nmcli c down enp2s0

Connection 'enp2s0' successfully deactivated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/439)

[root@centos222 ~]# nmcli c reload enp2s0

[root@centos222 ~]# nmcli c up enp2s0

Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/440)

[root@centos222 ~]# ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

       valid_lft forever preferred_lft forever

    inet6 ::1/128 scope host

       valid_lft forever preferred_lft forever

2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000

    link/ether 6c:92:bf:70:97:cc brd ff:ff:ff:ff:ff:ff

    inet 192.168.101.222/24 brd 192.168.101.255 scope global noprefixroute eno1

       valid_lft forever preferred_lft forever

    inet6 fe80::aad6:ae47:a954:b791/64 scope link noprefixroute

       valid_lft forever preferred_lft forever

3: enp4s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000

    link/ether 6c:92:bf:70:97:cd brd ff:ff:ff:ff:ff:ff

4: enp132s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000

    link/ether 68:91:d0:61:57:2e brd ff:ff:ff:ff:ff:ff

5: enp132s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000

    link/ether 68:91:d0:61:57:2f brd ff:ff:ff:ff:ff:ff

6: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000

    link/ether f4:52:14:88:07:60 brd ff:ff:ff:ff:ff:ff

    inet 10.10.10.222/24 brd 10.10.10.255 scope global noprefixroute enp2s0

       valid_lft forever preferred_lft forever

    inet6 fe80::da78:33ac:bf32:8856/64 scope link noprefixroute

       valid_lft forever preferred_lft forever

7: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000

    link/ether 52:54:00:53:81:7e brd ff:ff:ff:ff:ff:ff

    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0

       valid_lft forever preferred_lft forever

8: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master virbr0 state DOWN group default qlen 1000

    link/ether 52:54:00:53:81:7e brd ff:ff:ff:ff:ff:ff

四、QA;

在安装的过程中可能出现很多问题,最常见的就是缺少安装包,可先安装缺少的包,再次安装驱动;

下面是安装驱动必须的包:

tcl tcsh gcc-gfortran tk python36 perl

在centos 上直接联网进行更新安装即可;

yum install tcl tcsh gcc-gfortran tk python36 perl

posted @ 2020-07-09 14:20  橘子飞飞  阅读(2297)  评论(0编辑  收藏  举报