mst status
MST modules:
------------
    MST PCI module is not loaded
    MST PCI configuration module is not loaded

PCI Devices:
------------

	No devices were found.

因此，对于这些老的网卡，要安装旧的版本，如 4.24 版本。

对于 cx4 / cx5 等新一点的网卡，可以安装最新版本。

安装

windows安装

直接打开下载的 WinMFT_x64_4_24_0_72.exe 安装即可，默认安装路径为 C:\Program Files\Mellanox\WinMFT

linux安装

以 ubuntu 为例，

$ wget --no-check-certificate https://www.mellanox.com/downloads/MFT/mft-4.24.0-72-x86_64-deb.tgz
$ tar xvf mft-4.24.0-72-x86_64-deb.tgz
$ cd mft-4.24.0-72-x86_64-deb
# 如果缺少依赖包会导致安装失败，可以先apt命令安装以下包
$ sudo apt-get install gcc make dkms
$ sudo ./install.sh
-I- Removing mft external packages installed on the machine
-I- Installing package: /home/sky/hp544/mft-4.24.0-72-x86_64-deb/SDEBS/kernel-mft-dkms_4.24.0-72_all.deb
-I- Installing package: /home/sky/hp544/mft-4.24.0-72-x86_64-deb/DEBS/mft_4.24.0-72_amd64.deb
-I- In order to start mst, please run "mst start".

重新安装

在 pve 下遇到安装时正常使用，后续不知道为什么突然无法使用了，报错如下：

mst start  
Starting MST (Mellanox Software Tools) driver set
Loading MST PCI modulemodprobe: FATAL: Module mst_pci not found in directory /lib/modules/6.5.13-1-pve
 - Failure: 1
Loading MST PCI configuration modulemodprobe: FATAL: Module mst_pciconf not found in directory /lib/modules/6.5.13-1-pve
 - Failure: 1
Create devices

mst_pci driver not found
Unloading MST PCI module (unused)modprobe: FATAL: Module mst_pci not found.
 - Failure: 1
Unloading MST PCI configuration module (unused)modprobe: FATAL: Module mst_pciconf not found.
 - Failure: 1

解决的方式是重新安装 mft 工具，就可以恢复正常使用。

启动

在 linux 下需要运行如下命令启动 mst （windows 下不需要）：

sudo mst start

Starting MST (Mellanox Software Tools) driver set
Loading MST PCI module - Success
Loading MST PCI configuration module - Success
Create devices

使用

所有工具都有权限要求，在 windows 下打开 cmd 时必须选择以管理员身份运行 cmd ，linux下需要用到 sudo。

mst工具

mst status 用来查看当前的网卡情况，linux下是这样：

sudo mst status
MST modules:
------------
    MST PCI module loaded
    MST PCI configuration module loaded

MST devices:
------------
/dev/mst/mt4103_pciconf0         - PCI configuration cycles access.
                                   domain:bus:dev.fn=0000:04:00.0 addr.reg=88 data.reg=92 cr_bar.gw_offset=-1
                                   Chip revision is: 00
/dev/mst/mt4103_pciconf1         - PCI configuration cycles access.
                                   domain:bus:dev.fn=0000:14:00.0 addr.reg=88 data.reg=92 cr_bar.gw_offset=-1
                                   Chip revision is: 00
/dev/mst/mt4103_pciconf2         - PCI configuration cycles access.
                                   domain:bus:dev.fn=0000:1d:00.0 addr.reg=88 data.reg=92 cr_bar.gw_offset=-1
                                   Chip revision is: 00
/dev/mst/mt4103_pci_cr0          - PCI direct access.
                                   domain:bus:dev.fn=0000:04:00.0 bar=0xfe700000 size=0x100000
                                   Chip revision is: 00
/dev/mst/mt4103_pci_cr1          - PCI direct access.
                                   domain:bus:dev.fn=0000:14:00.0 bar=0xfd900000 size=0x100000
                                   Chip revision is: 00
/dev/mst/mt4103_pci_cr2          - PCI direct access.
                                   domain:bus:dev.fn=0000:1d:00.0 bar=0xfd100000 size=0x100000
                                   Chip revision is: 00

我这里插了三块卡，所以出来的信息比较多，如果只有一块网卡，则应该是这样：

sudo mst status
MST modules:
------------
    MST PCI module loaded
    MST PCI configuration module loaded

MST devices:
------------
/dev/mst/mt4103_pciconf0         - PCI configuration cycles access.
                                   domain:bus:dev.fn=0000:04:00.0 addr.reg=88 data.reg=92 cr_bar.gw_offset=-1
                                   Chip revision is: 00
                                   Chip revision is: 00
/dev/mst/mt4103_pci_cr0          - PCI direct access.
                                   domain:bus:dev.fn=0000:04:00.0

windows下输出内容类似，只是 devices 信息中，"/dev/mst/mt4103_pci_cr0" 会变成 “mt4103_pci_cr0”。

mst status

MST devices:
------------
  mt4103_pci_cr0
  mt4103_pciconf0

mlxfwmanager工具

mlxfwmanager工具可以用来查询当前固件信息：

sudo mlxfwmanager
Querying Mellanox devices firmware ...

Device #1:
----------

  Device Type:      ConnectX3Pro
  Part Number:      764285-B21_Ax
  Description:      HP InfiniBand FDR/Ethernet 10Gb/40Gb 2-port 544+FLR-QSFP Adapter
  PSID:             HP_1380110017
  PCI Device Name:  /dev/mst/mt4103_pci_cr2
  Port1 MAC:        e0071b783ea1
  Port2 MAC:        e0071b783ea2
  Versions:         Current        Available     
     FW             2.42.5700      N/A           
     CLP            8025           N/A           
     PXE            3.4.0754       N/A           
     UEFI           14.11.0049     N/A           

  Status:           No matching image found

FW 2.42.5700 N/A 这里显示的就是当前设置的固件版本。5700是最新的固件版本，有些老一点的固件会显示为类似 FW 2.42.5016 N/A。

注意：在 linux 下运行时要 sudo，不然没有权限会显示没有设备的信息，容易被误导：

mlxfwmanager  
-E- No devices found or specified, mst might be stopped, run 'mst start' to load MST modules
➜  mft-4.24.0-72-x86_64-deb sudo mlxfwmanager
Querying Mellanox devices firmware ...

类似的，在 windows 下打开 cmd 时必须选择以管理员身份运行 cmd 。

mlxconfig

mlxconfig工具用于修改网卡的设置，常见命令有:

查询设置

windows下:

mlxconfig -d mt4103_pci_cr0 query

linux下:

sudo mlxconfig -d /dev/mst/mt4103_pci_cr0 query

sudo mlxconfig -d /dev/mst/mt4103_pci_cr0 query

Device #1:
----------

Device type:    ConnectX3Pro    
Device:         /dev/mst/mt4103_pci_cr0

Configurations:                                      Next Boot
         SRIOV_EN                                    True(1)         
         NUM_OF_VFS                                  16              
         WOL_MAGIC_EN_P2                             True(1)         
         LINK_TYPE_P1                                ETH(2)          
         PHY_TYPE_P1                                 XFI(2)          
         XFI_MODE_P1                                 _10G(0)         
         FORCE_MODE_P1                               False(0)        
         LINK_TYPE_P2                                ETH(2)          
         PHY_TYPE_P2                                 XFI(2)          
         XFI_MODE_P2                                 _10G(0)         
         FORCE_MODE_P2                               False(0)        
         LOG_BAR_SIZE                                5               
         BOOT_PKEY_P1                                0               
         BOOT_PKEY_P2                                0               
         BOOT_OPTION_ROM_EN_P1                       True(1)         
         BOOT_VLAN_EN_P1                             False(0)        
         BOOT_RETRY_CNT_P1                           0               
         LEGACY_BOOT_PROTOCOL_P1                     PXE(1)          
         BOOT_VLAN_P1                                1               
         BOOT_OPTION_ROM_EN_P2                       True(1)         
         BOOT_VLAN_EN_P2                             False(0)        
         BOOT_RETRY_CNT_P2                           0               
         LEGACY_BOOT_PROTOCOL_P2                     PXE(1)          
         BOOT_VLAN_P2                                1               
         IP_VER_P1                                   IPv4(0)         
         IP_VER_P2                                   IPv4(0)         
         CQ_TIMESTAMP                                True(1)         
         STEER_FORCE_VLAN                            False(0)

重置设置

windows下:

mlxconfig -d mt4103_pci_cr0 reset

linux下:

sudo mlxconfig -d /dev/mst/mt4103_pci_cr0 reset

切换IB/Ethernet模式

windows下:

mlxconfig -d mt4103_pci_cr0 set LINK_TYPE_P1=2  # 1. ib模式 2. eth模式 3. vpi 模式 
mlxconfig -d mt4103_pci_cr0 set LINK_TYPE_P2=2

linux下：

sudo mlxconfig -d /dev/mst/mt4103_pciconf0 set LINK_TYPE_P1=2
sudo mlxconfig -d /dev/mst/mt4103_pciconf0 set LINK_TYPE_P2=2

flint工具

flint工具用于刷新网卡的固件，后续详细介绍。

2.2 - 下载固件

介绍如何下载hp544+网卡的固件

HPE 官方下载

HPE 官方下载页面

Product Detail - HPE FDR InfiniBand Adapters | HPE Support

Select Model 中选择 “HPE InfiniBand FDR/Ethernet 10Gb/40Gb 2-port 544+FLR-QSFP Adapter”，列出来的内容中找到

Firmware for HPE InfiniBand FDR/Ethernet 10Gb/40Gb 2-port 544+FLR-QSFP Adapter 764285-B21

这就是最新的 5700 版本的固件了。

直接下载链接为：

https://downloads.hpe.com/pub/softlib2/software1/pubsw-linux/p84948234/v182403/fw-ConnectX3Pro-rel-2_42_5700-764285-B21_Ax-CLP-8025-UEFI-14.11.49-FlexBoot-3.4.754.tgz

解压缩之后得到的 fw-ConnectX3Pro-rel-2_42_5700-764285-B21_Ax-CLP-8025-UEFI-14.11.49-FlexBoot-3.4.754.bin 文件就可以用于后续的固件刷新了。

2.3 - 刷新固件

介绍如何刷新hp544+网卡的固件

准备工作

首先安装前面的要求安装好 MFT 工具，并下载好需要进行刷新的固件文件。

刷新固件

MFT 中的 flint工具用于刷新网卡的固件。

注意：有权限要求，在 windows 下打开 cmd 时必须选择以管理员身份运行 cmd ，linux下需要用到 sudo。

windows刷新固件

windows 下可以将要刷新的固件文件(如 fw-ConnectX3Pro-rel-2_42_5700-764285-B21_Ax-CLP-8025-UEFI-14.11.49-FlexBoot-3.4.754.bin ) 放在mft的安装目录下，如 C:\Program Files\Mellanox\WinMFT。

执行 flint 命令刷新固件：

cd C:\Program Files\Mellanox\WinMFT

flint -d mt4103_pci_cr0 -i fw-ConnectX3Pro-rel-2_42_5700-764285-B21_Ax-CLP-8025-UEFI-14.11.49-FlexBoot-3.4.754.bin b

    Current FW version on flash:  2.42.5016
    New FW version:               2.42.5700

Burning FS2 FW image without signatures - OK
Restoring signature                     - OK

再次执行 mlxfwmanager 命令查看刷新之后的固件信息：

mlxfwmanager
Querying Mellanox devices firmware ...

Device #1:
----------

  Device Type:      ConnectX3Pro
  Part Number:      764285-B21_Ax
  Description:      HP InfiniBand FDR/Ethernet 10Gb/40Gb 2-port 544+FLR-QSFP Adapter
  PSID:             HP_1380110017
  PCI Device Name:  mt4103_pci_cr0
  Port1 GUID:       24be05ffffbd0801
  Port2 GUID:       24be05ffffbd0802
  Versions:         Current        Available
     FW             2.42.5700      2.42.5700
     CLP            8025           8025
     FW (Running)   2.42.5016      N/A
     PXE            3.4.0754       3.4.0754
     UEFI           14.11.0049     14.11.0049

  Status:           Up to date

可以看到固件已经被刷新到 2.42.5700 版本。

linux下刷新固件

linux 下类似，注意 -d 参数后面的设置信息需要是类似 “/dev/mst/mt4103_pci_cr0” 这样，具体参见 mft status 命令的输出：

sudo flint -d /dev/mst/mt4103_pci_cr0 -i fw-ConnectX3Pro-rel-2_42_5700-764285-B21_Ax-CLP-8025-UEFI-14.11.49-FlexBoot-3.4.754.bin b

重置设置

刷新固件之后，推荐重置网卡设置。windows下:

mlxconfig -d mt4103_pci_cr0 reset

linux下:

sudo mlxconfig -d /dev/mst/mt4103_pci_cr0 reset

切换Ethernet模式

hp544+支持 ib / eth / vpi 三种模式，但一般我固定用 eth 模式（ib模式暂时还不会设置软路由）。这种情况下可以考虑将网卡的工作模式固定为 eth。

windows下:

mlxconfig -d mt4103_pci_cr0 set LINK_TYPE_P1=2  # 1. ib模式 2. eth模式 3. vpi 模式 
mlxconfig -d mt4103_pci_cr0 set LINK_TYPE_P2=2

linux下：

sudo mlxconfig -d /dev/mst/mt4103_pciconf0 set LINK_TYPE_P1=2
sudo mlxconfig -d /dev/mst/mt4103_pciconf0 set LINK_TYPE_P2=2

移除 flexboot

flexboot 支持网卡启动，但目前我没有这方面的需求，而 flexboot 会影响开机启动速度，因此在不需要 flexboot 的情况下可以通过移除 flexboot 来加快开机启动速度。

windows下:

flint -d mt4103_pci_cr0 --allow_rom_change drom

linux下：

sudo flint -d /dev/mst/mt4103_pciconf0 --allow_rom_change drom

2.4 - 修改原厂固件

介绍如何修改原厂固件以便支持特性如单模光模块和56G以太网

说明

原文地址

原文请访问：

544+ flr 解锁56G直连 (github.com)

修改后的固件下载

修改好的固件请在此下载：

链接: https://pan.baidu.com/s/1uiebg1P-tTL1WIuxMgLblQ?pwd=tfgi

简单说明：

powerlevel：支持单模光模块（已验证，可用，但我只测试了40g的光模块）
56kr4: 支持56g 以太网（已验证，可用，但我只测试原厂dac线）
drom：去除 flexboot，这样开机不用在屏幕上显示网卡的字样，可以提供开机速度（应该bios里面也不会增加boot的内容，待验证）

有尝试这几个固件的同学请留言说明验证情况，谢谢！

复制一份原文在此以备不时之需：

原文

需要的工具

固件工具 NVIDIA Firmware Tools (MFT)
固件工具 4.3.0.25版本 NVIDIA Firmware Tools (MFT) 4.3.0.25
HPE固件 fw-ConnectX3Pro-rel-2_42_5700-764285-B21_Ax-CLP-8025-UEFI-14.11.49-FlexBoot-3.4.754.tgz
自定义固件 ConnectX3Pro-rel-2_40_5030.tgz
固件ini修改 mft-scripts

准备工作

下载所有需要的工具，并且安装 NVIDIA Firmware Tools (MFT) 和 NVIDIA Firmware Tools (MFT) 4.3.0.25 将 ConnectX3Pro-rel-2_40_5030.tgz 里的 fw-ConnectX3Pro-rel.mlx 和 MCX354A-FCC_Ax.ini 复制到 NVIDIA Firmware Tools (MFT) 4.3.0.25 安装目录我安装在了 C:\Program Files\Mellanox\WinMFT_x64_4_3_0_25 再复制一份 MCX354A-FCC_Ax.ini 命名为 MCX354A-FCC_Ax_56G.ini

再将 fw-ConnectX3Pro-rel-2_42_5700-764285-B21_Ax-CLP-8025-UEFI-14.11.49-FlexBoot-3.4.754.tgz 里的 fw-ConnectX3Pro-rel-2_42_5700-764285-B21_Ax-CLP-8025-UEFI-14.11.49-FlexBoot-3.4.754.bin 和 mft-scripts 里的 fs2_update_ini.py 复制到 NVIDIA Firmware Tools (MFT) 安装目录我安装在了 C:\Program Files\Mellanox\WinMFT

生成固件

修改 MCX354A-FCC_Ax_56G.ini 内 [IB] 部分

port1_802_3ap_56kr4_ability = true
port2_802_3ap_56kr4_ability = true

port1_802_3ap_cr4_enable = true
port2_802_3ap_cr4_enable = true
port1_802_3ap_cr4_ability = true
port2_802_3ap_cr4_ability = true

port1_802_3ap_kr4_enable = true
port2_802_3ap_kr4_enable = true
port1_802_3ap_kr4_ability = true
port2_802_3ap_kr4_ability = true

改为

port1_802_3ap_56kr4_enable = true
port2_802_3ap_56kr4_enable = true
port1_802_3ap_56kr4_ability = true
port2_802_3ap_56kr4_ability = true

port1_802_3ap_cr4_enable = true
port2_802_3ap_cr4_enable = true
port1_802_3ap_cr4_ability = true
port2_802_3ap_cr4_ability = true

port1_802_3ap_kr4_enable = true
port2_802_3ap_kr4_enable = true
port1_802_3ap_kr4_ability = true
port2_802_3ap_kr4_ability = true

分别生成两个固件, 开启56G和不开启的

C:\Program Files\Mellanox\WinMFT_x64_4_3_0_25> mlxburn -fw fw-ConnectX3Pro-rel.mlx -c MCX354A-FCC_Ax_56G.ini -wrimage MCX354A-FCC_Ax_56G.bin
-W- Removing parameter defined outside a group: "prepMLX version".
-I- Generating image ...
-I- Image generation completed successfully.

C:\Program Files\Mellanox\WinMFT_x64_4_3_0_25> mlxburn -fw fw-ConnectX3Pro-rel.mlx -c MCX354A-FCC_Ax.ini -wrimage MCX354A-FCC_Ax.bin
-W- Removing parameter defined outside a group: "prepMLX version".
-I- Generating image ...
-I- Image generation completed successfully.

分析固件

通过 UltraCompare 对比两个固件, 一共有四处改动

第一处在头部

MCX354A-FCC_Ax_56G.bin MCX354A-FCC_Ax.bin

00000020h: 00 00 68 E6 00 00 00 04 F5 00 00 0B FD 00 3B C8 ; 00000020h: 00 00 32 AE 00 00 00 04 F5 00 00 0B FD 00 3B C8 ;

00000030h: 00 0A 99 48 00 00 3B 84 00 10 00 40 00 00 01 ; 00000030h: 00 0A 99 44 00 00 3B 84 00 10 00 40 00 00 01 85 ;

MCX354A-FCC_Ax_56G.bin	MCX354A-FCC_Ax.bin
00000020h: 00 00 68 E6 00 00 00 04 F5 00 00 0B FD 00 3B C8 ;	00000020h: 00 00 32 AE 00 00 00 04 F5 00 00 0B FD 00 3B C8 ;
00000030h: 00 0A 99 48 00 00 3B 84 00 10 00 40 00 00 01 ;	00000030h: 00 0A 99 44 00 00 3B 84 00 10 00 40 00 00 01 85 ;

第二处在文件靠后位置

MCX354A-FCC_Ax_56G.bin MCX354A-FCC_Ax.bin

000a7bb0h: 1F 83 F9 00 7F 8F FF 20 00 01 F9 A0 00 8F F0 02 ; 000a7bb0h: 1F 03 F9 00 7F 8F FF 20 00 01 F9 A0 00 8F F0 02 ;

000a7bc0h: 03 8F F0 17 00 01 F9 A4 00 40 00 01 00 D3 01 FF ; 000a7bc0h: 03 8F F0 17 00 01 F9 A4 00 40 00 01 00 D3 01 FF ;

000a7bd0h: 00 01 F9 AC 1F 83 F9 00 7F 8F FF 20 00 01 F9 B0 ; 000a7bd0h: 00 01 F9 AC 1F 03 F9 00 7F 8F FF 20 00 01 F9 B0 ;

MCX354A-FCC_Ax_56G.bin	MCX354A-FCC_Ax.bin
000a7bb0h: 1F 83 F9 00 7F 8F FF 20 00 01 F9 A0 00 8F F0 02 ;	000a7bb0h: 1F 03 F9 00 7F 8F FF 20 00 01 F9 A0 00 8F F0 02 ;
000a7bc0h: 03 8F F0 17 00 01 F9 A4 00 40 00 01 00 D3 01 FF ;	000a7bc0h: 03 8F F0 17 00 01 F9 A4 00 40 00 01 00 D3 01 FF ;
000a7bd0h: 00 01 F9 AC 1F 83 F9 00 7F 8F FF 20 00 01 F9 B0 ;	000a7bd0h: 00 01 F9 AC 1F 03 F9 00 7F 8F FF 20 00 01 F9 B0 ;

第三处在文件末尾前

MCX354A-FCC_Ax_56G.bin MCX354A-FCC_Ax.bin

000a8fe0h: 00 00 96 1F 00 00 00 03 00 00 00 18 00 00 00 00 ; 000a8fe0h: 00 00 22 2D 00 00 00 03 00 00 00 18 00 00 00 ;

MCX354A-FCC_Ax_56G.bin	MCX354A-FCC_Ax.bin
000a8fe0h: 00 00 96 1F 00 00 00 03 00 00 00 18 00 00 00 00 ;	000a8fe0h: 00 00 22 2D 00 00 00 03 00 00 00 18 00 00 00 ;

第四处为文件末尾全部

第四处改动较多, 实际为ini压缩后数据, 第二处改了两个位置, 通过

port1_802_3ap_56kr4_enable = true
port2_802_3ap_56kr4_enable = true

可以猜测, 此处为两个端口的设置项而第一处和第三处则为校验位

继续分析 fw-ConnectX3Pro-rel.mlx, 找到 port1_802_3ap_56kr4_enable相关选项

scratchpad.eth.port[0].mode_40g_is_50g 0x1f99c.5 1 scratchpad.eth.port[0].b0_hw_eye_opener_cfg_measure_time 0x1f99c.8 4 scratchpad.eth.port[0].eth_802_3ap_56kr4_ability 0x1f99c.12 1 scratchpad.eth.port[0].eth_802_3ap_cr4_ability 0x1f99c.13 1 scratchpad.eth.port[0].eth_802_3ap_kr4_ability 0x1f99c.14 1 scratchpad.eth.port[0].eth_802_3ap_kr_ability 0x1f99c.15 1 scratchpad.eth.port[0].eth_802_3ap_kx_ability 0x1f99c.16 1 scratchpad.eth.port[0].eth_802_3ap_kx4_ability 0x1f99c.17 1 scratchpad.eth.port[0].eth_802_3ap_kr2_ability 0x1f99c.18 1 scratchpad.eth.port[0].eth_802_3ap_100M_ability 0x1f99c.19 1 scratchpad.eth.port[0].eth_802_3ap_56kr4_enable 0x1f99c.23 1 scratchpad.eth.port[0].eth_802_3ap_cr4_enable 0x1f99c.24 1 scratchpad.eth.port[0].eth_802_3ap_kr4_enable 0x1f99c.25 1 scratchpad.eth.port[0].eth_802_3ap_kr_enable 0x1f99c.26 1 scratchpad.eth.port[0].eth_802_3ap_kx_enable 0x1f99c.27 1 scratchpad.eth.port[0].eth_802_3ap_kx4_enable 0x1f99c.28 1 scratchpad.eth.port[0].eth_802_3ap_kr2_enable 0x1f99c.29 1 scratchpad.eth.port[0].eth_802_3ap_100M_enable 0x1f99c.30 1

可以看到此项位置为 0x1f99c.23 将第二处改动转为二进制进行对比

1F83 F900

31	30	29	28		27	26	25	24		23	22	21	20		19	18	17	16		15	14	13	12		11	10	9	8		7	6	5	4		3	2	1	0
0	0	0	1		1	1	1	1		1	0	0	0		0	0	1	1		1	1	1	1		1	0	0	1		0	0	0	0		0	0	0	0

1F03 F900

31	30	29	28		27	26	25	24		23	22	21	20		19	18	17	16		15	14	13	12		11	10	9	8		7	6	5	4		3	2	1	0
0	0	0	1		1	1	1	1		0	0	0	0		0	0	1	1		1	1	1	1		1	0	0	1		0	0	0	0		0	0	0	0

很明显, 第23位为 port1_802_3ap_56kr4_enable, 所以只要修改此位就可以不通过 mlxburn 解锁56G

制作固件

先到 NVIDIA Firmware Tools (MFT) 目录提取一份配置文件

C:\Program Files\Mellanox\WinMFT> flint -i fw-ConnectX3Pro-rel-2_42_5700-764285-B21_Ax-CLP-8025-UEFI-14.11.49-FlexBoot-3.4.754.bin dc HP_1380110017.ini

转到 [IB] 项

;;speed flags for port0
cx3_spec1_3_ib_support_port0 = 1
cx3_spec1_2_ib_support_port0 = 1
spec1_3_fdr10_ib_support_port0 = 1
spec1_3_fdr14_ib_support_port0 = 1
port1_802_3ap_56kr4_ability = 1
port1_802_3ap_cr4_ability = 1
port1_802_3ap_cr4_enable  = 1

可以看到缺少了以下三项

port1_802_3ap_56kr4_enable = true
port1_802_3ap_kr4_enable = true
port1_802_3ap_kr4_ability = true

根据上面分析, 缺失的部分为

scratchpad.eth.port[0].eth_802_3ap_kr4_ability 0x1f99c.14 1 scratchpad.eth.port[0].eth_802_3ap_56kr4_enable 0x1f99c.23 1 scratchpad.eth.port[0].eth_802_3ap_kr4_enable 0x1f99c.25 1

此时设置项应为

31	30	29	28		27	26	25	24		23	22	21	20		19	18	17	16		15	14	13	12		11	10	9	8		7	6	5	4		3	2	1	0
0	0	0	1		1	1	0	1		0	0	0	0		0	0	1	1		1	0	1	1		1	0	0	1		0	0	0	0		0	0	0	0

转换成16进制 1D03 B900 使用 UltraEdit 查找 1D03 B900

fw-ConnectX3Pro-rel-2_42_5700-764285-B21_Ax-CLP-8025-UEFI-14.11.49-FlexBoot-3.4.754.bin

000ed760h: 00 01 F9 9C 1D 03 B9 00 7F 8F FF 20 00 01 F9 A0 ;

000ed770h: 00 8F F0 02 03 8F F0 17 00 01 F9 A4 00 40 00 01 ;

000ed780h: 00 D3 01 FF 00 01 F9 AC 1D 03 B9 00 7F 8F FF 20 ;

fw-ConnectX3Pro-rel-2_42_5700-764285-B21_Ax-CLP-8025-UEFI-14.11.49-FlexBoot-3.4.754.bin
000ed760h: 00 01 F9 9C 1D 03 B9 00 7F 8F FF 20 00 01 F9 A0 ;
000ed770h: 00 8F F0 02 03 8F F0 17 00 01 F9 A4 00 40 00 01 ;
000ed780h: 00 D3 01 FF 00 01 F9 AC 1D 03 B9 00 7F 8F FF 20 ;

说明判断正确, 修改设置项

31	30	29	28		27	26	25	24		23	22	21	20		19	18	17	16		15	14	13	12		11	10	9	8		7	6	5	4		3	2	1	0
0	0	0	1		1	1	1	1		1	0	0	0		0	0	1	1		1	1	1	1		1	0	0	1		0	0	0	0		0	0	0	0

转换成16进制 1F83 F900, 与 MCX354A-FCC_Ax_56G.bin 一致使用 UltraEdit 更改 fw-ConnectX3Pro-rel-2_42_5700-764285-B21_Ax-CLP-8025-UEFI-14.11.49-FlexBoot-3.4.754.bin 并重命名为 fw-ConnectX3Pro-rel-2_42_5700-764285-B21_Ax-CLP-8025-UEFI-14.11.49-FlexBoot-3.4.754_56G.bin

fw-ConnectX3Pro-rel-2_42_5700-764285-B21_Ax-CLP-8025-UEFI-14.11.49-FlexBoot-3.4.754_56G.bin

000ed760h: 00 01 F9 9C 1F 83 F9 00 7F 8F FF 20 00 01 F9 A0 ;

000ed770h: 00 8F F0 02 03 8F F0 17 00 01 F9 A4 00 40 00 01 ;

000ed780h: 00 D3 01 FF 00 01 F9 AC 1F 83 F9 00 7F 8F FF 20 ;

fw-ConnectX3Pro-rel-2_42_5700-764285-B21_Ax-CLP-8025-UEFI-14.11.49-FlexBoot-3.4.754_56G.bin
000ed760h: 00 01 F9 9C 1F 83 F9 00 7F 8F FF 20 00 01 F9 A0 ;
000ed770h: 00 8F F0 02 03 8F F0 17 00 01 F9 A4 00 40 00 01 ;
000ed780h: 00 D3 01 FF 00 01 F9 AC 1F 83 F9 00 7F 8F FF 20 ;

此时并不能直接刷固件, 因为前面说了还有校验码

校验固件

C:\Program Files\Mellanox\WinMFT> flint -i fw-ConnectX3Pro-rel-2_42_5700-764285-B21_Ax-CLP-8025-UEFI-14.11.49-FlexBoot-3.4.754_56G.bin v

     FS2 failsafe image. Start address: 0x0. Chunk size 0x80000:

     NOTE: The addresses below are contiguous logical addresses. Physical addresses on
           flash may be different, based on the image start address and chunk size

     /0x00000038-0x0000065b (0x000624)/ (BOOT2) - OK
     /0x0000065c-0x0000297f (0x002324)/ (BOOT2) - OK
     /0x00002980-0x00003923 (0x000fa4)/ (Configuration) - OK
     /0x00003924-0x00047f5f (0x04463c)/ (ROM) - OK
     /0x00047f60-0x00047fa3 (0x000044)/ (GUID) - OK
     /0x00047fa4-0x0004812f (0x00018c)/ (Image Info) - OK
     /0x00048130-0x00055513 (0x00d3e4)/ (DDR) - OK
     /0x00055514-0x00056577 (0x001064)/ (DDR) - OK
     /0x00056578-0x00056967 (0x0003f0)/ (DDR) - OK
     /0x00056968-0x00094fab (0x03e644)/ (DDR) - OK
     /0x00094fac-0x00099e2f (0x004e84)/ (DDR) - OK
     /0x00099e30-0x0009e423 (0x0045f4)/ (DDR) - OK
     /0x0009e424-0x0009ef1b (0x000af8)/ (DDR) - OK
     /0x0009ef1c-0x000cf0ef (0x0301d4)/ (DDR) - OK
     /0x000cf0f0-0x000d2c9b (0x003bac)/ (DDR) - OK
     /0x000d2c9c-0x000e820f (0x015574)/ (DDR) - OK
     /0x000e8210-0x000e8317 (0x000108)/ (DDR) - OK
     /0x000e8318-0x000ed39b (0x005084)/ (DDR) - OK
     /0x000ed39c-0x000eeb97 (0x0017fc)/ (Configuration) /0x000ed39c/ - wrong CRC (exp:0x8e4c, act:0x9008)
-E- FW image verification failed: Bad CRC.. AN HCA DEVICE CAN NOT BOOT FROM THIS IMAGE.

使用 UltraEdit 定位到 0x000eeb96

fw-ConnectX3Pro-rel-2_42_5700-764285-B21_Ax-CLP-8025-UEFI-14.11.49-FlexBoot-3.4.754_56G.bin

000eeb90h: 00 00 00 7F 00 00 90 08 00 00 00 03 00 00 00 18 ;

fw-ConnectX3Pro-rel-2_42_5700-764285-B21_Ax-CLP-8025-UEFI-14.11.49-FlexBoot-3.4.754_56G.bin
000eeb90h: 00 00 00 7F 00 00 90 08 00 00 00 03 00 00 00 18 ;

按照提示修改为

fw-ConnectX3Pro-rel-2_42_5700-764285-B21_Ax-CLP-8025-UEFI-14.11.49-FlexBoot-3.4.754_56G.bin

000eeb90h: 00 00 00 7F 00 00 8E 4C 00 00 00 03 00 00 00 18 ;

fw-ConnectX3Pro-rel-2_42_5700-764285-B21_Ax-CLP-8025-UEFI-14.11.49-FlexBoot-3.4.754_56G.bin
000eeb90h: 00 00 00 7F 00 00 8E 4C 00 00 00 03 00 00 00 18 ;

现在可以把ini文件再更新进去复制 HP_1380110017.ini 为 HP_1380110017_56G.ini, 修改 [IB] 部分

;;speed flags for port0
cx3_spec1_3_ib_support_port0 = 1
cx3_spec1_2_ib_support_port0 = 1
spec1_3_fdr10_ib_support_port0 = 1
spec1_3_fdr14_ib_support_port0 = 1
port1_802_3ap_56kr4_ability = 1
port1_802_3ap_cr4_ability = 1
port1_802_3ap_cr4_enable  = 1

;;speed flags for port1
cx3_spec1_3_ib_support_port1 = 1
cx3_spec1_2_ib_support_port1 = 1
spec1_3_fdr10_ib_support_port1 = 1
spec1_3_fdr14_ib_support_port1 = 1
port2_802_3ap_56kr4_ability = 1
port2_802_3ap_cr4_ability = 1
port2_802_3ap_cr4_enable  = 1

改为

;;speed flags for port0
cx3_spec1_3_ib_support_port0 = 1
cx3_spec1_2_ib_support_port0 = 1
spec1_3_fdr10_ib_support_port0 = 1
spec1_3_fdr14_ib_support_port0 = 1
port1_802_3ap_56kr4_ability = 1
port1_802_3ap_56kr4_enable = 1
port1_802_3ap_cr4_ability = 1
port1_802_3ap_cr4_enable  = 1
port1_802_3ap_kr4_ability = 1
port1_802_3ap_kr4_enable = 1

;;speed flags for port1
cx3_spec1_3_ib_support_port1 = 1
cx3_spec1_2_ib_support_port1 = 1
spec1_3_fdr10_ib_support_port1 = 1
spec1_3_fdr14_ib_support_port1 = 1
port2_802_3ap_56kr4_ability = 1
port2_802_3ap_56kr4_enable = 1
port2_802_3ap_cr4_ability = 1
port2_802_3ap_cr4_enable  = 1
port2_802_3ap_kr4_ability = 1
port2_802_3ap_kr4_enable = 1

替换固件内的ini文件

C:\Program Files\Mellanox\WinMFT> python3 fs2_update_ini.py fw-ConnectX3Pro-rel-2_42_5700-764285-B21_Ax-CLP-8025-UEFI-14.11.49-FlexBoot-3.4.754_56G.bin HP_1380110017_56G.ini

再次校验固件

C:\Program Files\Mellanox\WinMFT> flint -i fw-ConnectX3Pro-rel-2_42_5700-764285-B21_Ax-CLP-8025-UEFI-14.11.49-FlexBoot-3.4.754_56G.bin v

     FS2 failsafe image. Start address: 0x0. Chunk size 0x80000:

     NOTE: The addresses below are contiguous logical addresses. Physical addresses on
           flash may be different, based on the image start address and chunk size

     /0x00000038-0x0000065b (0x000624)/ (BOOT2) - OK
     /0x0000065c-0x0000297f (0x002324)/ (BOOT2) - OK
     /0x00002980-0x00003923 (0x000fa4)/ (Configuration) - OK
     /0x00003924-0x00047f5f (0x04463c)/ (ROM) - OK
     /0x00047f60-0x00047fa3 (0x000044)/ (GUID) - OK
     /0x00047fa4-0x0004812f (0x00018c)/ (Image Info) - OK
     /0x00048130-0x00055513 (0x00d3e4)/ (DDR) - OK
     /0x00055514-0x00056577 (0x001064)/ (DDR) - OK
     /0x00056578-0x00056967 (0x0003f0)/ (DDR) - OK
     /0x00056968-0x00094fab (0x03e644)/ (DDR) - OK
     /0x00094fac-0x00099e2f (0x004e84)/ (DDR) - OK
     /0x00099e30-0x0009e423 (0x0045f4)/ (DDR) - OK
     /0x0009e424-0x0009ef1b (0x000af8)/ (DDR) - OK
     /0x0009ef1c-0x000cf0ef (0x0301d4)/ (DDR) - OK
     /0x000cf0f0-0x000d2c9b (0x003bac)/ (DDR) - OK
     /0x000d2c9c-0x000e820f (0x015574)/ (DDR) - OK
     /0x000e8210-0x000e8317 (0x000108)/ (DDR) - OK
     /0x000e8318-0x000ed39b (0x005084)/ (DDR) - OK
     /0x000ed39c-0x000eeb97 (0x0017fc)/ (Configuration) - OK
     /0x000eeb98-0x000eec0b (0x000074)/ (Jump addresses) - OK
     /0x000eec0c-0x000ef7d7 (0x000bcc)/ (FW Configuration) - OK
     /0x00000000-0x000ef7d7 (0x0ef7d8)/ (Full Image) - OK

-I- FW image verification succeeded. Image is bootable.

全部OK

刷入固件

获取网卡名

C:\Program Files\Mellanox\WinMFT> mst status -v
MST devices:
------------
  mt4103_pci_cr0         bus:dev.fn=2d:00.0
  mt4103_pciconf0        bus:dev.fn=2d:00.0

刷入修改好的固件

C:\Program Files\Mellanox\WinMFT> flint -d mt4103_pci_cr0 -i fw-ConnectX3Pro-rel-2_42_5700-764285-B21_Ax-CLP-8025-UEFI-14.11.49-FlexBoot-3.4.754_56G.bin b

    Current FW version on flash:  2.42.5700
    New FW version:               2.42.5700

    Note: The new FW version is the same as the current FW version on flash.

 Do you want to continue ? (y/n) [n] : y

Burning FS2 FW image without signatures - OK
Restoring signature                     - OK

(可选)删除 FlexBoot

C:\Program Files\Mellanox\WinMFT> flint -d mt4103_pci_cr0 --allow_rom_change drom

-I- Preparing to remove ROM ...
Removing ROM image    - OK
Restoring signature  - OK

重置网卡

C:\Program Files\Mellanox\WinMFT> mlxconfig -d mt4103_pci_cr0 reset

 Reset configuration for device mt4103_pci_cr0? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new configurations.

重启即可开启56G

参考

3 - HP544+ 网卡驱动

介绍HP544+ 网卡驱动的下载和安装

3.1 - windows驱动

下载安装hp544+网卡的windows驱动

下载

下载页面：

Mellanox OFED for Windows - WinOF / WinOF-2 (nvidia.com)

按照页面的提示：

For ConnectX-3 and ConnectX-3 Pro drivers download WinOF.

For ConnectX-4 and onwards adapter cards drivers download WinOF-2.

hp544+网卡属于 ConnectX-3 Pro，因此只能下载 WinOF. 驱动。

windows驱动目前有 53000 和 54000 两个版本，其中 54000 是 Windows Server 2019 专用的，其他操作系统请使用 53000

WinOF v5.50.54000 includes a driver for Windows Server 2019 only. For other OSes, please see WinOF v5.50.53000.

下载链接：

windows10: MLNX_VPI_WinOF-5_50_53000_All_Win2019_x64.exe
windows server 2019: MLNX_VPI_WinOF-5_50_54000_All_win2019_x64.exe
win8.1: MLNX_VPI_WinOF-5_50_53000_All_Win2012R2_x64.exe

安装

windows 10 安装

直接运行下载的 MLNX_VPI_WinOF-5_50_53000_All_Win2019_x64.exe 文件即可。

3.2 - linux驱动

下载安装hp544+网卡的linux驱动

3.2.1 - 下载linux驱动

下载hp544+网卡的linux驱动

下载驱动

下载页面：

Linux InfiniBand Drivers (nvidia.com)

按照页面的提示：

Note: MLNX_OFED 4.9-x LTS should be used by customers who would like to utilize one of the following:

NVIDIA ConnectX-3 Pro
NVIDIA ConnectX-3
NVIDIA Connect-IB
RDMA experimental verbs library (mlnx_lib)
OSs based on kernel version lower than 3.10

Note: All of the above are not available on MLNX_OFED 5.x branch.

hp544+网卡属于 ConnectX-3 Pro，因此只能下载 4.9 驱动。

windows驱动目前有 53000 和 54000 两个版本，其中 54000 是 Windows Server 2019 专用的，其他操作系统请使用 53000

WinOF v5.50.54000 includes a driver for Windows Server 2019 only. For other OSes, please see WinOF v5.50.53000.

下载链接：

Ubuntu 20.04 : MLNX_OFED_LINUX-4.9-6.0.6.0-ubuntu20.04-x86_64.tgz

3.2.2 - ubuntu20.04上安装驱动

在 ubuntu 20.04上安装hp544+网卡驱动

准备工作

检查当前默认驱动：

$ sudo su
$ modinfo mlx4_core | grep version
version:        4.0-0
srcversion:     CD88194143D98D15E719CD7
vermagic:       5.4.0-94-generic SMP mod_unload modversions

$ modinfo mlx4_core | grep ^version:|sed 's/version: * //g'
4.0-0

在操作前，网卡最好连接好网线，否则ifconfig会看不到网卡信息，或者需要加 -a 参数：

$ ifconfig -a

ens1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.0.0.20  netmask 255.255.255.0  broadcast 10.0.0.255
        inet6 fe80::4a0f:cfff:fef7:89c1  prefixlen 64  scopeid 0x20<link>
        ether 48:0f:cf:f7:89:c1  txqueuelen 1000  (Ethernet)
        RX packets 2196  bytes 166931 (166.9 KB)
        RX errors 0  dropped 1577  overruns 0  frame 0
        TX packets 418  bytes 48817 (48.8 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

$ ethtool -i ens1
driver: mlx4_en
version: 4.0-0
firmware-version: 2.42.5700
expansion-rom-version: 
bus-info: 0000:b3:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes

查看HP544+网卡的设备信息：

$ lspci -vvv
$ lspci | grep Mellanox
b3:00.0 Network controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]

$ lspci -vv -s b3:00.0 | grep "Part number" -A 3
			[PN] Part number: 764285-B21
			[EC] Engineering changes: A3
			[SN] Serial number: IL254902M0
			[V0] Vendor specific: Alom FDR x8 13W

更新驱动

从其他机器上复制文件:

scp ./MLNX_OFED_LINUX-4.9-6.0.6.0-ubuntu20.04-x86_64.tgz sky@192.168.0.10:/home/sky

解压缩：

tar -xvf MLNX_OFED_LINUX-4.9-6.0.6.0-ubuntu20.04-x86_64.tgz
cd MLNX_OFED_LINUX-4.9-6.0.6.0-ubuntu20.04-x86_64

执行安装命令：

$ sudo ./mlnxofedinstall --without-fw-update

Logs dir: /tmp/MLNX_OFED_LINUX.18504.logs
General log file: /tmp/MLNX_OFED_LINUX.18504.logs/general.log

Below is the list of MLNX_OFED_LINUX packages that you have chosen
(some may have been added by the installer due to package dependencies):

ofed-scripts
mlnx-ofed-kernel-utils
mlnx-ofed-kernel-dkms
rshim-dkms
iser-dkms
isert-dkms
srp-dkms
libibverbs1
ibverbs-utils
libibverbs-dev
libibverbs1-dbgmlnxofedinstall
libmlx4-1
libmlx4-dev
libmlx4-1-dbg
libmlx5-1
libmlx5-dev
libmlx5-1-dbg
libibumad
libibumad-static
libibumad-devel
ibacm
ibacm-dev
librdmacm1
librdmacm-utils
librdmacm-dev
mstflint
ibdump
libibmad
libibmad-static
libibmad-devel
libopensm
opensm
opensm-doc
libopensm-devel
infiniband-diags
infiniband-diags-compat
mft
kernel-mft-dkms
libibcm1
libibcm-dev
perftest
ibutils2
libibdm1
ibutils
ar-mgr
dump-pr
ibsim
ibsim-doc
ucx
sharp
hcoll
knem-dkms
knem
openmpi
mpitests
libdapl2
dapl2-utils
libdapl-dev
srptools
mlnx-ethtool
mlnx-iproute2

This program will install the MLNX_OFED_LINUX package on your machine.
Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed.
Those packages are removed due to conflicts with MLNX_OFED_LINUX, do not reinstall them.

Do you want to continue?[y/N]:y

Checking SW Requirements...
One or more required packages for installing MLNX_OFED_LINUX are missing.
Attempting to install the following missing packages:
quilt automake dpatch libgfortran4 flex dkms gcc make autoconf chrpath tcl gfortran swig autotools-dev graphviz tk m4 libnl-route-3-200 libltdl-dev debhelper bison pkg-config
Removing old packages...
Installing new packages
Installing ofed-scripts-4.9...
Installing mlnx-ofed-kernel-utils-4.9...
Installing mlnx-ofed-kernel-dkms-4.9...

Installing rshim-dkms-1.18...
Installing iser-dkms-4.9...
Installing isert-dkms-4.9...
Installing srp-dkms-4.9...
Installing libibverbs1-41mlnx1...
Installing ibverbs-utils-41mlnx1...
Installing libibverbs-dev-41mlnx1...
Installing libibverbs1-dbg-41mlnx1...
Installing libmlx4-1-41mlnx1...
Installing libmlx4-dev-41mlnx1...
Installing libmlx4-1-dbg-41mlnx1...
Installing libmlx5-1-41mlnx1...
Installing libmlx5-dev-41mlnx1...
Installing libmlx5-1-dbg-41mlnx1...
Installing libibumad-43.1.1.MLNX20200211.078947f...
Installing libibumad-static-43.1.1.MLNX20200211.078947f...
Installing libibumad-devel-43.1.1.MLNX20200211.078947f...
Installing ibacm-41mlnx1...
Installing ibacm-dev-41mlnx1...
Installing librdmacm1-41mlnx1...
Installing librdmacm-utils-41mlnx1...
Installing librdmacm-dev-41mlnx1...
Installing mstflint-4.14.0...
Installing ibdump-6.0.0...
Installing libibmad-5.4.0.MLNX20190423.1d917ae...
Installing libibmad-static-5.4.0.MLNX20190423.1d917ae...
Installing libibmad-devel-5.4.0.MLNX20190423.1d917ae...
Installing libopensm-5.7.2.MLNX20201014.9378048...
Installing opensm-5.7.2.MLNX20201014.9378048...
Installing opensm-doc-5.7.2.MLNX20201014.9378048...
Installing libopensm-devel-5.7.2.MLNX20201014.9378048...
Installing infiniband-diags-5.6.0.MLNX20200211.354e4b7...
Installing infiniband-diags-compat-5.6.0.MLNX20200211.354e4b7...
Installing mft-4.15.1...
Installing kernel-mft-dkms-4.15.1...
Installing libibcm1-41mlnx1...
Installing libibcm-dev-41mlnx1...
Installing perftest-4.5.0.mlnxlibs...
Installing ibutils2-2.1.1...
Installing libibdm1-1.5.7.1...
Installing ibutils-1.5.7.1...
Installing ar-mgr-1.0...
Installing dump-pr-1.0...
Installing ibsim-0.10...
Installing ibsim-doc-0.10...
Installing ucx-1.8.0...
Installing sharp-2.1.2.MLNX20200428.ddda184...
Installing hcoll-4.4.2968...
Installing knem-dkms-1.1.4.90mlnx1...
Installing knem-1.1.4.90mlnx1...
Installing openmpi-4.0.3rc4...
Installing mpitests-3.2.20...
Installing libdapl2-2.1.10.1.mlnx...
Installing dapl2-utils-2.1.10.1.mlnx...
Installing libdapl-dev-2.1.10.1.mlnx...
Installing srptools-41mlnx1...
Installing mlnx-ethtool-5.4...
Installing mlnx-iproute2-5.4.0...
Selecting previously unselected package mlnx-fw-updater.
(Reading database ... 87526 files and directories currently installed.)
Preparing to unpack .../mlnx-fw-updater_4.9-4.1.7.0_amd64.deb ...
Unpacking mlnx-fw-updater (4.9-4.1.7.0) ...
Setting up mlnx-fw-updater (4.9-4.1.7.0) ...

Added 'RUN_FW_UPDATER_ONBOOT=no to /etc/infiniband/openib.conf

Installation passed successfully
To load the new driver, run:
/etc/init.d/openibd restart

重启设备驱动：

/etc/init.d/openibd restart

Unloading HCA driver:                                      [  OK  ]
Loading HCA driver and Access Layer:                       [  OK  ]

重启机器，然后检验：

$ modinfo mlx4_core | grep version

version:        4.9-6.0.6
srcversion:     B7B1BFEEF8DC7BE5A999C14
vermagic:       5.4.0-94-generic SMP mod_unload modversions

可以看到现在的驱动版本已经从默认安装的 4.0-0 变成了 4.9-6.0.6 。

5.15内核

尝试在 ubuntu 20.04 上更新内核到 5.15.0-58，然后在安装 MLNX_OFED 驱动时报错：

Copying build sources from '/var/lib/dkms/mlnx-ofed-kernel/4.9/build/../build' to '/usr/src/ofa_kernel/5.15.0-58-generic' ...
/bin/cp: cannot stat 'Module*.symvers': No such file or directory

目前 MLNX_OFED 不支持高版本的内核，测试验证过的只有 5.4 内核（也就是 ubuntu 20.04 默认的内核）。如果更新内核，就只能使用默认的 4.0.0 版本的驱动了，无法更新 4.9 版本。貌似 nvidia 也不准备继续支持。

尝试了一下在 5.4 内核上，更新 MLNX_OFED_LINUX-4.9-6.0.6.0 非常顺利。

参考：

总结

hp544+ 这种 cx3 网卡，比较尴尬，驱动更新基本要停了，对新内核的支持也处于放弃状态。目前只能选择：

用 ubuntu 20.04 + 5.4 内核，驱动升级到 MLNX_OFED_LINUX-4.9-6.0.6.0
用 ubuntu 20.04 + 5.15 内核，驱动继续使用默认自带的 4.0.0 版本。

考虑到目前对最新版本内核没有特别的需求，我选择第一个方案，暂时维持系统在最稳定的状态：ubuntu 20.04 + 5.4 内核 + MLNX_OFED_LINUX-4.9-6.0.6.0 ，都是有官方支持做保障的。

参考资料

3.2.3 - linux-mint上安装驱动

在 linux-mint 上安装hp544+网卡驱动

在基于 ubuntu 20.04 内核的 linux mint 20.02 版本上更新驱动。但奈何没能搞定，各种错误，最后放弃，只能继续使用自带的驱动。

以下记录仅作为归档。

直接执行命令会报错，因为默认有linux发行版本的检查：

./mlnxofedinstall
Current operation system is not supported (linuxmint20.2)!

解决的方式之一是通过命令行参数 --distro 传递发行版本信息进去：

./mlnxofedinstall --distro ubuntu20.04

Removing old packages...
Installing new packages
Installing ofed-scripts-4.9...
Installing mlnx-ofed-kernel-utils-4.9...
Installing mlnx-ofed-kernel-dkms-4.9...
Failed to install mlnx-ofed-kernel-dkms DEB
Collecting debug info...
See /tmp/MLNX_OFED_LINUX.11004.logs/mlnx-ofed-kernel-dkms.debinstall.log

遇到报错，/tmp/MLNX_OFED_LINUX.11004.logs/mlnx-ofed-kernel-dkms.debinstall.log 中的信息为:

/usr/bin/dpkg -i --force-confnew --force-confmiss /home/sky/hp544/MLNX_OFED_LINUX-4.9-4.1.7.0-ubuntu20.04-x86_64/DEBS/MLNX_LIBS/mlnx-ofed-kernel-dkms_4.9-OFED.4.9.4.1.7.1_all.deb
Selecting previously unselected package mlnx-ofed-kernel-dkms.
(Reading database ... 322968 files and directories currently installed.)
Preparing to unpack .../mlnx-ofed-kernel-dkms_4.9-OFED.4.9.4.1.7.1_all.deb ...
Unpacking mlnx-ofed-kernel-dkms (4.9-OFED.4.9.4.1.7.1) ...
Setting up mlnx-ofed-kernel-dkms (4.9-OFED.4.9.4.1.7.1) ...
Loading new mlnx-ofed-kernel-4.9 DKMS files...
First Installation: checking all kernels...
Building only for 5.4.0-92-generic
Building for architecture x86_64
Building initial module for 5.4.0-92-generic
Error! Bad return status for module build on kernel: 5.4.0-92-generic (x86_64)
Consult /var/lib/dkms/mlnx-ofed-kernel/4.9/build/make.log for more information.
dpkg: error processing package mlnx-ofed-kernel-dkms (--install):
 installed mlnx-ofed-kernel-dkms package post-installation script subprocess returned error exit status 10
Errors were encountered while processing:
 mlnx-ofed-kernel-dkms

/var/lib/dkms/mlnx-ofed-kernel/4.9/build/make.log 文件中的信息：

Copying build sources from '/var/lib/dkms/mlnx-ofed-kernel/4.9/build/../build' to '/usr/src/ofa_kernel/5.4.0-92-generic' ...
/bin/cp: cannot stat 'Module*.symvers': No such file or directory

检查这台机器上的相关信息：

$ uname --all
Linux skyserver3 5.4.0-94-generic #106-Ubuntu SMP Thu Jan 6 23:58:14 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
$ ls /usr/src/
linux-headers-5.4.0-94  linux-headers-5.4.0-94-generic  mlnx-ofed-kernel-4.9  ofa_kernel  ofa_kernel-4.9
$ ls /lib/modules/
5.4.0-94-generic
$ locate Module.symvers
# 注意这里没有信息

对照成功安装驱动的ubuntu server 20.04 机器上的信息：

$ uname --all
Linux skywork2 5.4.0-94-generic #106-Ubuntu SMP Thu Jan 6 23:58:14 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
$ ls /usr/src/
iser-4.9                linux-headers-5.4.0-94          ofa_kernel-4.9
isert-4.9               linux-headers-5.4.0-94-generic  rshim-1.18
kernel-mft-dkms-4.15.1  mlnx-ofed-kernel-4.9            srp-4.9
knem-1.1.4.90mlnx1      ofa_kernel
$ ls /lib/modules/
5.4.0-94-generic
$ locate Module.symvers
/usr/src/linux-headers-5.4.0-94-generic/Module.symvers
/usr/src/ofa_kernel/5.4.0-94-generic/Module.symvers
/usr/src/ofa_kernel/5.4.0-94-generic/compat/build/Module.symvers

执行 locate Module.symvers 时发现没有列出信息：

$ locate Module.symvers

# 事实上python3已经安装好了
sudo apt-get install python3
# 但是python命令不存在，需要通过python-is-python3来把python命令按照python3来执行
sudo apt-get install python-is-python3
# 安装过程中需要用到 distutils
sudo apt-get install python3-distutils

$ 开始安装
$./mlnxofedinstall  --distro ubuntu20.04 --without-fw-update

Logs dir: /tmp/MLNX_OFED_LINUX.1976.logs
General log file: /tmp/MLNX_OFED_LINUX.1976.logs/general.log

Below is the list of MLNX_OFED_LINUX packages that you have chosen
(some may have been added by the installer due to package dependencies):

ofed-scripts
mlnx-ofed-kernel-utils
mlnx-ofed-kernel-dkms
......
This program will install the MLNX_OFED_LINUX package on your machine.
Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed.
Those packages are removed due to conflicts with MLNX_OFED_LINUX, do not reinstall them.

Do you want to continue?[y/N]:y

Checking SW Requirements...


One or more required packages for installing MLNX_OFED_LINUX are missing.
Attempting to install the following missing packages:
gfortran graphviz tcl swig chrpath dpatch debhelper libltdl-dev libgfortran4 tk automake quilt autotools-dev autoconf


Removing old packages...
Installing new packages
Installing ofed-scripts-4.9...
Installing mlnx-ofed-kernel-utils-4.9...
Installing mlnx-ofed-kernel-dkms-4.9...


Error: mlnx-ofed-kernel-dkms installation failed!
Collecting debug info...

See:
	/tmp/MLNX_OFED_LINUX.1976.logs/mlnx-ofed-kernel-dkms.debinstall.log
Removing newly installed packages...

但奇怪的是 /tmp/MLNX_OFED_LINUX.1976.logs/mlnx-ofed-kernel-dkms.debinstall.log 日志文件中并没有报错，反而是显示 DKMS: install completed.

./mlnxofedinstall  --distro ubuntu20.04 
./mlnxofedinstall  --distro ubuntu20.04 --without-fw-update
./mlnxofedinstall --add-kernel-support --distro ubuntu20.04 --skip-repo

参考 https://docs.nvidia.com/networking/display/MLNXOFEDv494080/Installing+Mellanox+OFED

./mlnxofedinstall --without-dkms --add-kernel-support --kernel 3.13.0-85-generic --without-fw-update --force

失败但是日志没有错误信息，实在无力再拍错了。

退回去用自带 4.0.0 版本驱动，好处就是可以使用比较新的 5.15 内核。

3.3 - esxi驱动

下载安装hp544+网卡的esxi驱动

3.3.1 - 更新esxi 6.7的驱动

下载更新hp544+网卡的esxi 6.7的驱动

现有驱动

esxi 6.7 下默认带有 mlx4 驱动。

在软件包中搜索 mlx 可以看到：

也可以 ssh 之后通过命令查询：

$ esxcli software vib list | grep mlx
net-mlx4-core                  1.9.7.0-1vmw.670.0.0.8169922          VMW      VMwareCertified     2023-03-31
net-mlx4-en                    1.9.7.0-1vmw.670.0.0.8169922          VMW      VMwareCertified     2023-03-31
nmlx4-core                     3.17.13.1-1vmw.670.2.48.13006603      VMW      VMwareCertified     2023-03-31
nmlx4-en                       3.17.13.1-1vmw.670.2.48.13006603      VMW      VMwareCertified     2023-03-31
nmlx4-rdma                     3.17.13.1-1vmw.670.2.48.13006603      VMW      VMwareCertified     2023-03-31
nmlx5-core                     4.17.13.1-1vmw.670.3.73.14320388      VMW      VMwareCertified     2023-03-31
nmlx5-rdma                     4.17.13.1-1vmw.670.2.48.13006603      VMW      VMwareCertified     2023-03-31

可以看到当前版本为 3.17.13.1-1vmw.670.2.48.13006603

下载新驱动

官方下载

mellannox 网站的 exsi 驱动下载页面：

ConnectX® Ethernet Driver for VMware® ESXi Server (nvidia.com)

找到 ESXi6.7 和 ConnectX-3 Pro：

在这里可以看到 ConnectX-3 Pro 最高只支持到 esxi 6.7, **nmlx4_en **驱动最高版本为 3.17.70.1，下载地址为：

Download VMware vCloud Suite - VMware Customer Connect

怎么找到最新版本的驱动？

而 esxi 7.0.2 自带的 nmlx4_en 驱动版本为 3.19.16.8。

我在网上无意间看到的一个版本号是 3.19.70.1，然后靠这个版本号通过google才搜索到 mellannox 网站的下载链接。这个 3.19.70.1 版本是 2020-09-08 发布的。

所以，问题来了？还有没有更新的 nmlx4_en 版本？怎么找出来？

mellannox网站自带的搜索功能实在是太烂了。

下载地址：

Download VMware vSphere - VMware Customer Connect

解压缩

下载下来的文件需要先解压缩，得到里面的 zip 文件，这个文件才可以用于后面的驱动更新，否则会报错。

注意文件名有 offline_bundle 字样：

Mellanox-nmlx4_3.19.70.1-1OEM.670.0.0.8169922-offline_bundle-17262032.zip

更新驱动

通过esxi 的控制台，用数据存储浏览器将下载的文件上传到 datastore1 下。

ssh 登录，然后执行命令：

$ esxcli software vib update -d /vmfs/volumes/d
atastore1/upload/Mellanox-nmlx4_3.19.70.1-1OEM.670.0.0.8169922-offline_bundle-17262032.zip 
Installation Result
   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
   Reboot Required: true
   VIBs Installed: MEL_bootbank_nmlx4-core_3.19.70.1-1OEM.670.0.0.8169922, MEL_bootbank_nmlx4-en_3.19.70.1-1OEM.670.0.0.8169922, MEL_bootbank_nmlx4-rdma_3.19.70.1-1OEM.670.0.0.8169922
   VIBs Removed: VMW_bootbank_nmlx4-core_3.17.13.1-1vmw.670.2.48.13006603, VMW_bootbank_nmlx4-en_3.17.13.1-1vmw.670.2.48.13006603, VMW_bootbank_nmlx4-rdma_3.17.13.1-1vmw.670.2.48.13006603
   VIBs Skipped:

提示需要重启才能生效。

特别注意：这里的文件路径必须是绝对路径，否则会报错说文件找到不到，错误提示如下：

$ cd /vmfs/volumes/datastore1/upload/
$ esxcli software vib update -d Mellanox-nmlx4_
3.19.70.1-1OEM.670.0.0.8169922-offline_bundle-17262032.zip 
 [MetadataDownloadError]
 Could not download from depot at zip:/var/log/vmware/Mellanox-nmlx4_3.19.70.1-1OEM.670.0.0.8169922-offline_bundle-17262032.zip?index.xml, skipping (('zip:/var/log/vmware/Mellanox-nmlx4_3.19.70.1-1OEM.670.0.0.8169922-offline_bundle-17262032.zip?index.xml', '', "Error extracting index.xml from /var/log/vmware/Mellanox-nmlx4_3.19.70.1-1OEM.670.0.0.8169922-offline_bundle-17262032.zip: [Errno 2] No such file or directory: '/var/log/vmware/Mellanox-nmlx4_3.19.70.1-1OEM.670.0.0.8169922-offline_bundle-17262032.zip'"))
        url = zip:/var/log/vmware/Mellanox-nmlx4_3.19.70.1-1OEM.670.0.0.8169922-offline_bundle-17262032.zip?index.xml
 Please refer to the log file for more details.

重启之后验证：

$ esxcli software vib list | grep mlx
nmlx4-core                     3.19.70.1-1OEM.670.0.0.8169922        MEL      VMwareCertified     2023-05-25
nmlx4-en                       3.19.70.1-1OEM.670.0.0.8169922        MEL      VMwareCertified     2023-05-25
nmlx4-rdma                     3.19.70.1-1OEM.670.0.0.8169922        MEL      VMwareCertified     2023-05-25
net-mlx4-core                  1.9.7.0-1vmw.670.0.0.8169922          VMW      VMwareCertified     2023-03-31
net-mlx4-en                    1.9.7.0-1vmw.670.0.0.8169922          VMW      VMwareCertified     2023-03-31
nmlx5-core                     4.17.13.1-1vmw.670.3.73.14320388      VMW      VMwareCertified     2023-03-31
nmlx5-rdma                     4.17.13.1-1vmw.670.2.48.13006603      VMW      VMwareCertified     2023-03-31

可以看到 nmlx4 驱动从 3.17.13.1 升级到了 3.19.70.1。

4 - HP544+ 网卡测速

对 HP544+ 网卡进行速度测试

使用的软件主要是 iperf 和 iperf3，主要原因是某些情况下 iperf3 无法跑满带宽，因此需要补充 iperf 的测试。

软交换机器的测速

有一台技嘉 x99 ud4 机器，插了四块 HP544+ 网卡，都跑在 pcie 3.0 8x 上，cpu 是 e5 2680 v4 14核28线，物理机方式安装 debian 12 操作系统。使用系统默认的驱动。机器 IP 地址为 192.168.0.99 。

安装 iperf 和 iperf3 并设置 iperf3 自动启动。

$ uname -a 
Linux switch99 6.1.0-18-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.76-1 (2024-02-01) x86_64 GNU/Linux

$ iperf --version                               
iperf version 2.1.8 (12 August 2022) pthreads

$ iperf3 --version                      
iperf 3.12 (cJSON 1.7.15)
Linux switch99 6.1.0-18-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.76-1 (2024-02-01) x86_64
Optional features available: CPU affinity setting, IPv6 flow label, SCTP, TCP congestion algorithm setting, sendfile / zerocopy, socket pacing, authentication, bind to device, support IPv4 don't fragment

本机测速

首先尝试 iperf3 访问本机的速度，即 iperf3 服务器端和 iperf3 客户端都在本地运行

$ iperf3 -c 192.168.0.99 -t 30 -i 1 -P 5
Connecting to host 192.168.0.99, port 5201
[  5] local 192.168.0.99 port 43692 connected to 192.168.0.99 port 5201
[  7] local 192.168.0.99 port 43702 connected to 192.168.0.99 port 5201
[  9] local 192.168.0.99 port 43710 connected to 192.168.0.99 port 5201
[ 11] local 192.168.0.99 port 43712 connected to 192.168.0.99 port 5201
[ 13] local 192.168.0.99 port 43716 connected to 192.168.0.99 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   751 MBytes  6.29 Gbits/sec    0    639 KBytes       
[  7]   0.00-1.00   sec   751 MBytes  6.29 Gbits/sec    0    639 KBytes       
[  9]   0.00-1.00   sec   751 MBytes  6.29 Gbits/sec    0    639 KBytes       
[ 11]   0.00-1.00   sec   751 MBytes  6.29 Gbits/sec    0    639 KBytes       
[ 13]   0.00-1.00   sec   751 MBytes  6.29 Gbits/sec    0    639 KBytes       
[SUM]   0.00-1.00   sec  3.67 GBytes  31.5 Gbits/sec    0 
......
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-30.00  sec  22.8 GBytes  6.54 Gbits/sec    0             sender
[  5]   0.00-30.00  sec  22.8 GBytes  6.54 Gbits/sec                  receiver
[  7]   0.00-30.00  sec  22.8 GBytes  6.54 Gbits/sec    0             sender
[  7]   0.00-30.00  sec  22.8 GBytes  6.54 Gbits/sec                  receiver
[  9]   0.00-30.00  sec  22.8 GBytes  6.54 Gbits/sec    0             sender
[  9]   0.00-30.00  sec  22.8 GBytes  6.54 Gbits/sec                  receiver
[ 11]   0.00-30.00  sec  22.8 GBytes  6.54 Gbits/sec    0             sender
[ 11]   0.00-30.00  sec  22.8 GBytes  6.54 Gbits/sec                  receiver
[ 13]   0.00-30.00  sec  22.8 GBytes  6.54 Gbits/sec    0             sender
[ 13]   0.00-30.00  sec  22.8 GBytes  6.54 Gbits/sec                  receiver
[SUM]   0.00-30.00  sec   114 GBytes  32.7 Gbits/sec    0             sender
[SUM]   0.00-30.00  sec   114 GBytes  32.7 Gbits/sec                  receiver

测试出来的带宽只有 32.7 Gbits/sec ，非常的低。

改用 iperf ，启动服务器端

iperf -s -p 10001

开始客户端测试：

iperf -c 192.168.0.99 -t 30 -i 1 -P 10 -p 10001

测试出来的带宽是 286 Gbits/sec，远远大于 iperf3 测试出来的 32.7:

iperf -c 192.168.0.99 -t 10 -i 1 -P 10 -p 10001
------------------------------------------------------------
Client connecting to 192.168.0.99, TCP port 10001
TCP window size: 2.50 MByte (default)
------------------------------------------------------------
[  2] local 192.168.0.99 port 54650 connected with 192.168.0.99 port 10001 (icwnd/mss/irtt=319/32741/20)
[  4] local 192.168.0.99 port 54656 connected with 192.168.0.99 port 10001 (icwnd/mss/irtt=319/32741/24)
[  1] local 192.168.0.99 port 54672 connected with 192.168.0.99 port 10001 (icwnd/mss/irtt=319/32741/26)
[ 10] local 192.168.0.99 port 54716 connected with 192.168.0.99 port 10001 (icwnd/mss/irtt=319/32741/17)
[  3] local 192.168.0.99 port 54666 connected with 192.168.0.99 port 10001 (icwnd/mss/irtt=319/32741/23)
[  5] local 192.168.0.99 port 54680 connected with 192.168.0.99 port 10001 (icwnd/mss/irtt=319/32741/24)
[  6] local 192.168.0.99 port 54690 connected with 192.168.0.99 port 10001 (icwnd/mss/irtt=319/32741/25)
[  8] local 192.168.0.99 port 54724 connected with 192.168.0.99 port 10001 (icwnd/mss/irtt=319/32741/24)
[  7] local 192.168.0.99 port 54700 connected with 192.168.0.99 port 10001 (icwnd/mss/irtt=319/32741/24)
[  9] local 192.168.0.99 port 54730 connected with 192.168.0.99 port 10001 (icwnd/mss/irtt=319/32741/24)
[ ID] Interval       Transfer     Bandwidth
[  5] 0.0000-1.0000 sec  3.36 GBytes  28.8 Gbits/sec
[  7] 0.0000-1.0000 sec  3.33 GBytes  28.6 Gbits/sec
[  4] 0.0000-1.0000 sec  3.26 GBytes  28.0 Gbits/sec
[  2] 0.0000-1.0000 sec  3.33 GBytes  28.6 Gbits/sec
[  3] 0.0000-1.0000 sec  3.23 GBytes  27.7 Gbits/sec
[  9] 0.0000-1.0000 sec  3.25 GBytes  27.9 Gbits/sec
[  1] 0.0000-1.0000 sec  3.24 GBytes  27.8 Gbits/sec
[  8] 0.0000-1.0000 sec  3.35 GBytes  28.8 Gbits/sec
[  6] 0.0000-1.0000 sec  3.28 GBytes  28.2 Gbits/sec
[ 10] 0.0000-1.0000 sec  3.22 GBytes  27.7 Gbits/sec
[SUM] 0.0000-1.0000 sec  32.8 GBytes   282 Gbits/sec
......
[  5] 0.0000-10.0046 sec  33.9 GBytes  29.1 Gbits/sec
[  4] 0.0000-10.0044 sec  33.0 GBytes  28.4 Gbits/sec
[  2] 0.0000-10.0045 sec  33.0 GBytes  28.3 Gbits/sec
[  9] 0.0000-10.0043 sec  33.3 GBytes  28.6 Gbits/sec
[  8] 0.0000-10.0041 sec  33.1 GBytes  28.5 Gbits/sec
[ 10] 0.0000-10.0050 sec  34.1 GBytes  29.3 Gbits/sec
[  3] 0.0000-10.0044 sec  33.0 GBytes  28.3 Gbits/sec
[  6] 0.0000-10.0041 sec  33.1 GBytes  28.4 Gbits/sec
[  1] 0.0000-10.0042 sec  33.3 GBytes  28.6 Gbits/sec
[  7] 0.0000-10.0045 sec  33.5 GBytes  28.7 Gbits/sec
[SUM] 0.0000-10.0007 sec   333 GBytes   286 Gbits/sec

备注

注意并发线程即 -P 参数的影响很大，需要反复测试才能找到测试结果最好的参数。比如同样这台机器测试，-P 不同的结果是这样的：

-P 1: 33.1 Gbits/sec

-P 2: 61.3 Gbits/sec

-P 2: 147 Gbits/sec

-P 8: 242 Gbits/sec

-P 10: 288 Gbits/sec

-P 12: 225 Gbits/sec

-P 20: 163 Gbits/sec

能看到 10 个测试线程可以跑到最高带宽 288 G，之后加大线程反而跑的更慢。

从其他机器访问

从其他机器访问，测试的这几台机器都是安装的 pve 8.1，linux 6.5 内核，同样是采用的操作系统自带的驱动。

物理网卡访问

在 pve 下直接访问软交换机器，此时可以视为两台物理机用网卡直连。

iperf -c 192.168.0.99 -P 10 -t 10 -i 1 -p 10001

测试结果为 47.6 Gbits/sec，根据经验，这是物理机模式下两块 hp544+ 直连能跑出来的最大带宽了。

iperf -c 192.168.0.99 -P 10 -t 10 -i 1 -p 10001 
------------------------------------------------------------
Client connecting to 192.168.0.99, TCP port 10001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  4] local 192.168.0.19 port 54790 connected with 192.168.0.99 port 10001 (icwnd/mss/irtt=14/1448/176)
[  2] local 192.168.0.19 port 54788 connected with 192.168.0.99 port 10001 (icwnd/mss/irtt=14/1448/149)
[  1] local 192.168.0.19 port 54798 connected with 192.168.0.99 port 10001 (icwnd/mss/irtt=14/1448/124)
[  7] local 192.168.0.19 port 54806 connected with 192.168.0.99 port 10001 (icwnd/mss/irtt=14/1448/113)
[  5] local 192.168.0.19 port 54808 connected with 192.168.0.99 port 10001 (icwnd/mss/irtt=14/1448/81)
[  3] local 192.168.0.19 port 54800 connected with 192.168.0.99 port 10001 (icwnd/mss/irtt=14/1448/106)
[ 10] local 192.168.0.19 port 54840 connected with 192.168.0.99 port 10001 (icwnd/mss/irtt=14/1448/91)
[  9] local 192.168.0.19 port 54830 connected with 192.168.0.99 port 10001 (icwnd/mss/irtt=14/1448/49)
[  6] local 192.168.0.19 port 54810 connected with 192.168.0.99 port 10001 (icwnd/mss/irtt=14/1448/86)
[  8] local 192.168.0.19 port 54824 connected with 192.168.0.99 port 10001 (icwnd/mss/irtt=14/1448/71)
[ ID] Interval       Transfer     Bandwidth
[ 10] 0.0000-1.0000 sec   805 MBytes  6.75 Gbits/sec
[  4] 0.0000-1.0000 sec   189 MBytes  1.59 Gbits/sec
[  9] 0.0000-1.0000 sec   804 MBytes  6.75 Gbits/sec
[  1] 0.0000-1.0000 sec   194 MBytes  1.63 Gbits/sec
[  8] 0.0000-1.0000 sec   798 MBytes  6.70 Gbits/sec
[  6] 0.0000-1.0000 sec   362 MBytes  3.04 Gbits/sec
[  7] 0.0000-1.0000 sec   448 MBytes  3.75 Gbits/sec
[  5] 0.0000-1.0000 sec   808 MBytes  6.78 Gbits/sec
[  3] 0.0000-1.0000 sec   445 MBytes  3.74 Gbits/sec
[  2] 0.0000-1.0000 sec   815 MBytes  6.84 Gbits/sec
[SUM] 0.0000-1.0000 sec  5.54 GBytes  47.6 Gbits/sec
......
[ 10] 0.0000-10.0117 sec  7.91 GBytes  6.79 Gbits/sec
[  9] 0.0000-10.0115 sec  7.92 GBytes  6.79 Gbits/sec
[  1] 0.0000-10.0116 sec  2.00 GBytes  1.72 Gbits/sec
[  6] 0.0000-10.0115 sec  3.84 GBytes  3.29 Gbits/sec
[  5] 0.0000-10.0115 sec  7.81 GBytes  6.70 Gbits/sec
[  2] 0.0000-10.0117 sec  7.93 GBytes  6.80 Gbits/sec
[  8] 0.0000-10.0117 sec  7.87 GBytes  6.75 Gbits/sec
[  3] 0.0000-10.0113 sec  3.97 GBytes  3.41 Gbits/sec
[  7] 0.0000-10.0116 sec  4.01 GBytes  3.44 Gbits/sec
[  4] 0.0000-10.0116 sec  2.08 GBytes  1.79 Gbits/sec
[SUM] 0.0000-10.0048 sec  55.3 GBytes  47.5 Gbits/sec

多台类似配置的机器，分别用同样的测试方法，得到的最高测试速度分别是：

skyserver：47.5 Gbits/sec
Skyserver2：43.6 Gbits/sec
skyserver3：47.7 Gbits/sec
Skyserver4：42.2 Gbits/sec
Skyserver5：42.6 Gbits/sec
Skyserver6：42.4 Gbits/sec

同样用 iperf3 测试:

iperf3 -c 192.168.0.99 -P 10 -t 10 -i 1

测试结果是 39.3 Gbits/sec :

iperf3 -c 192.168.0.99 -P 10 -t 10 -i 1        
Connecting to host 192.168.0.99, port 5201
[  5] local 192.168.0.19 port 40664 connected to 192.168.0.99 port 5201
[  7] local 192.168.0.19 port 40672 connected to 192.168.0.99 port 5201
[  9] local 192.168.0.19 port 40674 connected to 192.168.0.99 port 5201
[ 11] local 192.168.0.19 port 40678 connected to 192.168.0.99 port 5201
[ 13] local 192.168.0.19 port 40688 connected to 192.168.0.99 port 5201
[ 15] local 192.168.0.19 port 40700 connected to 192.168.0.99 port 5201
[ 17] local 192.168.0.19 port 40706 connected to 192.168.0.99 port 5201
[ 19] local 192.168.0.19 port 40716 connected to 192.168.0.99 port 5201
[ 21] local 192.168.0.19 port 40730 connected to 192.168.0.99 port 5201
[ 23] local 192.168.0.19 port 40734 connected to 192.168.0.99 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   432 MBytes  3.62 Gbits/sec    0    266 KBytes       
[  7]   0.00-1.00   sec   432 MBytes  3.62 Gbits/sec    0    267 KBytes       
[  9]   0.00-1.00   sec   432 MBytes  3.62 Gbits/sec    0    205 KBytes       
[ 11]   0.00-1.00   sec   431 MBytes  3.61 Gbits/sec    0    214 KBytes       
[ 13]   0.00-1.00   sec   431 MBytes  3.61 Gbits/sec    0    314 KBytes       
[ 15]   0.00-1.00   sec   431 MBytes  3.61 Gbits/sec    0    293 KBytes       
[ 17]   0.00-1.00   sec   432 MBytes  3.62 Gbits/sec    0    233 KBytes       
[ 19]   0.00-1.00   sec   431 MBytes  3.61 Gbits/sec    0    187 KBytes       
[ 21]   0.00-1.00   sec   431 MBytes  3.61 Gbits/sec    0    310 KBytes       
[ 23]   0.00-1.00   sec   431 MBytes  3.61 Gbits/sec    0    315 KBytes       
[SUM]   0.00-1.00   sec  4.22 GBytes  36.1 Gbits/sec    0   
......
[SUM]   0.00-10.00  sec  45.8 GBytes  39.3 Gbits/sec    0             sender
[SUM]   0.00-10.00  sec  45.8 GBytes  39.3 Gbits/sec                  receiver

还是比 iperf 速度慢不少。

sriov 虚拟网卡测试

vmbr 网卡测试

交换速度

其他两台机器通过软交换机相互访问的速度。

skyserver：充当 iperf 服务器端
Skyserver2：42.4 Gbits/sec
skyserver3：36.2 Gbits/sec （和 skyserver 共用同一块网卡，受限于pcie3.0 8x 64g的总带宽）
Skyserver4：42.3 Gbits/sec
Skyserver5：42.2 Gbits/sec
Skyserver6：42.2 Gbits/sec