Mellanox cx6 ocp3网卡
Mellanox cx6 ocp3 200g网卡
曾经购入过两块cx6 200g网卡，ocp3 接口，用 ocp3转pcie4 转换卡安装到 z690 主板。结果因为 pcie 信号强度问题降速，可惜了。
记录一下当时测试的一些情况，以后买 ocp 网卡需要谨慎。
lspci
插在 pcie 4.0 4x 槽上：
sudo lspci -s 01:00.0 -vvv | grep Width
[sudo] password for sky:
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM not supported
LnkSta: Speed 8GT/s (ok), Width x4 (ok)
插在 pcie 4.0 16x 槽上：
mlx5_core 0000:01:00.0 enp1s0np0: Link up
➜ ~ sudo lspci -s 01:00.0 -vvv | grep Width
LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM not supported
LnkSta: Speed 8GT/s (downgraded), Width x16 (ok)
dmesg
插在 pcie 4.0 4x 槽上：
sudo dmesg | grep mlx5_core
[ 1.437917] mlx5_core 0000:04:00.0: enabling device (0000 -> 0002)
[ 1.438118] mlx5_core 0000:04:00.0: firmware version: 20.27.446
[ 1.438142] mlx5_core 0000:04:00.0: 8.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x4 link at 0000:00:1b.4 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)
[ 1.731931] mlx5_core 0000:04:00.0: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps
[ 1.732211] mlx5_core 0000:04:00.0: E-Switch: Total vports 10, per vport: max uc(1024) max mc(16384)
[ 1.734873] mlx5_core 0000:04:00.0: Port module event: module 0, Cable plugged
[ 1.735122] mlx5_core 0000:04:00.0: mlx5_pcie_event:299:(pid 324): Detected insufficient power on the PCIe slot (25W).
[ 1.741863] mlx5_core 0000:04:00.0: MLX5E: StrdRq(0) RqSz(1024) StrdSz(2048) RxCqeCmprss(1 basic)
[ 1.989891] mlx5_core 0000:04:00.0 eth0: Disabling rxhash, not supported when CQE compress is active
[ 1.990170] mlx5_core 0000:04:00.0 enp4s0np0: renamed from eth0
[ 5.203057] mlx5_core 0000:04:00.0 enp4s0np0: Link down
插在 pcie 4.0 16x 槽上：
sudo dmesg | grep mlx5_core
[ 1.444801] mlx5_core 0000:01:00.0: enabling device (0000 -> 0002)
[ 1.445005] mlx5_core 0000:01:00.0: firmware version: 20.27.446
[ 1.445026] mlx5_core 0000:01:00.0: 126.016 Gb/s available PCIe bandwidth, limited by 8.0 GT/s PCIe x16 link at 0000:00:01.0 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)
[ 1.684398] mlx5_core 0000:01:00.0: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps
[ 1.684600] mlx5_core 0000:01:00.0: E-Switch: Total vports 10, per vport: max uc(1024) max mc(16384)
[ 1.686818] mlx5_core 0000:01:00.0: Port module event: module 0, Cable plugged
[ 1.687055] mlx5_core 0000:01:00.0: mlx5_pcie_event:304:(pid 328): PCIe slot advertised sufficient power (75W).
[ 1.693066] mlx5_core 0000:01:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0 basic)
[ 1.801740] mlx5_core 0000:01:00.0 enp1s0np0: renamed from eth0
[ 5.189354] mlx5_core 0000:01:00.0 enp1s0np0: Link down
[ 21.129045] mlx5_core 0000:01:00.0 enp1s0np0: Link up
mst
插在 pcie 4.0 4x 槽上：
sudo mst status
MST modules:
------------
MST PCI module is not loaded
MST PCI configuration module loaded
MST devices:
------------
/dev/mst/mt4123_pciconf0 - PCI configuration cycles access.
domain:bus:dev.fn=0000:04:00.0 addr.reg=88 data.reg=92 cr_bar.gw_offset=-1
Chip revision is: 00
插在 pcie 4.0 16x 槽上：
sudo mst status
MST modules:
------------
MST PCI module is not loaded
MST PCI configuration module loaded
MST devices:
------------
/dev/mst/mt4123_pciconf0 - PCI configuration cycles access.
domain:bus:dev.fn=0000:01:00.0 addr.reg=88 data.reg=92 cr_bar.gw_offset=-1
Chip revision is: 00
mlxlink
插在 pcie 4.0 4x 槽上：
sudo mlxlink -d /dev/mst/mt4123_pciconf0 --port_type PCIE -e
PCIe Operational (Enabled) Info
-------------------------------
Depth, pcie index, node : 0, 0, 0
Link Speed Active (Enabled) : 2.5G-Gen 1 (16G-Gen 4)
Link Width Active (Enabled) : 4X (16X)
Errors
------
Showing Eye via SLRG raised the following exception: Eye information available for Gen3 and above
降速严重。
插在 pcie 4.0 16x 槽上：
sudo mlxlink -d /dev/mst/mt4123_pciconf0 --port_type PCIE -e
PCIe Operational (Enabled) Info
-------------------------------
Depth, pcie index, node : 0, 0, 0
Link Speed Active (Enabled) : 8G-Gen 3 (16G-Gen 4)
Link Width Active (Enabled) : 16X (16X)
EYE Opening Info (PCIe)
-----------------------
Physical Grade : 1848, 2220, 2146, 2301, 2067, 2052, 1944, 2135, 2035, 1980, 2090, 2340, 1980, 2170, 2196, 2072
Height Eye Opening [mV] : 147, 177, 171, 184, 165, 164, 155, 170, 162, 158, 167, 187, 158, 173, 175, 165
Phase Eye Opening [psec] : 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16
设置 mtu
sudo ifconfig enp1s0np0 mtu 9000 up