This post explains how to install the mlx4 driver into ESXi 5.1 in order that it can see and communicate with the NC542m 2-port Ethernet adapter card. The card was fairly new (at the time) and ESXi 5.1 could not see it by default.
ESXi 5.1 Fails to See the Card
I was commissioning an HP BL460c G7 blade equipped with an NC542m mezzanine card for a recent client. This card is a 2 port Ethernet adapter including Flex10 technology. In combination with a Virtual Connect Flex 10 module, it provides the blade with 2 x 4 = 8 Ethernet NICs. The card was physically installed in mezzanine slot 2, ie. the slot furthest from the motherboard.
I installed ESXi 5.1 on the blade and used the simple ESXi console interface to look at the available NICs. It could see only 8, not 16 as expected. There ought to be 16, because 8 are provided by the 2-port on board LAN module on the motherboard (LOM), and another 8 by the NC542m card.
Missing Driver in ESXi 5.1
Logging directly into the blade IP address with ssh gives a simple Linux/busybox command shell. Grepping /var/log/messages for the word “claimed” returned only 8 messages, showing a driver claiming the on-board LOM NICs. Nothing about the NICs on the NC542m card. That’s evidence of a missing driver. Under /dev, there were 8 devices (/dev/vmnic0, vmnic1, …vmnic7) for the LOM NICs, but non for the card NICs, which further suggested a missing driver.
The ESXi 5.1 command “esxcfg-nics -l” returned only 8 NICs too, further confirming that ESXi could indeed see only 8 NICs.
However the “lspci” command returned all 16 NICs, showing that blade could see all of them at the hardware level. It showed the 8 Emulex NICs (as seen in the esxcfg-nics -l output), and another 8 “Mellanox Technologies” NICs. These last are on the Mezzanine NC542m card.
NOTE: There is a switch “-n” that will cause the lspci command to show vendor and product codes instead of expanding them into strings. The codes can be useful when looking for drivers, as some sites ask for them. In this example, 153b is the vendor code, which corresponds to Giganet.
Eg.
# lspci -n | grep 00:09
00:09:00.0 Class 0200: 15b3:6764 [vmnic8]
00:09:00.1 Class 0200: 15b3:6764 [vmnic9]
00:09:00.2 Class 0200: 15b3:6764 [vmnic10]
…etc
mlx4 Driver
Internet research showed that ESXi requires a driver called mlx4 to communicate with the NC542m card. The driver was present on some of the client’s other BL460c blades. Listing the drivers on the blade of interest (in the Busybox shell) did not show mlx4 loaded:
# vmkload_mod -l | grep mlx
Trying to load the driver (vmkload_mod) resulted in a message saying it was not found.
Installing the mlx4 Driver
I downloaded the driver from HP. The file name was mlx4_en-mlnx-1.6.1.2-471530.zip
I put the zip file on the system and unzipped it. Then performed the installation by carefully following the short procedure in the README file, which amounted to:
1. Copy driver file to the ESXi host of interest (ie the blade), say into /tmp
2. Unzip it
3. Install with command
# esxcli software vib install -v /tmp/net-mlx4-en-1.6.1.2-1OEM.500.0.0.406165.x86_64.vib
Installation Result
Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
Reboot Required: true
VIBs Installed: Mellanox_bootbank_net-mlx4-en_1.6.1.2-1OEM.500.0.0.406165
VIBs Removed:
VIBs Skipped:
4. Reboot the blade.
Which fixed the problem. All NICs visible. The commands above that showed lack of NICs, driver etc all now all show full complement. The previously missing NICs now correctly report as Mellanox under ESXi, as they did for lspci.
esxcfg-nics -l now showed all 16 NICs instead of just 8. Also there were more /dev/vmnic files, and the kernel log showed the mlx4 driver claiming the hardware on boot.
Name PCI Driver Link Speed Duplex MAC Address MTU Description vmnic0 0000:02:00.00 be2net Up 5000Mbps Full 00:17:a4:76:44:10 1500 Emulex Corporation NC553i 10Gb 2-port FlexFabric Converged Network Adapter vmnic1 0000:02:00.01 be2net Up 5000Mbps Full 00:17:a4:76:44:12 1500 Emulex Corporation NC553i 10Gb 2-port FlexFabric Converged Network Adapter vmnic10 0000:09:00.02 mlx4_en Up 5000Mbps Full 00:17:a4:76:44:1c 1500 Mellanox Technologies MT26468 [ConnectX EN 10GigE, PCIe 2.0 5GT/s Virtualization+] vmnic11 0000:09:00.03 mlx4_en Up 5000Mbps Full 00:17:a4:76:44:1e 1500 Mellanox Technologies MT26468 [ConnectX EN 10GigE, PCIe 2.0 5GT/s Virtualization+] vmnic12 0000:09:00.04 mlx4_en Down 0Mbps Half 78:e7:d1:68:be:52 1500 Mellanox Technologies MT26468 [ConnectX EN 10GigE, PCIe 2.0 5GT/s Virtualization+] vmnic13 0000:09:00.05 mlx4_en Down 0Mbps Half 78:e7:d1:68:be:5a 1500 Mellanox Technologies MT26468 [ConnectX EN 10GigE, PCIe 2.0 5GT/s Virtualization+] vmnic14 0000:09:00.06 mlx4_en Down 0Mbps Half 78:e7:d1:68:be:53 1500 Mellanox Technologies MT26468 [ConnectX EN 10GigE, PCIe 2.0 5GT/s Virtualization+] vmnic15 0000:09:00.07 mlx4_en Down 0Mbps Half 78:e7:d1:68:be:5b 1500 Mellanox Technologies MT26468 [ConnectX EN 10GigE, PCIe 2.0 5GT/s Virtualization+] vmnic2 0000:02:00.02 be2net Up 5000Mbps Full 00:17:a4:76:44:18 1500 Emulex Corporation NC553i 10Gb 2-port FlexFabric Converged Network Adapter vmnic3 0000:02:00.03 be2net Up 5000Mbps Full 00:17:a4:76:44:1a 1500 Emulex Corporation NC553i 10Gb 2-port FlexFabric Converged Network Adapter vmnic4 0000:02:00.04 be2net Down 0Mbps Half 80:c1:6e:62:f3:92 1500 Emulex Corporation NC553i 10Gb 2-port FlexFabric Converged Network Adapter vmnic5 0000:02:00.05 be2net Down 0Mbps Half 80:c1:6e:62:f3:96 1500 Emulex Corporation NC553i 10Gb 2-port FlexFabric Converged Network Adapter vmnic6 0000:02:00.06 be2net Down 0Mbps Half 80:c1:6e:62:f3:93 1500 Emulex Corporation NC553i 10Gb 2-port FlexFabric Converged Network Adapter vmnic7 0000:02:00.07 be2net Down 0Mbps Half 80:c1:6e:62:f3:97 1500 Emulex Corporation NC553i 10Gb 2-port FlexFabric Converged Network Adapter vmnic8 0000:09:00.00 mlx4_en Up 5000Mbps Full 00:17:a4:76:44:14 1500 Mellanox Technologies MT26468 [ConnectX EN 10GigE, PCIe 2.0 5GT/s Virtualization+] vmnic9 0000:09:00.01 mlx4_en Up 5000Mbps Full 00:17:a4:76:44:16 1500 Mellanox Technologies MT26468 [ConnectX EN 10GigE, PCIe 2.0 5GT/s Virtualization+]