BL460c G7 Blade Cannot see NC542m Mezzanine Card

This post explains how to install the mlx4 driver into ESXi 5.1 in order that it can see and communicate with the NC542m 2-port Ethernet adapter card.  The card was fairly new (at the time) and ESXi 5.1 could not see it by default.

ESXi 5.1 Fails to See the Card

I was commissioning an HP BL460c G7 blade equipped with an NC542m mezzanine card for a recent client.  This card is a 2 port Ethernet adapter including Flex10 technology.  In combination with a Virtual Connect Flex 10 module, it provides the blade with 2 x 4 = 8 Ethernet NICs.  The card was physically installed in mezzanine slot 2, ie. the slot furthest from the motherboard.

I installed ESXi 5.1 on the blade and used the simple ESXi console interface to look at the available NICs.  It could see only 8, not 16 as expected.  There ought to be 16, because 8 are provided by the 2-port on board LAN module on the motherboard (LOM), and another 8 by the NC542m card.

Missing Driver in ESXi 5.1

Logging directly into the blade IP address with ssh gives a simple Linux/busybox command shell.  Grepping /var/log/messages for the word “claimed” returned only 8 messages, showing a driver claiming the on-board LOM NICs.  Nothing about the NICs on the NC542m card.  That’s evidence of a missing driver.  Under /dev, there were 8 devices (/dev/vmnic0, vmnic1, …vmnic7) for the LOM NICs, but non for the card NICs, which further suggested a missing driver.

The ESXi 5.1 command “esxcfg-nics -l” returned only 8 NICs too, further confirming that ESXi could indeed see only 8 NICs.

However the “lspci” command returned all 16 NICs, showing that blade could see all of them at the hardware level.  It showed the 8 Emulex NICs (as seen in the esxcfg-nics -l output), and another 8 “Mellanox Technologies” NICs.  These last are on the Mezzanine NC542m card.

NOTE: There is a switch “-n” that will cause the lspci command to show vendor and product codes instead of expanding them into strings.  The codes can be useful when looking for drivers, as some sites ask for them.  In this example, 153b is the vendor code, which corresponds to Giganet.

Eg.

# lspci -n | grep 00:09
00:09:00.0 Class 0200: 15b3:6764 [vmnic8]
00:09:00.1 Class 0200: 15b3:6764 [vmnic9]
00:09:00.2 Class 0200: 15b3:6764 [vmnic10]
…etc

mlx4 Driver

Internet research showed that ESXi requires a driver called mlx4 to communicate with the NC542m card.  The driver was present on some of the client’s other BL460c blades. Listing the drivers on the blade of interest (in the Busybox shell) did not show mlx4 loaded:

# vmkload_mod -l | grep mlx

Trying to load the driver (vmkload_mod) resulted in a message saying it was not found.

Installing the mlx4 Driver

I downloaded the driver from HP. The file name was mlx4_en-mlnx-1.6.1.2-471530.zip

I put the zip file on the system and unzipped it.  Then performed the installation by carefully following the short procedure in the README file, which amounted to:

1. Copy driver file to the ESXi host of interest (ie the blade), say into /tmp
2. Unzip it
3. Install with command

# esxcli software vib install -v  /tmp/net-mlx4-en-1.6.1.2-1OEM.500.0.0.406165.x86_64.vib
Installation Result
Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
Reboot Required: true
VIBs Installed: Mellanox_bootbank_net-mlx4-en_1.6.1.2-1OEM.500.0.0.406165
VIBs Removed:
VIBs Skipped:

4. Reboot the blade.

Which fixed the problem. All NICs visible.  The commands above that showed lack of NICs, driver etc all now all show full complement. The previously missing NICs now correctly report as Mellanox under ESXi, as they did for lspci.

esxcfg-nics -l now showed all 16 NICs instead of just 8.  Also there were more /dev/vmnic files, and the kernel log showed the mlx4 driver claiming the hardware on boot.

Name    PCI           Driver      Link Speed     Duplex MAC Address       MTU    Description
vmnic0  0000:02:00.00 be2net      Up   5000Mbps  Full   00:17:a4:76:44:10 1500   Emulex Corporation NC553i 10Gb 2-port FlexFabric Converged Network Adapter
vmnic1  0000:02:00.01 be2net      Up   5000Mbps  Full   00:17:a4:76:44:12 1500   Emulex Corporation NC553i 10Gb 2-port FlexFabric Converged Network Adapter
vmnic10 0000:09:00.02 mlx4_en     Up   5000Mbps  Full   00:17:a4:76:44:1c 1500   Mellanox Technologies MT26468 [ConnectX EN 10GigE, PCIe 2.0 5GT/s Virtualization+]
vmnic11 0000:09:00.03 mlx4_en     Up   5000Mbps  Full   00:17:a4:76:44:1e 1500   Mellanox Technologies MT26468 [ConnectX EN 10GigE, PCIe 2.0 5GT/s Virtualization+]
vmnic12 0000:09:00.04 mlx4_en     Down 0Mbps     Half   78:e7:d1:68:be:52 1500   Mellanox Technologies MT26468 [ConnectX EN 10GigE, PCIe 2.0 5GT/s Virtualization+]
vmnic13 0000:09:00.05 mlx4_en     Down 0Mbps     Half   78:e7:d1:68:be:5a 1500   Mellanox Technologies MT26468 [ConnectX EN 10GigE, PCIe 2.0 5GT/s Virtualization+]
vmnic14 0000:09:00.06 mlx4_en     Down 0Mbps     Half   78:e7:d1:68:be:53 1500   Mellanox Technologies MT26468 [ConnectX EN 10GigE, PCIe 2.0 5GT/s Virtualization+]
vmnic15 0000:09:00.07 mlx4_en     Down 0Mbps     Half   78:e7:d1:68:be:5b 1500   Mellanox Technologies MT26468 [ConnectX EN 10GigE, PCIe 2.0 5GT/s Virtualization+]
vmnic2  0000:02:00.02 be2net      Up   5000Mbps  Full   00:17:a4:76:44:18 1500   Emulex Corporation NC553i 10Gb 2-port FlexFabric Converged Network Adapter
vmnic3  0000:02:00.03 be2net      Up   5000Mbps  Full   00:17:a4:76:44:1a 1500   Emulex Corporation NC553i 10Gb 2-port FlexFabric Converged Network Adapter
vmnic4  0000:02:00.04 be2net      Down 0Mbps     Half   80:c1:6e:62:f3:92 1500   Emulex Corporation NC553i 10Gb 2-port FlexFabric Converged Network Adapter
vmnic5  0000:02:00.05 be2net      Down 0Mbps     Half   80:c1:6e:62:f3:96 1500   Emulex Corporation NC553i 10Gb 2-port FlexFabric Converged Network Adapter
vmnic6  0000:02:00.06 be2net      Down 0Mbps     Half   80:c1:6e:62:f3:93 1500   Emulex Corporation NC553i 10Gb 2-port FlexFabric Converged Network Adapter
vmnic7  0000:02:00.07 be2net      Down 0Mbps     Half   80:c1:6e:62:f3:97 1500   Emulex Corporation NC553i 10Gb 2-port FlexFabric Converged Network Adapter
vmnic8  0000:09:00.00 mlx4_en     Up   5000Mbps  Full   00:17:a4:76:44:14 1500   Mellanox Technologies MT26468 [ConnectX EN 10GigE, PCIe 2.0 5GT/s Virtualization+]
vmnic9  0000:09:00.01 mlx4_en     Up   5000Mbps  Full   00:17:a4:76:44:16 1500   Mellanox Technologies MT26468 [ConnectX EN 10GigE, PCIe 2.0 5GT/s Virtualization+]

Leave a Reply

Your email address will not be published. Required fields are marked *