Update: ESXi 5.0 on HP G7 blades, now a Go!

About three weeks back I reported on Emulex firmware problems that prevented the use of ESXi 5.0 on HP G7 blade hardware. This was fixed now, somehow...

HP has now updated the advisory that describes the issue and published an updated firmware that fixes the VLAN handling problems with ESXi 5.0 if it is used together with the be2net driver 4.0.355.1.

Be sure that you read the release notes of the firmware! It looks like it is an emergency/workaround release that leaves many issues unresolved. A firmware version that you can really trust for production will probably be available mid-November.

Update (2012-12-09): HP and Emulex published the final version of the OneConnect firmware (4.0.360.15a) on Nov 19th. VMware's KB2007397 also lists the recommended drivers to use with this firmware for both ESXi 4.1 and 5.0.

Update (2012-03-09): HP has published yet another firmware update on March 5th. Download version 4.0.360.15b. The previous link has become invalid.

Update (2012-04-16): Please refer to my HP & VMware links page to find the download for the latest version of the firmware.

VMware finally released the Open Source Code of vSphere 5.0!

Great news! Today VMware finally made the vSphere v5.0 Open Source code archives available for download.

Why is that important?

Since the release of VMware's ESXi 5.0 (Aug 24, 2011) many people are asking for the development of drivers for hardware devices that are not supported by ESXi 5.0 out-of-the-box.

ESXi device drivers are based on Linux device drivers (which lead to the persistent misunderstanding that ESXi itself is based on Linux), but the stock Linux driver code must be modified in a specific way to be compatible with ESXi.

With past versions of ESXi (up to 4.1) it was possible to study and reproduce these required modifications, because VMware published the source code of the ESXi device drivers (the original Linux code plus their modifications). The reason for this is that most Linux drivers are licensed under the GPL (General Public License), and the GPL requires that derived works are also published under the GPL and their source code is made freely available (aka the "Copyleft" principle).

Now, that VMware also published the Open Source Code of ESXi 5.0 (including the device drivers that it contains) it will be possible (or at least much easier) to develop custom ESXi 5.0 drivers for devices that are not officially supported by VMware.

HP Virtual Connect profile not applied ...

When I recently rebooted one of our BL620c G7 blades with ESXi 4.1 installed on it I found that the server had suddenly lost network connectivity after the reboot.
A quick check on the console revealed that the Virtual Connect profile that was defined for that blade had not been applied to it. I realized that because the MAC addresses of the NICs had not been overwritten with the virtual addresses of the Virtual Connect profile.

I tried powering down and up the blade, re-assigning the Virtual Connect profile multiple times, all to no avail ... Then I had the idea that it might be related to the iLO-board of the blade, and - yes, indeed - after resetting the iLO3-board of the blade the Virtual Connect profile was properly applied and all was fine again.

While later looking on hp.com for some related information I stumbled over the Customer advisory c02820591 that described an issue with the Virtual Connect profile being lost upon an iLO3 reset. Not exactly the issue that I had, and the advisory also stated that this is fixed with iLO3 firmware version 1.20, and that is already installed on our iLOs. However, the advisory confirmed my assumption that the Virtual Connect profile is applied by the iLO-board.

So, if you have similar problems try resetting the iLO-board before you start pulling your hair out, or the blade out of the chassis ...

Currently a No-Go: ESXi 5.0 on HP G7 blades

Back in May I reported on problems with ESXi 4.1 and the Emulex OneConnect CNA that is built into HP's G7 blade servers.
If you now try to install ESXi 5.0 on such a hardware you will have a strong déjà vu: The be2net driver that is available right now for ESXi 5.0 is not really functioning due to "VLAN tagging issues". HP has published an advisory on this stating that an updated driver (that should fix these issues) is "currently in the certification process" and will be made available in "Q4 2011".

Okay, I won't update our production hosts to ESXi 5.0 that soon anyway, but I just wanted to install it on some spare blades for testing and evaluation. Too bad ... waiting for a fix again ...

Update (2011-10-27):
HP has now updated the advisory and published an updated firmware that fixes the VLAN handling problems with ESXi 5.0 if it is used together with the be2net driver 4.0.355.1.
Be sure that you read the release notes of the firmware! It looks like it is an emergency/workaround release that leaves many issues unresolved. A firmware version that you can really trust for production will probably be available mid-November.

Update (2012-04-16):
In the meantime it looks like all problems have been fixed with newer firmware and driver versions. Please refer to this newer post of mine!

Unable to assign license after installing a server with the HP ESXi 5.0 ISO

With the availability of vSphere 5.0 HP published a customized ESXi installation ISO for HP servers.
There have been reports that this build includes an annoying bug: HP has included a license file that has wrong permissions set. That potentially causes errors once you want to assign an own license to the host.

You can fix this by removing the offending license file with the following commands (to be executed on the ESXi host after directly after it has been installed):

  esxcli software vib remove -n hp-esx-license --no-live-install

and reboot the host. The command will remove the HP license and restore the original state of the host being in evaluation mode.

HP has also published an advisory describing the problem and providing a way to update their license package to fix the problem.

[Announcement] ESXi-Customizer 2.5 ...

... is out now. See the prior post for what's new. Download it from the project page!

VIB files, Offline-Bundles and ESXi-Customizer 2.5

The current version 2.0 of my ESXi-Customizer script is able to add OEM.tgz-style driver packages to ESXi 4.1 and ESXi 5.0 installation ISOs. Tgz (short for tar.gz) is the format that is used to distribute all community-developed drivers for ESXi 4.1 (and prior versions). So these can currently be added to ESXi 4.1 with my script.

However, for ESXi 5.0 you cannot use the driver packages for ESXi 4.1, they need to be re-engineered and re-compiled starting with the stock Linux driver code that is the basis for ESXi drivers. It looks like - so far - nobody in the user community has figured out how to compile drivers for ESXi 5.0. I hope that this will change in the near future, but for now the current version of the ESXi-Customizer script is pretty much useless for ESXi 5.0 ...

On the other hand, there are new and updated driver packages for ESXi 5.0 already available directly from VMware or 3rd party hardware vendors. These drivers are distributed in VIB format and as so-called Offline-Bundles (in zip-format). I wondered if my script could also support these "official" VMware package format and had a closer look at the format of these files.

(Note: To fully understand the following, it is helpful to read my Anatomy post about the structure of the ESXi 5.0 ISO first!)

What is a VIB file?
VIB stands for "VMware Installation Bundle". If you just open a vib-file in a text editor you will see that it starts with the (somewhat well known) header "!<arch>". This means that the file is in the Unix ar format. Wikipedia has a detailed description of this file format, and there you can learn that VIB files use the common ar format, and that you can use the Unix ar command to handle this kind of archive files.
Fortunately, 7-zip - the Swiss army knife of packaging tools - is also able to handle ar-files, and it is also available for Windows. Since I already use it with ESXi-Customizer to unpack ISO-files I was delighted to realize that it is also able to unpack a vib-file ...

So, what's in the VIB file? Exactly three files:

1. A file named "descriptor.xml": The name says it all. This file describes the contents and dependencies of the driver package. Later you will find that again in the IMGDB.TGZ file. For more details see section 4 of my anatomy-post. There you will find the descriptor.xml file of the e1000-driver as an example.
2. A file named "sig.pkcs7": VMware certified drivers need to be electronically signed, and this file includes the uuenconded PKCS7 signature.
3. The payload file: You will find the exact name of this file in the name-attribute of the <payload> tag in the descriptor.xml file. It does not have any file extension.

By design a VIB file can contain multiple payload files. However, each VIB file I looked at contained only exactly one. The payload file's type was always VGZ (short for vmtar.gz), but TGZ-format should also be possible. In fact the payload file is just the archive that makes up the actual driver package. Section 3 of my anatomy post describes that in more detail.

What is an Offline-Bundle?
Offline-Bundles come in ZIP-format and are just a collection of one or multiple VIB files. An Offline Bundle contains additional meta data in an included archive named metadata.zip, the VIB-file(s) are stored in the archive's sub directory vib20\.

Is every ZIP-file an Offline-Bundle?
At the VMware site you can download new and updated drivers for ESXi 5.0 from the Drivers & Tools / Driver CDs section of the vSphere 5 download page. These downloads are also in ZIP-format. However, if you look at the contents of such a downloaded file you will notice that it is not an Offline-Bundle itself, but includes another ZIP-file that actually is the Offline-Bundle!

So, please be careful... The real Offline-Bundle-ZIP files have the string offline_bundle or just bundle in their names (e.g. LSI_5_34-offline_bundle-455140.zip). If you are unsure then look into the zip file and check if it includes a metadata.zip file and a vib20\ sub-directory.

ESXi-Customizer 2.5
... will be out soon, and it will support adding VIB files and Offline-Bundles to an ESXi 5.0 media! Watch this space for the announcement.