ESXi-Customizer-PS script updated to be compatible with PowerCLI 5.1

I was made aware that my ESXi-Customizer-PS script fails when running with the new PowerCLI version 5.1. With this version VMware made changes to the ImageBuilder object model (dropping the SoftwareChannel object) that broke my script.
Read more »

Updated ESXi-Customizer-PS script allows to create ESXi 5.1 Offline bundle

Following the availability of the ESXi 5.1 packages in the VMware Online depot I have updated my ESXi-Customizer-PS script and made two minor, but important changes:
Read more »

How to simplify and automate VMware ImageBuilder with the ESXi-Customizer-PS script

I published a new and improved version of my ESXi-Customizer-PS script today. This PowerCLI script greatly simplifies and automates the process of creating fully patched and customized ESXi 5.x installation ISOs using the VMware PowerCLI ImageBuilder snapin.

Over time I added more and more options to the script, and I think it is time now to provide a detailed tutorial to show how to make efficient use of it. Grab the latest version (1.3) from my Google Code page and give it a try. The only prerequisite you need is an installation of the current VMware vSphere PowerCLI.
Read more »

[Update] ESXi-Customizer Powershell script version 1.2

Following up on some user feedback I updated my ESXi-Customizer-PS (Powershell) script to version 1.2. I introduced this script in the last part of my ImageBuilder Deep Dive series (see also part 1 and 2). This version adds advanced features by providing the following new optional command line parameters:

• -ipname  : provide a name for the customized ImageProfile (the default is derived from the cloned VMware ImageProfile)
• -sip : manually select an ImageProfile from the current list (default = auto-select latest available ImageProfile)
• -hprel <mon>  : select HP packages from release dated <mon> (e.g. jun2012) (default = select latest available HP packages)

Please download the latest version of the script from my Google Code page.

Use the command line parameter -help for complete usage instructions:

ESXi-Customizer-PS v1.2 help screen

[Update] ESXi-Customizer Powershell script version 1.1

With the last part of my ImageBuilder Deep Dive series (also don't miss part 1 and 2) I published a Powershell script to build a customized ESXi 5.0 installation ISO, optionally adding the contents of the HP Online VIBs depot.

HP has recently updated the contents of their Online depots adding the latest releases of their bundles (see this post). I tested the first version of my script and found that it would incorrectly add older versions of the HP packages. So, I have fixed it now to add only the newest version of each available package. Other changes:

• Skip the hp-smx-limited package. It is not included in HP's customized images and conflicts with the full CIM provider package hp-smx-provider. In fact it is only useful for the new ProLiant Gen8 servers if you want to monitor these through the Agentless Management Service (AMS) only.
• Added a new command line switch -test. It will skip the actual package download and image building - useful for testing the script and seeing what it would add without wasting any download bandwidth.

The new version 1.1 is available for download now.

ESXi-Customizer-PS v1.1 help screen

ESXi-Customizer 2.6 and Tgz2Vib5 1.0

I just published the new version 2.6 of my ESXi-Customizer script.

What's new:

• With this version you are able to optionally create an (U)EFI-bootable ISO file for the installation of ESXi 5.0. (U)EFI stands for (Universal) Extensible Firmware Interface. This is going to replace the current BIOS firmware interface on modern PCs. Please note that the original VMware ESXi 5.0 ISO is already UEFI-capable, the new version of my script is just able to keep this possibility in the customized ISO.
• The new version includes an additional utility script called Tgz2Vib5. With this script you are able to convert an OEM.tgz-style driver package (for ESXi 5.0 only!) into a VIB file. That is the "official" VMware format for software packages - read more about it in this earlier post!

I'd like to encourage the developers of community supported ESXi 5.0 drivers to convert their packages into VIB format (using Tgz2Vib5) before publishing them! The VIB format has several advantages over the traditional OEM.tgz format:

• You can add descriptive meta data (like vendor/author name, version and detailed description) to the driver package
• Unlike an OEM.tgz file a VIB file can be easily installed into an already running ESXi 5.0 system by running the following commands inside a local or remote ESXi shell:
    esxcli software acceptance set --level=CommunitySupported
    esxcli software vib install -v VIB-URL
  (The host must not run any VMs at install time, because it needs to be rebooted after the installation.)
• A VIB file can also be updated with a newer version without having to re-install the whole system. This can be achieved by running the following command inside a local or remote ESXi shell:
    esxcli software vib update -v VIB_URL

VMware finally released the Open Source Code of vSphere 5.0!

Great news! Today VMware finally made the vSphere v5.0 Open Source code archives available for download.

Why is that important?

Since the release of VMware's ESXi 5.0 (Aug 24, 2011) many people are asking for the development of drivers for hardware devices that are not supported by ESXi 5.0 out-of-the-box.

ESXi device drivers are based on Linux device drivers (which lead to the persistent misunderstanding that ESXi itself is based on Linux), but the stock Linux driver code must be modified in a specific way to be compatible with ESXi.

With past versions of ESXi (up to 4.1) it was possible to study and reproduce these required modifications, because VMware published the source code of the ESXi device drivers (the original Linux code plus their modifications). The reason for this is that most Linux drivers are licensed under the GPL (General Public License), and the GPL requires that derived works are also published under the GPL and their source code is made freely available (aka the "Copyleft" principle).

Now, that VMware also published the Open Source Code of ESXi 5.0 (including the device drivers that it contains) it will be possible (or at least much easier) to develop custom ESXi 5.0 drivers for devices that are not officially supported by VMware.

[Announcement] ESXi-Customizer 2.5 ...

... is out now. See the prior post for what's new. Download it from the project page!

VIB files, Offline-Bundles and ESXi-Customizer 2.5

The current version 2.0 of my ESXi-Customizer script is able to add OEM.tgz-style driver packages to ESXi 4.1 and ESXi 5.0 installation ISOs. Tgz (short for tar.gz) is the format that is used to distribute all community-developed drivers for ESXi 4.1 (and prior versions). So these can currently be added to ESXi 4.1 with my script.

However, for ESXi 5.0 you cannot use the driver packages for ESXi 4.1, they need to be re-engineered and re-compiled starting with the stock Linux driver code that is the basis for ESXi drivers. It looks like - so far - nobody in the user community has figured out how to compile drivers for ESXi 5.0. I hope that this will change in the near future, but for now the current version of the ESXi-Customizer script is pretty much useless for ESXi 5.0 ...

On the other hand, there are new and updated driver packages for ESXi 5.0 already available directly from VMware or 3rd party hardware vendors. These drivers are distributed in VIB format and as so-called Offline-Bundles (in zip-format). I wondered if my script could also support these "official" VMware package format and had a closer look at the format of these files.

(Note: To fully understand the following, it is helpful to read my Anatomy post about the structure of the ESXi 5.0 ISO first!)

What is a VIB file?
VIB stands for "VMware Installation Bundle". If you just open a vib-file in a text editor you will see that it starts with the (somewhat well known) header "!<arch>". This means that the file is in the Unix ar format. Wikipedia has a detailed description of this file format, and there you can learn that VIB files use the common ar format, and that you can use the Unix ar command to handle this kind of archive files.
Fortunately, 7-zip - the Swiss army knife of packaging tools - is also able to handle ar-files, and it is also available for Windows. Since I already use it with ESXi-Customizer to unpack ISO-files I was delighted to realize that it is also able to unpack a vib-file ...

So, what's in the VIB file? Exactly three files:

1. A file named "descriptor.xml": The name says it all. This file describes the contents and dependencies of the driver package. Later you will find that again in the IMGDB.TGZ file. For more details see section 4 of my anatomy-post. There you will find the descriptor.xml file of the e1000-driver as an example.
2. A file named "sig.pkcs7": VMware certified drivers need to be electronically signed, and this file includes the uuenconded PKCS7 signature.
3. The payload file: You will find the exact name of this file in the name-attribute of the <payload> tag in the descriptor.xml file. It does not have any file extension.

By design a VIB file can contain multiple payload files. However, each VIB file I looked at contained only exactly one. The payload file's type was always VGZ (short for vmtar.gz), but TGZ-format should also be possible. In fact the payload file is just the archive that makes up the actual driver package. Section 3 of my anatomy post describes that in more detail.

What is an Offline-Bundle?
Offline-Bundles come in ZIP-format and are just a collection of one or multiple VIB files. An Offline Bundle contains additional meta data in an included archive named metadata.zip, the VIB-file(s) are stored in the archive's sub directory vib20\.

Is every ZIP-file an Offline-Bundle?
At the VMware site you can download new and updated drivers for ESXi 5.0 from the Drivers & Tools / Driver CDs section of the vSphere 5 download page. These downloads are also in ZIP-format. However, if you look at the contents of such a downloaded file you will notice that it is not an Offline-Bundle itself, but includes another ZIP-file that actually is the Offline-Bundle!

So, please be careful... The real Offline-Bundle-ZIP files have the string offline_bundle or just bundle in their names (e.g. LSI_5_34-offline_bundle-455140.zip). If you are unsure then look into the zip file and check if it includes a metadata.zip file and a vib20\ sub-directory.

ESXi-Customizer 2.5
... will be out soon, and it will support adding VIB files and Offline-Bundles to an ESXi 5.0 media! Watch this space for the announcement.

How ESXi-Customizer supports ESXi 5.0 - FAQ

I got a lot of feedback after posting the new ESXi-Customizer (with support for ESXi 5.0) and the "anatomy"-article explaining its technical background. It looks like I haven't been clear enough on some points and need to provide some additional information. So here is a list of frequently asked questions (FAQ), I might update it from time to time, so stay tuned.

1. Can I use existing drivers (made for ESXi 4.x) for customizing ESXi 5.0?

No, you can't. Driver binaries compiled for ESXi 4.x are not compatible with ESXi 5.0. They just won't be loaded. Instead vmkload_mod will throw the error message "Module does not provide a license tag".

2. What input does ESXi-Customizer expect for customizing ESXi 5.0?

It expects a gzip-compessed tar-file (with extension .tgz) that includes exactly three files:

• /usr/lib/vmware/vmkmod/<driver-module> (the binary driver module)


• /etc/vmware/driver.map.d/<driver-name>.map (maps PCI device IDs to the binary module)


• /usr/share/hwdata/driver.pciids.d/<driver-name>.ids (maps PCI device IDs to display names)

Nothing more is needed. All other steps outlined in the "anatomy"-post will be done by ESXi-Customizer.

3. When and where will ESXi 5.0 compatible community drivers be available?

ESXi device drivers are derived from device drivers written for the Linux kernel. However, it is necessary to make specific changes to the source code of a stock Linux driver to turn it into an ESXi driver. An experienced Linux developer can find out what changes are necessary by studying the complete source of the existing ESXi drivers that are shipped with ESXi 5.0.

The source code of these drivers has not yet been published by VMware. However, they are obliged to do this (sooner or later), because most of the original Linux drivers are licensed under the GNU GPL requiring that the source code of derived works also needs to be publicly available.
So, we need to wait for VMware to publish the OpenSource code of its drivers (we can expect it here), and for some knowledgeable people to compile new ESXi 5.0 compatible drivers then.

I am confident that this will happen in the near future. And I expect the new drivers to become available at Dave Mishchenko's vm-help.com, the home of the ESXi Whitebox HCL.

4. Does ESX-Customizer support creating a bootable USB-key with ESXi 5.0?

No, it does not. If the machine that you want to install ESXi on does not have a CD-ROM drive, you can help yourself by installing ESXi 5.0 using any other machine (that has a CD-ROM drive) onto a USB key drive. Once you have a bootable USB key you can use that to also boot any other machine!
The easiest and safest method is to use a virtual machine provided by VMware Workstation or VMware player to do the initial install. Yes, ESXi 5.0 can be installed in a VMware Workstation VM - just select "ESX Server 4" as the guest OS type.

The anatomy of the ESXi 5.0 installation CD - and how to customize it

1. Introduction

With vSphere 5 VMware introduced the Auto Deploy Server and the Image Builder that allow to customize the ESXi installation ISO with partner supplied driver and tools packages.
The Image Builder is a Powershell snapin that comes with the latest version of the PowerCLI package. It allows to add software packages to a pre-defined set of packages (a so-called ImageProfile) and even lets you create an installation ISO from such a baseline making it easier than ever to customize the ESXi installation.

However, doing this is not a straight-forward task. It requires a working installation of the Powershell, plus the PowerCLI software, access to the offline-bundle that makes up the base installation (which is not included with the free version of ESXi!), a custom driver in VIB format, and some guidance on what Powershell-cmdlets you need to use to add the custom driver package and build an ISO from it.
For the developers of custom drivers it requires to supply their packages in VIB format, and it's not trivial and costs extra effort to build such a package (compared to a simple OEM.TGZ file).

I wondered if it is still possible to customize the ESXi 5.0 install ISO with a simple OEM.TGZ file like you can do with ESXi 4.1, e.g. with my ESXi-Customizer script. And yes, it is possible - but it's very different now! I want to provide some background information here on how this works:

2. The contents of the ESXi 5.0 installation ISO

First let's have a look at the root directory of the ESXi 5.0 install ISO:

Contents of the ESXi 4.0 install CD root directory

Unlike the ESXi 4.1 ISO you can see lots of ISO9660-compatible file names here (all capitals and 8.3-format). You can guess from their names that the files with the V00 (and V01, V02, etc.) extensions are device driver archives. The original  type of these files is VGZ, the short form of VMTAR.GZ. That means that they are gzip'ed vmtar-files.

vmtar is a VMware proprietary variant of tar, and you need the vmtar-tool to pack and unpack vmtar archives. It is part of ESXi 5.0 and also ESXi 4.x. Other files have the extensions TGZ and T00 (like TOOLS.T00). These files are gzip'ed standard tar files that the boot loader can also handle. Good.

Comparing with the ESXi 4.1 media you will notice that there is no ddimage.bz2 file any more. In earlier versions of ESXi this is a compressed image that is written to the installation target disk and contains the whole installed ESXi system. Actually you can write this image to a USB key drive to produce a bootable ESXi system without ever booting the install CD. You cannot do this with ESXi 5.0 any more. However, customizing the install CD has become easier this way, because you do not need to add a second copy of your oem.tgz file to this system image.

There are also files named ISOLINUX.BIN and ISOLINUX.CFG in the ISO root. That means that ESXi 5.0 still uses the isolinux boot loader to make the installation CD bootable. If you look into ISOLINUX.CFG it includes a reference to the file BOOT.CFG, and in BOOT.CFG you find references to all the VGZ and TGZ files:

Contents of the BOOT.CFG file

A second copy of the BOOT.CFG file is in the directory \EFI\BOOT. The ESXi 5.0 install ISO (and ESXi 5.0 itself) was built to boot not only on a standard x86 BIOS, but also on new (U)EFI enabled BIOS versions. Just one thing to remember: If you change the one BOOT.CFG you better make the same change to the other.

Now let's have a closer look at a driver VGZ package.

3. What's in a driver's vgz-file?

As mentioned before you need the vmtar-tool to look into a VGZ-file. Since it is only part of ESXi itself you need to have access to an installed copy of ESXi (either 4.1 or 5.0). Luckily you are able to install ESXi 4.1 (and also 5.0!) inside a VMware Workstation 7 VM.

I did this by creating a VM of type "ESX Server 4" with typical settings except for the size of the virtual disk (2GB is enough for ESXi) and installing ESXi 5.0 in it. During installation the driver files from the CD root are uncompressed and copied to the directory /tardisks, so here is where you can find them again. After enabling the local shell (luckily still available with 5.0) I logged in and was finally able to look inside and unpack such a driver archive using the vmtar tool:

Unpacking NET-E100.V00 with the vmtar tool

So there are basically three files in the archive:

1. The driver binary module (with no file name extension, e1000 in this example) that will be unpacked to the well known location /usr/lib/vmware/vmkmod.

2. A text file that maps PCI device IDs to the included driver:

Contents of /etc/vmware/driver.map.d/e1000.map

3. Another text file that maps PCI IDs to vendor and device descriptive names:

Contents of /usr/share/hwdata/driver.pciids.d/e1000.ids

It is good to know that the PCI ID mapping files are now separated by driver. In ESXi 4.1 there is a single pci.ids file and a single simple.map file for all drivers which raised the potential of having conflicting copies of these files in case you merged multiple OEM drivers into the image.

It looks easy now to add a custom driver to the install CD: Just create a tgz-file containing the three files mentioned above, copy it to the ISO root directory and add its name to the two BOOT.CFG files. And yes, this will indeed work for the CD boot! The custom driver will be loaded and you will be able to install ESXi, ... but the installation routine will not copy the tgz-file to the install media, and if you boot the installed system the first time it will behave like a regular install without the custom driver.

So, there is more to it...

4. The image database IMGDB.TGZ

There is a file named IMGDB.TGZ in the root directory of the CD that is also listed in the BOOT.CFG files and has the following contents:

Unpacking the IMGDB.TGZ file

It contains files that will be unpacked to the directory /var/db/esximg. For each driver (or other software package) an XML-file is created under the vibs sub directory. There are a lot more of these files than shown here (I fiddled the output with "..."), one example is net-e1000--925314997.xml for the e1000 driver. Let's look into this file:

The contents of net-e1000--925314997.xml

The xml-file contains information about the package including possible dependencies on other packages and a list of all included files. Its file name ("net-e1000--925314997.xml") consists of the name element plus a (probably) unique number with 9 or 10 digits. The list of payloads is the list of included archive files (either of type vgz or tgz), in most cases it's just one. The name of the payload is limited to 8 characters ("net-e100" in this case) and is the name of the corresponding file in the CD's root directory. The extension of this file is expected to be ".v00" if the file is of type vgz and ".t00" if the file is of type tgz. If there are name conflicts with other packages the number in the extension is counted up. E.g. the payload file for the e1000e driver is "net-e100.v01".

Then there is the host image profile XML file in the directory /var/db/esximg/profiles. In our example this is the file ESXi-5.0.0-381646-standard1293795055. Let's look into this one:

... ... ... (lot more <vib></vib> entries cutted) ... ... ...

Contents of the host image profile XML file

Here we find a list of all vib-packages that make up the currently installed system. Please note that the vib-id of a package strictly corresponds to the element values that are in the associated vib xml file (see picture before), it is composed the following way:

<vendor>_<type>_<name>_<version>

So the vib-id element of the net-e1000 driver e.g. is

VMware_bootbank_net-e1000_8.0.3.1-2vmw.0.0.383646

The payload names that are listed in the image profile file are the same as in the distinct vib xml files with the exception that here the exact file names (e.g. "net-e100.v00") are listed rather than just the file type (vgz or tgz).

Conclusion: If we want to add a custom driver to the install CD we need to do the following (in addition to the steps described in section 3.): modify the contents of IMGDB.TGZ, add a vib xml file for the driver (similar to net-e1000...xml) to it and update the contained image profile file to include the driver as an additional <vib>-entry.

There is another particular XML element in both the vib files and image profile file that we need to take care of: the <acceptancelevel>. VMware distinguishes four different acceptance levels: VMwareCertified, VMwareAccepted, PartnerSupported and CommunitySupported, in the XML files they are coded as certified, vmware, partner and community. The names are pretty self-explanatory, and one can easily guess that certified is stricter than vmware that is stricter than partner that in turn is stricter than community. In other words: If the host image profile is of acceptance level certified only packages of the same acceptance level can be part of it. If it is of acceptance level vmware only VMware certified and VMware accepted packages can be installed. If it is of acceptance level partner (and this is the default!) partner supported packages can be installed in addition to that. The least restrictive level is community that would accept all four types of packages.

My expectation is that custom drivers for whitebox hardware are community supported (unless they are published by a hardware vendor company). However, if the driver's vib file contains the acceptance level community the image profile's acceptance level must also be changed to community. Otherwise the installation of the package will fail.

5. Can we automate it?

Yes, we can! The latest version of ESXi-Customizer does automate all the steps described here to add custom drivers in tgz-format to an ESXi 5.0 install ISO. You only need to feed it with a tgz-file that contains the three files listed in section 3 of this post.

Please note: Packages made for earlier ESXi versions will not work with ESXi 5.0, not only because the directory structure has changed, but also because the earlier versions' driver modules won't be loaded by the new version! And - at the time of this writing - there are probably no oem.tgz-style driver packages available that are compatible with ESXi 5.0!
Hopefully, this will soon change. If you are looking for a driver of a device that does not work out-of-the-box with ESXi 5.0 check the Unofficial Whitebox HCL at vm-help.com.

[Update] ESXi-Customizer 1.2 - another bugfix release

If you used the Advanced edit mode with ESXi-Customizer 1.0 or 1.1 and got a "Corrupt boot image" message in ESXi (either when booting the customized ISO or after having installed with it) ... this was caused by a corruption of the OEM.tgz file while re-packaging it.

It was very hard to find a Windows version of tar that produces tar archives which are fully compatible with ESXi. But (I hope) I have finally found one: a Windows port of busybox. Since ESXi uses busybox, too, this should guarantee maximum compatibility. If you ever wondered what a Windows port of busybox could be good for ... now you know ;-)

Please update to version 1.2 that incorporates this fix, and let me know if you are still struck by this bug! Please download it from the project page!

[Update] ESXi-Customizer 1.1 - bugfix release

I published an updated version of my ESXi-Customizer script. There was an annoying bug with the "Advanced edit"-mode causing the oem.tgz file to become corrupted during re-packaging. This has been fixed, and I also added an update check feature to let the script check for newer versions of itself.

Download it from the project page.

[Release] ESXi-Customizer

Have you ever tried installing ESXi on a hardware box that is not explicitly supported by VMware? If you try this you will often run into the problem that the original VMware install ISO does not include a driver for the storage controller or network card that is in your box.

There is a community that works on building drivers for such devices (so called Whitebox hardware, see www.vm-help-com), and there are instructions and scripts available for adding these drivers to the original install ISO. However, they all require some (at least basic) Linux knowledge and access to a Linux system.

This has changed now ... I have written a script that automates this task and runs entirely on Windows 7. Visit the project page on this blog site to learn more!