How to build device drivers for ESXi 5.x

Since I published my ESXi-Customizer and ESXi-Customizer-PS tools to slipstream driver packages into an ESXi installation ISO I was asked dozens of times if I can provide a driver for the unsupported device xyz or whether I can give instructions on building such a driver. So far my answer was always No, because - although I'm quite familiar with using Linux - I do not have any of the Linux kernel hacking skills that I thought would be required to do this.

Read more »

How to clone ESXi 5

I recently received an e-mail with the following question: How do I copy an ESXi installation from a USB key drive on to the internal hard disk of my computer? Normally you would answer like this: You don't do it. You backup the ESXi configuration (e.g. with the vicfg-backup tool), install a fresh copy of ESXi on your hard disk and then restore the saved ESXi configuration.
Read more »

A Daemon's VIB - Part 3 (Building a software package for VMware ESXi)

Part 3: Packaging, SecPolicy and VisorFS

This is the third and last part of the "Daemon's VIB" series about building a software package for ESXi, using the example of the ProFTPD Offline Bundle that I recently released. In the first part I explained how to get (or create yourself) suitable binaries for ESXi, in the second part we cared about the Daemon's configuration, firewall settings and automatic startup. In this last part we will
   a) build and install the actual software package
   b) deal with the VMkernel access security policies and
   c) make the service configurable to adapt to your own needs.

Read more »

A Daemon's VIB - Part 2 (Building a software package for VMware ESXi)

Part 2: Service configuration and startup

This is the second part of the "Daemon's VIB" series about building a software package for ESXi, using the example of the ProFTPD Offline Bundle that I recently released. In the first part I explained how to get (or create yourself) suitable binaries for ESXi. In this part I will focus on what is necessary
   a) to make the service start up automatically on system boot and
   b) to make it fully manageable via the vSphere Client.
Read more »

A Daemon's VIB: Building a software package for VMware ESXi (Part 1/3)

Part 1: Binaries

I recently released an Offline Bundle for installing the ProFTP Daemon in ESXi 5.x. This was the result of a self-educational project: The subject of this project was not really how to develop software for ESXi, but how to nicely package software for ESXi in a way that makes it easy to distribute and easy to deploy and use for the average VMware administrator. I chose the example of an FTP daemon, because I already stumbled several times over requests for having this available in ESXi, and I know that there were some hacks for it around since the days of ESXi 3.5.

I promised to share what I have learnt in this project, and I will deliver on this promise in three parts. This first part is about binaries and binary compatibility.

Read more »

[Release] ProFTPD (FTP server) for VMware ESXi 5.x

ProFTPD is a GPL-licensed, feature rich and highly configurable FTP server. Today I have made available an Offline Bundle to install ProFTPD in VMware ESXi 5.x. Installing it will enable you to transfer files from and to an ESXi host using an FTP client program.

Read more »

[Update] ESXi5 Community Packaging Tools 2.1

I just published updated versions of my ESXi5 Community Packaging Tools. What's new:

• The TGZ2VIB5 tool now allows to build the VIB payload file from a directory structure (rather than using a pre-packaged tgz file as input)
• You can now change the Acceptance Level of the VIB package

Read more »

VMware Labs' latest fling: VIB Author and how it compares to the ESXi5 Community Packaging Tools

The VMware Labs have published a new tool called VIB Author that "allows ESXi administrator to create custom VIBs at the CommunitySupported level". When I read this news my first thought was: Hey, that's exactly what my ESXi5 Community Packaging Tools (ESXi5-CPT, consisting of tgz2vib and vib2zip) are supposed to do! So, how do these two compare?
Read more »

ESXi-Customizer-PS script updated to be compatible with PowerCLI 5.1

I was made aware that my ESXi-Customizer-PS script fails when running with the new PowerCLI version 5.1. With this version VMware made changes to the ImageBuilder object model (dropping the SoftwareChannel object) that broke my script.
Read more »

vSphere 5.1 is generally available (incl. the free and the HP customized version of ESXi)

Late yesterday VMware made vSphere 5.1 generally available. Go to
   http://www.vmware.com/go/download-vsphere
to download. The free version of ESXi 5.1 is also already available:
   http://www.vmware.com/go/get-free-esxi

And HP hardware owners will be happy to find that there is also an HP customized version of the ESXi 5.1 ISO available for download. The shortcut http://www.vmware.com/go/downloads-image-hp-esxi still points to the 5.0 version, but HP's ESXi 5.1 GA is available here:
   https://my.vmware.com/web/vmware/details?downloadGroup=HP-ESXI-5.1.0-GA-10SEP2012&productId=285

Right now it is not yet possible to update an existing ESXi 5.0 installation to 5.1 using the well known methods (via Update Manager and the VMware Online Depot). You will probably be able to update by booting the host with the ESXi 5.1 ISO, but the other methods should also be available shortly.

Sep2012 release of the HP Customized ESXi 5.0 ISO

HP has just published updated software packages and device drivers for VMware ESXi 5.0 (at http://vibsdepot.hp.com). With this Sep2012 release HP changes the way how they provide this software: Up to now they have made a complete customized installation ISO (including all the packages) available for download on their own web site. Now they have replaced this download link with a shortcut URL that redirects to a VMware download page: http://www.vmware.com/go/downloads-image-hp-esxi.
Read more »

How to simplify and automate VMware ImageBuilder with the ESXi-Customizer-PS script

I published a new and improved version of my ESXi-Customizer-PS script today. This PowerCLI script greatly simplifies and automates the process of creating fully patched and customized ESXi 5.x installation ISOs using the VMware PowerCLI ImageBuilder snapin.

Over time I added more and more options to the script, and I think it is time now to provide a detailed tutorial to show how to make efficient use of it. Grab the latest version (1.3) from my Google Code page and give it a try. The only prerequisite you need is an installation of the current VMware vSphere PowerCLI.
Read more »

VMware vCenter Update 1a and ESXi 5.0 Build 768111

VMware has released Update 1a for vCenter and important patches for ESXi 5.0 on July 12th 2012.

What's new and fixed in vCenter Update 1a:

• This is the first update that is also available for the vCenter (Linux based) appliance
• For the appliance it switches the embedded DB2 database to VMware's own vPostgres database.
• The problem with excessive memory consumption of the tomcat6 service (KB2013890) has been resolved.
• The problem with HA restarts failing for VMs that were migrated with Storage VMotion (KB2013639) is fixed.
• For a full list of fixes see Release notes.

What's new in the ESXi 5.0 Patch Release ESXi500-201207001:

• Fixes the problem with syslogging stopping when the syslog server becomes unreachable  (KB2003127).
• Fixes the VM autostart feature that was broken in the free version of 5.0 (see my earlier post about this, and the fix announcement of Kyle Gleed)
• For a full list of fixes see KB2019107 and further KB articles that are referenced there.

Here is a link to the full ESXi 5.0 update bundle: ESXi500-201207001.

Explaining the CX + vSphere5 + VAAI support riddle

I recently blogged about the oddity of EMC CX arrays not being supported with VAAI on vSphere 5.0 anymore and - like many others - I wondered what are the real reasons behind this. Today Chad Sakac (EMC) explained the riddle in a web cast covering a VNX engineering update and - well - exactly this issue.

If you missed it: he just published the presentation that he showed on his blog. Here is a summary:
Read more »

[Update] ESXi-Customizer Powershell script version 1.2

Following up on some user feedback I updated my ESXi-Customizer-PS (Powershell) script to version 1.2. I introduced this script in the last part of my ImageBuilder Deep Dive series (see also part 1 and 2). This version adds advanced features by providing the following new optional command line parameters:

• -ipname  : provide a name for the customized ImageProfile (the default is derived from the cloned VMware ImageProfile)
• -sip : manually select an ImageProfile from the current list (default = auto-select latest available ImageProfile)
• -hprel <mon>  : select HP packages from release dated <mon> (e.g. jun2012) (default = select latest available HP packages)

Please download the latest version of the script from my Google Code page.

Use the command line parameter -help for complete usage instructions:

ESXi-Customizer-PS v1.2 help screen

[Update] ESXi-Customizer Powershell script version 1.1

With the last part of my ImageBuilder Deep Dive series (also don't miss part 1 and 2) I published a Powershell script to build a customized ESXi 5.0 installation ISO, optionally adding the contents of the HP Online VIBs depot.

HP has recently updated the contents of their Online depots adding the latest releases of their bundles (see this post). I tested the first version of my script and found that it would incorrectly add older versions of the HP packages. So, I have fixed it now to add only the newest version of each available package. Other changes:

• Skip the hp-smx-limited package. It is not included in HP's customized images and conflicts with the full CIM provider package hp-smx-provider. In fact it is only useful for the new ProLiant Gen8 servers if you want to monitor these through the Agentless Management Service (AMS) only.
• Added a new command line switch -test. It will skip the actual package download and image building - useful for testing the script and seeing what it would add without wasting any download bandwidth.

The new version 1.1 is available for download now.

ESXi-Customizer-PS v1.1 help screen

How to disable Storage I/O Control for an unavailable datastore

If you run your VMs on FC based storage then you occasionally have the necessity to unmap a storage LUN from one or more ESX(i) hosts (e.g. when retiring a storage array or re-organizing your storage layout). It is important to do this in the right way using the procedures that are documented in

• KB1029786 for ESX(i) 4.1
• KB2004605 for ESXi 5.0

The procedure is very complex for vSphere 4.1, but fortunately is has become much easier with vSphere 5.0.

What happens if you fail to follow these procedures and just unpresent the LUN on the storage array, so that the hosts can not access it anymore? The ESX(i) hosts will detect an APD (all paths down) condition in this case, and - particularly ESX(i) 4.1 hosts - can become very unhappy about this (see KB1016626 and KB1030980). Again, ESXi 5.0 hosts are much more resilient to APD conditions: they will eventually turn them into PDL (= permanent device loss, s. KB2004684) conditions and will completely recover from the LUN loss after rescanning the HBAs ... unless you have Storage I/O control (SIOC) enabled on the lost datastores ... and this is what happened to us today :-( The vmkernel.log files were flooded with the following messages, because SIOC was trying to access the lost datastore:

Permanent Device Loss (PDL) with SIOC enabled

Now the problem is: You cannot just disable SIOC on a lost datastore using the vSphere client - you should have done this before unmapping the storage LUN!

One way to recover from this situation is to reboot any of the affected hosts. However, I really wanted to save the time to put all the hosts in maintenance mode and reboot them one after the other. So I looked for a way to forcibly unmount the datastore directly on the host via esxcli or similar. All the ways that are documented in the VMware KB articles and docs did not work in my case, but I finally stumbled over this wonderful blog post by William Lam:

Does SIOC actually require Enterprise Plus and vCenter?

It is a bit old and refers to ESX(i) 4.1, but it is still valid for ESXi 5.0! Here William describes a way to enable (and disable!) SIOC for a datastore directly on a host without using (even without having available!) a vCenter server.

And it is really easy: All you need to know is the device ID of the datastore/LUN. In ESXi 5.0 you can find it out by using the command

   # esxcli storage vmfs extent list

It will list all datastores with their labels and device IDs (starting with naa.). And then you can use the following vsish command to disable SIOC for the device

   # vsish -e set /storage/scsifw/devices/<naa-id>/iormState 1496

That's it! After waiting a few seconds the affected datastore suddenly disappeared from the list of mounted datastores in the vSphere client, and the VMkernel.log error messages also stopped.

Please note: vsish is a powerful but largely undocumented utility to query and set VMkernel parameters. It is only available in an ESXi local or remote shell. William has quite a few posts about it on his virtuallyGhetto blog. This time it saved us the trouble and time of rebooting a whole cluster of hosts ...

Updated: What's the deal with EMC CX arrays not supporting VAAI with vSphere 5.0?

(If you have read this post before then skip to the bottom for recent updates)

We are in the process of upgrading our production environment to vSphere 5.0 U1. A new vCenter 5.0 server is already in place, and we attached the production ESXi hosts (that are still on 4.1 U2) to it.
The next step would be to upgrade the hosts. In the meantime, since this is a qualified production environment running more than 1.500 virtual servers I became a bit paranoid about hardware and software/firmware compatibility, and decided to double check if the environment is fully supported (or if we should upgrade any firmware first). I was pretty confident that we are on the safe side because we never had any issues with vSphere 4.1 and always kept the environment up to date.

But then I stumbled over this VMware KB article: EMC CX and VNX Firmware and ESX requirements for vStorage APIs for Array Integration (VAAI) support. It states that ESXi 5.0 does not support VAAI on our Clariion CX4 arrays even although they have the latest FLARE code:

	vSphere 4.1	vSphere 5.0
CX4 Series Flare 30/29/28+	VAAI Supported	Not Supported
VNX Series OE 31 or Later *	VAAI Supported	VAAI Supported

I checked the test hosts that we already updated and they showed VAAI being supported on the CX4 LUNs in the vSphere client. However, the KB article recommends to disable VAAI on these hosts ...

I quickly searched the Internet for relevant posts and statements and browsed through the VMware Community forums. There I found this post where people complain about the VAAI status shown as unsupported. It looks like there are dependencies to the LUN size (smaller or larger than 2TB) and to the array's failover mode (sounds like you need to use ALUA / failover mode = 4 which we are already using). But from this post I get the impression that everything is fixed with the latest 30.x FLARE code releases).

We contacted both VMware and EMC to get some clarification now, and I will keep this post updated with any new information. In the meantime I ask everyone using ESXi 5.0 with EMC CX storage to comment on this post whether you are using VAAI, if you have any issues with it, and what FLARE code you have on the arrays. BTW, you can check what VAAI primitives are in use with your storage LUNs by running the following esxcli command:
   esxcli storage core device vaai status get

Thanks in advance for any helpful comments!

---

Update (2012-05-20): I got some feedback from other users, VMware and EMC on this issue, and used my best Google-Fu to get related information about vSphere 5, VAAI, VMFS-5 and/or EMC CX support. Here are the results (somewhat unsorted):

• Several users use this unsupported setup, but do not have any issues with it (so far).
• pape pointed to some useful VMware KB articles in the comments of this post. Here is an overview of these and some more related articles:
• KB2006858: ESXi 5.0 hosts fail to mount VMFS5 volumes that are formatted with ATS-only capabilities (describes an issue with VMFS-5 volumes and ATS locking seen on EMC arrays, but this is solved with ESXi 5.0 Update 1)
• KB1033665: Disabling the VAAI functionality in ESX/ESXi
• KB2012967: Disabling VAAI functionality for a specific array (Now that would be interesting, but the link doesn't work as of now)
• KB1021976: vStorage APIs for Array Integration FAQ (this FAQ also covers differences in ESXi 4.1 and 5.0 resp. VMFS-3 vs. VMFS-5)
• KB2003813: vSphere 5 FAQ: VMFS-5
• The following VMware blog posts of Cormac Hogan are interesting in the context of VAAI usage with VMFS-3 vs. VMFS-5:
• vSphere 5.0 Storage Features Part 1 - VMFS-5
• vSphere 5.0 Storage Features Part 3 - VAAI
• VMFS Locking Uncovered (About the improvements that were made with file locking during VMFS evolution)

Regarding my specific issue (lack of official support for VAAI with EMC CX arrays and ESXi 5.0) I finally got one really detailed and helpful comment from an EMC representative: EMC tested the VAAI functionality of ESXi 5.0 with their EMC CX4 arrays (I specifically asked for the CX4). It has been "tested as functional", but it only passed the certification tests of the ATS and Zero primitives, but failed the certification test of the XCopy primitive.
I wonder why this happened, because VAAI was fully supported with ESXi 4.1 (i.e. it must have also passed the XCopy test), and with ESXi 5.0 there were no changes to the XCopy primitive (at least I could not find any information about such a change).

Anyway, this is probably the reason why EMC and VMware do not officially support VAAI with ESXi 5.0 on CX arrays. From a customer's point of view this is disappointing, but from EMC's point of view this decision is understandable, because they want to focus on their current products and not waste support resources on somewhat outdated arrays like the CX ones. However, according to this EMC representative it may be possible to come to an individiual support agreement with EMC and VMware (EMC calls this RPQ = Request for Product Qualification) if you nevertheless must or want to have an official support statement for whatever reason.

I also had the idea of disabling only the XCopy primitive to be on the safe side, but I don't want to do this globally, because we also use a fully VAAI supported VNX array with our production hosts. So I was glad to find a link to the promising KB article KB2012967 (see list above), but this link currently doesn't work. I will ask VMware about this ...

---

Update (2012-06-20): In the meantime an EMC representative confirmed to me that all three VAAI primitives work fine with vSphere 5.0 and CX arrays, and that they even support it, but you need to file an RPQ with them in order to get formal support.

We have this configuration (VAAI turned on for both our CX4 and VNX arrays) running with vSphere 5.0 in production for about 6 weeks now without any problems.

The link to VMware KB article KB2012967 still doesn't work, but it is now listed to as being "[Archived]" in the reference section of KB1033665 ...

---

Update (2012-07-11): It turned out that you no longer need an RPQ with EMC. They now officially support the vSphere 5's VAAI implementation on the CX4. The combination is listed in their latest Simple Support Matrix for VMware vSphere 5 (Powerlink account needed for download):

Snippet from the EMC Simple Support Matrix for vSphere 5 (of July 2012)

VMware though does still not officially support it (KB2008822 is unchanged as of today). In another blog post I already explained the reason for this.

HP VMware ESXi 5.0 U1 Customized Image

While searching for an updated HP Customized ESXi 5.0 ISO that would include Update 1 I noticed that the HP download page for ESXi still shows only the October release (ESXi 5.0 GA) as of today, March 25th 2012.
However, I stumbled over a document titled HP VMware ESXi 5.0 U1 Customized Image Release Notes for March 2012 that is apparently available since March 16th. That means that the release of this customized image is imminent. Here are some highlights from the release notes:

• Support for the new HP Gen8 servers incl. support for the new Agentless Management Service
• Installation conflict of HP WBEM providers and Emulex CIM providers resolved
• Now includes the hpacucli utility (so you no longer need the workaround of running the Linux version of this tool that I described earlier). 

I will update this post once the new HP Customized ESXi 5.0 U1 Image is finally published.

Update (2012-03-27):  The HP VMware ESXi 5.0 U1 Customized Image is now available for download!

ESXi 5.0 Update 1 breaks VM Autostarts on free ESXi

In the VMware Communities an issue with ESXi 5.0 Update 1 was reported that is important for everyone who uses the free (VMware vSphere Hypervisor) version of ESXi 5.0:  Apparently VM autostarts do not work anymore after installing Update 1 if the host runs with a free license. The VM autostart function will automatically power on a defined list of VMs in a defined order when the system boots. This is probably used by a lot of people who run standalone ESXi hosts with the free license.

I was able to reproduce this issue in my home lab, and currently there is no workaround or solution. So if you rely on this functionality then you should postpone the installation of Update 1 until a fix for this is released. If you have already installed Update 1 and want to make it undone due to running into this issue then you can roll back to the previous installation (without Update 1) by pressing Shift-R at the ESXi boot prompt.

Update (2012-07-14): VMware has released a patch for ESXi 5.0 on July 12th that fixes this problem.