Explaining the CX + vSphere5 + VAAI support riddle

I recently blogged about the oddity of EMC CX arrays not being supported with VAAI on vSphere 5.0 anymore and - like many others - I wondered what are the real reasons behind this. Today Chad Sakac (EMC) explained the riddle in a web cast covering a VNX engineering update and - well - exactly this issue.

If you missed it: he just published the presentation that he showed on his blog. Here is a summary:
Read more »

[Update] ESXi-Customizer Powershell script version 1.2

Following up on some user feedback I updated my ESXi-Customizer-PS (Powershell) script to version 1.2. I introduced this script in the last part of my ImageBuilder Deep Dive series (see also part 1 and 2). This version adds advanced features by providing the following new optional command line parameters:

• -ipname  : provide a name for the customized ImageProfile (the default is derived from the cloned VMware ImageProfile)
• -sip : manually select an ImageProfile from the current list (default = auto-select latest available ImageProfile)
• -hprel <mon>  : select HP packages from release dated <mon> (e.g. jun2012) (default = select latest available HP packages)

Please download the latest version of the script from my Google Code page.

Use the command line parameter -help for complete usage instructions:

ESXi-Customizer-PS v1.2 help screen

How to list vSphere 5.0 HA restarts with PowerCLI

VMware HA usually does a good job restarting VMs in case of an ESX(i) host failure. Imagine this happened at night - you come into office the next morning and want to know what VMs were restarted (or eventually failed to restart) because of the HA event.

You can find that out by looking at the vCenter event log, but if this happened several hours (or even longer) ago the vSphere client will no longer display the associated events. Even if the events are still displayed you will have a hard time to find them and compile a list of restarted VMs.

To cope with this situation I wrote a small PowerCLI script that searches the vCenter event log, finds the relevant entries and prints the list of VMs that were restarted during the latest HA event. It works with vSphere 5.0 only. Here it is:

param(
    [string]$vcenter = "localhost",
    [int]$last = 24,
    [switch]$help = $false
)

$maxevents = 250000

"`nScript to generate list of successful and failed VM restart attempts after an HA host failure."
if ($help) {
"Optional parameters:
-help:               display this help
-vcenter servername: connect to vCenter server servername (default is localhost)
-last n:             analyze events from the last n hours (default is 24)
"
exit
} else {
"(Use -help for list of parameters)"
}

$stop = get-date
$start = $stop - (New-TimeSpan -Hours $last)

if (!(get-pssnapin -name "VMware.VimAutomation.Core" -ErrorAction SilentlyContinue )) { add-pssnapin "VMware.VimAutomation.Core" }

write-host "`nConnecting to vCenter server $vcenter ..."
Connect-VIServer $vcenter | out-null

write-host "`nGetting all events from $start to $stop (max. $maxevents) ..."
$events = Get-VIEvent -Start $start -Finish $stop -MaxSamples $maxevents

write-host Got $events.Length events ...

write-host -nonewline "`nSearching for host failure events ..."
$ha = @()
$events | where-object { $_.EventTypeID -eq "com.vmware.vc.HA.DasHostFailedEvent" } | foreach { $ha += $_ }
write-host (" found " + $ha.Length + " event(s).")
if ($ha.Length -eq 0) {
    write-host "`nNo host failure events found in the last $last hours."
    write-host "Use parameter -last to specify number of hours to look back.`n"
    exit
} else {
    write-host ("`nLatest host failure event was " + $ha[0].ObjectName + " at " + $ha[0].CreatedTime + ".")
}

$events = $events | where-object { $_.CreatedTime -ge $ha[0].CreatedTime }

write-host "`nList of successful VM restarts:"
$events | where-object { $_.EventTypeID -eq "com.vmware.vc.ha.VmRestartedByHAEvent" } | foreach {
    write-host $_.CreatedTime: $_.ObjectName
}

write-host "`nList of failed VM restarts:"
$failures = @{}
$events | where-object { $_.FullFormattedMessage -like "vSphere HA stopped trying*" } | foreach {
    $vmname = $_.FullFormattedMessage.Split(" ")[6]
    if (!($failures.ContainsKey($vmname))) {
        $failures.Add($vmname,$_.CreatedTime)
        write-host $_.CreatedTime: $vmname
    }
}

Disconnect-VIServer -Force -Confirm:$false

The script takes three optional parameters:

• -vcenter servername: Connect to the vCenter server named servername (default is localhost)
• -last n: Analyze events of the last n hours (default is 24). If e.g. the HA event was during a weekend, and you run this script on Monday you may need to raise this to as much as 72. Please note: The higher n is the longer the script will run and the more memory it will consume!
• -help: display help on parameters

Usually you may want to specify at least the vCenter server name. If you have only one you can of course hard code it into the script as the default value (instead of localhost) in line 2.

How does it work?

As a first step the script reads all events of the last n (default: 24) hours and searches them for host failure events (event id "com.vmware.vc.HA.DasHostFailedEvent", see line 36). If there were multiple host failures in this time frame it will only look at the latest one and discard all earlier events (line 46).

To find out what VMs were restarted the script looks for events of id "com.vmware.vc.ha.VmRestartedByHAEvent". It will print the time stamps of these events and the names of the VMs that were restarted (line 49 to 51).

At last the script looks for events messages that start with the text "vSphere HA stopped trying". These events are thrown when HA fails to restart a VM multiple times in a row. With vSphere 5 HA will try very hard and repeatedly to restart a VM, so you might see this event multiple times for each failing VM. The script records the failing VMs in a hashed array in order to print their names only once, together with the time stamp of the latest failing restart attempt.

[Update] ESXi-Customizer Powershell script version 1.1

With the last part of my ImageBuilder Deep Dive series (also don't miss part 1 and 2) I published a Powershell script to build a customized ESXi 5.0 installation ISO, optionally adding the contents of the HP Online VIBs depot.

HP has recently updated the contents of their Online depots adding the latest releases of their bundles (see this post). I tested the first version of my script and found that it would incorrectly add older versions of the HP packages. So, I have fixed it now to add only the newest version of each available package. Other changes:

• Skip the hp-smx-limited package. It is not included in HP's customized images and conflicts with the full CIM provider package hp-smx-provider. In fact it is only useful for the new ProLiant Gen8 servers if you want to monitor these through the Agentless Management Service (AMS) only.
• Added a new command line switch -test. It will skip the actual package download and image building - useful for testing the script and seeing what it would add without wasting any download bandwidth.

The new version 1.1 is available for download now.

ESXi-Customizer-PS v1.1 help screen

HP ProLiant Firmware and Driver updates

Today HP released a new Service Pack for ProLiant (SPP): Version 2012.06.0(B). The SPP is a DVD image that includes the latest firmwares and drivers for all ProLiant servers (and associated components like Blade enclosures, iLO boards, NICs, CNAs and Smart Array controllers). Also included is the HP Smart Update Manager (updated to version 5.1.0) that you can use either on a Windows OS or by booting a ProLiant server off the DVD into Linux.

At the same time HP also published updates for their customized ESXi (4.1 U2 and 5.0 U1) installation CDs that include updated Offline bundles and updated drivers (both are also separately available).

Last but not least a new Virtual Connect firmware (version 3.60) hits the street today.

You can find download links to all the mentioned components on my HP & VMware Links page.

ImageBuilder Deep Dive, Part 3: The Power of the Shell

Let's start this third and last part of my ImageBuilder series with a short retrospective summary: In part one we introduced an easily adaptable script for building a customized ESXi 5.0 installation ISO. In the second part we looked closer at the basic cmdlets and learnt some new advanced ones.

In this final post we will walk through an improved and more universal version of the first script. It is more about Powershell in general than about ImageBuilder, so it will definitely help if you are already somewhat familiar with Powershell. But even if this is not the case you can still benefit by just using the improved script as it is. It took me some time to write it, because - according to Powershell - I was an absolute beginner when starting. I learnt a lot while developing it, but it is probably still not perfect ...

You can just download a copy of the script and use it. For the curious here is the listing:

#
# ESXi-Customizer-PS.ps1 - a script to build a customized ESXi installation ISO using ImageBuilder
# Version: 1.0
# Author: Andreas Peetz (ESXi-Customizer@v-front.de)
#

param(
    [string]$obDir = $(Split-Path $MyInvocation.MyCommand.Path),
    [string]$isoDir = $(Split-Path $MyInvocation.MyCommand.Path),
    [switch]$hp = $false,
    [switch]$help = $false
)

$AccLevel = @{"VMwareCertified" = 1; "VMwareAccepted" = 2; "PartnerSupported" = 3; "CommunitySupported" = 4}

function AddVIB2Profile($vib) {
    if ($AccLevel[$vib.AcceptanceLevel.ToString()] -gt $AccLevel[$MyProfile.AcceptanceLevel.ToString()]) {
        write-host -nonewline (" [New AcceptanceLevel: " + $vib.AcceptanceLevel + "]")
        $MyProfile.AcceptanceLevel = $vib.AcceptanceLevel
    }
    Add-EsxSoftwarePackage -SoftwarePackage $vib -ImageProfile $MyProfile | Out-Null 
    if ($?) { " [OK]" } else { " [FAILED]" }
}

# Write info and help if requested
write-host "`nScript to build a customized ESXi installation ISO using ImageBuilder"
if ($help) {
    write-host "Optional parameters:"
    write-host "   -help         : display this help"
    write-host "   -obDir <dir>  : directory of Offline bundles to add (default = script directory)"
    write-host "   -isoDir <dir> : directory to store the customized ISO (default = script directory)"
    write-host "   -hp           : add packages from the HP VIBs Depot (default = no)`n"
    exit
} else {
    write-host "(Call with -help for instructions)"
}

# Load the ImageBuilder Snapin (if not already loaded)
if (!(Get-PSSnapin -name VMware.ImageBuilder -ErrorAction:SilentlyContinue)) {
    if (!(Add-PSSnapin -PassThru VMware.ImageBuilder)) {
        # Error out if loading fails
        write-host "FATAL ERROR: Cannot load the ImageBuilder Snapin. Is the PowerCLI installed?"
        exit
    }
}

# Add the VMware ESXi base depot
write-host -nonewline "`nConnecting the VMware ESXi base depot ..."
if ($baseDepot = Add-EsxSoftwareDepot https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml) {
    write-host " [OK]"
} else {
    write-host "`n   FATAL ERROR: Cannot add VMware base Online depot. Please check your internet connectivity and/or proxy settings!"
    exit
} 

# Get the newest ImageProfile from the base depot
$LatestIP = (Get-EsxImageProfile "ESXi-5.0.0-20*-standard" | Sort-Object -Descending Name)[0]
write-host ("Using latest ImageProfile " + $LatestIP.Name)
write-host ("(dated " + $LatestIP.CreationTime + ", AcceptanceLevel: " + $LatestIP.AcceptanceLevel + ",")
write-host ($LatestIP.Description + ")")

# Create your own Imageprofile
$MyProfile = New-EsxImageProfile -CloneProfile $LatestIP -Name ($LatestIP.Name + "-customized") -Description ($LatestIP.Description + " + customizations")

if ($hp) {
    # Add the HP VIBs depot
    write-host -nonewline "`nConnecting the HP Online Depot ..."
    if ($hpDepot = Add-EsxSoftwareDepot http://vibsdepot.hp.com) {
        write-host " [OK]"
    } else {
        write-host "`n   FATAL ERROR: Cannot add HP Online depot. Please check your internet connectivity and/or proxy settings!"
        exit
    }
    $hpDepot.Channels[0] | Get-EsxSoftwarePackage | foreach {
        write-host -nonewline ("   Add VIB " + $_.Name + $_.Version )
        AddVIB2Profile $_
    }
}

# Loop over Offline bundles
write-host "`nAdding Offline bundles from" $obDir ...
foreach ($oBundle in Get-Item $obDir\*.zip) {
    write-host -nonewline "   Adding" $oBundle ...
    if ($ob = Add-EsxSoftwareDepot $oBundle) {
        write-host " [OK]"
        $ob.Channels[0] | Get-EsxSoftwarePackage | foreach {
            write-host -nonewline "      Add VIB" $_.Name $_.Version
            AddVIB2Profile $_
        }
    } else {
        write-host " [FAILED]`n      Probably not a valid Offline bundle, ignoring."
    }
}

# Export the Imageprofile into an installation ISO file
$isoFile = $isoDir + "\" + $MyProfile.Name + ".iso"
write-host -nonewline ("`nExporting to ISO file " + $isoFile + ". Please be patient ...")
Export-EsxImageProfile -ImageProfile $MyProfile -ExportToIso -FilePath $isoFile -Force
if ($?) { " [OK]" } else { " [FAILED]" }

write-host "`nAll done."

Let's look at the script line by line:

Line 7 - 12: The script accepts command line parameters that are defined inside a param() statement. One of the possible parameters is the switch -help. If specified the script will print a help text (explaining the rest of the parameters) and exit (see line 25ff.). Another optional switch is -hp which will add the HP Online VIBs depot (s. line 65ff.). Using -obDir you can specify a directory where the script will look for Offline bundle zip files to be added. And with -isoDir you can specify the directory to store the created ISO file. The default for both directories is the path where the script itself is stored (which is determined by $(Split-Path $MyInvocation.MyCommand.Path)).

Line 14: One of the advanced Powershell features is using hash tables like the one that is created here. It contains the string representations of the four possible acceptance levels that a package or an image profile can have. They are assigned a ranking value (1 to 4) that is used to determine what the most restrictive acceptance level is ("VMwareCertified" = 1) as opposed to the least restrictive ("CommunitySupported" = 4).

Line 16 - 23: In these lines we define a Powershell function named AddVIB2Profile that adds a VIB package ($vib) that is passed as an argument to the custom image profile. As a first step it compares the acceptance level of the software package with that of the image profile (line 17) to determine if the latter needs to be lowered in order to accept the package (line 18 - 19). This is where the above mentioned hash table is used. Then it adds the package using the Add-EsxSoftwarePackage cmdlet. As I want the output of the script to be pretty and not garbled by useless information I pipe the output of the cmdlet into out-null. That means that it is just discarded. In line 22 I use the special Powershell variable $? to test whether the previous command was successful or not. This is a basic and the easiest way to add error handling to a Powershell script, and I will use that a lot here.

Line 25 - 36: This is actually the start of the main script. It prints a title line (line 26), then it will either display a help text and exit the script if -help was specified (line 27 - 33), or it will display only a short line explaining that -help can be used for instructions (line 35) and continue. BTW the special characters `n represent a new line when used in Powershell strings.

Line 39 to 45: Here we will check whether the required snapin VMware.ImageBuilder was already loaded (e.g. because you used the PowerCLI desktop shortcut to start your Powershell session). If not then we try to load it with Add-PSSnapin (line 40). If this fails (e.g. because PowerCLI is not installed on the computer) then the script errors out, because it depends on the ImageBuilder snapin being available and loaded.

Line 47 to 54: Now we add the VMware ESXi Online depot to the ImageBuilder session (also with some error handling). Nothing spectacular, we've seen this before.

Line 57 to 60: In line 57 we grab all standard image profiles from the VMware depot, create a sorted array of them and just pick the first one (with index [0]). This is the newest one, because we sorted the array in descending order. In the following lines we output some information about this image profile that is stored in its attributes.

Line 63: Here we create our custom image profile by cloning the latest one from the VMware depot. Its name and description are derived from the original VMware profile's properties.

Line 65 to 78: If the command line switch -hp was specified then we connect the HP Online VIBs depot to the ImageBuilder session (line 68 to 73) and add all included software packages using the AddVIB2Profile function that we defined at the beginning of the script.

Line 81 to 93: The foreach loop starting at line 82 is another essential part of the script code: Here we loop over all zip files that are located in the obDir directory and try to add them as Offline bundles to the ImageBuilder session (line 84). If it succeeds because a zip file is a valid Offline bundle (and not some other sort of file archive) then all software packages that are included in the bundle will be added to the custom image profile, again using our self-defined AddVIB2Profile function (line 88).

Line 96 to 99: Finally we export the custom image profile into an installation ISO. Its file name is derived from the image profile's name.

How do you run this Powershell script (and others)?

Open a plain Powershell window from the Start menu, or use the PowerCLI desktop shortcut to start an already customized Powershell session.

By default Powershell will only execute scripts that were electronically signed for security reasons. This script (and most other useful scripts that you will find in other places) is not signed, so you will need to relax the security check by using the Set-ExecutionPolicy cmdlet like shown in the following screenshot. You can do this for the current user only by specifying -Scope CurrentUser. This will also work if you do not have administrative permissions on the computer. If you have administrative permissions then you can leave the -Scope parameter and make the setting system global. Setting the execution policy needs to be done only once - it will be saved to the registry and reused for future Powershell sessions.

You can just call the script by its name (adding parameters as required). Since the script is from an "untrusted" source you still need to confirm its execution every time you start it:

ESXi-Customizer-PS help screen

Finally this is an example output from a complete script run:

ESXi-Customizer-PS example run

The script that I provide here is fully functional and flexible enough to meet most requirements. However, it is certainly not perfect. E.g. it could have better error handling. If you have any comments or ideas on how to improve it (or if you run into errors when trying it) then please add a comment to this post!

Update (2012-06-07): In the meantime I updated this script to fix errors. Please watch out for the latest version using this download link!

How to disable Storage I/O Control for an unavailable datastore

If you run your VMs on FC based storage then you occasionally have the necessity to unmap a storage LUN from one or more ESX(i) hosts (e.g. when retiring a storage array or re-organizing your storage layout). It is important to do this in the right way using the procedures that are documented in

• KB1029786 for ESX(i) 4.1
• KB2004605 for ESXi 5.0

The procedure is very complex for vSphere 4.1, but fortunately is has become much easier with vSphere 5.0.

What happens if you fail to follow these procedures and just unpresent the LUN on the storage array, so that the hosts can not access it anymore? The ESX(i) hosts will detect an APD (all paths down) condition in this case, and - particularly ESX(i) 4.1 hosts - can become very unhappy about this (see KB1016626 and KB1030980). Again, ESXi 5.0 hosts are much more resilient to APD conditions: they will eventually turn them into PDL (= permanent device loss, s. KB2004684) conditions and will completely recover from the LUN loss after rescanning the HBAs ... unless you have Storage I/O control (SIOC) enabled on the lost datastores ... and this is what happened to us today :-( The vmkernel.log files were flooded with the following messages, because SIOC was trying to access the lost datastore:

Permanent Device Loss (PDL) with SIOC enabled

Now the problem is: You cannot just disable SIOC on a lost datastore using the vSphere client - you should have done this before unmapping the storage LUN!

One way to recover from this situation is to reboot any of the affected hosts. However, I really wanted to save the time to put all the hosts in maintenance mode and reboot them one after the other. So I looked for a way to forcibly unmount the datastore directly on the host via esxcli or similar. All the ways that are documented in the VMware KB articles and docs did not work in my case, but I finally stumbled over this wonderful blog post by William Lam:

Does SIOC actually require Enterprise Plus and vCenter?

It is a bit old and refers to ESX(i) 4.1, but it is still valid for ESXi 5.0! Here William describes a way to enable (and disable!) SIOC for a datastore directly on a host without using (even without having available!) a vCenter server.

And it is really easy: All you need to know is the device ID of the datastore/LUN. In ESXi 5.0 you can find it out by using the command

   # esxcli storage vmfs extent list

It will list all datastores with their labels and device IDs (starting with naa.). And then you can use the following vsish command to disable SIOC for the device

   # vsish -e set /storage/scsifw/devices/<naa-id>/iormState 1496

That's it! After waiting a few seconds the affected datastore suddenly disappeared from the list of mounted datastores in the vSphere client, and the VMkernel.log error messages also stopped.

Please note: vsish is a powerful but largely undocumented utility to query and set VMkernel parameters. It is only available in an ESXi local or remote shell. William has quite a few posts about it on his virtuallyGhetto blog. This time it saved us the trouble and time of rebooting a whole cluster of hosts ...