Thursday, May 10, 2012

Updated: What's the deal with EMC CX arrays not supporting VAAI with vSphere 5.0?

(If you have read this post before then skip to the bottom for recent updates)

We are in the process of upgrading our production environment to vSphere 5.0 U1. A new vCenter 5.0 server is already in place, and we attached the production ESXi hosts (that are still on 4.1 U2) to it.
The next step would be to upgrade the hosts. In the meantime, since this is a qualified production environment running more than 1.500 virtual servers I became a bit paranoid about hardware and software/firmware compatibility, and decided to double check if the environment is fully supported (or if we should upgrade any firmware first). I was pretty confident that we are on the safe side because we never had any issues with vSphere 4.1 and always kept the environment up to date.

But then I stumbled over this VMware KB article: EMC CX and VNX Firmware and ESX requirements for vStorage APIs for Array Integration (VAAI) support. It states that ESXi 5.0 does not support VAAI on our Clariion CX4 arrays even although they have the latest FLARE code:


vSphere 4.1vSphere 5.0
CX4 Series Flare 30/29/28+VAAI SupportedNot Supported
VNX Series OE 31 or Later *VAAI SupportedVAAI Supported

I checked the test hosts that we already updated and they showed VAAI being supported on the CX4 LUNs in the vSphere client. However, the KB article recommends to disable VAAI on these hosts ...

I quickly searched the Internet for relevant posts and statements and browsed through the VMware Community forums. There I found this post where people complain about the VAAI status shown as unsupported. It looks like there are dependencies to the LUN size (smaller or larger than 2TB) and to the array's failover mode (sounds like you need to use ALUA / failover mode = 4 which we are already using). But from this post I get the impression that everything is fixed with the latest 30.x FLARE code releases).

We contacted both VMware and EMC to get some clarification now, and I will keep this post updated with any new information. In the meantime I ask everyone using ESXi 5.0 with EMC CX storage to comment on this post whether you are using VAAI, if you have any issues with it, and what FLARE code you have on the arrays. BTW, you can check what VAAI primitives are in use with your storage LUNs by running the following esxcli command:
   esxcli storage core device vaai status get

Thanks in advance for any helpful comments!


Update (2012-05-20): I got some feedback from other users, VMware and EMC on this issue, and used my best Google-Fu to get related information about vSphere 5, VAAI, VMFS-5 and/or EMC CX support. Here are the results (somewhat unsorted):
Regarding my specific issue (lack of official support for VAAI with EMC CX arrays and ESXi 5.0) I finally got one really detailed and helpful comment from an EMC representative: EMC tested the VAAI functionality of ESXi 5.0 with their EMC CX4 arrays (I specifically asked for the CX4). It has been "tested as functional", but it only passed the certification tests of the ATS and Zero primitives, but failed the certification test of the XCopy primitive.
I wonder why this happened, because VAAI was fully supported with ESXi 4.1 (i.e. it must have also passed the XCopy test), and with ESXi 5.0 there were no changes to the XCopy primitive (at least I could not find any information about such a change).

Anyway, this is probably the reason why EMC and VMware do not officially support VAAI with ESXi 5.0 on CX arrays. From a customer's point of view this is disappointing, but from EMC's point of view this decision is understandable, because they want to focus on their current products and not waste support resources on somewhat outdated arrays like the CX ones. However, according to this EMC representative it may be possible to come to an individiual support agreement with EMC and VMware (EMC calls this RPQ = Request for Product Qualification) if you nevertheless must or want to have an official support statement for whatever reason.

I also had the idea of disabling only the XCopy primitive to be on the safe side, but I don't want to do this globally, because we also use a fully VAAI supported VNX array with our production hosts. So I was glad to find a link to the promising KB article KB2012967 (see list above), but this link currently doesn't work. I will ask VMware about this ...



Update (2012-06-20): In the meantime an EMC representative confirmed to me that all three VAAI primitives work fine with vSphere 5.0 and CX arrays, and that they even support it, but you need to file an RPQ with them in order to get formal support.

We have this configuration (VAAI turned on for both our CX4 and VNX arrays) running with vSphere 5.0 in production for about 6 weeks now without any problems.

The link to VMware KB article KB2012967 still doesn't work, but it is now listed to as being "[Archived]" in the reference section of KB1033665 ...





Update (2012-07-11): It turned out that you no longer need an RPQ with EMC. They now officially support the vSphere 5's VAAI implementation on the CX4. The combination is listed in their latest Simple Support Matrix for VMware vSphere 5 (Powerlink account needed for download):
Snippet from the EMC Simple Support Matrix for vSphere 5 (of July 2012)
VMware though does still not officially support it (KB2008822 is unchanged as of today). In another blog post I already explained the reason for this.







Monday, May 7, 2012

ImageBuilder Deep Dive, Part 2: A closer look and advanced functions

In the first part of this Image Builder Deep Dive I introduced a script that you can use to build your own customized installation ISO and can easily be adapted to your own needs. In this second part we will take a closer look at the basic cmdlets we used and get to know some more cmdlets with advanced functionality.

To make best use of the following explanations I suggest that you load the complete script that we go through in this part into the PowerShell ISE and run it line by line while reading the post. You can also experiment with the various code samples by pasting them into the command prompt of the ISE.

Using Online depots with proxy servers

At the beginning of the first part's script we added two Online depots to the ImageBuilder session using the Add-EsxSoftwareDepot cmdlet. I stated that this is not possible when using a proxy server, and  - after some testing - I need to correct this statement: I got it working with a proxy server that does not require authentication. With a proxy server requiring authentication it worked on the first try and failed when adding the second online depot (?!). Originally I had a proxy auto-configuration URL entered in the Internet Explorer settings, and this made adding an Online depot consequently fail. Your mileage may vary ...

By default PowerCLI will use the Windows proxy settings (that means what you have configured in Internet Explorer). You can change that by using the Set-PowerCLIConfiguration cmdlet. However, the possibilities are somewhat limited. You can either use the system proxy settings or no proxy at all:
# Load the VimAutomation Snapin
Add-PSSnapin VMware.VimAutomation.Core
#
# Set and get the proxy configuration
Set-PowerCLIConfiguration -ProxyPolicy NoProxy
Set-PowerCLIConfiguration -ProxyPolicy UseSystemProxy
Get-PowerCLIConfiguration
#
Please note: You need to add the VMware.VimAutomation.Core snapin first (if not already added) to use these cmdlets. To display the current setting use the Get-PowerCLIConfiguration cmdlet.

How to "download" the VMware Online base depot

Now you may wonder: What if your computer has no Internet connection (or only via a proxy configuration that causes the above mentioned problems)? Can I somewhat "download" an Online depot and re-use it later on the same or another computer? Yes, you can!

To be exact: You cannot create an offline copy of a complete Online depot, but you can export a specific ImageProfile not only to an installation ISO, but also into a re-usable Offline bundle:
# Load the ImageBuilder Snapin
Add-PSSnapin VMware.ImageBuilder
#
# Reference the VMware ESXi base depot
$baseDepot = Add-EsxSoftwareDepot https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml
#
# List available ImageProfiles sorted by name (filtered on updated standard profiles)
Get-EsxImageProfile "ESXi-5.0.0-2012*-standard" | Sort-Object -Descending Name
#
# The previous command outputs "ESXi-5.0.0-20120504001-standard" as the newest profile (as of May 5th 2012)
# Export this Imageprofile into an Offline bundle zip file
Export-EsxImageProfile -ImageProfile ESXi-5.0.0-20120504001-standard -ExportToBundle -FilePath 'U:\ESXi-5.0.0-20120504001-standard.zip'
#
Now you can re-use this exported zip file any time and on any other computer by adding it with the Add-EsxSoftwareDepot cmdlet. Obviously this Offline bundle zip file only contains the ImageProfile that you exported and only the packages that are part of this ImageProfile.

Please note: The HP Online depot cannot be "downloaded" in this way, because it does not expose any ImageProfiles that you can export. However, in this special case you can also open the http://vibsdepot.hp.com URL in a browser and download the Offline bundles that it includes from the linked web pages.

Looking closer at the ImageProfile object

Apropos ImageProfiles: So far we only looked at the names of these objects. Obviously VMware codes the release date of an ImageProfile (= patch level) into its name using the format YYYYMMDD. Hopefully they will stick with this naming standard - it makes it easy to identify the newest ImageProfile at first sight and programmatically.

The ImageProfile object has more interesting attributes though:
# Examining the ImageProfile's attributes
$ip = Get-EsxImageProfile ESXi-5.0.0-20120504001-standard
$ip | format-list
$ip.VibList
#
By piping the object through the format-list cmdlet you can display all its attributes in a nicely formatted list view:
Name            : ESXi-5.0.0-20120504001-standard
Vendor : VMware, Inc.
Author :
Description : For more information, see http://kb.vmware.com/kb/2019863.
CreationTime : 30.04.2012 21:47:06
ModifiedTime : 30.04.2012 21:47:06
ReadOnly : False
VibList : {ata-pata-atiixp 0.4.6-3vmw.500.0.0.469512, net-nx-nic 4.0.557-3vmw.500.1.11.623860, scsi-rste 2.0.2.0088-1vmw.500.1.11.623860, net-e1000 8.0.3.1-2vmw.500.0.7.515841...}
AcceptanceLevel : PartnerSupported
Guid : 076da48a930e11e1b58d0017a477682c
Rules :
StatelessReady : True

The Description attribute is interesting, because its value includes a URL to a VMware KB article with further information about this ESXi 5.0 patch release.

The VibList attribute is an array of all included software packages. The list is shortened in the formatted object view to retain readability of the output. The third command above ($ip.VibList) will output the complete list of packages with names and versions.

The AcceptanceLevel attribute is something that you normally don't need to take care of, but we will come back to it at the end of this post.

How to compare two ImageProfiles

If you want to quickly check the differences of two ImageProfiles you could list their attributes one after the other like shown before and compare the outputs, especially the VibLists. However, there is a more elegant way to do this - you can use the Compare-EsxImageProfile cmdlet:
# Comparing two ImageProfiles
Compare-EsxImageProfile ESXi-5.0.0-20120504001-standard -ReferenceProfile ESXi-5.0.0-20120404001-standard
#
This command will create and output an ImageProfile comparison object. In this example it looks like this:
Equal               : False
PackagesEqual : False
RefAcceptanceLevel : PartnerSupported
CompAcceptanceLevel : PartnerSupported
OnlyInRef : {}
OnlyInComp : {}
UpgradeFromRef : {VMware_bootbank_esx-base_5.0.0-1.13.702118}
DowngradeFromRef : {}
What do we see here? The two ImageProfiles are not equal (Equal = False), they do not contain the exact same list of packages (PackagesEqual = False), but they have the same AcceptanceLevel (PartnerSupported). The remaining four attributes summarize the differences in the software package list of the two ImageProfiles. Their names are pretty self-explanatory, and in this example the only difference is that the comparison profile (ESXi-5.0.0-20120504001-standard) contains a newer version of the esx-base package than the reference profile (ESXi-5.0.0-20120404001-standard).

Examining, adding and removing single software packages from a depot

In the first part of this series of posts we used a pipeline of commands to add all packages of a depot to our custom profile. Now we are going to take a closer look at a single package using the Get-EsxSoftwarePackage cmdlet:
# Reference downloaded HP offline bundle for be2net driver
$be2net = Add-EsxSoftwareDepot "U:\HP-ESXi5-Drivers\be2net-4.0.355.1-offline_bundle-487292.zip"
#
# Look at all versions of a single software package (net-be2net)
Get-EsxSoftwarePackage net-be2net
# Look at all properties of a package's specific version
Get-EsxSoftwarePackage net-be2net -Version 4.0.88.0-1vmw.500.0.7.515841 | fl
#
The output of the command in line 34 looks like this:
Name                     Version                        Vendor     Release Date    
---- ------- ------ ------------
net-be2net 4.0.88.0-1vmw.500.0.7.515841 VMware 15.12.2011 00...
net-be2net 4.0.88.0-1vmw.500.0.0.469512 VMware 19.08.2011 01...
net-be2net 4.0.355.1-1OEM.500.0.0.406165 Emulex 16.09.2011 00...
That means that there are three different versions of the net-ne2net package, the first two are in the VMware base depot, and the third one is in the Offline bundle we downloaded from HP.

In line 36 we examine the properties of a specific version of a package. We will see that the SoftwarePackage object has some interesting attributes:
Name            : net-be2net
Version : 4.0.88.0-1vmw.500.0.7.515841
Vendor : VMware
Summary : Updates the ESX 5.0.0 net-be2net
Description : For build informatin, see KB http://kb.vmware.com/kb/2007675
ReleaseDate : 15.12.2011 00:00:00
Depends : {vmkapi_2_0_0_0, com.vmware.driverAPI-9.2.0.0}
Conflicts : {}
Replaces : {}
AcceptanceLevel : VMwareCertified
MaintenanceMode : False
LiveInstallOk : False
LiveRemoveOk : False
SourceUrls : {https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/net-be2net/VMware_bootbank_net-be2net_4.0.88.0-1vmw.500.0.7.515841.vib}
Guid : VMware_bootbank_net-be2net_4.0.88.0-1vmw.500.0.7.515841
StatelessReady : True
Tags : {driver, module, category:bugfix, severity:general}
Like with the ImageProfile object the Description property contains a link to a VMware KB article with further information on this package. The SourceUrl indicates from where the package will be downloaded. And the package's Tags provides additional classification categories.

The three properties Depends, Conflicts and Replaces describe relationships to other packages (or features that are provided by other packages). Their names are self-explanatory, and ImageBuilder will take care of them for us. You will e.g. not be allowed to add a package that conflicts with another package that is already part of the profile.

Now let's add this single package to a custom profile using the Add-EsxSoftwarePackage cmdlet:
# Create a new ImageProfile
$MyProfile = New-EsxImageProfile -CloneProfile ESXi-5.0.0-20120504001-standard -Name MyProfile
#
# Add an older version of the net-be2net package to the custom profile
Add-EsxSoftwarePackage -ImageProfile $MyProfile "net-be2net 4.0.88.0-1vmw.500.0.7.515841"
# Add the neweset net-be2net package to the custom profile
Add-EsxSoftwarePackage -ImageProfile $MyProfile net-be2net
#
If you just specify the name of the software package (net-be2net in this example), and there are multiple versions of this package available then Add-EsxSoftwarePackage will automatically add the newest version. If you want to add an older version then you can specify it like shown in line 42. Whenever the ImageProfile already includes a package of the same name, but a different version, it will be replaced by the added package, even if it is older! If the ImageProfile already includes the exact same version then the command will fail, or in other words: you cannot add the same version twice.

Finally, you can also remove a package from an ImageProfile using the Remove-EsxSoftwarePackage cmdlet:
# Remove a package from the custom profile
Remove-EsxSoftwarePackage -ImageProfile $MyProfile net-bnx2
#
You won't be able to remove packages that are required by other packages of the ImageProfile. In some case you can override this dependency check by using the parameter -Force, but you really shouldn't do that, because it will most likely result in an invalid ImageProfile.

Adding community developed packages

If you have ever used my ESXi-Customizer script to add a Community developed software package to an ESXi 5.0 installation ISO then you may wonder: Can you also do that with ImageBuilder? Yes, you can, and here is how:
# Add an Offline Bundle with a community developed software package
Add-EsxSoftwareDepot 'U:\$Download\fwenable-ntpd-1.2.0-0-offline_bundle.zip'
# Change the AcceptanceLevel of the custom profile to "CommunitySupported"
$MyProfile.AcceptanceLevel = "CommunitySupported"
# Add the package to the profile
Add-EsxSoftwarePackage -ImageProfile $MyProfile fwenable-ntpd
In this example we add a software package that includes a custom firewall rule to allow incoming NTP queries (this way you can use the ESXi host as NTP server. Read this post for background information). The Offline bundle was created using my ESXi5 Community Packaging Tools by converting a standard tar.gz file to a VIB file first, and then packaging the VIB file into an Offline bundle zip. The project page of the tools contains detailed instructions on how to do this.

There is only one particularity that you need to be aware off when adding Community developed packages this way: Software packages that were created (and certified) by VMware or trusted partners contain an electronic signature that is checked when adding (or installing) the package. This qualifies them for one of the following three so-called AcceptanceLevels: VMwareCertified, VMwareAccepted or PartnerSupported. Of course, Community developed packages do not have trusted electronic signatures, which means that you can only install them if they have the least restrictive (and least trustworthy) AcceptanceLevel called CommunitySupported. The ImageProfile object also has an AcceptanceLevel, and that must be lowered to the same level before you are able to add a Community developed package. This happens in line 52.

This ends the second part of the ImageBuilder Deep Dive series. The next and last part will not deal with the ImageBuilder snapin specifically, but with Powershell coding in general. We will then develop the script from the first part to a more general and versatile solution.


Sunday, April 29, 2012

ImageBuilder Deep Dive, Part 1: Building your own customized ESXi ISO

I know that my ESXi-Customizer script has gotten some popularity and a lot of people use it for merging community developed or commercial third-party hardware drivers into the ESXi installation ISO. It has some limitations though - some of you might have stumbled over them already, at least I have described them in the "Known issues" section. Probably the worst is the fact that it cannot handle complex software bundles like ESXi patches (e.g. the ESXi 5.0 Update 1 package).

My recommendation for such cases is to use the VMware supplied and supported method to create your own customized installation ISOs: The ImageBuilder PowerCLI snapin. With this post I'm going to start a series of blog posts that will cover ImageBuilder in detail and will help you to make effective use of it.

This first post will cover the prerequisites and installation of the PowerCLI and will introduce a script that will help you to get the task done. You won't need in-depth PowerShell knowledge for this, but it will definitely help if you are also interested in the remaining parts of the series that will go into the details of the ImageBuilder cmdlets and finally refine the first script to a more universal solution.

Prerequisites and installation

Obviously you need a Windows computer with the current version (2.0) of Powershell installed. Powershell is Microsoft's state-of-the-art scripting language - it is already included with Windows 7 and Windows 2008 R2 server, for earlier versions of Windows it is available as part of the Windows Management Framework. If you are a serious Windows admin you probably have looked at it before. If not then this is a good opportunity to do it. You can start learning e.g. at Microsoft's Script Center, but this is not a prerequisite for now!

The functionality of Powershell can be extended through so-called snapins. VMware makes available such snapins to add functions (so-called cmdlets) that you can use to manage vCenter servers and ESX(i) hosts. The thing is called PowerCLI, and once you have downloaded and installed the current version you are ready to go!

Following is a Powershell script that will create an ESXi 5.0 installation ISO with the current patch level, the HP Offline bundles and some HP drivers. I suggest that you do the following to walk through the explanations below:
  • Download a copy of the script to your computer
  • In Explorer right-click on the file and choose "Edit" from the context menu. This will open it in the Powershell ISE (Integrated Scripting Environment) editor.
  • Within the ISE you can select single (or multiple) lines of the script and execute them separately from the rest:
Run selected lines of code in the Powershell ISE

The first script
# Load the ImageBuilder Snapin
Add-PSSnapin VMware.ImageBuilder

# Reference the VMware ESXi base depot
$baseDepot = Add-EsxSoftwareDepot https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml

# Reference the HP VIBs depot
$hpDepot = Add-EsxSoftwareDepot http://vibsdepot.hp.com

# List the VIB packages of HP depot
$hpDepot.Channels[0] | Get-EsxSoftwarePackage

# Reference downloaded HP driver offline bundles
$be2iscsi = Add-EsxSoftwareDepot "U:\HP-ESXi5-Drivers\be2iscsi-4.0.317.0-offline_bundle-469760.zip"
$be2net = Add-EsxSoftwareDepot "U:\HP-ESXi5-Drivers\be2net-4.0.355.1-offline_bundle-487292.zip"
$lpfc820 = Add-EsxSoftwareDepot "U:\HP-ESXi5-Drivers\lpfc820-8.2.2.105.36-offline_bundle-489567.zip"

# List available Imageprofiles sorted by creation date (newest first)
Get-EsxImageProfile | Sort-Object -Descending CreationTime | Format-List Name,CreationTime

# Create your own Imageprofile
$MyProfile = New-EsxImageProfile -CloneProfile ESXi-5.0.0-20120404001-standard -Name MyProfile -Description "ESXi-5.0.0-20120404001-standard + HP components"

# Add all the HP VIB packages to MyProfile
$hpDepot.Channels[0] | Get-EsxSoftwarePackage | Add-EsxSoftwarePackage -ImageProfile $MyProfile
$be2iscsi.Channels[0] | Get-EsxSoftwarePackage | Add-EsxSoftwarePackage -ImageProfile $MyProfile
$be2net.Channels[0] | Get-EsxSoftwarePackage | Add-EsxSoftwarePackage -ImageProfile $MyProfile
$lpfc820.Channels[0] | Get-EsxSoftwarePackage | Add-EsxSoftwarePackage -ImageProfile $MyProfile

# Export the Imageprofile into an installation ISO file
Export-EsxImageProfile -ImageProfile $MyProfile -ExportToIso -FilePath "U:\MyProfile.iso"
Line 2: This command will add the ImageBuilder snapin to the current Powershell session. Please note: If you start your session with the PowerCLI shortcut on your desktop then the snapin will automatically be loaded and you can skip this line.

Line 5: The Add-ESXSoftwareDepot cmdlet adds an Online depot or a downloaded Offline bundle as a package source to the current ImageBuilder session. The VMware depot referenced here includes the base ESXi 5.0 sources and is needed in any case. Obviously adding an Online depot requires a working connection to the Internet. I could get this to work with a direct connection only, but not through a proxy server. If someone knows how to make this work with a proxy then please comment here!

Line 8: This adds an additional Online depot that was made available by HP and includes their Offline bundles for ESXi 5.0.

Line 11: A software depot object like $hpDepot that was created through the Add-EsxSoftwareDepot cmdlet can be organized in multiple channels (but mostly it's only one channel). By piping the first channel into the Get-EsxSoftwarePackage cmdlet we can list the software packages that are included in this depot. The output looks like this:
Name                     Version                        Vendor     Release Date    
---- ------- ------ ------------
hpnmi 2.0.11-434156 hp 29.07.2011 20...
char-hpilo 500.9.0.0.9-1OEM.500.0.0.43... Hewlett... 07.10.2011 14...
hp-ams 500.9.0.0-55.434156 Hewlett... 09.03.2012 23...
char-hpcru 5.0.0.8-1OEM.500.0.0.434156 Hewlett... 15.07.2011 17...
hpbootcfg 01-00.10 Hewlett... 15.07.2011 16...
hpacucli 9.0-24.0 Hewlett... 02.02.2012 06...
hponcfg 04-00.10 Hewlett... 13.11.2011 02...
hp-smx-provider 500.02.10.61.43-434156 Hewlett... 18.11.2011 02...


This step is not really needed to build the new ISO. So you can safely skip it if you already know or are not interested in the contents of a depot.

Lines 14-16: In these lines we add three additional depots, but this time these are not Online depots, but Offline bundles that we downloaded before. You can find the download links on my HP & VMware links page (see section ESXi 5.0 drivers for the HP Emulex 10GbE Converged Network Adapters). But please note: The files that you download from VMware's driver pages are in zip format, but they are not Offline bundles. They include Offline bundles though, so you need to extract the *offline_bundle*.zip files from the downloaded zip-files first. And of course you need to change the file paths used in the script to your own download directory.

Line 19: The base depot that we added in line 5 includes not only software packages, but also so-called image profiles. An image profile is a grouping of software packages out of the depot. In this case each image profile makes up a specific ESXi patch level. The Get-EsxImageProfile cmdlet will list all available image profiles. We pipe it through the Sort-Object cmdlet here to sort the output by creation date. The output will show the newest image profile first and looks like this:
Name         : ESXi-5.0.0-20120404001-no-tools
CreationTime : 16.03.2012 22:59:09

Name : ESXi-5.0.0-20120404001-standard
CreationTime : 16.03.2012 22:59:09

Name : ESXi-5.0.0-20120302001-no-tools
CreationTime : 16.03.2012 22:59:08

Name : ESXi-5.0.0-20120302001-standard
CreationTime : 16.03.2012 22:59:08

Name : ESXi-5.0.0-20120301001s-standard
CreationTime : 16.03.2012 22:59:08

Name : ESXi-5.0.0-20120301001s-no-tools
CreationTime : 16.03.2012 22:59:08

Name : ESXi-5.0.0-20111204001-standard
CreationTime : 31.10.2011 11:24:00

(... output shortened for readability ...)

Each image profile comes in two different flavors: The -standard one includes all packages that make up an ESXi 5 installation, the -no-tools version is the same without the VMware Tools package. We will choose the standard version in our example.

Line 22: The image profiles of the Online depot are read-only. Since we want to modify one of these and add more packages, we need to create a copy first. With the New-EsxImageProfile cmdlet we create a new image profile ($MyProfile) by cloning the newest standard image profile (ESXi-5.0.0-20120404001-standard in this case). We choose the newest one, because it represents the most recent patch level of ESXi! We also assign a new name to the cloned profile (this is mandatory) and a new description (optional).

Lines 25-28: In line 11 we already listed the software packages that are included in the $hpDepot. By piping the output of this command to the Add-EsxSoftwarePackage cmdlet we add all the included packages to our newly created image profile $MyProfile. And we do the same for the three Offline bundles that we added in the lines 14 to 16.

Line 31: In the last line we use the Export-EsxImageProfile cmdlet to "export" our customized image profile into an ISO file. Now guess what: This ISO file is a complete ESXi 5.0 installation media! You can use it to install a machine with ESXi 5.0 with the current patch level and all the HP packages that you added before.

Booting an ImageBuilder customized ESXi 5.0 installation ISO
Wrap-up

In this first part of my ImageBuilder Deep Dive series I introduced a script that you can use to build an up-to-date ESXi 5.0 installation ISO with additional drivers. You learnt some basic terms, the most important cmdlets that are necessary to get the job done, and you will (hopefully) be able to modify the script and adapt it to your own needs.

In the next part I will introduce some more useful ImageBuilder cmdlets, and I will explain how you can integrate community developed software packages to the installation media (a job that you can also do with my ESXi-Customizer script). Stay tuned!

Monday, April 16, 2012

New page: HP & VMware links

I had this plan for some time now: Providing a list with links to HP drivers and firmware downloads and useful documents for VMware ESX(i).

Why? Because it is so hard to find this stuff on the HP pages ... Currently they are revamping their web pages, and I thought maybe this is getting better now, and that they would finally begin to provide their downloads in a well structured way making it easy to find the stuff again and link to it. But they didn't - it looks like they are just changing the layout of the web pages and are trying to break their own ridiculous world record for the longest URLs.

Okay, enough grumbling. Here is the list. Of course it is far from complete and mainly covers the hardware that I'm using myself (that makes sure that I will keep it up to date), but I believe that everyone using VMware on HP hardware will find it useful.

Monday, April 9, 2012

Review: PHD Virtual Backup and Replication for VMware vSphere

One reason why I consider virtual servers to be superior to physical servers is that you have advanced hypervisor based technologies and functionalities available that let you manage virtual servers more efficiently. A good example for this is doing backups.

PHD Virtual was among the first to provide a virtualization specific backup solution. They use a Virtual Appliance approach without the need for physical hardware and make use of VMware's Change Block Tracking (CBT) feature to enable efficient block level incremental backups. Their backup product has evolved since 2006 and is now available as PHD Virtual Backup and Replication v5.4. It is compatible with VMware vSphere 4.1 and 5.0 (I used both for testing) and also works with Citrix XenServer.

How does it work?

PHD Virtual Backup architecture
The main engine of the product is a Linux based Virtual Appliance that is deployed to the virtualization hosts that also run the virtual machines that are to be backed up. For easy scale-up you can have multiple copies of such Virtual Backup Appliances (VBA). They connect to a single ESX(i) host or to a vCenter server managing multiple hosts.

For backing up a VM the VBA first creates a snapshot for it through this connection. Creating a snapshot means that each virtual hard disk of the VM will be frozen into a read-only base disk, and all changes to the disk (done by the guest OS) will be recorded in an extra delta file. The VBA will then hot-add the read-only base disk to itself and do a block level backup of it to a backup store. The backup store can be a locally attached virtual disk or a network share that was exported via CIFS or NFS.

The very first backup will copy all blocks of the disk, but to minimize the overall backup data and time the VBA will use smart deduplication and compression algorithms. For subsequent backups of the same disk you can make use of CBT, and that means that only the blocks that were changed or added since the last backup will be stored. Every backup is a full backup though, because the unchanged blocks are also linked into each new backup set. This strategy is also known as incremental forever. Together with the applied deduplication and compression this is probably the most efficient backup method that you can implement these days.

After all (changed) blocks have been backed up the VBA will hot-remove the disk again from itself, and - as a last step - initiate the deletion of the VM's snapshot. Needless to say the VBA will of course also save the VM's configuration, so that it can later be completely re-created and restored from the backup set.

Ease of installation and configuration

One promise that PHD Virtual does for its product is that you can have it up and running in 5 minutes. So, how long did it take me? Well, in fact it was less than 5 minutes!

The software download includes the VBA in OVF format and a Windows based management console software. In a minute I imported the appliance to the vCenter server of my test environment, it's ~800 MB in size and inflates to 8 GB if thick provisioned. Before powering it on I attached an additional 100 GB disk to the VBA that I planned to use as backup storage. When powered on the VBA configured its network through DHCP (manually setting an IP address is also possible on the console).

While the import was running I also installed the management software on my workstation. A typical Windows setup wizard did the job with five mouse clicks and left me no chance to make a mistake.

Installation done! I then launched the management console and entered the name of the vCenter server and login credentials for it. The application started with a Dashboard view where the VBA was already listed and shown as waiting to be configured:

PHD Virtual Backup console - first launch
In the Configuration dialog you need to define some General Settings and assign a Backup Storage to the VBA:
VBA General Settings
Time sync through NTP should be properly set up in this dialog to ensure correct handling of backup schedules and time stamping of backup sets! The Hypervisor Credentials will be used by the VBA to connect to the vCenter server (or to a single ESX host). The number of Data Streams (i.e. the maximum number of simultaneous backup and restore jobs) is limited to 4 in the Trial version. In the Enterprise version the maximum is 8.

VBA Backup Storage settings
I selected the Attached Virtual Disk as Backup Storage Type in the next tab. Other choices are a NFS or CIFS share. By using local disks you can implement a LAN free backup eliminating any possible network bottlenecks. Backup data is compressed by default, but this option is also configurable here.

After saving the configuration a restart of the VBA is necessary for the settings to become effective. Actually these steps are enough to get you ready for your first backup job! To complete the picture here are the remaining configuration options:
  • Network settings: DHCP or manual IP address, DNS servers
  • Email: SMTP settings used for sending messages (of selectable severity levels)
  • Backup Retention: lets you define how long you want to keep backup sets. Out-dated sets will be automatically deleted to save backup storage space.
  • Replication and Connectors: See "Beyond Backup: DR Replication" below
  • Support options
A while later I noticed that the installation of the console software also included a plugin that nicely integrates the console's dialogs into the vSphere client. This integration saves you launching an extra program and entering your credentials a second time:

PHD Virtual Backup - vSphere Client integration
Let's back up!

At the beginning of this post I already explained how the backup process works technically. Scheduling it is easy with the management console: Obviously the first step is to select the VM(s) to back up. You can pick individual VMs here or a complete VM folder from the vCenter inventory view. If you schedule a job for a folder then it will also automatically apply to any VMs that are added later to this folder - a nice way to implement a "set and forget" backup schedule.

The next steps are to select a VBA that will do the job (if you have multiple VBAs in the same cluster) and the backup schedule: The choices are "Now", "Once", "Daily" and "Weekly" here. In the Options dialog you can define additional parameters for the job:

Backup job options
  • Verify backup: None (fastest), New blocks only (safe mode), All blocks (paranoia mode)
  • Backup powered off machines: Not by default
  • Set backups as archived: Allows for archived backups that will never be deleted regardless of retention time configuration
  • Quiesce the VM before backing up (Windows only): Will initiate a Windows Volume Shadow Copy Service (VSS) snapshot through the VMware Tools which allows for application consistent backups
  • Use Changed Block Tracking: Use the VMware CBT feature to identify disk blocks that were changed since the last backup
Once a backup job runs you can watch its progress in two different places: In the Jobs view of the Console program, and - using the vSphere client - on the console of the VBA:

VBA console showing backup progress
In both places you can also see statistics about how much data was backed up, how much was actually written, and the deduplication ratio.

I was very much pleased with the speed of the differential backups using CBT. Backing up a VM this way would often take only a couple of minutes. If you are not satisfied with the speed of backups one thing to check is the CPU and memory load of the VBA(s). By default they have minimum resources (1 vCPU and 1 GB RAM) - upgrading them with more CPUs and RAM can significantly improve their throughput.

Backups are boring, Restores are exciting?!

When using traditional backup methods (with an agent running inside the guest OS and doing a file level backup) restoring a machine can be a time-consuming and thrilling experience. In this case a restore can be an error-prone process that needs to be carefully tested for each OS version and special applications like SQL that need special agents.

With the image based backup approach that PHD Virtual is using this is somewhat different. Usually you do not need to worry about a restored machine not being bootable, the OS will always be in a consistent state. If the applications that runs inside the VM benefit from the pre-snapshot Windows VSS quiescing (mentioned above) then they will also be in a consistent state after recovery. However, not all applications support this - if in doubt ask your vendor and do backup/restore tests to safeguard yourself from unpleasant surprises.

When doing a full restore with the backup console you have the following options available (besides from the inevitable selection of the VBA and the VMs' backup sets to restore):

Restore job options
  • You can change the VMs' names by appending a suffix to their display names (if you want to keep the original VMs)
  • Default VM Storage: Choose a datastore to store the recovered VM
  • Network Settings: The default is to use network settings from the backup. If the original VM is still around you may change the MAC address of the restored VM to avoid any conflicts. Finally you must choose a virtual port group to attach the restored VM to (Distributed Virtual Switches are also fully supported).
Once the restore job is started the VBA will recreate the VMs with the original configuration plus the selected restore settings, and their virtual disks will be filled with the backup data.

A special way of doing File Level Restore (FLR)

You might run into situations where you don't want a VM to be completely restored, but only some of its files or application records (e.g. for restoring files from a virtual file server or single mailbox items from a virtual Exchange server). This is also possible with PHD Virtual Backup, and it uses a quite unique approach to accomplish this.

After starting the File Recovery wizard you can select a single virtual disk from a VM's backup set and export that via iSCSI:

Initiating a File Level Restore (FLR)

In the Options dialog you can define a custom CHAP secret for mounting the iSCSI target (or accept system generated credentials). If you check the Add target to iSCSI initiator on this computer option then this will automatically mount the iSCSI target on the Windows machine that runs the Backup Console. A prerequisite for this is an installed iSCSI initiator. Since Windows 2008 an iSCSI initiator is included in Windows, for earlier versions of Windows it is available as a free download from Microsoft.

The Restore iSCSI target can be mounted from an arbitrary computer (iSCSI initiators are available not only for Windows, but literally every Guest OS including Linux) by specifying the VBA's IP address for iSCSI discovery and the defined CHAP secrets for mounting the target.

The big advantage of this method is that the original virtual disk can be mounted directly from the backup as a block device - that means it will appear in the same way as the original disk, but with the data of the selected backup set. This way File Level Restore (FLR) is completely Guest OS and file system agnostic.

This unique FLR method enables not only a straightforward single file restore, but also interesting opportunities for seamless application item restore. On the PHD Virtual web site you can watch videos that demo e.g. the restore of a single SQL server table and a granular Exchange item recovery.

Beyond backup: DR replication

DR Replication architecture
Besides from Backup and Restore the product also features a special Replication mode for Disaster Recovery (DR) from one site to another. The way that it is implemented a Replication job is just a special case of a Restore job, with the following differences:
  • You can schedule replications to take place at defined intervals
  • A VBA can replicate a VM from a backup that was created by another VBA
The latter means that multiple VBAs can share a backup storage as a replication source. An easy way to accomplish this is to export the local backup storage of one VBA through CIFS. The replicating VBA (that would typically be part of another VMware cluster at a DR site) can mount this backup storage and replicate a VM from it into its own environment.

The initial replication can be "seeded" by transporting offline copies of the VMs physically to the DR site (in case there is only a low capacity WAN link between sites). Subsequent replications will only synchronize data that was changed since the last replication and will hence require only little bandwidth.

DR replication is a nice add-on of the Virtual Backup product, but the use cases are limited to non-critical workloads, because replication is not done directly, but only from backups. The RPO (Recovery Point Objective) time needed for business critical VMs (less than an hour) can hardly be met with this method. However, most people have already protected such workloads by using different technologies (e.g. online data mirroring, OS or application based clustering), so that they can still benefit from PHD Virtual's DR Replication.

Conclusion

The PHD Virtual Backup product offers a simple, robust and scalable architecture that fully benefits from the virtualization intrinsic advantages (e.g. VMware DRS and HA). It combines an efficient backup data handling with a unique and universal method of File Level Restore. Because of its unlimited scalability it is suitable not only for small and medium business customers, but also for large enterprises.

The simple architecture has a drawback though if you need to handle a very large environment with thousands of VMs and dozens of backup appliances: Each VBA needs to be configured and managed separately, and you need to think very carefully about how to distribute all the backup jobs among the VBAs - in the end the backup schedules are static. Wouldn't it be good to have a central management instance that will configure, manage and monitor all VBAs and dynamically orchestrate the backup jobs among them? Wouldn't it be good to have at least programmatic interfaces to third party tools that could potentially overtake these responsibilities?
It looks like PHD Virtual is aware of this issue and is going to address it: With the latest version 5.4 of the product they make an API and SDK available that can be used by customers to extend its functionality and integrate it into third party management workflows.

Availability

PHD Virtual's VMware vSphere Backup and Replication is available for download as a 15-day-trial version. It is licensed per virtualization host (thank God not by CPU sockets ;-)). Purchases can be made through authorized resellers. According to PHD Virtual the API and SDK for the product are available on request to "anyone having a good use case for it".


Sunday, March 25, 2012

HP VMware ESXi 5.0 U1 Customized Image

While searching for an updated HP Customized ESXi 5.0 ISO that would include Update 1 I noticed that the HP download page for ESXi still shows only the October release (ESXi 5.0 GA) as of today, March 25th 2012.
However, I stumbled over a document titled HP VMware ESXi 5.0 U1 Customized Image Release Notes for March 2012 that is apparently available since March 16th. That means that the release of this customized image is imminent. Here are some highlights from the release notes:


I will update this post once the new HP Customized ESXi 5.0 U1 Image is finally published.

Update (2012-03-27):  The HP VMware ESXi 5.0 U1 Customized Image is now available for download!

Monday, March 19, 2012

ESXi 5.0 Update 1 breaks VM Autostarts on free ESXi

In the VMware Communities an issue with ESXi 5.0 Update 1 was reported that is important for everyone who uses the free (VMware vSphere Hypervisor) version of ESXi 5.0:  Apparently VM autostarts do not work anymore after installing Update 1 if the host runs with a free license. The VM autostart function will automatically power on a defined list of VMs in a defined order when the system boots. This is probably used by a lot of people who run standalone ESXi hosts with the free license.

I was able to reproduce this issue in my home lab, and currently there is no workaround or solution. So if you rely on this functionality then you should postpone the installation of Update 1 until a fix for this is released. If you have already installed Update 1 and want to make it undone due to running into this issue then you can roll back to the previous installation (without Update 1) by pressing Shift-R at the ESXi boot prompt.

Update (2012-07-14): VMware has released a patch for ESXi 5.0 on July 12th that fixes this problem.