Last week I published an esxcli plugin that allows to run any shell command through esxcli. In this post I want to share what you need to know about the esxcli plugin system to be able to write your own plugins.
Read more »
Showing posts with label howto. Show all posts
Showing posts with label howto. Show all posts
Friday, January 18, 2013
Wednesday, January 2, 2013
How to use the latest VMware Tools with older vSphere versions
With vSphere 5.0 VMware made an important change regarding the compatibility and supportability of the VMware Tools: The Tools that come with vSphere 5.0 (and newer releases) can also be used with earlier vSphere releases (down to vSphere 4.0). For e.g. the latest VMware Tools of ESXi 5.1 the VMware Product Interoperability Matrix shows the following:
Read more »
 |
| VMware Tools of ESXi 5.1 interoperability |
Saturday, August 18, 2012
How do you stay up-to-date on VMware product patches and updates?
A while ago I posted the news about vCenter Server Update 1a and the latest ESXi patch being released, and in the comments section a reader asked some questions about updating VMware ESXi specifically and other VMware products in general.
His first question was: How do you know that a vCenter/ESXi/any VMware product patch/update is released?As a VMware admin, how do you stay up-to-date on this matter? My first thought was: Now, this is easy to answer, but in the end I was not able to find a single definitive answer to this question ...
Read more »
His first question was: How do you know that a vCenter/ESXi/any VMware product patch/update is released?As a VMware admin, how do you stay up-to-date on this matter? My first thought was: Now, this is easy to answer, but in the end I was not able to find a single definitive answer to this question ...
Read more »
Thursday, May 31, 2012
ImageBuilder Deep Dive, Part 3: The Power of the Shell
Let's start this third and last part of my ImageBuilder series with a short retrospective summary: In part one we introduced an easily adaptable script for building a customized ESXi 5.0 installation ISO. In the second part we looked closer at the basic cmdlets and learnt some new advanced ones.
In this final post we will walk through an improved and more universal version of the first script. It is more about Powershell in general than about ImageBuilder, so it will definitely help if you are already somewhat familiar with Powershell. But even if this is not the case you can still benefit by just using the improved script as it is. It took me some time to write it, because - according to Powershell - I was an absolute beginner when starting. I learnt a lot while developing it, but it is probably still not perfect ...
You can just download a copy of the script and use it. For the curious here is the listing:
Line 7 - 12: The script accepts command line parameters that are defined inside a param() statement. One of the possible parameters is the switch -help. If specified the script will print a help text (explaining the rest of the parameters) and exit (see line 25ff.). Another optional switch is -hp which will add the HP Online VIBs depot (s. line 65ff.). Using -obDir you can specify a directory where the script will look for Offline bundle zip files to be added. And with -isoDir you can specify the directory to store the created ISO file. The default for both directories is the path where the script itself is stored (which is determined by $(Split-Path $MyInvocation.MyCommand.Path)).
Line 14: One of the advanced Powershell features is using hash tables like the one that is created here. It contains the string representations of the four possible acceptance levels that a package or an image profile can have. They are assigned a ranking value (1 to 4) that is used to determine what the most restrictive acceptance level is ("VMwareCertified" = 1) as opposed to the least restrictive ("CommunitySupported" = 4).
Line 16 - 23: In these lines we define a Powershell function named AddVIB2Profile that adds a VIB package ($vib) that is passed as an argument to the custom image profile. As a first step it compares the acceptance level of the software package with that of the image profile (line 17) to determine if the latter needs to be lowered in order to accept the package (line 18 - 19). This is where the above mentioned hash table is used. Then it adds the package using the Add-EsxSoftwarePackage cmdlet. As I want the output of the script to be pretty and not garbled by useless information I pipe the output of the cmdlet into out-null. That means that it is just discarded. In line 22 I use the special Powershell variable $? to test whether the previous command was successful or not. This is a basic and the easiest way to add error handling to a Powershell script, and I will use that a lot here.
Line 25 - 36: This is actually the start of the main script. It prints a title line (line 26), then it will either display a help text and exit the script if -help was specified (line 27 - 33), or it will display only a short line explaining that -help can be used for instructions (line 35) and continue. BTW the special characters `n represent a new line when used in Powershell strings.
Line 39 to 45: Here we will check whether the required snapin VMware.ImageBuilder was already loaded (e.g. because you used the PowerCLI desktop shortcut to start your Powershell session). If not then we try to load it with Add-PSSnapin (line 40). If this fails (e.g. because PowerCLI is not installed on the computer) then the script errors out, because it depends on the ImageBuilder snapin being available and loaded.
Line 47 to 54: Now we add the VMware ESXi Online depot to the ImageBuilder session (also with some error handling). Nothing spectacular, we've seen this before.
Line 57 to 60: In line 57 we grab all standard image profiles from the VMware depot, create a sorted array of them and just pick the first one (with index [0]). This is the newest one, because we sorted the array in descending order. In the following lines we output some information about this image profile that is stored in its attributes.
Line 63: Here we create our custom image profile by cloning the latest one from the VMware depot. Its name and description are derived from the original VMware profile's properties.
Line 65 to 78: If the command line switch -hp was specified then we connect the HP Online VIBs depot to the ImageBuilder session (line 68 to 73) and add all included software packages using the AddVIB2Profile function that we defined at the beginning of the script.
Line 81 to 93: The foreach loop starting at line 82 is another essential part of the script code: Here we loop over all zip files that are located in the obDir directory and try to add them as Offline bundles to the ImageBuilder session (line 84). If it succeeds because a zip file is a valid Offline bundle (and not some other sort of file archive) then all software packages that are included in the bundle will be added to the custom image profile, again using our self-defined AddVIB2Profile function (line 88).
Line 96 to 99: Finally we export the custom image profile into an installation ISO. Its file name is derived from the image profile's name.
How do you run this Powershell script (and others)?
Open a plain Powershell window from the Start menu, or use the PowerCLI desktop shortcut to start an already customized Powershell session.
By default Powershell will only execute scripts that were electronically signed for security reasons. This script (and most other useful scripts that you will find in other places) is not signed, so you will need to relax the security check by using the Set-ExecutionPolicy cmdlet like shown in the following screenshot. You can do this for the current user only by specifying -Scope CurrentUser. This will also work if you do not have administrative permissions on the computer. If you have administrative permissions then you can leave the -Scope parameter and make the setting system global. Setting the execution policy needs to be done only once - it will be saved to the registry and reused for future Powershell sessions.
You can just call the script by its name (adding parameters as required). Since the script is from an "untrusted" source you still need to confirm its execution every time you start it:
Finally this is an example output from a complete script run:
The script that I provide here is fully functional and flexible enough to meet most requirements. However, it is certainly not perfect. E.g. it could have better error handling. If you have any comments or ideas on how to improve it (or if you run into errors when trying it) then please add a comment to this post!
Update (2012-06-07): In the meantime I updated this script to fix errors. Please watch out for the latest version using this download link!
In this final post we will walk through an improved and more universal version of the first script. It is more about Powershell in general than about ImageBuilder, so it will definitely help if you are already somewhat familiar with Powershell. But even if this is not the case you can still benefit by just using the improved script as it is. It took me some time to write it, because - according to Powershell - I was an absolute beginner when starting. I learnt a lot while developing it, but it is probably still not perfect ...
You can just download a copy of the script and use it. For the curious here is the listing:
#Let's look at the script line by line:
# ESXi-Customizer-PS.ps1 - a script to build a customized ESXi installation ISO using ImageBuilder
# Version: 1.0
# Author: Andreas Peetz (ESXi-Customizer@v-front.de)
#
param(
[string]$obDir = $(Split-Path $MyInvocation.MyCommand.Path),
[string]$isoDir = $(Split-Path $MyInvocation.MyCommand.Path),
[switch]$hp = $false,
[switch]$help = $false
)
$AccLevel = @{"VMwareCertified" = 1; "VMwareAccepted" = 2; "PartnerSupported" = 3; "CommunitySupported" = 4}
function AddVIB2Profile($vib) {
if ($AccLevel[$vib.AcceptanceLevel.ToString()] -gt $AccLevel[$MyProfile.AcceptanceLevel.ToString()]) {
write-host -nonewline (" [New AcceptanceLevel: " + $vib.AcceptanceLevel + "]")
$MyProfile.AcceptanceLevel = $vib.AcceptanceLevel
}
Add-EsxSoftwarePackage -SoftwarePackage $vib -ImageProfile $MyProfile | Out-Null
if ($?) { " [OK]" } else { " [FAILED]" }
}
# Write info and help if requested
write-host "`nScript to build a customized ESXi installation ISO using ImageBuilder"
if ($help) {
write-host "Optional parameters:"
write-host " -help : display this help"
write-host " -obDir <dir> : directory of Offline bundles to add (default = script directory)"
write-host " -isoDir <dir> : directory to store the customized ISO (default = script directory)"
write-host " -hp : add packages from the HP VIBs Depot (default = no)`n"
exit
} else {
write-host "(Call with -help for instructions)"
}
# Load the ImageBuilder Snapin (if not already loaded)
if (!(Get-PSSnapin -name VMware.ImageBuilder -ErrorAction:SilentlyContinue)) {
if (!(Add-PSSnapin -PassThru VMware.ImageBuilder)) {
# Error out if loading fails
write-host "FATAL ERROR: Cannot load the ImageBuilder Snapin. Is the PowerCLI installed?"
exit
}
}
# Add the VMware ESXi base depot
write-host -nonewline "`nConnecting the VMware ESXi base depot ..."
if ($baseDepot = Add-EsxSoftwareDepot https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml) {
write-host " [OK]"
} else {
write-host "`n FATAL ERROR: Cannot add VMware base Online depot. Please check your internet connectivity and/or proxy settings!"
exit
}
# Get the newest ImageProfile from the base depot
$LatestIP = (Get-EsxImageProfile "ESXi-5.0.0-20*-standard" | Sort-Object -Descending Name)[0]
write-host ("Using latest ImageProfile " + $LatestIP.Name)
write-host ("(dated " + $LatestIP.CreationTime + ", AcceptanceLevel: " + $LatestIP.AcceptanceLevel + ",")
write-host ($LatestIP.Description + ")")
# Create your own Imageprofile
$MyProfile = New-EsxImageProfile -CloneProfile $LatestIP -Name ($LatestIP.Name + "-customized") -Description ($LatestIP.Description + " + customizations")
if ($hp) {
# Add the HP VIBs depot
write-host -nonewline "`nConnecting the HP Online Depot ..."
if ($hpDepot = Add-EsxSoftwareDepot http://vibsdepot.hp.com) {
write-host " [OK]"
} else {
write-host "`n FATAL ERROR: Cannot add HP Online depot. Please check your internet connectivity and/or proxy settings!"
exit
}
$hpDepot.Channels[0] | Get-EsxSoftwarePackage | foreach {
write-host -nonewline (" Add VIB " + $_.Name + $_.Version )
AddVIB2Profile $_
}
}
# Loop over Offline bundles
write-host "`nAdding Offline bundles from" $obDir ...
foreach ($oBundle in Get-Item $obDir\*.zip) {
write-host -nonewline " Adding" $oBundle ...
if ($ob = Add-EsxSoftwareDepot $oBundle) {
write-host " [OK]"
$ob.Channels[0] | Get-EsxSoftwarePackage | foreach {
write-host -nonewline " Add VIB" $_.Name $_.Version
AddVIB2Profile $_
}
} else {
write-host " [FAILED]`n Probably not a valid Offline bundle, ignoring."
}
}
# Export the Imageprofile into an installation ISO file
$isoFile = $isoDir + "\" + $MyProfile.Name + ".iso"
write-host -nonewline ("`nExporting to ISO file " + $isoFile + ". Please be patient ...")
Export-EsxImageProfile -ImageProfile $MyProfile -ExportToIso -FilePath $isoFile -Force
if ($?) { " [OK]" } else { " [FAILED]" }
write-host "`nAll done."
Line 7 - 12: The script accepts command line parameters that are defined inside a param() statement. One of the possible parameters is the switch -help. If specified the script will print a help text (explaining the rest of the parameters) and exit (see line 25ff.). Another optional switch is -hp which will add the HP Online VIBs depot (s. line 65ff.). Using -obDir you can specify a directory where the script will look for Offline bundle zip files to be added. And with -isoDir you can specify the directory to store the created ISO file. The default for both directories is the path where the script itself is stored (which is determined by $(Split-Path $MyInvocation.MyCommand.Path)).
Line 14: One of the advanced Powershell features is using hash tables like the one that is created here. It contains the string representations of the four possible acceptance levels that a package or an image profile can have. They are assigned a ranking value (1 to 4) that is used to determine what the most restrictive acceptance level is ("VMwareCertified" = 1) as opposed to the least restrictive ("CommunitySupported" = 4).
Line 16 - 23: In these lines we define a Powershell function named AddVIB2Profile that adds a VIB package ($vib) that is passed as an argument to the custom image profile. As a first step it compares the acceptance level of the software package with that of the image profile (line 17) to determine if the latter needs to be lowered in order to accept the package (line 18 - 19). This is where the above mentioned hash table is used. Then it adds the package using the Add-EsxSoftwarePackage cmdlet. As I want the output of the script to be pretty and not garbled by useless information I pipe the output of the cmdlet into out-null. That means that it is just discarded. In line 22 I use the special Powershell variable $? to test whether the previous command was successful or not. This is a basic and the easiest way to add error handling to a Powershell script, and I will use that a lot here.
Line 25 - 36: This is actually the start of the main script. It prints a title line (line 26), then it will either display a help text and exit the script if -help was specified (line 27 - 33), or it will display only a short line explaining that -help can be used for instructions (line 35) and continue. BTW the special characters `n represent a new line when used in Powershell strings.
Line 39 to 45: Here we will check whether the required snapin VMware.ImageBuilder was already loaded (e.g. because you used the PowerCLI desktop shortcut to start your Powershell session). If not then we try to load it with Add-PSSnapin (line 40). If this fails (e.g. because PowerCLI is not installed on the computer) then the script errors out, because it depends on the ImageBuilder snapin being available and loaded.
Line 47 to 54: Now we add the VMware ESXi Online depot to the ImageBuilder session (also with some error handling). Nothing spectacular, we've seen this before.
Line 57 to 60: In line 57 we grab all standard image profiles from the VMware depot, create a sorted array of them and just pick the first one (with index [0]). This is the newest one, because we sorted the array in descending order. In the following lines we output some information about this image profile that is stored in its attributes.
Line 63: Here we create our custom image profile by cloning the latest one from the VMware depot. Its name and description are derived from the original VMware profile's properties.
Line 65 to 78: If the command line switch -hp was specified then we connect the HP Online VIBs depot to the ImageBuilder session (line 68 to 73) and add all included software packages using the AddVIB2Profile function that we defined at the beginning of the script.
Line 81 to 93: The foreach loop starting at line 82 is another essential part of the script code: Here we loop over all zip files that are located in the obDir directory and try to add them as Offline bundles to the ImageBuilder session (line 84). If it succeeds because a zip file is a valid Offline bundle (and not some other sort of file archive) then all software packages that are included in the bundle will be added to the custom image profile, again using our self-defined AddVIB2Profile function (line 88).
Line 96 to 99: Finally we export the custom image profile into an installation ISO. Its file name is derived from the image profile's name.
How do you run this Powershell script (and others)?
Open a plain Powershell window from the Start menu, or use the PowerCLI desktop shortcut to start an already customized Powershell session.
By default Powershell will only execute scripts that were electronically signed for security reasons. This script (and most other useful scripts that you will find in other places) is not signed, so you will need to relax the security check by using the Set-ExecutionPolicy cmdlet like shown in the following screenshot. You can do this for the current user only by specifying -Scope CurrentUser. This will also work if you do not have administrative permissions on the computer. If you have administrative permissions then you can leave the -Scope parameter and make the setting system global. Setting the execution policy needs to be done only once - it will be saved to the registry and reused for future Powershell sessions.
You can just call the script by its name (adding parameters as required). Since the script is from an "untrusted" source you still need to confirm its execution every time you start it:
 |
| ESXi-Customizer-PS help screen |
 |
| ESXi-Customizer-PS example run |
Update (2012-06-07): In the meantime I updated this script to fix errors. Please watch out for the latest version using this download link!
Monday, May 7, 2012
ImageBuilder Deep Dive, Part 2: A closer look and advanced functions
In the first part of this Image Builder Deep Dive I introduced a script that you can use to build your own customized installation ISO and can easily be adapted to your own needs. In this second part we will take a closer look at the basic cmdlets we used and get to know some more cmdlets with advanced functionality.
To make best use of the following explanations I suggest that you load the complete script that we go through in this part into the PowerShell ISE and run it line by line while reading the post. You can also experiment with the various code samples by pasting them into the command prompt of the ISE.
Using Online depots with proxy servers
At the beginning of the first part's script we added two Online depots to the ImageBuilder session using the Add-EsxSoftwareDepot cmdlet. I stated that this is not possible when using a proxy server, and - after some testing - I need to correct this statement: I got it working with a proxy server that does not require authentication. With a proxy server requiring authentication it worked on the first try and failed when adding the second online depot (?!). Originally I had a proxy auto-configuration URL entered in the Internet Explorer settings, and this made adding an Online depot consequently fail. Your mileage may vary ...
By default PowerCLI will use the Windows proxy settings (that means what you have configured in Internet Explorer). You can change that by using the Set-PowerCLIConfiguration cmdlet. However, the possibilities are somewhat limited. You can either use the system proxy settings or no proxy at all:
How to "download" the VMware Online base depot
Now you may wonder: What if your computer has no Internet connection (or only via a proxy configuration that causes the above mentioned problems)? Can I somewhat "download" an Online depot and re-use it later on the same or another computer? Yes, you can!
To be exact: You cannot create an offline copy of a complete Online depot, but you can export a specific ImageProfile not only to an installation ISO, but also into a re-usable Offline bundle:
Please note: The HP Online depot cannot be "downloaded" in this way, because it does not expose any ImageProfiles that you can export. However, in this special case you can also open the http://vibsdepot.hp.com URL in a browser and download the Offline bundles that it includes from the linked web pages.
Looking closer at the ImageProfile object
Apropos ImageProfiles: So far we only looked at the names of these objects. Obviously VMware codes the release date of an ImageProfile (= patch level) into its name using the format YYYYMMDD. Hopefully they will stick with this naming standard - it makes it easy to identify the newest ImageProfile at first sight and programmatically.
The ImageProfile object has more interesting attributes though:
The VibList attribute is an array of all included software packages. The list is shortened in the formatted object view to retain readability of the output. The third command above ($ip.VibList) will output the complete list of packages with names and versions.
The AcceptanceLevel attribute is something that you normally don't need to take care of, but we will come back to it at the end of this post.
How to compare two ImageProfiles
If you want to quickly check the differences of two ImageProfiles you could list their attributes one after the other like shown before and compare the outputs, especially the VibLists. However, there is a more elegant way to do this - you can use the Compare-EsxImageProfile cmdlet:
Examining, adding and removing single software packages from a depot
In the first part of this series of posts we used a pipeline of commands to add all packages of a depot to our custom profile. Now we are going to take a closer look at a single package using the Get-EsxSoftwarePackage cmdlet:
In line 36 we examine the properties of a specific version of a package. We will see that the SoftwarePackage object has some interesting attributes:
The three properties Depends, Conflicts and Replaces describe relationships to other packages (or features that are provided by other packages). Their names are self-explanatory, and ImageBuilder will take care of them for us. You will e.g. not be allowed to add a package that conflicts with another package that is already part of the profile.
Now let's add this single package to a custom profile using the Add-EsxSoftwarePackage cmdlet:
Finally, you can also remove a package from an ImageProfile using the Remove-EsxSoftwarePackage cmdlet:
Adding community developed packages
If you have ever used my ESXi-Customizer script to add a Community developed software package to an ESXi 5.0 installation ISO then you may wonder: Can you also do that with ImageBuilder? Yes, you can, and here is how:
There is only one particularity that you need to be aware off when adding Community developed packages this way: Software packages that were created (and certified) by VMware or trusted partners contain an electronic signature that is checked when adding (or installing) the package. This qualifies them for one of the following three so-called AcceptanceLevels: VMwareCertified, VMwareAccepted or PartnerSupported. Of course, Community developed packages do not have trusted electronic signatures, which means that you can only install them if they have the least restrictive (and least trustworthy) AcceptanceLevel called CommunitySupported. The ImageProfile object also has an AcceptanceLevel, and that must be lowered to the same level before you are able to add a Community developed package. This happens in line 52.
This ends the second part of the ImageBuilder Deep Dive series. The next and last part will not deal with the ImageBuilder snapin specifically, but with Powershell coding in general. We will then develop the script from the first part to a more general and versatile solution.
To make best use of the following explanations I suggest that you load the complete script that we go through in this part into the PowerShell ISE and run it line by line while reading the post. You can also experiment with the various code samples by pasting them into the command prompt of the ISE.
Using Online depots with proxy servers
At the beginning of the first part's script we added two Online depots to the ImageBuilder session using the Add-EsxSoftwareDepot cmdlet. I stated that this is not possible when using a proxy server, and - after some testing - I need to correct this statement: I got it working with a proxy server that does not require authentication. With a proxy server requiring authentication it worked on the first try and failed when adding the second online depot (?!). Originally I had a proxy auto-configuration URL entered in the Internet Explorer settings, and this made adding an Online depot consequently fail. Your mileage may vary ...
By default PowerCLI will use the Windows proxy settings (that means what you have configured in Internet Explorer). You can change that by using the Set-PowerCLIConfiguration cmdlet. However, the possibilities are somewhat limited. You can either use the system proxy settings or no proxy at all:
# Load the VimAutomation SnapinPlease note: You need to add the VMware.VimAutomation.Core snapin first (if not already added) to use these cmdlets. To display the current setting use the Get-PowerCLIConfiguration cmdlet.
Add-PSSnapin VMware.VimAutomation.Core
#
# Set and get the proxy configuration
Set-PowerCLIConfiguration -ProxyPolicy NoProxy
Set-PowerCLIConfiguration -ProxyPolicy UseSystemProxy
Get-PowerCLIConfiguration
#
How to "download" the VMware Online base depot
Now you may wonder: What if your computer has no Internet connection (or only via a proxy configuration that causes the above mentioned problems)? Can I somewhat "download" an Online depot and re-use it later on the same or another computer? Yes, you can!
To be exact: You cannot create an offline copy of a complete Online depot, but you can export a specific ImageProfile not only to an installation ISO, but also into a re-usable Offline bundle:
# Load the ImageBuilder SnapinNow you can re-use this exported zip file any time and on any other computer by adding it with the Add-EsxSoftwareDepot cmdlet. Obviously this Offline bundle zip file only contains the ImageProfile that you exported and only the packages that are part of this ImageProfile.
Add-PSSnapin VMware.ImageBuilder
#
# Reference the VMware ESXi base depot
$baseDepot = Add-EsxSoftwareDepot https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml
#
# List available ImageProfiles sorted by name (filtered on updated standard profiles)
Get-EsxImageProfile "ESXi-5.0.0-2012*-standard" | Sort-Object -Descending Name
#
# The previous command outputs "ESXi-5.0.0-20120504001-standard" as the newest profile (as of May 5th 2012)
# Export this Imageprofile into an Offline bundle zip file
Export-EsxImageProfile -ImageProfile ESXi-5.0.0-20120504001-standard -ExportToBundle -FilePath 'U:\ESXi-5.0.0-20120504001-standard.zip'
#
Please note: The HP Online depot cannot be "downloaded" in this way, because it does not expose any ImageProfiles that you can export. However, in this special case you can also open the http://vibsdepot.hp.com URL in a browser and download the Offline bundles that it includes from the linked web pages.
Looking closer at the ImageProfile object
Apropos ImageProfiles: So far we only looked at the names of these objects. Obviously VMware codes the release date of an ImageProfile (= patch level) into its name using the format YYYYMMDD. Hopefully they will stick with this naming standard - it makes it easy to identify the newest ImageProfile at first sight and programmatically.
The ImageProfile object has more interesting attributes though:
# Examining the ImageProfile's attributesBy piping the object through the format-list cmdlet you can display all its attributes in a nicely formatted list view:
$ip = Get-EsxImageProfile ESXi-5.0.0-20120504001-standard
$ip | format-list
$ip.VibList
#
Name : ESXi-5.0.0-20120504001-standardThe Description attribute is interesting, because its value includes a URL to a VMware KB article with further information about this ESXi 5.0 patch release.
Vendor : VMware, Inc.
Author :
Description : For more information, see http://kb.vmware.com/kb/2019863.
CreationTime : 30.04.2012 21:47:06
ModifiedTime : 30.04.2012 21:47:06
ReadOnly : False
VibList : {ata-pata-atiixp 0.4.6-3vmw.500.0.0.469512, net-nx-nic 4.0.557-3vmw.500.1.11.623860, scsi-rste 2.0.2.0088-1vmw.500.1.11.623860, net-e1000 8.0.3.1-2vmw.500.0.7.515841...}
AcceptanceLevel : PartnerSupported
Guid : 076da48a930e11e1b58d0017a477682c
Rules :
StatelessReady : True
The VibList attribute is an array of all included software packages. The list is shortened in the formatted object view to retain readability of the output. The third command above ($ip.VibList) will output the complete list of packages with names and versions.
The AcceptanceLevel attribute is something that you normally don't need to take care of, but we will come back to it at the end of this post.
How to compare two ImageProfiles
If you want to quickly check the differences of two ImageProfiles you could list their attributes one after the other like shown before and compare the outputs, especially the VibLists. However, there is a more elegant way to do this - you can use the Compare-EsxImageProfile cmdlet:
# Comparing two ImageProfilesThis command will create and output an ImageProfile comparison object. In this example it looks like this:
Compare-EsxImageProfile ESXi-5.0.0-20120504001-standard -ReferenceProfile ESXi-5.0.0-20120404001-standard
#
Equal : False
PackagesEqual : False
RefAcceptanceLevel : PartnerSupported
CompAcceptanceLevel : PartnerSupported
OnlyInRef : {}
OnlyInComp : {}
UpgradeFromRef : {VMware_bootbank_esx-base_5.0.0-1.13.702118}
DowngradeFromRef : {}
What do we see here? The two ImageProfiles are not equal (Equal = False), they do not contain the exact same list of packages (PackagesEqual = False), but they have the same AcceptanceLevel (PartnerSupported). The remaining four attributes summarize the differences in the software package list of the two ImageProfiles. Their names are pretty self-explanatory, and in this example the only difference is that the comparison profile (ESXi-5.0.0-20120504001-standard) contains a newer version of the esx-base package than the reference profile (ESXi-5.0.0-20120404001-standard).Examining, adding and removing single software packages from a depot
In the first part of this series of posts we used a pipeline of commands to add all packages of a depot to our custom profile. Now we are going to take a closer look at a single package using the Get-EsxSoftwarePackage cmdlet:
# Reference downloaded HP offline bundle for be2net driverThe output of the command in line 34 looks like this:
$be2net = Add-EsxSoftwareDepot "U:\HP-ESXi5-Drivers\be2net-4.0.355.1-offline_bundle-487292.zip"
#
# Look at all versions of a single software package (net-be2net)
Get-EsxSoftwarePackage net-be2net
# Look at all properties of a package's specific version
Get-EsxSoftwarePackage net-be2net -Version 4.0.88.0-1vmw.500.0.7.515841 | fl
#
Name Version Vendor Release Date
---- ------- ------ ------------
net-be2net 4.0.88.0-1vmw.500.0.7.515841 VMware 15.12.2011 00...
net-be2net 4.0.88.0-1vmw.500.0.0.469512 VMware 19.08.2011 01...
net-be2net 4.0.355.1-1OEM.500.0.0.406165 Emulex 16.09.2011 00...
That means that there are three different versions of the net-ne2net package, the first two are in the VMware base depot, and the third one is in the Offline bundle we downloaded from HP.In line 36 we examine the properties of a specific version of a package. We will see that the SoftwarePackage object has some interesting attributes:
Name : net-be2net
Version : 4.0.88.0-1vmw.500.0.7.515841
Vendor : VMware
Summary : Updates the ESX 5.0.0 net-be2net
Description : For build informatin, see KB http://kb.vmware.com/kb/2007675
ReleaseDate : 15.12.2011 00:00:00
Depends : {vmkapi_2_0_0_0, com.vmware.driverAPI-9.2.0.0}
Conflicts : {}
Replaces : {}
AcceptanceLevel : VMwareCertified
MaintenanceMode : False
LiveInstallOk : False
LiveRemoveOk : False
SourceUrls : {https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/net-be2net/VMware_bootbank_net-be2net_4.0.88.0-1vmw.500.0.7.515841.vib}
Guid : VMware_bootbank_net-be2net_4.0.88.0-1vmw.500.0.7.515841
StatelessReady : True
Tags : {driver, module, category:bugfix, severity:general}
Like with the ImageProfile object the Description property contains a link to a VMware KB article with further information on this package. The SourceUrl indicates from where the package will be downloaded. And the package's Tags provides additional classification categories.The three properties Depends, Conflicts and Replaces describe relationships to other packages (or features that are provided by other packages). Their names are self-explanatory, and ImageBuilder will take care of them for us. You will e.g. not be allowed to add a package that conflicts with another package that is already part of the profile.
Now let's add this single package to a custom profile using the Add-EsxSoftwarePackage cmdlet:
# Create a new ImageProfileIf you just specify the name of the software package (net-be2net in this example), and there are multiple versions of this package available then Add-EsxSoftwarePackage will automatically add the newest version. If you want to add an older version then you can specify it like shown in line 42. Whenever the ImageProfile already includes a package of the same name, but a different version, it will be replaced by the added package, even if it is older! If the ImageProfile already includes the exact same version then the command will fail, or in other words: you cannot add the same version twice.
$MyProfile = New-EsxImageProfile -CloneProfile ESXi-5.0.0-20120504001-standard -Name MyProfile
#
# Add an older version of the net-be2net package to the custom profile
Add-EsxSoftwarePackage -ImageProfile $MyProfile "net-be2net 4.0.88.0-1vmw.500.0.7.515841"
# Add the neweset net-be2net package to the custom profile
Add-EsxSoftwarePackage -ImageProfile $MyProfile net-be2net
#
Finally, you can also remove a package from an ImageProfile using the Remove-EsxSoftwarePackage cmdlet:
# Remove a package from the custom profileYou won't be able to remove packages that are required by other packages of the ImageProfile. In some case you can override this dependency check by using the parameter -Force, but you really shouldn't do that, because it will most likely result in an invalid ImageProfile.
Remove-EsxSoftwarePackage -ImageProfile $MyProfile net-bnx2
#
Adding community developed packages
If you have ever used my ESXi-Customizer script to add a Community developed software package to an ESXi 5.0 installation ISO then you may wonder: Can you also do that with ImageBuilder? Yes, you can, and here is how:
# Add an Offline Bundle with a community developed software packageIn this example we add a software package that includes a custom firewall rule to allow incoming NTP queries (this way you can use the ESXi host as NTP server. Read this post for background information). The Offline bundle was created using my ESXi5 Community Packaging Tools by converting a standard tar.gz file to a VIB file first, and then packaging the VIB file into an Offline bundle zip. The project page of the tools contains detailed instructions on how to do this.
Add-EsxSoftwareDepot 'U:\$Download\fwenable-ntpd-1.2.0-0-offline_bundle.zip'
# Change the AcceptanceLevel of the custom profile to "CommunitySupported"
$MyProfile.AcceptanceLevel = "CommunitySupported"
# Add the package to the profile
Add-EsxSoftwarePackage -ImageProfile $MyProfile fwenable-ntpd
There is only one particularity that you need to be aware off when adding Community developed packages this way: Software packages that were created (and certified) by VMware or trusted partners contain an electronic signature that is checked when adding (or installing) the package. This qualifies them for one of the following three so-called AcceptanceLevels: VMwareCertified, VMwareAccepted or PartnerSupported. Of course, Community developed packages do not have trusted electronic signatures, which means that you can only install them if they have the least restrictive (and least trustworthy) AcceptanceLevel called CommunitySupported. The ImageProfile object also has an AcceptanceLevel, and that must be lowered to the same level before you are able to add a Community developed package. This happens in line 52.
This ends the second part of the ImageBuilder Deep Dive series. The next and last part will not deal with the ImageBuilder snapin specifically, but with Powershell coding in general. We will then develop the script from the first part to a more general and versatile solution.
Sunday, April 29, 2012
ImageBuilder Deep Dive, Part 1: Building your own customized ESXi ISO
I know that my ESXi-Customizer script has gotten some popularity and a lot of people use it for merging community developed or commercial third-party hardware drivers into the ESXi installation ISO. It has some limitations though - some of you might have stumbled over them already, at least I have described them in the "Known issues" section. Probably the worst is the fact that it cannot handle complex software bundles like ESXi patches (e.g. the ESXi 5.0 Update 1 package).
My recommendation for such cases is to use the VMware supplied and supported method to create your own customized installation ISOs: The ImageBuilder PowerCLI snapin. With this post I'm going to start a series of blog posts that will cover ImageBuilder in detail and will help you to make effective use of it.
This first post will cover the prerequisites and installation of the PowerCLI and will introduce a script that will help you to get the task done. You won't need in-depth PowerShell knowledge for this, but it will definitely help if you are also interested in the remaining parts of the series that will go into the details of the ImageBuilder cmdlets and finally refine the first script to a more universal solution.
Prerequisites and installation
Obviously you need a Windows computer with the current version (2.0) of Powershell installed. Powershell is Microsoft's state-of-the-art scripting language - it is already included with Windows 7 and Windows 2008 R2 server, for earlier versions of Windows it is available as part of the Windows Management Framework. If you are a serious Windows admin you probably have looked at it before. If not then this is a good opportunity to do it. You can start learning e.g. at Microsoft's Script Center, but this is not a prerequisite for now!
The functionality of Powershell can be extended through so-called snapins. VMware makes available such snapins to add functions (so-called cmdlets) that you can use to manage vCenter servers and ESX(i) hosts. The thing is called PowerCLI, and once you have downloaded and installed the current version you are ready to go!
Following is a Powershell script that will create an ESXi 5.0 installation ISO with the current patch level, the HP Offline bundles and some HP drivers. I suggest that you do the following to walk through the explanations below:
The first script
Line 5: The Add-ESXSoftwareDepot cmdlet adds an Online depot or a downloaded Offline bundle as a package source to the current ImageBuilder session. The VMware depot referenced here includes the base ESXi 5.0 sources and is needed in any case. Obviously adding an Online depot requires a working connection to the Internet. I could get this to work with a direct connection only, but not through a proxy server. If someone knows how to make this work with a proxy then please comment here!
Line 8: This adds an additional Online depot that was made available by HP and includes their Offline bundles for ESXi 5.0.
Line 11: A software depot object like $hpDepot that was created through the Add-EsxSoftwareDepot cmdlet can be organized in multiple channels (but mostly it's only one channel). By piping the first channel into the Get-EsxSoftwarePackage cmdlet we can list the software packages that are included in this depot. The output looks like this:
This step is not really needed to build the new ISO. So you can safely skip it if you already know or are not interested in the contents of a depot.
Lines 14-16: In these lines we add three additional depots, but this time these are not Online depots, but Offline bundles that we downloaded before. You can find the download links on my HP & VMware links page (see section ESXi 5.0 drivers for the HP Emulex 10GbE Converged Network Adapters). But please note: The files that you download from VMware's driver pages are in zip format, but they are not Offline bundles. They include Offline bundles though, so you need to extract the *offline_bundle*.zip files from the downloaded zip-files first. And of course you need to change the file paths used in the script to your own download directory.
Line 19: The base depot that we added in line 5 includes not only software packages, but also so-called image profiles. An image profile is a grouping of software packages out of the depot. In this case each image profile makes up a specific ESXi patch level. The Get-EsxImageProfile cmdlet will list all available image profiles. We pipe it through the Sort-Object cmdlet here to sort the output by creation date. The output will show the newest image profile first and looks like this:
Each image profile comes in two different flavors: The -standard one includes all packages that make up an ESXi 5 installation, the -no-tools version is the same without the VMware Tools package. We will choose the standard version in our example.
Line 22: The image profiles of the Online depot are read-only. Since we want to modify one of these and add more packages, we need to create a copy first. With the New-EsxImageProfile cmdlet we create a new image profile ($MyProfile) by cloning the newest standard image profile (ESXi-5.0.0-20120404001-standard in this case). We choose the newest one, because it represents the most recent patch level of ESXi! We also assign a new name to the cloned profile (this is mandatory) and a new description (optional).
Lines 25-28: In line 11 we already listed the software packages that are included in the $hpDepot. By piping the output of this command to the Add-EsxSoftwarePackage cmdlet we add all the included packages to our newly created image profile $MyProfile. And we do the same for the three Offline bundles that we added in the lines 14 to 16.
Line 31: In the last line we use the Export-EsxImageProfile cmdlet to "export" our customized image profile into an ISO file. Now guess what: This ISO file is a complete ESXi 5.0 installation media! You can use it to install a machine with ESXi 5.0 with the current patch level and all the HP packages that you added before.
Wrap-up
In this first part of my ImageBuilder Deep Dive series I introduced a script that you can use to build an up-to-date ESXi 5.0 installation ISO with additional drivers. You learnt some basic terms, the most important cmdlets that are necessary to get the job done, and you will (hopefully) be able to modify the script and adapt it to your own needs.
In the next part I will introduce some more useful ImageBuilder cmdlets, and I will explain how you can integrate community developed software packages to the installation media (a job that you can also do with my ESXi-Customizer script). Stay tuned!
My recommendation for such cases is to use the VMware supplied and supported method to create your own customized installation ISOs: The ImageBuilder PowerCLI snapin. With this post I'm going to start a series of blog posts that will cover ImageBuilder in detail and will help you to make effective use of it.
This first post will cover the prerequisites and installation of the PowerCLI and will introduce a script that will help you to get the task done. You won't need in-depth PowerShell knowledge for this, but it will definitely help if you are also interested in the remaining parts of the series that will go into the details of the ImageBuilder cmdlets and finally refine the first script to a more universal solution.
Prerequisites and installation
Obviously you need a Windows computer with the current version (2.0) of Powershell installed. Powershell is Microsoft's state-of-the-art scripting language - it is already included with Windows 7 and Windows 2008 R2 server, for earlier versions of Windows it is available as part of the Windows Management Framework. If you are a serious Windows admin you probably have looked at it before. If not then this is a good opportunity to do it. You can start learning e.g. at Microsoft's Script Center, but this is not a prerequisite for now!
The functionality of Powershell can be extended through so-called snapins. VMware makes available such snapins to add functions (so-called cmdlets) that you can use to manage vCenter servers and ESX(i) hosts. The thing is called PowerCLI, and once you have downloaded and installed the current version you are ready to go!
Following is a Powershell script that will create an ESXi 5.0 installation ISO with the current patch level, the HP Offline bundles and some HP drivers. I suggest that you do the following to walk through the explanations below:
- Download a copy of the script to your computer
- In Explorer right-click on the file and choose "Edit" from the context menu. This will open it in the Powershell ISE (Integrated Scripting Environment) editor.
- Within the ISE you can select single (or multiple) lines of the script and execute them separately from the rest:
 |
| Run selected lines of code in the Powershell ISE |
The first script
# Load the ImageBuilder SnapinLine 2: This command will add the ImageBuilder snapin to the current Powershell session. Please note: If you start your session with the PowerCLI shortcut on your desktop then the snapin will automatically be loaded and you can skip this line.
Add-PSSnapin VMware.ImageBuilder
# Reference the VMware ESXi base depot
$baseDepot = Add-EsxSoftwareDepot https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml
# Reference the HP VIBs depot
$hpDepot = Add-EsxSoftwareDepot http://vibsdepot.hp.com
# List the VIB packages of HP depot
$hpDepot.Channels[0] | Get-EsxSoftwarePackage
# Reference downloaded HP driver offline bundles
$be2iscsi = Add-EsxSoftwareDepot "U:\HP-ESXi5-Drivers\be2iscsi-4.0.317.0-offline_bundle-469760.zip"
$be2net = Add-EsxSoftwareDepot "U:\HP-ESXi5-Drivers\be2net-4.0.355.1-offline_bundle-487292.zip"
$lpfc820 = Add-EsxSoftwareDepot "U:\HP-ESXi5-Drivers\lpfc820-8.2.2.105.36-offline_bundle-489567.zip"
# List available Imageprofiles sorted by creation date (newest first)
Get-EsxImageProfile | Sort-Object -Descending CreationTime | Format-List Name,CreationTime
# Create your own Imageprofile
$MyProfile = New-EsxImageProfile -CloneProfile ESXi-5.0.0-20120404001-standard -Name MyProfile -Description "ESXi-5.0.0-20120404001-standard + HP components"
# Add all the HP VIB packages to MyProfile
$hpDepot.Channels[0] | Get-EsxSoftwarePackage | Add-EsxSoftwarePackage -ImageProfile $MyProfile
$be2iscsi.Channels[0] | Get-EsxSoftwarePackage | Add-EsxSoftwarePackage -ImageProfile $MyProfile
$be2net.Channels[0] | Get-EsxSoftwarePackage | Add-EsxSoftwarePackage -ImageProfile $MyProfile
$lpfc820.Channels[0] | Get-EsxSoftwarePackage | Add-EsxSoftwarePackage -ImageProfile $MyProfile
# Export the Imageprofile into an installation ISO file
Export-EsxImageProfile -ImageProfile $MyProfile -ExportToIso -FilePath "U:\MyProfile.iso"
Line 5: The Add-ESXSoftwareDepot cmdlet adds an Online depot or a downloaded Offline bundle as a package source to the current ImageBuilder session. The VMware depot referenced here includes the base ESXi 5.0 sources and is needed in any case. Obviously adding an Online depot requires a working connection to the Internet. I could get this to work with a direct connection only, but not through a proxy server. If someone knows how to make this work with a proxy then please comment here!
Line 8: This adds an additional Online depot that was made available by HP and includes their Offline bundles for ESXi 5.0.
Line 11: A software depot object like $hpDepot that was created through the Add-EsxSoftwareDepot cmdlet can be organized in multiple channels (but mostly it's only one channel). By piping the first channel into the Get-EsxSoftwarePackage cmdlet we can list the software packages that are included in this depot. The output looks like this:
Name Version Vendor Release Date
---- ------- ------ ------------
hpnmi 2.0.11-434156 hp 29.07.2011 20...
char-hpilo 500.9.0.0.9-1OEM.500.0.0.43... Hewlett... 07.10.2011 14...
hp-ams 500.9.0.0-55.434156 Hewlett... 09.03.2012 23...
char-hpcru 5.0.0.8-1OEM.500.0.0.434156 Hewlett... 15.07.2011 17...
hpbootcfg 01-00.10 Hewlett... 15.07.2011 16...
hpacucli 9.0-24.0 Hewlett... 02.02.2012 06...
hponcfg 04-00.10 Hewlett... 13.11.2011 02...
hp-smx-provider 500.02.10.61.43-434156 Hewlett... 18.11.2011 02...
This step is not really needed to build the new ISO. So you can safely skip it if you already know or are not interested in the contents of a depot.
Lines 14-16: In these lines we add three additional depots, but this time these are not Online depots, but Offline bundles that we downloaded before. You can find the download links on my HP & VMware links page (see section ESXi 5.0 drivers for the HP Emulex 10GbE Converged Network Adapters). But please note: The files that you download from VMware's driver pages are in zip format, but they are not Offline bundles. They include Offline bundles though, so you need to extract the *offline_bundle*.zip files from the downloaded zip-files first. And of course you need to change the file paths used in the script to your own download directory.
Line 19: The base depot that we added in line 5 includes not only software packages, but also so-called image profiles. An image profile is a grouping of software packages out of the depot. In this case each image profile makes up a specific ESXi patch level. The Get-EsxImageProfile cmdlet will list all available image profiles. We pipe it through the Sort-Object cmdlet here to sort the output by creation date. The output will show the newest image profile first and looks like this:
Name : ESXi-5.0.0-20120404001-no-tools
CreationTime : 16.03.2012 22:59:09
Name : ESXi-5.0.0-20120404001-standard
CreationTime : 16.03.2012 22:59:09
Name : ESXi-5.0.0-20120302001-no-tools
CreationTime : 16.03.2012 22:59:08
Name : ESXi-5.0.0-20120302001-standard
CreationTime : 16.03.2012 22:59:08
Name : ESXi-5.0.0-20120301001s-standard
CreationTime : 16.03.2012 22:59:08
Name : ESXi-5.0.0-20120301001s-no-tools
CreationTime : 16.03.2012 22:59:08
Name : ESXi-5.0.0-20111204001-standard
CreationTime : 31.10.2011 11:24:00
(... output shortened for readability ...)Each image profile comes in two different flavors: The -standard one includes all packages that make up an ESXi 5 installation, the -no-tools version is the same without the VMware Tools package. We will choose the standard version in our example.
Line 22: The image profiles of the Online depot are read-only. Since we want to modify one of these and add more packages, we need to create a copy first. With the New-EsxImageProfile cmdlet we create a new image profile ($MyProfile) by cloning the newest standard image profile (ESXi-5.0.0-20120404001-standard in this case). We choose the newest one, because it represents the most recent patch level of ESXi! We also assign a new name to the cloned profile (this is mandatory) and a new description (optional).
Lines 25-28: In line 11 we already listed the software packages that are included in the $hpDepot. By piping the output of this command to the Add-EsxSoftwarePackage cmdlet we add all the included packages to our newly created image profile $MyProfile. And we do the same for the three Offline bundles that we added in the lines 14 to 16.
Line 31: In the last line we use the Export-EsxImageProfile cmdlet to "export" our customized image profile into an ISO file. Now guess what: This ISO file is a complete ESXi 5.0 installation media! You can use it to install a machine with ESXi 5.0 with the current patch level and all the HP packages that you added before.
 |
| Booting an ImageBuilder customized ESXi 5.0 installation ISO |
In this first part of my ImageBuilder Deep Dive series I introduced a script that you can use to build an up-to-date ESXi 5.0 installation ISO with additional drivers. You learnt some basic terms, the most important cmdlets that are necessary to get the job done, and you will (hopefully) be able to modify the script and adapt it to your own needs.
In the next part I will introduce some more useful ImageBuilder cmdlets, and I will explain how you can integrate community developed software packages to the installation media (a job that you can also do with my ESXi-Customizer script). Stay tuned!
Thursday, August 25, 2011
The anatomy of the ESXi 5.0 installation CD - and how to customize it
1. Introduction
With vSphere 5 VMware introduced the Auto Deploy Server and the Image Builder that allow to customize the ESXi installation ISO with partner supplied driver and tools packages.
The Image Builder is a Powershell snapin that comes with the latest version of the PowerCLI package. It allows to add software packages to a pre-defined set of packages (a so-called ImageProfile) and even lets you create an installation ISO from such a baseline making it easier than ever to customize the ESXi installation.
The Image Builder is a Powershell snapin that comes with the latest version of the PowerCLI package. It allows to add software packages to a pre-defined set of packages (a so-called ImageProfile) and even lets you create an installation ISO from such a baseline making it easier than ever to customize the ESXi installation.
However, doing this is not a straight-forward task. It requires a working installation of the Powershell, plus the PowerCLI software, access to the offline-bundle that makes up the base installation (which is not included with the free version of ESXi!), a custom driver in VIB format, and some guidance on what Powershell-cmdlets you need to use to add the custom driver package and build an ISO from it.
For the developers of custom drivers it requires to supply their packages in VIB format, and it's not trivial and costs extra effort to build such a package (compared to a simple OEM.TGZ file).
For the developers of custom drivers it requires to supply their packages in VIB format, and it's not trivial and costs extra effort to build such a package (compared to a simple OEM.TGZ file).
I wondered if it is still possible to customize the ESXi 5.0 install ISO with a simple OEM.TGZ file like you can do with ESXi 4.1, e.g. with my ESXi-Customizer script. And yes, it is possible - but it's very different now! I want to provide some background information here on how this works:
2. The contents of the ESXi 5.0 installation ISO
First let's have a look at the root directory of the ESXi 5.0 install ISO:
 |
| Contents of the ESXi 4.0 install CD root directory |
Unlike the ESXi 4.1 ISO you can see lots of ISO9660-compatible file names here (all capitals and 8.3-format). You can guess from their names that the files with the V00 (and V01, V02, etc.) extensions are device driver archives. The original type of these files is VGZ, the short form of VMTAR.GZ. That means that they are gzip'ed vmtar-files.
vmtar is a VMware proprietary variant of tar, and you need the vmtar-tool to pack and unpack vmtar archives. It is part of ESXi 5.0 and also ESXi 4.x. Other files have the extensions TGZ and T00 (like TOOLS.T00). These files are gzip'ed standard tar files that the boot loader can also handle. Good.
Comparing with the ESXi 4.1 media you will notice that there is no ddimage.bz2 file any more. In earlier versions of ESXi this is a compressed image that is written to the installation target disk and contains the whole installed ESXi system. Actually you can write this image to a USB key drive to produce a bootable ESXi system without ever booting the install CD. You cannot do this with ESXi 5.0 any more. However, customizing the install CD has become easier this way, because you do not need to add a second copy of your oem.tgz file to this system image.
There are also files named ISOLINUX.BIN and ISOLINUX.CFG in the ISO root. That means that ESXi 5.0 still uses the isolinux boot loader to make the installation CD bootable. If you look into ISOLINUX.CFG it includes a reference to the file BOOT.CFG, and in BOOT.CFG you find references to all the VGZ and TGZ files:
A second copy of the BOOT.CFG file is in the directory \EFI\BOOT. The ESXi 5.0 install ISO (and ESXi 5.0 itself) was built to boot not only on a standard x86 BIOS, but also on new (U)EFI enabled BIOS versions. Just one thing to remember: If you change the one BOOT.CFG you better make the same change to the other.
 |
| Contents of the BOOT.CFG file |
Now let's have a closer look at a driver VGZ package.
3. What's in a driver's vgz-file?
As mentioned before you need the vmtar-tool to look into a VGZ-file. Since it is only part of ESXi itself you need to have access to an installed copy of ESXi (either 4.1 or 5.0). Luckily you are able to install ESXi 4.1 (and also 5.0!) inside a VMware Workstation 7 VM.
I did this by creating a VM of type "ESX Server 4" with typical settings except for the size of the virtual disk (2GB is enough for ESXi) and installing ESXi 5.0 in it. During installation the driver files from the CD root are uncompressed and copied to the directory /tardisks, so here is where you can find them again. After enabling the local shell (luckily still available with 5.0) I logged in and was finally able to look inside and unpack such a driver archive using the vmtar tool:
 |
| Unpacking NET-E100.V00 with the vmtar tool |
2. A text file that maps PCI device IDs to the included driver:
 |
| Contents of /etc/vmware/driver.map.d/e1000.map |
 |
| Contents of /usr/share/hwdata/driver.pciids.d/e1000.ids |
It is good to know that the PCI ID mapping files are now separated by driver. In ESXi 4.1 there is a single pci.ids file and a single simple.map file for all drivers which raised the potential of having conflicting copies of these files in case you merged multiple OEM drivers into the image.
It looks easy now to add a custom driver to the install CD: Just create a tgz-file containing the three files mentioned above, copy it to the ISO root directory and add its name to the two BOOT.CFG files. And yes, this will indeed work for the CD boot! The custom driver will be loaded and you will be able to install ESXi, ... but the installation routine will not copy the tgz-file to the install media, and if you boot the installed system the first time it will behave like a regular install without the custom driver.
So, there is more to it...
4. The image database IMGDB.TGZ
There is a file named IMGDB.TGZ in the root directory of the CD that is also listed in the BOOT.CFG files and has the following contents:
It contains files that will be unpacked to the directory /var/db/esximg. For each driver (or other software package) an XML-file is created under the vibs sub directory. There are a lot more of these files than shown here (I fiddled the output with "..."), one example is net-e1000--925314997.xml for the e1000 driver. Let's look into this file:
The xml-file contains information about the package including possible dependencies on other packages and a list of all included files. Its file name ("net-e1000--925314997.xml") consists of the name element plus a (probably) unique number with 9 or 10 digits. The list of payloads is the list of included archive files (either of type vgz or tgz), in most cases it's just one. The name of the payload is limited to 8 characters ("net-e100" in this case) and is the name of the corresponding file in the CD's root directory. The extension of this file is expected to be ".v00" if the file is of type vgz and ".t00" if the file is of type tgz. If there are name conflicts with other packages the number in the extension is counted up. E.g. the payload file for the e1000e driver is "net-e100.v01".
Then there is the host image profile XML file in the directory /var/db/esximg/profiles. In our example this is the file ESXi-5.0.0-381646-standard1293795055. Let's look into this one:
There is a file named IMGDB.TGZ in the root directory of the CD that is also listed in the BOOT.CFG files and has the following contents:
 |
| Unpacking the IMGDB.TGZ file |
 |
| The contents of net-e1000--925314997.xml |
Then there is the host image profile XML file in the directory /var/db/esximg/profiles. In our example this is the file ESXi-5.0.0-381646-standard1293795055. Let's look into this one:
... ... ... (lot more <vib></vib> entries cutted) ... ... ...
 |
| Contents of the host image profile XML file |
Here we find a list of all vib-packages that make up the currently installed system. Please note that the vib-id of a package strictly corresponds to the element values that are in the associated vib xml file (see picture before), it is composed the following way:
<vendor>_<type>_<name>_<version>
So the vib-id element of the net-e1000 driver e.g. is
VMware_bootbank_net-e1000_8.0.3.1-2vmw.0.0.383646
The payload names that are listed in the image profile file are the same as in the distinct vib xml files with the exception that here the exact file names (e.g. "net-e100.v00") are listed rather than just the file type (vgz or tgz).
Conclusion: If we want to add a custom driver to the install CD we need to do the following (in addition to the steps described in section 3.): modify the contents of IMGDB.TGZ, add a vib xml file for the driver (similar to net-e1000...xml) to it and update the contained image profile file to include the driver as an additional <vib>-entry.
There is another particular XML element in both the vib files and image profile file that we need to take care of: the <acceptancelevel>. VMware distinguishes four different acceptance levels: VMwareCertified, VMwareAccepted, PartnerSupported and CommunitySupported, in the XML files they are coded as certified, vmware, partner and community. The names are pretty self-explanatory, and one can easily guess that certified is stricter than vmware that is stricter than partner that in turn is stricter than community. In other words: If the host image profile is of acceptance level certified only packages of the same acceptance level can be part of it. If it is of acceptance level vmware only VMware certified and VMware accepted packages can be installed. If it is of acceptance level partner (and this is the default!) partner supported packages can be installed in addition to that. The least restrictive level is community that would accept all four types of packages.
My expectation is that custom drivers for whitebox hardware are community supported (unless they are published by a hardware vendor company). However, if the driver's vib file contains the acceptance level community the image profile's acceptance level must also be changed to community. Otherwise the installation of the package will fail.
5. Can we automate it?
Yes, we can! The latest version of ESXi-Customizer does automate all the steps described here to add custom drivers in tgz-format to an ESXi 5.0 install ISO. You only need to feed it with a tgz-file that contains the three files listed in section 3 of this post.
Please note: Packages made for earlier ESXi versions will not work with ESXi 5.0, not only because the directory structure has changed, but also because the earlier versions' driver modules won't be loaded by the new version! And - at the time of this writing - there are probably no oem.tgz-style driver packages available that are compatible with ESXi 5.0!
Hopefully, this will soon change. If you are looking for a driver of a device that does not work out-of-the-box with ESXi 5.0 check the Unofficial Whitebox HCL at vm-help.com.
Please note: Packages made for earlier ESXi versions will not work with ESXi 5.0, not only because the directory structure has changed, but also because the earlier versions' driver modules won't be loaded by the new version! And - at the time of this writing - there are probably no oem.tgz-style driver packages available that are compatible with ESXi 5.0!
Hopefully, this will soon change. If you are looking for a driver of a device that does not work out-of-the-box with ESXi 5.0 check the Unofficial Whitebox HCL at vm-help.com.
Tuesday, August 23, 2011
How to throttle that disk I/O hog
We are in the middle of a large server virtualization project and are utilizing two Clariion CX-400 arrays as target storage systems. The load on these arrays is increasing while we are putting more and more VMs on them. This is somewhat expected, but recently we noticed an unusual and unexpected drop of performance on one of the CX-400s. The load on its storage processors went way up and its cache was quickly and repeatedly filled up to 100% causing so-called forced flushes: That means the array needs to shortly stop any I/O coming in while it is staging the cache contents down to the hard disks in order to free the cache up again. As a result overall latency went up and throughput went down, and this affected every VM on every LUN of this array!
As the root cause of this we identified a single VM that fired up to 50.000(!) write I/Os per second. It was a MS SQL server machine that we recently virtualized. When it was on physical hardware it used locally attached hard disks that were never able to provide this amount of I/O capacity, but now - being a VM on high-performance SAN storage - it took every I/O it could have, monopolizing the storage array's cache and bringing it to its knees.
We found that we urgently needed to throttle that disk I/O hog, or it would severely impact the whole environment's performance. There are several means to prioritize disk I/O in a vSphere environment: You can use disk shares to distribute available I/Os among VMs running on the same host. This did not not help here: the host that ran the VM had no reason to throttle it, because the other VMs it was running did not require lots of I/Os at the same time. So, for the host there was no real need to fairly distribute the available resources.
Storage I/O Control (SIOC) is a rather new feature that allows for I/O prioritization at the datastore level. It utilizes the vCenter server's view on datastore performance (rather than a single host's view) and kicks in when a datastore's latency raises over a defined threshold (30ms by default). It will then adapt the I/O queue depth's of all VMs that are on this datastore according to the shares you have defined for them. Nice feature, but it did not help here either, because the I/O hog had a datastore on its own and was not competing with other VMs from a SIOC perspective ...
We needed a way to throttle the VM's I/O absolutely, not relatively to other VMs. Luckily there really is a way to do exactly this: It is documented in KB1038241 "Limiting disk I/O from a specific virtual machine". There are VM advanced-configuration parameters described here that allow to set absolute throughput caps and bandwidth caps on a VM's virtual disks. We did this and it really helped to throttle the VM and restore overall system performance!
By the way, the KB article describes how to change the VM's advanced configuration by using the vSphere client which requires that the VM is powered off. However, there is a way to do this without powering the VM off. Since this can be handy in a lot of situations I added a description of how to do this on the HowTo page.
Update (2011-08-30): In the comments of this post Didier Pironet pointed out that there are some oddities with using this feature and refers to his blog post Limiting Disk I/O From A Specific Virtual Machine. It features a nice video demonstrating the effect of disk throttling. Let me summarize his findings and add another interesting information that was clarified and confirmed by VMware Support:
As the root cause of this we identified a single VM that fired up to 50.000(!) write I/Os per second. It was a MS SQL server machine that we recently virtualized. When it was on physical hardware it used locally attached hard disks that were never able to provide this amount of I/O capacity, but now - being a VM on high-performance SAN storage - it took every I/O it could have, monopolizing the storage array's cache and bringing it to its knees.
We found that we urgently needed to throttle that disk I/O hog, or it would severely impact the whole environment's performance. There are several means to prioritize disk I/O in a vSphere environment: You can use disk shares to distribute available I/Os among VMs running on the same host. This did not not help here: the host that ran the VM had no reason to throttle it, because the other VMs it was running did not require lots of I/Os at the same time. So, for the host there was no real need to fairly distribute the available resources.
Storage I/O Control (SIOC) is a rather new feature that allows for I/O prioritization at the datastore level. It utilizes the vCenter server's view on datastore performance (rather than a single host's view) and kicks in when a datastore's latency raises over a defined threshold (30ms by default). It will then adapt the I/O queue depth's of all VMs that are on this datastore according to the shares you have defined for them. Nice feature, but it did not help here either, because the I/O hog had a datastore on its own and was not competing with other VMs from a SIOC perspective ...
We needed a way to throttle the VM's I/O absolutely, not relatively to other VMs. Luckily there really is a way to do exactly this: It is documented in KB1038241 "Limiting disk I/O from a specific virtual machine". There are VM advanced-configuration parameters described here that allow to set absolute throughput caps and bandwidth caps on a VM's virtual disks. We did this and it really helped to throttle the VM and restore overall system performance!
By the way, the KB article describes how to change the VM's advanced configuration by using the vSphere client which requires that the VM is powered off. However, there is a way to do this without powering the VM off. Since this can be handy in a lot of situations I added a description of how to do this on the HowTo page.
Update (2011-08-30): In the comments of this post Didier Pironet pointed out that there are some oddities with using this feature and refers to his blog post Limiting Disk I/O From A Specific Virtual Machine. It features a nice video demonstrating the effect of disk throttling. Let me summarize his findings and add another interesting information that was clarified and confirmed by VMware Support:
- Unlike stated in KB1038241 you can also specify IOps or Bps values (not only K, M or GIOps resp. K, M or GBps) for the caps (e.g. "500IOps"). If you do not specify a unit at all IOps resp. Bps is assumed, not KIOps/KBps like stated in the article.
- The throughput cap can also be specified through the vSphere client (see VM properties / Resources / Disk), but not the bandwidth cap. This can even be done while the machine is powered on, and the change will become immediately effective.
- And now the part that is the least intuitive: Although you specify the limits per virtual disk the scheduler will manage and enforce the limits on a per datastore(!) basis. That means:
- If the VM has multiple virtual disks on the same datastore, and you want to limit one of them, then you must specify limits (of the same type, throughput or bandwidth) for all the virtual disks that are on the same datastore. If you don't do this, no limit will be enforced.
- The scheduler will add up the limits of all virtual disks that are on the same datastore and will then limit them altogether by this sum of their limits. This explains Didier's finding that a single disk is limited to 150IOps although he defined a limit of 100IOps for this disk, but another limit of 50IOps for a second disk that was on the same datastore.
- So, if you want to enforce a specific limit to only a single virtual disk then you need to put that disk on a datastore where no other disks of the VM are stored.
Saturday, June 18, 2011
How to hide unused FlexNICs
When I configured an HP Blade Enclosure with VirtualConnect modules for the first time I stumbled over an issue that probably has bothered most of the people doing this, especially if they run ESX(i) on the blade servers:
The BL620c G7 blade servers we are using have four built in 10Gbit-ports, and each of them can be partitioned into up to four so-called FlexNICs (or FlexHBAs for FCoE if you use them together with FlexFabric VirtualConnect modules like we do). The overall 10GBit bandwidth of one port will be split among its FlexNICs in a configurable way. You could e.g. have four FlexNICs with 2,5 GBit each, two with 6 and 4 GBit, or any combination of one to four FlexNICs with their bandwidth adding up to 10GBit.
For the OS (e.g. VMware ESXi) that is installed on the blade server each FlexNIC appears as a separate PCI device. So an ESX(i) host installed on a BL620c G7 can have up to 16 NICs. Cool, eh?
However, we did not really want to use too much of that feature and divided the first two 10Gbit-ports in a 4Gbit-FlexHBA and a 6GBit-FlexNIC each. The third and fourth port we even configured as single 10GBit-FlexNICs.
Now, the problem is that every 10Gbit-port will show up as four PCI devices even if you have configured less than four FlexNICs for it. Even if you have not partitioned it at all, but use it as a single 10Gbit-NIC, it will show up as four NICs with the unconfigured ones being displayed as disconnected!
In our case we ended up with ESXi seeing (and complaining about) 10 disconnected NICs. Since we monitor the blades with HP Insight Manager it also constantly warned us about the disconnected NICs.
So, we thought about a method to get rid of the unused FlexNICs. If we had Windows running directly on the blades this would have been easy: We would just disable the devices and Windows (and also HP Insight Manager) would not be bothered by them. However, in ESX(i) you cannot just disable a device ... but you can configure it for "VMDirectPath":
This dialog can be found in the Advanced Hardware Settings of a host's configuration. What does it do?
With VMDirectPath you can make a host's PCI device available to a single VM. It will be passed through to the VM, and the guest OS will then be able to see and use that device in addition to its virtual devices.
This way it is possible to present a physical device to a VM that you normally would not be able to add.
In the dialog shown above you configure which devices are available for VMDirectPath (also called PCI Passthrough). You can then add all the selected devices to the hardware of individual VMs.
We really did not want to do the latter... but there is one desirable side effect of this configuration: A device that is configured for VMDirectPath becomes invisible for the VMkernel. And this is exactly what we wanted to achieve for the unused FlexNICs!
So we configured all unused FlexNICs for VMDirectPath, and they were no longer being displayed as (disconnected) vmnics. If you want to do the same you need to know what PCI device a vmnic corresponds to. In the screenshot I posted you will notice that for some of the PCI devices the vmnic name is displayed in brackets, but not for all. So, it can be hard to figure out what devices need to be selected, but it's worth it!
The BL620c G7 blade servers we are using have four built in 10Gbit-ports, and each of them can be partitioned into up to four so-called FlexNICs (or FlexHBAs for FCoE if you use them together with FlexFabric VirtualConnect modules like we do). The overall 10GBit bandwidth of one port will be split among its FlexNICs in a configurable way. You could e.g. have four FlexNICs with 2,5 GBit each, two with 6 and 4 GBit, or any combination of one to four FlexNICs with their bandwidth adding up to 10GBit.
For the OS (e.g. VMware ESXi) that is installed on the blade server each FlexNIC appears as a separate PCI device. So an ESX(i) host installed on a BL620c G7 can have up to 16 NICs. Cool, eh?
However, we did not really want to use too much of that feature and divided the first two 10Gbit-ports in a 4Gbit-FlexHBA and a 6GBit-FlexNIC each. The third and fourth port we even configured as single 10GBit-FlexNICs.
Now, the problem is that every 10Gbit-port will show up as four PCI devices even if you have configured less than four FlexNICs for it. Even if you have not partitioned it at all, but use it as a single 10Gbit-NIC, it will show up as four NICs with the unconfigured ones being displayed as disconnected!
In our case we ended up with ESXi seeing (and complaining about) 10 disconnected NICs. Since we monitor the blades with HP Insight Manager it also constantly warned us about the disconnected NICs.
So, we thought about a method to get rid of the unused FlexNICs. If we had Windows running directly on the blades this would have been easy: We would just disable the devices and Windows (and also HP Insight Manager) would not be bothered by them. However, in ESX(i) you cannot just disable a device ... but you can configure it for "VMDirectPath":
 |
| PCI Passthrough configuration of a BL620c G7 |
With VMDirectPath you can make a host's PCI device available to a single VM. It will be passed through to the VM, and the guest OS will then be able to see and use that device in addition to its virtual devices.
This way it is possible to present a physical device to a VM that you normally would not be able to add.
In the dialog shown above you configure which devices are available for VMDirectPath (also called PCI Passthrough). You can then add all the selected devices to the hardware of individual VMs.
We really did not want to do the latter... but there is one desirable side effect of this configuration: A device that is configured for VMDirectPath becomes invisible for the VMkernel. And this is exactly what we wanted to achieve for the unused FlexNICs!
So we configured all unused FlexNICs for VMDirectPath, and they were no longer being displayed as (disconnected) vmnics. If you want to do the same you need to know what PCI device a vmnic corresponds to. In the screenshot I posted you will notice that for some of the PCI devices the vmnic name is displayed in brackets, but not for all. So, it can be hard to figure out what devices need to be selected, but it's worth it!
Saturday, May 28, 2011
Network troubleshooting, Part II: What physical switch port does the pNIC connect to?
When you have found out what physical NIC (pNIC) a VM is actually using (see my previous post) you may want to check the external switch port that this pNIC connects to (Is it properly configured, what about the error counters?). Okay, what switch port do you need to check?
It is considered good data center practice to have every connection from every server's pNIC to the switch ports carefully documented. Do you? Do you trust the documentation? Is it up to date? If yes you are fine and can stop reading here...
If you want to be sure, and if you use Cisco switches in your data centers then there is a much more reliable way to track these connections: The Cisco Discovery Protocol (CDP). On Cisco devices this is enabled by default, and it periodically broadcasts interface information to the devices attached to its ports (like your ESX hosts).
By default ESX(i) (version 3.5 and above) will receive these broadcasts and display the information in it through the VI client. In the Hosts and Clusters-view select Networking in the Configuration tab of a host. This will display your virtual switches with their physical up-links (vmnic0, vmnic1, etc.). Now click on the little speech bubbles next to the Physical Adapters and a window like the following will pop up:
You can find a lot useful information here. The Device ID is the name of the Cisco switch. And Port ID shows the number/name of the switch module and the port number on that module. So you can tell your network admins exactly what switch port they need to check.
If CDP information is not available for a physical adapter the pop-up window will also tell you this. Possible reasons: You don't use Cisco switches or have CDP broadcasts disabled on them, or the ESX(i) host's interfaces are not in CDP listen mode.
For more detailed information on CDP and how to configure it in ESX see the VMware KB: Cisco Discovery Protocol (CDP) network information.
It is considered good data center practice to have every connection from every server's pNIC to the switch ports carefully documented. Do you? Do you trust the documentation? Is it up to date? If yes you are fine and can stop reading here...
If you want to be sure, and if you use Cisco switches in your data centers then there is a much more reliable way to track these connections: The Cisco Discovery Protocol (CDP). On Cisco devices this is enabled by default, and it periodically broadcasts interface information to the devices attached to its ports (like your ESX hosts).
By default ESX(i) (version 3.5 and above) will receive these broadcasts and display the information in it through the VI client. In the Hosts and Clusters-view select Networking in the Configuration tab of a host. This will display your virtual switches with their physical up-links (vmnic0, vmnic1, etc.). Now click on the little speech bubbles next to the Physical Adapters and a window like the following will pop up:
 |
| CDP information shown in the VI client |
You can find a lot useful information here. The Device ID is the name of the Cisco switch. And Port ID shows the number/name of the switch module and the port number on that module. So you can tell your network admins exactly what switch port they need to check.
If CDP information is not available for a physical adapter the pop-up window will also tell you this. Possible reasons: You don't use Cisco switches or have CDP broadcasts disabled on them, or the ESX(i) host's interfaces are not in CDP listen mode.
For more detailed information on CDP and how to configure it in ESX see the VMware KB: Cisco Discovery Protocol (CDP) network information.
Thursday, May 26, 2011
Network troubleshooting, Part I: What physical NIC does the VM use?
If you encounter a network issue in a VM (like bad performance or packet drops) a good first question to ask yourself is: Is this issue limited to the VM or can it be pinned to one of the host's physical NICs?
So, you need to find out what physical NIC (pNIC) the VM is actually using. In most environment this is not obvious, because the virtual switch that the VM connects to typically has multiple physical up-links (for redundancy) that are all active (to maximize bandwidth).
Unfortunately, it is not possible to find this out by using the VI client. It does not reveal this information regardless whether you use standard or distributed virtual switches.
You need to log in to the host that runs the VM (see the HowTos section for instructions) and run esxtop.
Press n to switch to the network view, and you will see a picture like this one:
Find the VM's display name in the USED-BY column and look to the corresponding TEAM-PNIC column then. In this example the VM FRASINT215 uses vmnic1.
So, you need to find out what physical NIC (pNIC) the VM is actually using. In most environment this is not obvious, because the virtual switch that the VM connects to typically has multiple physical up-links (for redundancy) that are all active (to maximize bandwidth).
Unfortunately, it is not possible to find this out by using the VI client. It does not reveal this information regardless whether you use standard or distributed virtual switches.
You need to log in to the host that runs the VM (see the HowTos section for instructions) and run esxtop.
Press n to switch to the network view, and you will see a picture like this one:
 |
| Network view of esxtop (click to enlarge) |
Subscribe to:
Posts (Atom)