Yesterday VMware published minor updates to their Cloud Infrastructure suite products including vCenter and ESXi 5.0 Update 1. The news is already well covered, e.g. here in the VMware vSphere blog by Duncan Epping.
It looks like there are only few new features in there. For me the most exiting one is that VAAI Thin Provisioning Block Reclaim/UNMAP is back, not inline, but on demand though. However, the list of bug fixes is very long, and this is good! I have the feeling that many enterprise customers that are still on vSphere 4.x were waiting for this update and will now more seriously attack the migration to vSphere 5.0 (following the conservative rule "Don't trust a .0 release"). At least this is what we were thinking ...
Friday, March 16, 2012
Thursday, March 15, 2012
Watch out if you used the ESXi 5.0 Driver Rollup ISO 1 or 2 to install your hosts!
If you have installed your hosts with the VMware ESXi 5.0 Driver Rollup ISO (version 1 or 2) then you may run into the following issue sooner or later:
The Rollup ISOs contain newer versions of the QLogic ima-qla4xxx driver package and they include some superfluous files that will be stored in a directory called /VIB once the installed system is booted. The presence of these files does not harm the functionality of ESXi 5.0, but it prevents the successful installation of certain other software packages (that also include a /VIB directory). The problem is known to occur with the vShield App 5.0 agent package for ESXi 5.0, but may potentially occur with other packages as well.
You can consider this a bug in both the ima-qla4xxx driver and the vShield App package, because the use of the /VIB directory (for storing pre- and post-installation scripts) is deprecated in ESXi 5.0. Obviously for a good reason ...
The following versions of the ima-qla4xxx driver packages do have this issue:
Workaround / Solution:
If you do not need the ima-qla4xxx package for your host's hardware then just remove it by running the following command in an ESXi shell:
esxcli software vib remove -n ima-qla4xxx
If you have a device in your host that needs this driver to work properly then you need to wait for an updated and fixed version of this driver and/or the vShield App agent package.
If you are unsure whether you are affected by this issue then check your ESXi host's file system in an ESXi shell for the presence of the directory /VIB:
ls /VIB
If the command returns a file listing (of the files postinst, postrm, preinst and prerm) then you are affected by this issue.
And now the fun part of the story: How was this issue discovered?
Recently I was contacted by a VMware Escalation Engineer who told me that a VMware customer ran into the issue described above with the vShield App agent package. This customer had installed his hosts with an ESXi 5.0 Driver Rollup ISO that he customized with my ESXi-Customizer script (to inject some newer Brocade CNA drivers into it). This is why the Engineer believed that the issue was caused by ESXi-Customizer and contacted me.
Well, we both did some tests and could finally prove that the issue was not caused by ESXi-Customizer, but solely by the ima-qla4xxx package.
Anyway, I want to take this incident as an opportunity to remind all users of my ESXi-Customizer script of an important and obvious fact: My tool is not supported by VMware, and VMware will never endorse using it (although I learnt that there are quite a few VMware employees using it at home ;-).
On the other hand it is good to know that
a) I am not aware of a single case where my tool caused an issue, and
b) VMware Support will come to your rescue even if you used it and run into a problem afterwards.
The Rollup ISOs contain newer versions of the QLogic ima-qla4xxx driver package and they include some superfluous files that will be stored in a directory called /VIB once the installed system is booted. The presence of these files does not harm the functionality of ESXi 5.0, but it prevents the successful installation of certain other software packages (that also include a /VIB directory). The problem is known to occur with the vShield App 5.0 agent package for ESXi 5.0, but may potentially occur with other packages as well.
You can consider this a bug in both the ima-qla4xxx driver and the vShield App package, because the use of the /VIB directory (for storing pre- and post-installation scripts) is deprecated in ESXi 5.0. Obviously for a good reason ...
The following versions of the ima-qla4xxx driver packages do have this issue:
- ima-qla4xxx version 500.2.01.20-1vmw.0.0.081609 (included in the Rollup 2 ISO, and separately available here)
- ima-qla4xxx version 500.2.01.29-1vmw.0.0.112306 (available here)
- ima-qla4xxx version 500.2.01.16-1vmw.0.0.051901 (was included in the Rollup 1 ISO which is not available for download anymore)
Workaround / Solution:
If you do not need the ima-qla4xxx package for your host's hardware then just remove it by running the following command in an ESXi shell:
esxcli software vib remove -n ima-qla4xxx
If you have a device in your host that needs this driver to work properly then you need to wait for an updated and fixed version of this driver and/or the vShield App agent package.
If you are unsure whether you are affected by this issue then check your ESXi host's file system in an ESXi shell for the presence of the directory /VIB:
ls /VIB
If the command returns a file listing (of the files postinst, postrm, preinst and prerm) then you are affected by this issue.
And now the fun part of the story: How was this issue discovered?
Recently I was contacted by a VMware Escalation Engineer who told me that a VMware customer ran into the issue described above with the vShield App agent package. This customer had installed his hosts with an ESXi 5.0 Driver Rollup ISO that he customized with my ESXi-Customizer script (to inject some newer Brocade CNA drivers into it). This is why the Engineer believed that the issue was caused by ESXi-Customizer and contacted me.
Well, we both did some tests and could finally prove that the issue was not caused by ESXi-Customizer, but solely by the ima-qla4xxx package.
Anyway, I want to take this incident as an opportunity to remind all users of my ESXi-Customizer script of an important and obvious fact: My tool is not supported by VMware, and VMware will never endorse using it (although I learnt that there are quite a few VMware employees using it at home ;-).
On the other hand it is good to know that
a) I am not aware of a single case where my tool caused an issue, and
b) VMware Support will come to your rescue even if you used it and run into a problem afterwards.
Saturday, March 10, 2012
How to run the HP Online ACU CLI for Linux in ESXi 4.x
Please note: I noticed that this post still gets a lot of attention... Before reading through the whole post you should know: the workaround described here is only necessary for ESXi 4.x. For ESXi 5.x there is an officially supported Online ACU CLI available from HP (see the update at the end of this post)!A while ago I posted about the Offline version of HP's Array Configuration Utility (ACU). HP does not provide an Online version of this tool for VMware ESXi, so expanding a local RAID array can only be done with this Offline version which requires a downtime of the host for the whole duration of the expansion.
Somehow I did not want to accept this, and - by just trying it out - I finally discovered that you can actually run the Linux version of the ACU CLI tool (hpacucli) in ESXi! hpacucli is not the web interface version of the ACU (which is called cpqacuxe) that you may be used to from Windows systems. cpqacuxe is also available in a Linux version, but I could not get it to run in the ESXi shell.
hpacucli is surely less intuitive to use, but it offers the same functionality than the web tool and allows you to do all controller configuration and volume expansion tasks that are supported with SmartArray controllers. Before trying it be sure to look at the Configuring Arrays on HP Smart Array Controllers Reference Guide (section "Using the ACU CLI", p51 f.) to learn how to use it. The reference guide also describes another utility named hpacuscripting that is also included in this package and works with ESXi. I haven't tried that, but it looks like it is mainly meant for the initial RAID configuration on a newly deployed host.
The Linux version of hpacucli requires some script modifications to make it run in the ESXi shell. First I thought about providing a modified version of it that you could just install and run in ESXi, but then I noticed that HP's license terms do not allow to redistribute the software package. So I will provide step-by-step instructions instead for how to do the required modifications on your own using a Windows machine. Don't worry, it's easy:
1. Download the file hpacucli-8.75-12.0.noarch.rpm from HP.
2. Download 7-zip and install it.
3. Open the downloaded rpm-file in 7-zip:
4. Inside the 7-zip window navigate to the embedded directory \opt\compaq\hpacucli\bld by double-clicking on the displayed file and directory names:
5. Select all the displayed files and click on the "Extract" or "Copy" button to extract them to a new empty sub-directory (U:\$Download\hpacucli in this example):
6. Download the patch script fix4esxi.sh that I prepared and put it into the same directory.
7. Now use the vSphere Client to upload the complete directory to a datastore that is accessible by the host that you want to run the tool on. You can store the directory on a shared datastore to make the same copy accessible to multiple hosts:
8. Now open a shell as root in the ESXi host that you want to run the tool on. Change to the datastore directory that you uploaded and run the patch script like this:
. ./fix4esxi.sh
You need to run the script only once! It will patch the shell scripts that are included and restore the executable permission bits that got lost when we extracted the files in Windows.
9. Now you are ready to run the hpacucli utility in ESXi, either by running ./hpacucli after changing to the installation directory or by calling it by its full file path:
The command controller all show config will immediately show you if the utility has detected your SmartArray controller and display its configuration. With the help command you will get an overview of all available commands and how to use them. However, I strongly recommend reading the above mentioned reference guide, because it includes more detailed information and usage examples.
Now at the end ... a big fat warning and some more hints: Running hpacucli inside ESXi is (of course) not supported by HP or VMware! I can tell you that it just works for me in the configurations that I was able to test, but it could as well fail completely for you or blow up your host!
The tool was able to detect SmartArray P400 and P410i controllers with ESXi 4.1 and 5.0. On an ESXi 5.0 host with a P400 controller I was able to break a RAID1 mirror and concatenate the disks to a RAID0 volume of double size. However, ESXi would not pick up the changed disk size, so I was not able to grow the VMFS volume without rebooting the host. At least I only needed to reboot (causing a very short downtime) and did not have to take the host offline for the whole volume conversion process (which can take very long depending on hard disk sizes).
I wonder if ESXi not detecting the disk size change is due to the old cciss driver that the P400 controller is using. Other SmartArray controllers use the newer hpsa driver that might not show this issue.
Please provide feedback by commenting on this post if you manage to successfully make use of hpacucli in ESXi like described here! This will help to get me and others an overview of what works and what not.
Update (2012-04-18): In the meantime HP has officially made available hpacucli for ESXi. It is part of the HP ESXi Utilities Offline Bundle for VMware ESXi 5.0 that I reference on my HP & VMware Links page. For earlier versions of ESXi you still depend on the workaround that I described here.
Wednesday, March 7, 2012
How to subscribe to HP Software Alerts
I often complain about the hp.com web pages and how challenging it is to find the right information there. And this is a pity, because there really is quite a lot of important and useful information available here, e.g. regarding BIOS, firmware and driver updates for the HP hardware that you are using and advisories about known problems and how to avoid or fix them.
So it is good to know that you can actually subscribe to this information and have e-mails sent to you whenever there are important updates or alerts for your hardware.
Click on the Subscriber's choice for business log in page to register for and configure this service. If you already have an account with HP you can directly sign in with it, otherwise click on the "Register" button and create a new account (It's free and just requires a valid email address).
Once you are logged in you can select the products that you want to receive alerts for by clicking on the "Customize" link for Driver and Support alerts:
I recommend that you carefully think about what HP hardware components you use and want to add here. If e.g. you are using c7000 blade enclosures with BL620c G7 servers then add every single component that has its own firmware and/or drivers: the BL620s, but also their iLO boards and CNAs, the enclosure, Onboard Administrator and Virtual Connect modules (see picture). Add this list for any type of HP server model you have.
In the next steps you choose how often and in which format (HTML or text e-mail or RSS feed) you want to receive the alerts and what operating systems you are interested in.
Submit you registration and you will soon receive e-mail alerts like this one:
Highly recommended to pro-actively inform yourself about possible hardware problems (and there is a plenty of them)!
So it is good to know that you can actually subscribe to this information and have e-mails sent to you whenever there are important updates or alerts for your hardware.
Click on the Subscriber's choice for business log in page to register for and configure this service. If you already have an account with HP you can directly sign in with it, otherwise click on the "Register" button and create a new account (It's free and just requires a valid email address).
Once you are logged in you can select the products that you want to receive alerts for by clicking on the "Customize" link for Driver and Support alerts:
 |
| Subscribe to HP alerts |
In the next steps you choose how often and in which format (HTML or text e-mail or RSS feed) you want to receive the alerts and what operating systems you are interested in.
Submit you registration and you will soon receive e-mail alerts like this one:
 |
| HP Alerts example e-mail |
Sunday, February 26, 2012
How to smoothly migrate a VM from one vDS to another
If you are using virtual Distributed Switches (vDS) in your environment you may have stumbled in a situation where you wanted to migrate a VM from one cluster to another using different vDS and couldn't do it: VMotion is not possible in this scenario. Even if the source and target host are CPU-compatible and both have access to the VM's storage, vCenter will deny a live migration whenever the target host is not part of the vDS that the VM is connected to. This is somewhat understandable, because the VM would lose its network port while running, and this is probably not what you intended.
So, let's power off the VM and try a cold migration ... but - surprise(?) - even this will be rejected by vCenter, with the same error message:
So, let's power off the VM and try a cold migration ... but - surprise(?) - even this will be rejected by vCenter, with the same error message:
 |
| vCenter rejecting an inter-vDS migration |
One way to cope with that is to remove the vNIC from the VM, do the migration and then add a new vNIC to the VM. However, this has consequences that might be undesirable: The new NIC will have a new MAC address and will be detected as a new device by the guest OS which might led to even more problems, another reboot being necessary etc.
There is another, smoother way to do it: In our environment we have created an extra vDS that serves the only purpose of enabling inter-vDS migrations. We called it TransferSwitch, another good name would be MigrationSwitch, and we have added all hosts of all clusters that are managed by vCenter to this vDS, but without any physical uplinks. A migration of a VM from one vDS to another then becomes a three steps process:
- Connect the VM's NIC to the TransferSwitch
- Migrate the VM to the target host
- Connect the VM's NIC to the destination vDS
Monday, February 20, 2012
About the VMware Tools of ESXi 5.0 and why you should install them on vSphere 4.x
There is a rather new VMware KB article available that describes an interesting problem with the VMware Tools version of ESX(i) 4.1 Update 2: If the clock resolution of a Windows VM has been changed from the default then the VMware Tools service will continually consume 15% CPU performance (in a 1 vCPU VM, for 2 vCPU VMs it will be 7%, etc.).
We have seen this problem on few of our VMs, it looks like there are certain Windows applications around that change the clock resolution thus causing the problem. Detailed background information about the Windows clock resolution (and why it is not a good idea to change it) is available from Microsoft.
The resolution documented in the KB article is to downgrade the VMware Tools to an earlier version or - and this is probably surprising for most of us - to install the VMware Tools version of ESXi 5.0 instead.
This reminds us of the fact that VMware has changed their VMware Tools support policy with the introduction of vSphere 5.0: The VMware Product Interoperability Matrixes now include a selection for the VMware Tools, and it shows that the Tools of ESXi 5.0 are "interoperable" not only with ESXi 5.0 but also with ESX(i) 4.1 and even ESX(i) 4.0:
... whereas earlier versions were only interoperable with the corresponding ESX(i) version.
So, if you are still on vSphere version 4.1 or 4.0 and are planning to upgrade to vSphere 5 sooner or later then you can start deploying the VMware Tools of ESXi 5.0 now, and avoid the effort of future tools upgrades.
You can download the latest version of the ESXi 5.0 tools here at packages.vmware.com.
If you run a manual custom installation of the ESXi 5.0 tools in a Windows VM you will notice that there are some new components included:
The default selection of components (this is what you get when doing an automatic install or upgrade) is now more suitable for VMware ESXi than it was with earlier versions of the tools, but it still includes two components that are useful for VMware Workstation and completely useless when running ESXi: the Record/Replay Driver and the Audio Driver. Earlier versions of the Tools would also install the Shared Folders component by default, although it is also only useful with VMware Workstation.
A last hint: There is still another "feature" in the VMware Tools package for Windows that I personally find very annoying: Once you have installed the Tools you are by default not able to modify or repair the installation through the "Add or Remove Programs" control panel applet. To fix this find the GUID key for the VMware Tools package in the registry under
HKLM\Software\Microsoft\Windows\CurrentVersion\Uninstall
and change the NoModify and NoRepair values there to 0.
We have seen this problem on few of our VMs, it looks like there are certain Windows applications around that change the clock resolution thus causing the problem. Detailed background information about the Windows clock resolution (and why it is not a good idea to change it) is available from Microsoft.
The resolution documented in the KB article is to downgrade the VMware Tools to an earlier version or - and this is probably surprising for most of us - to install the VMware Tools version of ESXi 5.0 instead.
This reminds us of the fact that VMware has changed their VMware Tools support policy with the introduction of vSphere 5.0: The VMware Product Interoperability Matrixes now include a selection for the VMware Tools, and it shows that the Tools of ESXi 5.0 are "interoperable" not only with ESXi 5.0 but also with ESX(i) 4.1 and even ESX(i) 4.0:
 |
| VMware Tools Interoperability Matrix |
So, if you are still on vSphere version 4.1 or 4.0 and are planning to upgrade to vSphere 5 sooner or later then you can start deploying the VMware Tools of ESXi 5.0 now, and avoid the effort of future tools upgrades.
You can download the latest version of the ESXi 5.0 tools here at packages.vmware.com.
If you run a manual custom installation of the ESXi 5.0 tools in a Windows VM you will notice that there are some new components included:
 |
| VMware Tools 5.0 components default selection |
A last hint: There is still another "feature" in the VMware Tools package for Windows that I personally find very annoying: Once you have installed the Tools you are by default not able to modify or repair the installation through the "Add or Remove Programs" control panel applet. To fix this find the GUID key for the VMware Tools package in the registry under
HKLM\Software\Microsoft\Windows\CurrentVersion\Uninstall
and change the NoModify and NoRepair values there to 0.
Sunday, February 19, 2012
vSphere Hypervisor ESXi 5.0 Driver Rollup 2
Since the release of ESXi 5.0 on August 24th 2011 VMware and 3rd party hardware vendors have updated and added lots of device drivers that can be downloaded in the Drivers & Tools section of VMware's vSphere 5 download center. These drivers are not included in the original ESXi 5.0 installation ISO.
However, at the same place you can download something that is called the vSphere Hypervisor ESXi 5.0 Driver Rollup 2. In fact this is the original installation ISO plus all drivers that were updated or added up to the release date of the rollup (for the Rollup 2 this is Feb 8 2012). So, if you are going to install a new server you should choose this ISO (instead of the GA release ISO) to make use of the latest drivers.
There have been reports that you cannot download the Rollup ISO if you have the free Hypervisor license only, but this is not true (at least not anymore): If you have registered for the Free VMware vSphere Hypervisor you will also be able to download the Rollup ISO.
Please be aware that the Rollup ISO does only contain updated drivers, but no Hypervisor patches, so after deploying a machine with it you still need to apply the latest VMware patches in order to have a system that is fully up to date. There are instructions available on how to do this if you do not have vCenter Update Manager available: See e.g. a post by Chris Colotti.
Update (2012-03-15): There is an issue with the current Rollup 2 ISO that you need to be aware of, especially if you are considering the installation of VMware vShield 5.0. Read more in this blog post.
However, at the same place you can download something that is called the vSphere Hypervisor ESXi 5.0 Driver Rollup 2. In fact this is the original installation ISO plus all drivers that were updated or added up to the release date of the rollup (for the Rollup 2 this is Feb 8 2012). So, if you are going to install a new server you should choose this ISO (instead of the GA release ISO) to make use of the latest drivers.
There have been reports that you cannot download the Rollup ISO if you have the free Hypervisor license only, but this is not true (at least not anymore): If you have registered for the Free VMware vSphere Hypervisor you will also be able to download the Rollup ISO.
Please be aware that the Rollup ISO does only contain updated drivers, but no Hypervisor patches, so after deploying a machine with it you still need to apply the latest VMware patches in order to have a system that is fully up to date. There are instructions available on how to do this if you do not have vCenter Update Manager available: See e.g. a post by Chris Colotti.
Update (2012-03-15): There is an issue with the current Rollup 2 ISO that you need to be aware of, especially if you are considering the installation of VMware vShield 5.0. Read more in this blog post.
Subscribe to:
Posts (Atom)