vSphere 5 licensing - check your environment now to see how it affects you

There has been a lot of rant about the new licensing model of vSphere 5 (see my previous post), because for certain customers (specifically those with a very high RAM per CPU ratio which is more and more common with recent server hardware) will need to buy more vSphere 5 licenses to cover the vRAM usage as they had vSphere 4 licenses before.

Before you start complaining yourself check your environment now to find out how it will affect you. There are a number of PowerCLI scripts available now for doing this. I personally like LucD's the most, get it here: http://www.lucd.info/2011/07/13/query-vram/.

For my production environment it outputs the following:

  vCenter        : [MyVC-FQDN]
  vRAMConfigured : 2732.2
  vRAMUsed       : 2624.8
  vRAMEntitled   : 6000
  LicenseType    : vSphere 4 Enterprise Plus

Note that the used vRAM is lower than the configured vRAM, because it only takes into account the total RAM of all running VMs (and I also have some that are powered off).
The current version of the script also counts the assigned licenses only. However, if you have spare licenses that are currently unassigned, they will also add up to vRAM entitlements once they are upgraded to vSphere 5 (I ask Luc to fix that, maybe there is a new version of his script soon).

Anyway, as you can see, I am lucky with the new licensing model and would (yet) have plenty of unused vRAM in my pool if I upgraded today.

Update: There is now an even better script available by Virtu-Al: It can also handle ESX versions earlier than 4.1, looks for unassigned licenses and has a nice HTML output:
  http://www.virtu-al.net/2011/07/14/vsphere-5-license-entitlements/
It is referenced in an official VMware Blog post that tries to better explain the new licensing model and the motivation behind it.

VMware raised the bar - Announcement of vSphere 5 and other new products

Today VMware made some major announcements of new products and product versions that are planned to be available in Q3 2011 (see original press release):

vSphere 5 includes the following improvements and new features (compared to vSphere 4.1):

• Improved VM scalability (up to 32 vCPUs and 1 TB RAM) and performance (3x to 4x I/O improvements)
• New and improved HA architecture (easier to set up and more scalable)
• Autodeploy: On-the-fly deployment of ESXi hosts through PXE-boot
• Profile driven storage: allows to define classes of storage (distinguished e.g. by performance and availability) and tie VMs to it by defining "storage policies"
• Storage DRS: Automatic initial storage placement and balancing of VMs
• vSphere 5 hosts are ESXi only. No more classic ESX (like previously announced)
• Change in licensing model: vSphere 5 will still be licensed per physical CPU socket, but introduces another component: vRAM, which is the amount of RAM configured for VMs. Each CPU license entitles for the use of a specific amount of vRAM (dependent on the vSphere edition, e.g. 48 GB for Enterprise Plus). vRAM entitlements can be pooled among all hosts managed by a vCenter instance. For details see the new Licensing Whitepaper.

vCenter Site Recovery Manager (SRM) 5 introduces "vSphere Replication" a.k.a. host-based mirroring and a new "automatic failback" feature. For details see VMware's official product page.

The new vSphere Storage Appliance turns the local hard disks in your ESXi hosts into mirrored, highly available NFS datastores. This way you can use VMotion, DRS and HA without the need for additional shared storage hardware. See the product overview and the technical whitepaper.

vShield 5 introduces new sensitive data discovery and intrusion detection capabilities.

vCloud Director 1.5 now supports fast provisioning with linked clones (a feature that was already available with the Lab Manager product that is now obsoleted by vCloud Director) and supports SQL as its database.

Using hardware-assisted virtualization in Windows Server 2003 32-bit virtual machines

This is the title of a VMware KB article (KB2001372) that was recently posted, and it includes very interesting information for anyone running virtualized Windows 2003 servers on vSphere (so, probably all of us).

ESX(i) is able to use different methods for virtualizing the CPU and associated MMU (memory management unit) instruction sets. You can configure that for a VM in its Advanced Options / CPU/MMU virtualization:

CPU/MMU virtualization settings

In the Binary Translation (BT) mode software emulation is used for both CPU and MMU instructions (the second choice in the picture). For a long time this was the only option, until the CPU vendors Intel and AMD started building virtualization functions into their processors.
Choosing the third option will enable these hardware functions for the CPU instruction set virtualization (if available), but will remain using software virtualization for MMU instructions.
The fourth option will enable hardware virtualization for both types of instructions if available.
It depends on the CPU generation whether none, only the first or both hardware virtualization options are available. Since quite a few years Intel's and AMD's processors support CPU as well as MMU virtualization.

The default in the above dialog is "Automatic". This means that ESX(i) will choose what it considers to be the best option for the type of operating system that you have selected for the VM.
With Windows 2003 this is the "Software" mode. The reason for this is that Windows 2003 with SP1 in fact performs better with software emulation than with hardware virtualization. However, this changed with code changes introduced by Microsoft with SP2. Windows 2003 with SP2 performs better with hardware virtualization in almost any case.
Today, most Windows 2003 servers should have been updated with SP2. So, to ensure best performance you should go and change the virtualization mode of these VMs to one of the hardware-assisted ones.

For more details see the KB-article mentioned above.

Raising the bar, Part V - vSphere 5 is near

If you look at VMware's homepage these days you will notice an announcement of a live event on July 12th. It is titled "Raising the bar, Part V".
You don't need to be a visionary to figure out that this can only mean that VMware will announce the long-awaited new major release of its virtualization platform: vSphere 5.

This does not necessarily mean that vSphere 5 will become generally available on July 12th. However, once it is available I will post a list of at least the most important new features of it. So, stay tuned!

Mysterious port 903

I recently investigated what network ports are used by ESXi 4.1, because I had to compile the firewall requirements for a new deployment of ESXi hosts in a DMZ. There is a detailed source available for that in the VMware KB:

• KB1012382: TCP and UDP Ports required to access vCenter Server, ESX hosts, and other network components

And there are numerous other sources available (even nice diagrams like this one). In most cases it is obvious that their authors referred to and relied on the above mentioned official VMware KB source.

I'm usually not paranoid, but maybe I talked too much with the IT security guys (who tend to be extremely paranoid ;-)). Anyway, following the rule "Trust no one" I started looking at the network ports that are really used in our current production environment and compared them to the list in the KB article.

So I stumbled over port 903... According to the list both the vCenter server and any vSphere Client connect to an ESXi 4.1 host on that port for accessing the VM remote console. However, when I checked the network connections on the vCenter server and my Windows Desktop running the vSphere Client (with "netstat -an") I was not able to see any connection to an ESXi host's port 903, even when I opened multiple VM consoles. Instead it was obvious that port 902 is used for console connections.

This made me really curious, so I logged on to an ESXi host (in Tech support mode) and checked the open network connections there. In ESXi you use the command "esxcli network connection list" for that which produces an output that is quite similar to the netstat output (With classic ESX the netstat command is still available in the service console).

This command will also list all ports that are opened in LISTEN mode, that means there is some process waiting for connections on that port. But there was no listening process for port 903, and that means that no one and nothing would be able to connect to that port!

I opened a support request with VMware asking for clarification on the mysterious port 903 and was very curious about their answer. Of course, they quoted their own KB article first, insisted on that the port was actually used for this and that, but finally - after raising the issue to engineering - they admitted that "ESXi does not use port 903".

Also a request was made to update the KB article accordingly. So, when you read this it might already have been corrected to not include port 903 anymore, but the numerous third party documents based on KB1012382 will take some more time to be updated ...

Bottom line: Information is good. Correct information is better. Try to verify it if it is really important to you.

A quick primer on Changed Block Tracking (CBT)

We are about to implement a new backup solution that is based on Symantec Netbackup 7, and - like any modern VMware backup solution - it leverages a very cool feature named Changed Block Tracking (CBT) that was introduced in vSphere 4.0 to enable efficient block level incremental backups.

Since it has been around for a while there are numerous good articles around about that topic (see references). I will not just reproduce them here, but summarize the most important key facts you need to know if you come into touch with it for the first time.

1. How does CBT work and what is it good for?
If CBT is enabled for a virtual disk the VMkernel will create an additional file (named ...-ctk.vmdk) in the same directory where it stores a map of all the virtual disk's blocks. Once a block is changed it will be recorded in this map file. This way the VMkernel can easily tell a backup application what blocks of a file have changed since a certain point in time. The application can then perform an incremental backup by saving only these changed blocks.
CBT is also used by Storage VMotion that is able to move a virtual machine's disk files from one datastore to another while it is running.

2. How do you enable CBT?
CBT is to be enabled per virtual disk, and VMware's KB1031873 describes how to do this via editing a VM's advanced configuration parameters through the VI client. Unfortunately this requires the VM to be powered off. However, you can also change the setting while the VM is running by using an appropriate script like the one published here. To make the change effective then you need to perform a so called stun/unstun-cycle on the VM (i.e. power on/off, suspend/resume, create/delete snapshot).
It is important to know that CBT is not enabled by default, because it introduces a small overhead in virtual disk processing.

3. How do CBT and snapshots play together?
When you create a snapshot of a VM's virtual disk an additional ctk-file is created for the delta-disk file of the snapshot. Once this snapshot is deleted again the delta-ctk will be merged with the base-ctk, just like the delta-disk is merged with the base-disk.

4. Important notes and references

• KB1020128: Changed Block Tracking (CBT) on virtual machines
• KB1031873: Enabling Changed Block Tracking (CBT) on virtual machines
• While an application backs up a VM using CBT the VM cannot be vMotioned: KB2001004
• Inconsistency resolved in vSphere 4.0 U3 and vSphere 4.1: KB1021607
• KB1031106: Virtual machine freezes temporarily during snapshot removal on an NFS datastore in a ESX/ESXi 4.1 host
• Eric Siebert on CBT: A detailed introduction
• Additions by Duncan Epping: Even more details...

How to hide unused FlexNICs

When I configured an HP Blade Enclosure with VirtualConnect modules for the first time I stumbled over an issue that probably has bothered most of the people doing this, especially if they run ESX(i) on the blade servers:

The BL620c G7 blade servers we are using have four built in 10Gbit-ports, and each of them can be partitioned into up to four so-called FlexNICs (or FlexHBAs for FCoE if you use them together with FlexFabric VirtualConnect modules like we do). The overall 10GBit bandwidth of one port will be split among its FlexNICs in a configurable way. You could e.g. have four FlexNICs with 2,5 GBit each, two with 6 and 4 GBit, or any combination of one to four FlexNICs with their bandwidth adding up to 10GBit.
For the OS (e.g. VMware ESXi) that is installed on the blade server each FlexNIC appears as a separate PCI device. So an ESX(i) host installed on a BL620c G7 can have up to 16 NICs. Cool, eh?

However, we did not really want to use too much of that feature and divided the first two 10Gbit-ports in a 4Gbit-FlexHBA and a 6GBit-FlexNIC each. The third and fourth port we even configured as single 10GBit-FlexNICs.

Now, the problem is that every 10Gbit-port will show up as four PCI devices even if you have configured less than four FlexNICs for it. Even if you have not partitioned it at all, but use it as a single 10Gbit-NIC, it will show up as four NICs with the unconfigured ones being displayed as disconnected!
In our case we ended up with ESXi seeing (and complaining about) 10 disconnected NICs. Since we monitor the blades with HP Insight Manager it also constantly warned us about the disconnected NICs.

So, we thought about a method to get rid of the unused FlexNICs. If we had Windows running directly on the blades this would have been easy: We would just disable the devices and Windows (and also HP Insight Manager) would not be bothered by them. However, in ESX(i) you cannot just disable a device ... but you can configure it for "VMDirectPath":

PCI Passthrough configuration of a BL620c G7

This dialog can be found in the Advanced Hardware Settings of a host's configuration. What does it do?
With VMDirectPath you can make a host's PCI device available to a single VM. It will be passed through to the VM, and the guest OS will then be able to see and use that device in addition to its virtual devices.
This way it is possible to present a physical device to a VM that you normally would not be able to add.

In the dialog shown above you configure which devices are available for VMDirectPath (also called PCI Passthrough). You can then add all the selected devices to the hardware of individual VMs.
We really did not want to do the latter... but there is one desirable side effect of this configuration: A device that is configured for VMDirectPath becomes invisible for the VMkernel. And this is exactly what we wanted to achieve for the unused FlexNICs!

So we configured all unused FlexNICs for VMDirectPath, and they were no longer being displayed as (disconnected) vmnics. If you want to do the same you need to know what PCI device a vmnic corresponds to. In the screenshot I posted you will notice that for some of the PCI devices the vmnic name is displayed in brackets, but not for all. So, it can be hard to figure out what devices need to be selected, but it's worth it!