Explaining the CX + vSphere5 + VAAI support riddle

I recently blogged about the oddity of EMC CX arrays not being supported with VAAI on vSphere 5.0 anymore and - like many others - I wondered what are the real reasons behind this. Today Chad Sakac (EMC) explained the riddle in a web cast covering a VNX engineering update and - well - exactly this issue.

If you missed it: he just published the presentation that he showed on his blog. Here is a summary:
Read more »

Updated: What's the deal with EMC CX arrays not supporting VAAI with vSphere 5.0?

(If you have read this post before then skip to the bottom for recent updates)

We are in the process of upgrading our production environment to vSphere 5.0 U1. A new vCenter 5.0 server is already in place, and we attached the production ESXi hosts (that are still on 4.1 U2) to it.
The next step would be to upgrade the hosts. In the meantime, since this is a qualified production environment running more than 1.500 virtual servers I became a bit paranoid about hardware and software/firmware compatibility, and decided to double check if the environment is fully supported (or if we should upgrade any firmware first). I was pretty confident that we are on the safe side because we never had any issues with vSphere 4.1 and always kept the environment up to date.

But then I stumbled over this VMware KB article: EMC CX and VNX Firmware and ESX requirements for vStorage APIs for Array Integration (VAAI) support. It states that ESXi 5.0 does not support VAAI on our Clariion CX4 arrays even although they have the latest FLARE code:

	vSphere 4.1	vSphere 5.0
CX4 Series Flare 30/29/28+	VAAI Supported	Not Supported
VNX Series OE 31 or Later *	VAAI Supported	VAAI Supported

I checked the test hosts that we already updated and they showed VAAI being supported on the CX4 LUNs in the vSphere client. However, the KB article recommends to disable VAAI on these hosts ...

I quickly searched the Internet for relevant posts and statements and browsed through the VMware Community forums. There I found this post where people complain about the VAAI status shown as unsupported. It looks like there are dependencies to the LUN size (smaller or larger than 2TB) and to the array's failover mode (sounds like you need to use ALUA / failover mode = 4 which we are already using). But from this post I get the impression that everything is fixed with the latest 30.x FLARE code releases).

We contacted both VMware and EMC to get some clarification now, and I will keep this post updated with any new information. In the meantime I ask everyone using ESXi 5.0 with EMC CX storage to comment on this post whether you are using VAAI, if you have any issues with it, and what FLARE code you have on the arrays. BTW, you can check what VAAI primitives are in use with your storage LUNs by running the following esxcli command:
   esxcli storage core device vaai status get

Thanks in advance for any helpful comments!

---

Update (2012-05-20): I got some feedback from other users, VMware and EMC on this issue, and used my best Google-Fu to get related information about vSphere 5, VAAI, VMFS-5 and/or EMC CX support. Here are the results (somewhat unsorted):

• Several users use this unsupported setup, but do not have any issues with it (so far).
• pape pointed to some useful VMware KB articles in the comments of this post. Here is an overview of these and some more related articles:
• KB2006858: ESXi 5.0 hosts fail to mount VMFS5 volumes that are formatted with ATS-only capabilities (describes an issue with VMFS-5 volumes and ATS locking seen on EMC arrays, but this is solved with ESXi 5.0 Update 1)
• KB1033665: Disabling the VAAI functionality in ESX/ESXi
• KB2012967: Disabling VAAI functionality for a specific array (Now that would be interesting, but the link doesn't work as of now)
• KB1021976: vStorage APIs for Array Integration FAQ (this FAQ also covers differences in ESXi 4.1 and 5.0 resp. VMFS-3 vs. VMFS-5)
• KB2003813: vSphere 5 FAQ: VMFS-5
• The following VMware blog posts of Cormac Hogan are interesting in the context of VAAI usage with VMFS-3 vs. VMFS-5:
• vSphere 5.0 Storage Features Part 1 - VMFS-5
• vSphere 5.0 Storage Features Part 3 - VAAI
• VMFS Locking Uncovered (About the improvements that were made with file locking during VMFS evolution)

Regarding my specific issue (lack of official support for VAAI with EMC CX arrays and ESXi 5.0) I finally got one really detailed and helpful comment from an EMC representative: EMC tested the VAAI functionality of ESXi 5.0 with their EMC CX4 arrays (I specifically asked for the CX4). It has been "tested as functional", but it only passed the certification tests of the ATS and Zero primitives, but failed the certification test of the XCopy primitive.
I wonder why this happened, because VAAI was fully supported with ESXi 4.1 (i.e. it must have also passed the XCopy test), and with ESXi 5.0 there were no changes to the XCopy primitive (at least I could not find any information about such a change).

Anyway, this is probably the reason why EMC and VMware do not officially support VAAI with ESXi 5.0 on CX arrays. From a customer's point of view this is disappointing, but from EMC's point of view this decision is understandable, because they want to focus on their current products and not waste support resources on somewhat outdated arrays like the CX ones. However, according to this EMC representative it may be possible to come to an individiual support agreement with EMC and VMware (EMC calls this RPQ = Request for Product Qualification) if you nevertheless must or want to have an official support statement for whatever reason.

I also had the idea of disabling only the XCopy primitive to be on the safe side, but I don't want to do this globally, because we also use a fully VAAI supported VNX array with our production hosts. So I was glad to find a link to the promising KB article KB2012967 (see list above), but this link currently doesn't work. I will ask VMware about this ...

---

Update (2012-06-20): In the meantime an EMC representative confirmed to me that all three VAAI primitives work fine with vSphere 5.0 and CX arrays, and that they even support it, but you need to file an RPQ with them in order to get formal support.

We have this configuration (VAAI turned on for both our CX4 and VNX arrays) running with vSphere 5.0 in production for about 6 weeks now without any problems.

The link to VMware KB article KB2012967 still doesn't work, but it is now listed to as being "[Archived]" in the reference section of KB1033665 ...

---

Update (2012-07-11): It turned out that you no longer need an RPQ with EMC. They now officially support the vSphere 5's VAAI implementation on the CX4. The combination is listed in their latest Simple Support Matrix for VMware vSphere 5 (Powerlink account needed for download):

Snippet from the EMC Simple Support Matrix for vSphere 5 (of July 2012)

VMware though does still not officially support it (KB2008822 is unchanged as of today). In another blog post I already explained the reason for this.

QUADStor delivers Storage Virtualization and VAAI - for free!

Are you a VMware ESXi small business (or even home) user and do you enviously look at Enterprise and high budget customers who can afford fancy high performance SAN boxes with storage virtualization features and VAAI support?
Stop being envious and have a look at QUADstor's storage virtualization software. It is currently in beta and offered for free. And it has all the features that are normally only available with high priced enterprise solutions:

• Thin provisioning
• Transparent compression
• Block level deduplication (inline and post-process)
• Support for NAS, iSCSI and even Fibre Channel (with target capable QLogic adapters)
• VAAI support for VMware ESXi hosts with iSCSI and FC

QUADStor architecture

QUADStor adds a layer on top of the traditional Linux or FreeBSD block device layer and transforms a pool of block devices (which could also be software RAID protected volumes) into thin provisioned so called VDisks that are carved out of this pool. Each VDisk can be deduplicated and/or compressed independently from other VDisks.

VDisks can be used in (at least) three different ways. They can be

• locally formatted with a file system (like Linux ext3) and exported as NAS shares via Samba, NFS etc.
• exported as an iSCSI block device via a modified version of IET (iSCSI Enterprise Target)
• exported as a Fibre Channel block device via a modifed version of the target capable QLogic qla2xxx device driver (so this only works with QLogic FC adapters)

If a VDisk is exported as an iSCSI or FC target that is used by VMware ESX(i) hosts then QUADStor also implements VAAI support. VAAI stands for vSphere APIs for Array Integration. It was introduced with vSphere 4.1 and allows to offload certain disk operations to the storage array. It is based on SCSI open standards and basically has these functionalities (also called primitives): ATS (Atomic Test and Set for hardware assisted locking), XCOPY (Full copy to offload file copy tasks) and Block ZEROing (for initializing disk regions with zeros). The most impressive one is XCOPY, because it allows VM clone or Storage VMotion tasks to be completely offloaded to the storage array. Please note that with QUADStor the XCOPY primitive is only supported for ESXi 5.0.
For more information on VAAI please read the vStorage APIs for Array Integration FAQ article in the VMware KB.

Following is a quick guide for setting up the QUADStor software and configuring it as an iSCSI target for VMware ESX(i) hosts:

1. Prerequisites

The QUADStor software is to be installed on an existing FreeBSD or Linux (RHEL/CentOS 5.0 and 6.0 or SLES11 SP1) box that serves as the storage server. I chose an ESXi 4.1 VM with 2 vCPUs and 4 GB RAM that I set up with SLES for VMware. The following software packages need to be installed with the OS: apache2 (used for the administration web GUI), gcc and kernel-default-devel (the compiler and Linux kernel sources are needed for the compilation of various QUADstor kernel modules). In SLES you can install them with the yast administration and setup tool.

2. Installation

For downloading the QUADStor software you need to register at their web site. The download consists of two rpm packages: A Core package and an ITF package. They need to be installed by running the following commands in a shell:

rpm -ivh quadstor-core-xxx.rpm
rpm -ivh quadstor-itf-xxx.rpm

(xxx is the current build number)

The core package contains the software's back-end including a postgresql database and the web administration GUI. The ITF package contains the iSCSI stack (a modified version of IET) and a modified target-capable version of the qla2xxx driver for QLogic Fibre Channel adapters. They need to be compiled to fit to the current kernel of the server by running the command

/quadstor/bin/builditf

Now you are almost ready to go ... Launch a browser to connect to the web administration GUI (it listens on the standard http port):

QUADStor management web GUI

and add the license key that you received when registering in the menu Management / Licensing:

Adding the QUADStor license key

3. Adding and exporting storage

Attach a new blank disk to your QUADStor VM and add it to the QUADStor pool in the Physical Storage / View Storage menu:

Physical disk view

Click on the "Add" link to add the disk, and in the next menu choose whether you want to use compression with this disk:

Enable Compression for physical disk

After pressing "Submit" you are ready to create you first VDisk from the pool in the Virtual Disks / Add VDisk menu:

Add a QUADStor VDisk

Important note: If you want to access the VDisk with VMware ESX(i) hosts you need to check the "512 byte emulation" option, because ESX(i) is not able to handle other block sizes!
In the Virtual Disks / View/Modify menu you can then list, modify and delete existing VDisks:

Configured VDisks menu

Click on the "Modify" link to configure the VDisk settings:

Modify VDisk properties

Each VDisk is automatically exported via iSCSI. If you click on the "iSCSI" link in the Virtual Disks menu you can configure CHAP authentication for the selected VDisk:

Configure iSCSI CHAP authentication

Finally you can view several statistics counters for a VDisk if you click on the "Statistics" link in the Virtual Disks menu:

VDisk statistics

4. Attaching an iSCSI exported VDisk to an ESX(i) host

I used the ESXi software iSCSI initiator to access the QUADStor target. Its setup and configuration is beyond the scope of this article, but there are sources with detailed explanations available, e.g. the VMware KB article Configuring and troubleshooting basic software iSCSI setup.

After you have attached the iSCSI target to the host you can format it as a VMFS datastore, and it will be listed in the Datastores view of the vSphere client:

View of a VAAI enabled QUADStor datastore

The "Supported" in the Hardware Acceleration column denotes that ESXi has detected and is using the VAAI extensions with the QUADStor datastore.

Now you are ready to create VMs on the QUADStor datastore and experiment with the features it offers. If you run into trouble do not hesitate to contact the QUADStor support. It is very responsive and quick in bug fixing!

Conclusion: The QUADStor storage virtualization software is a work in progress, but definitively a great and affordable opportunity to try out advanced storage virtualization features, even in your home lab. Don't hesitate and get started today!