[Release] ESXi5 Community Packaging Tools v1.0

I just released the first version of my ESXi5 Community Packaging Tools. These are two scripts:

• tgz2vib5.cmd (to convert a TGZ file to a VIB file, formerly included with ESXi-Customizer)
• vib2zip.cmd (to combine one or more VIB files into an Offline Bundle ZIP)

These scripts can be used by Community developers (who provide e.g. drivers for unsupported Whitebox hardware) to package their software into VMware proprietary formats.

Go, read anything you want to know about these tools (and the different formats that they handle), and download them on the project page.

QUADStor delivers Storage Virtualization and VAAI - for free!

Are you a VMware ESXi small business (or even home) user and do you enviously look at Enterprise and high budget customers who can afford fancy high performance SAN boxes with storage virtualization features and VAAI support?
Stop being envious and have a look at QUADstor's storage virtualization software. It is currently in beta and offered for free. And it has all the features that are normally only available with high priced enterprise solutions:

• Thin provisioning
• Transparent compression
• Block level deduplication (inline and post-process)
• Support for NAS, iSCSI and even Fibre Channel (with target capable QLogic adapters)
• VAAI support for VMware ESXi hosts with iSCSI and FC

QUADStor architecture

QUADStor adds a layer on top of the traditional Linux or FreeBSD block device layer and transforms a pool of block devices (which could also be software RAID protected volumes) into thin provisioned so called VDisks that are carved out of this pool. Each VDisk can be deduplicated and/or compressed independently from other VDisks.

VDisks can be used in (at least) three different ways. They can be

• locally formatted with a file system (like Linux ext3) and exported as NAS shares via Samba, NFS etc.
• exported as an iSCSI block device via a modified version of IET (iSCSI Enterprise Target)
• exported as a Fibre Channel block device via a modifed version of the target capable QLogic qla2xxx device driver (so this only works with QLogic FC adapters)

If a VDisk is exported as an iSCSI or FC target that is used by VMware ESX(i) hosts then QUADStor also implements VAAI support. VAAI stands for vSphere APIs for Array Integration. It was introduced with vSphere 4.1 and allows to offload certain disk operations to the storage array. It is based on SCSI open standards and basically has these functionalities (also called primitives): ATS (Atomic Test and Set for hardware assisted locking), XCOPY (Full copy to offload file copy tasks) and Block ZEROing (for initializing disk regions with zeros). The most impressive one is XCOPY, because it allows VM clone or Storage VMotion tasks to be completely offloaded to the storage array. Please note that with QUADStor the XCOPY primitive is only supported for ESXi 5.0.
For more information on VAAI please read the vStorage APIs for Array Integration FAQ article in the VMware KB.

Following is a quick guide for setting up the QUADStor software and configuring it as an iSCSI target for VMware ESX(i) hosts:

1. Prerequisites

The QUADStor software is to be installed on an existing FreeBSD or Linux (RHEL/CentOS 5.0 and 6.0 or SLES11 SP1) box that serves as the storage server. I chose an ESXi 4.1 VM with 2 vCPUs and 4 GB RAM that I set up with SLES for VMware. The following software packages need to be installed with the OS: apache2 (used for the administration web GUI), gcc and kernel-default-devel (the compiler and Linux kernel sources are needed for the compilation of various QUADstor kernel modules). In SLES you can install them with the yast administration and setup tool.

2. Installation

For downloading the QUADStor software you need to register at their web site. The download consists of two rpm packages: A Core package and an ITF package. They need to be installed by running the following commands in a shell:

rpm -ivh quadstor-core-xxx.rpm
rpm -ivh quadstor-itf-xxx.rpm

(xxx is the current build number)

The core package contains the software's back-end including a postgresql database and the web administration GUI. The ITF package contains the iSCSI stack (a modified version of IET) and a modified target-capable version of the qla2xxx driver for QLogic Fibre Channel adapters. They need to be compiled to fit to the current kernel of the server by running the command

/quadstor/bin/builditf

Now you are almost ready to go ... Launch a browser to connect to the web administration GUI (it listens on the standard http port):

QUADStor management web GUI

and add the license key that you received when registering in the menu Management / Licensing:

Adding the QUADStor license key

3. Adding and exporting storage

Attach a new blank disk to your QUADStor VM and add it to the QUADStor pool in the Physical Storage / View Storage menu:

Physical disk view

Click on the "Add" link to add the disk, and in the next menu choose whether you want to use compression with this disk:

Enable Compression for physical disk

After pressing "Submit" you are ready to create you first VDisk from the pool in the Virtual Disks / Add VDisk menu:

Add a QUADStor VDisk

Important note: If you want to access the VDisk with VMware ESX(i) hosts you need to check the "512 byte emulation" option, because ESX(i) is not able to handle other block sizes!
In the Virtual Disks / View/Modify menu you can then list, modify and delete existing VDisks:

Configured VDisks menu

Click on the "Modify" link to configure the VDisk settings:

Modify VDisk properties

Each VDisk is automatically exported via iSCSI. If you click on the "iSCSI" link in the Virtual Disks menu you can configure CHAP authentication for the selected VDisk:

Configure iSCSI CHAP authentication

Finally you can view several statistics counters for a VDisk if you click on the "Statistics" link in the Virtual Disks menu:

VDisk statistics

4. Attaching an iSCSI exported VDisk to an ESX(i) host

I used the ESXi software iSCSI initiator to access the QUADStor target. Its setup and configuration is beyond the scope of this article, but there are sources with detailed explanations available, e.g. the VMware KB article Configuring and troubleshooting basic software iSCSI setup.

After you have attached the iSCSI target to the host you can format it as a VMFS datastore, and it will be listed in the Datastores view of the vSphere client:

View of a VAAI enabled QUADStor datastore

The "Supported" in the Hardware Acceleration column denotes that ESXi has detected and is using the VAAI extensions with the QUADStor datastore.

Now you are ready to create VMs on the QUADStor datastore and experiment with the features it offers. If you run into trouble do not hesitate to contact the QUADStor support. It is very responsive and quick in bug fixing!

Conclusion: The QUADStor storage virtualization software is a work in progress, but definitively a great and affordable opportunity to try out advanced storage virtualization features, even in your home lab. Don't hesitate and get started today!

Feature request: Add sudo to ESXi to make AD integration a success story

Recently I posted about (undocumented) improvements in the area of AD integration, but it looks like I missed a very important point:

You can log on to a local or remote console using an AD account that has administrative rights, but you won't have root privileges in this session, e.g. you cannot edit any configuration files, restart services etc. To gain root rights you need to use the su command, but that means that you still need to know and enter the password of the root user! From a compliance standpoint this is not acceptable, because the whole point of AD integration is that each VMware administrator uses his AD account for administration and does not even know the root password - to make sure that each change to the system can easily be related to a personal account (Well, for emergency cases e.g. when AD authentication is not available you still need someone who knows the root password or e.g. has it written down on a piece of paper in a sealed envelope).

The easiest way to achieve this would be to use the sudo command in the ESXi shell to run commands in root context without the need to know root's password. This is common practice when managing Unix/Linux servers. Now the point is: sudo used to be available in ESX, but it is not available in ESXi.

So I have a simple feature request for VMware: Add sudo to ESXi! It is the missing piece that would make AD integration a success story, finally.

If you agree and also feel bothered by this, please vote for this feature request in the VMware Community forums, where I opened this thread for that. Thank you all for voting/commenting and special thanks to Masa who brought this to my attention in the comments of my above mentioned post!

How to use ESXi 5 as an NTP server - OR - How to permanently add custom firewall rules?

Recently my attention was caught by a question posted to the VMware Community forums that sounds odd at first sight: Is it possible to configure ESXi 5.0 to act as a NTP server?

I wondered why should you try to do this? On the one hand it is not recommended to use ESXi for anything else than the task that it was designed for: being a hypervisor. On the other hand it is not recommended to run a VM as NTP server, because exact timekeeping can be quite a challenge in VMs as they do not own a real hardware clock timer. So, should you run a physical box just for NTP? Small shops that have reached 100% virtualization run only ESXi on their remaining physical servers. So I can understand people considering an exception and wanting to run an ESXi host as NTP server - it is a very lightweight service anyway ...

Now back to the question ..., and the answer is: Yes, it is. In fact it is very easy to do this, because once you have configured ESXi 5.0 to act as a NTP client it will also automatically act as a NTP server! The NTP daemon (/sbin/ntpd) does both at the same time, and its configuration file (/etc/ntp.conf) even allows any other machine to query it by default. There is only one hurdle: the ESXi 5.0 firewall.
By default it blocks the port for incoming NTP queries (UDP port 123). We need to create a custom firewall extension to open that port. KB2005304 explains how to do this. Basically you need to create a custom XML configuration file in the directory /etc/vmware/firewall, e.g. /etc/vmware/firewall/ntpd.xml with the following contents:

<!-- Firewall configuration information for NTP Daemon -->
<ConfigRoot>
  <service>
      <id>NTP Daemon</id>
      <rule id='0000'>
          <direction>inbound</direction>
          <protocol>udp</protocol>
          <porttype>dst</porttype>
          <port>123</port>
      </rule>
      <enabled>false</enabled>
      <required>false</required>
  </service>
</ConfigRoot>

(Take care when you copy or modify this: The XML tags are case sensitive!)

Then load the new configuration by running the following command inside a ESXi shell:
  esxcli network firewall refresh

After that you can see the custom firewall rule in the firewall properties dialog of the vSphere client:

Custom "NTP Daemon" firewall rule

 Enable the rule, and you are done ...
... until the next reboot of the host, because User defined xml firewall configurations are not persistent across ESXi host reboots. The KB article that describes this problem also includes a work-around to resolve it: Put the XML file on a shared datastore and modify the /etc/rc.local boot script to copy the file to the correct location on every reboot.

This works, but I personally consider this an ugly hack, because this modification is not inherent in the system but introduces a dependency to an external resource (the datastore). So I created a VIB file that you can effectively install on ESXi and that will permanently add the XML file to the system.
Run the following commands inside an ESXi shell to install the VIB file:

   esxcli software acceptance set --level CommunitySupported
   esxcli software vib install -v http://files.v-front.de/fwenable-ntpd-1.2.0.x86_64.vib

The first command  is needed for ESXi to accept the custom VIB, because it does not include a trusted signature file. The second command will download and install the VIB file (Note: you can also download the file with a browser, store it on a local datastore and reference the local file with the install command).

The installation will not require the host to be in maintenance mode and it will be immediately effective without the need to reboot the host! It will also automatically reload the firewall rules, so the only step left is to enable the rule in the vSphere client.

By the way, I created this VIB file with a new and improved version of my TGZ2VIB5 script that I currently work on. Once I have finished this new version and made it available here I will also post a detailed description of how I created the VIB file.

Top VMware and Virtualization Blog voting 2012 now open

Just a short note: Eric Siebert has just opened this year's voting for the Top VMware and Virtualization Blogs. This blog is listed in the categories "Independent Blogger" and "New Blog" (and among "all" of course). Make yourself heard and vote here!

Hard to find HP tools: The Offline Array Configuration Utility (ACU)

If you have ever looked for a solution to a specific problem or the download page for a certain tool on www.hp.com then you probably know: Searching (and finding) something is a pain on these pages, and the more desperate you need it the longer it will take you ...
So maybe I will even make a series of "Hard to find HP tools" posts. Anyway I will start with the Offline ACU tool today.

So, what do you need this tool for? I had this challenge before and I reminded that when I came across this VMware Community forums post: Imagine you have an HP based ESXi host with VMs running on local disks attached to a Smart Array RAID Controller. You have run out of disk space and decide to add an additional hard disk to the server. Instead of creating a new (unprotected) RAID volume on this single disk you prefer to expand an existing RAID volume with it. This will give you more disk space and keep the current RAID protection level. How do you do that?
No problem, if you had Windows (or Linux) running directly on the box, because HP made available the Array Configuration Utility (ACU) for these operating systems. It will allow you to do the RAID expansion online while the OS is running. However, for ESXi this tool is not available as an online version.
This is why you need to use the Offline ACU tool. This is just a bootable CD with Linux and the Linux ACU tool on it. So, you need to schedule a downtime for the host (and the VMs running on it) and reboot with that CD to make the required changes to your RAID volumes. Not online, but better than nothing ...

You can find the download link to the current version of the HP ProLiant Offline Array Configuration Utility on my HP & VMware links page (in the General section).

Once you have successfully expanded your RAID volume (and booted into ESXi again) you just need to do the same with the VMFS datastore that resides on it. Please note that since vSphere 4.0 you can grow a VMFS datastore online, and you do not need to use VMFS extents. Choose "Increase..." from the datastore's properties menu:

Undocumented parameters for ESXi 5.0 Active Directory integration

Since vSphere version 4.1 it is possible to integrate an ESXi host into a Microsoft Active Directory (AD). After the host is joined to the domain you can assign permissions to AD groups and users by connecting directly to the host with the vSphere client.
Instructions on how to do this (with ESXi 5.0) is available e.g. here in the VMware Online Documentation.

I first looked at AD integration when vSphere 4.1 was released and found one really annoying drawback in it that ruled it out from a possible implementation in our environment: When an ESXi 4.1 host is joined to a domain it will automatically (and repeatedly!) look up an AD group called "ESX Admins", and as soon as it finds this group it will grant this group Administrator permissions on the ESXi host. The real problem here is that the name ("ESX Admins") of this AD group is hard coded and can not be configured.
This may be a nice feature for small environments - you just need to create this group, fill in the necessary people and you are done. But if you think about an enterprise environment of a large company with lots of different sites, IT teams and vSphere installations, but only one Active Directory, you can not assume that all ESXi hosts in this company are managed by the same group of people.

When vSphere 5.0 was released I looked at the release notes and documentation to find out if this drawback was removed, but I did not find any positive information. Tests I did also showed that an ESXi 5.0 host behaves the same way, looks up the "ESX Admins" group and adds it with Administrator permissions.

However, recently I stumbled over the following when browsing the advanced configuration parameters of an ESXi 5.0 host:

Configuring the "ESX Admins" group

Yes, with ESXi 5.0 it is possible to change the name of the AD group that is automatically added by setting the advanced configuration option Config.HostAgent.plugins.hostsvc.esxAdminsGroup. You can even completely disable this functionality by setting the option  Config.HostAgent.plugins.hostsvc.esxAdminsGroupAutoAdd to false.
I searched for this again in the VMware documentation and the Knowledge Base, but did not find it being mentioned anywhere. So it looks like at the time this is completely undocumented, but it works as expected (I could not resist from immediately trying this out)!