Showing posts with label HP. Show all posts
Showing posts with label HP. Show all posts

Friday, March 29, 2013

How to prevent HP SIM from scanning the VMotion port

If you run your ESXi hosts on HP hardware and have a look at the vmkernel.log files from time to time then you have probably already stumbled over messages like these:
vmkernel: cpu4:2741)MigrateNet: vm 2741: 1982: Accepted connection from <x.x.x.x>
vmkernel: cpu4:2741)MigrateNet: vm 2741: 2052: dataSocket 0x41002750c600 receive buffer size is 563560
vmkernel: cpu4:2741)WARNING: Migrate: 215: Invalid message type for new connection: 542393671. Expecting message of type INIT (0).
Instead of x.x.x.x you will see an IP address that might be familiar to you: It belongs to a server that has the HP System Insight Manager (SIM) software installed for inventorying / managing / monitoring hardware devices in your network. This software is commonly used in HP shops, because it does a fairly good job with monitoring the hardware health of your servers and alerting if something is wrong, and this basic functionality is for free.

Read more »

Wednesday, February 20, 2013

HP releases Service Pack 2013.02.0 for ProLiant and updated software for ESXi 5.x

Yesterday HP released version 2013.02.0 of their Service Pack for ProLiant (SPP). This is a (bootable) ISO file that contains the latest firmware and driver packages for ProLiant servers. What's new:
  • Added online deployment of a select set of firmware components to systems running VMware ESXi 5.x (that means you can now deploy Smart Components through HP SUM!)
  • Support for new hardware devices and OS versions
  • Support for PXE booting the SPP ISO
  • Updated HP Smart Update Manager (SUM) to version 5.3.5
Additional information:
On http://vibsdepot.hp.com/ there are updated Offline bundles and drivers for VMware ESXi 5.x available (see the feb2013 directory for downloads).

The latest versions of the HP customized installation ISOs are available for ESXi 5.0 U2 and ESXi 5.1. As usual they do not include the latest ESXi patch level. For instructions on how to build a custom ISO with the latest VMware patch level and the latest HP stuff please see this blog post.

I have updated my HP & VMware Links page to include the new releases.


Monday, December 10, 2012

Finally...: HP Online Firmware Updates for ESXi

If you have VMware ESXi running on an HP ProLiant server then you had to take it down for updating the firmware of its hardware components and boot it with the Service Pack for ProLiant Boot CD ... until now!

Read more »

Monday, October 29, 2012

HP releases Service Pack 2012.10.0 for ProLiant and updated drivers for ESXi 5.0 und 5.1

HP has released version 2012.10.0 of their Service Pack for ProLiant (SPP). This is a (bootable) ISO file that contains the latest firmware and driver packages for ProLiant servers. What's new:
Read more »

Tuesday, September 11, 2012

vSphere 5.1 is generally available (incl. the free and the HP customized version of ESXi)

Late yesterday VMware made vSphere 5.1 generally available. Go to
   http://www.vmware.com/go/download-vsphere
to download. The free version of ESXi 5.1 is also already available:
   http://www.vmware.com/go/get-free-esxi

And HP hardware owners will be happy to find that there is also an HP customized version of the ESXi 5.1 ISO available for download. The shortcut http://www.vmware.com/go/downloads-image-hp-esxi still points to the 5.0 version, but HP's ESXi 5.1 GA is available here:
   https://my.vmware.com/web/vmware/details?downloadGroup=HP-ESXI-5.1.0-GA-10SEP2012&productId=285

Right now it is not yet possible to update an existing ESXi 5.0 installation to 5.1 using the well known methods (via Update Manager and the VMware Online Depot). You will probably be able to update by booting the host with the ESXi 5.1 ISO, but the other methods should also be available shortly.


Wednesday, September 5, 2012

HP releases new Service Pack for ProLiant and new drivers for ESXi 5.0 und 5.1

HP has released version 2012.08.0 of their Service Pack for ProLiant (SPP). This is a (bootable) ISO file that contains the latest firmware and driver packages for ProLiant servers. What's new:
Read more »

Tuesday, September 4, 2012

Sep2012 release of the HP Customized ESXi 5.0 ISO

HP has just published updated software packages and device drivers for VMware ESXi 5.0 (at http://vibsdepot.hp.com). With this Sep2012 release HP changes the way how they provide this software: Up to now they have made a complete customized installation ISO (including all the packages) available for download on their own web site. Now they have replaced this download link with a shortcut URL that redirects to a VMware download page: http://www.vmware.com/go/downloads-image-hp-esxi.
Read more »

Friday, August 10, 2012

How to simplify and automate VMware ImageBuilder with the ESXi-Customizer-PS script

I published a new and improved version of my ESXi-Customizer-PS script today. This PowerCLI script greatly simplifies and automates the process of creating fully patched and customized ESXi 5.x installation ISOs using the VMware PowerCLI ImageBuilder snapin.

Over time I added more and more options to the script, and I think it is time now to provide a detailed tutorial to show how to make efficient use of it. Grab the latest version (1.3) from my Google Code page and give it a try. The only prerequisite you need is an installation of the current VMware vSphere PowerCLI.
Read more »

Wednesday, June 27, 2012

[Update] ESXi-Customizer Powershell script version 1.2

Following up on some user feedback I updated my ESXi-Customizer-PS (Powershell) script to version 1.2. I introduced this script in the last part of my ImageBuilder Deep Dive series (see also part 1 and 2). This version adds advanced features by providing the following new optional command line parameters:
  • -ipname  : provide a name for the customized ImageProfile (the default is derived from the cloned VMware ImageProfile)
  • -sip : manually select an ImageProfile from the current list (default = auto-select latest available ImageProfile)
  • -hprel <mon>  : select HP packages from release dated <mon> (e.g. jun2012) (default = select latest available HP packages)
Please download the latest version of the script from my Google Code page.

Use the command line parameter -help for complete usage instructions:

ESXi-Customizer-PS v1.2 help screen


Thursday, June 7, 2012

[Update] ESXi-Customizer Powershell script version 1.1

With the last part of my ImageBuilder Deep Dive series (also don't miss part 1 and 2) I published a Powershell script to build a customized ESXi 5.0 installation ISO, optionally adding the contents of the HP Online VIBs depot.

HP has recently updated the contents of their Online depots adding the latest releases of their bundles (see this post). I tested the first version of my script and found that it would incorrectly add older versions of the HP packages. So, I have fixed it now to add only the newest version of each available package. Other changes:
  • Skip the hp-smx-limited package. It is not included in HP's customized images and conflicts with the full CIM provider package hp-smx-provider. In fact it is only useful for the new ProLiant Gen8 servers if you want to monitor these through the Agentless Management Service (AMS) only.
  • Added a new command line switch -test. It will skip the actual package download and image building - useful for testing the script and seeing what it would add without wasting any download bandwidth.
The new version 1.1 is available for download now.

ESXi-Customizer-PS v1.1 help screen


Monday, June 4, 2012

HP ProLiant Firmware and Driver updates

Today HP released a new Service Pack for ProLiant (SPP): Version 2012.06.0(B). The SPP is a DVD image that includes the latest firmwares and drivers for all ProLiant servers (and associated components like Blade enclosures, iLO boards, NICs, CNAs and Smart Array controllers). Also included is the HP Smart Update Manager (updated to version 5.1.0) that you can use either on a Windows OS or by booting a ProLiant server off the DVD into Linux.

At the same time HP also published updates for their customized ESXi (4.1 U2 and 5.0 U1) installation CDs that include updated Offline bundles and updated drivers (both are also separately available).

Last but not least a new Virtual Connect firmware (version 3.60) hits the street today.

You can find download links to all the mentioned components on my HP & VMware Links page.


Sunday, May 13, 2012

What's new with HP ProLiant Gen8 servers

I recently visited an HP roadshow presentation of their new ProLiant server generation Gen8. This has been around for a while, and you might already have seen some announcements, slides etc. Sometimes it can be hard to really learn about what is actually innovative, new and better (compared to older generations) with Gen8, because of all the buzz words and marketing going on about initiatives like "Voyager", "ProActive Insight Architecture" and so on. So I took some notes about "the hard facts" and decided to turn them into this blog post.

According to HP Gen8 is basically about reducing the following three cost factors:

  • Manual actions
  • Energy costs
  • Unplanned downtimes

Before I focus on these three items, let's quickly summarize what hardware changes come with Gen8:

  • Support for Intel Sandy Bridge EP (Xeon E5-2600) processors. These new CPUs have
    • the I/O hub integrated on the CPU die
    • four (formerly three) memory channels per CPU socket
    • two (formerly one) Quick Path Interlinks (QPIs) per socket
    • up to 8 CPU cores
    • Support for PCIe 3.0 (formerly 2.0)
  • The new iLO generation 4
    • incorporating a new 4GB flash RAM (for storing drivers, firmwares etc.)
    • removing the necessity to deploy hardware management agents into the OS (everything is monitored by and through iLO)
    • introducing a mobile app (for Android and iOS) for remote management
  • New SmartArray controllers (22x and 42x models) with FBWC (= Flash Backed Write Cache) only (no more BBWC = Battery Backed Write Cache)

Reducing manual actions

Together with Gen8 HP introduced a new version (5.0) of their Smart Update Manager (SUM). It was completely rewritten to be more intelligent and shall be able to completely automate the update process of complex hardware setups with multiple dependencies. If you have ever planned and executed firmware updates for HP Blade enclosures then you know that this was a very time consuming and error prone process so far with lots of manual tasks. HP SUM 5.0 now promises to make it a no-brainer and a streamlined, completely automated experience without the requirement of planned downtimes. I am very curious to try this out ...

At the same time HP announced a new support policy for their Service Packs for ProLiant. These are bootable ISOs containing certified sets of drivers and firmwares for ProLiant hardware and the new HP SUM tool. They are the successors of the Firmware maintenance CDs/DVDs that you may already know. HP plans to release a new Service Pack every four months and will support at least three consecutive releases - that means you may update all your firmwares only every 12 months while keeping your environment fully supported by HP. Of course there will be hotfixes between releases to address urgent issues. You can find the relevant links on my HP & VMware links page.

Reducing Energy costs and unplanned downtimes

HP has greatly increased the number of hardware sensors in the new ProLiants, they are distributed all over the chassis on multiple levels forming a "3D sea of sensors". With "HP ActiveHealth" (another great marketing term) there is also a new system for detecting and pro-actively mitigating hardware failures. This intelligence is part of the new iLO4 board that will also store a complete history of more than 1.600 system parameters and measured values. This data can be exported and uploaded to HP SnS for hardware analysis and troubleshooting.

They also claim to have made their power supplies even more efficient: The best efficiency value is still 94%, but it is now kept over a broader range of output powers.

Availability and next steps

At the time there are Gen8 version of the two socket models ML350p, DL160, DL360p and DL380p, BL460c, SL230s and SL250s. Although one of the main drivers for the new platform is the new Intel CPU architecture there will also be Gen8 models of the AMD based ProLiants available later this year.

One last note: If you already have gotten your hands on a new Gen8 ProLiant and want to install VMware vSphere on it then be sure that you use the HP Customized ESXi ISOs. The VMware vanilla ISOs do not yet include the required hardware drivers.


Sunday, March 25, 2012

HP VMware ESXi 5.0 U1 Customized Image

While searching for an updated HP Customized ESXi 5.0 ISO that would include Update 1 I noticed that the HP download page for ESXi still shows only the October release (ESXi 5.0 GA) as of today, March 25th 2012.
However, I stumbled over a document titled HP VMware ESXi 5.0 U1 Customized Image Release Notes for March 2012 that is apparently available since March 16th. That means that the release of this customized image is imminent. Here are some highlights from the release notes:


I will update this post once the new HP Customized ESXi 5.0 U1 Image is finally published.

Update (2012-03-27):  The HP VMware ESXi 5.0 U1 Customized Image is now available for download!

Saturday, March 10, 2012

How to run the HP Online ACU CLI for Linux in ESXi 4.x

Please note: I noticed that this post still gets a lot of attention... Before reading through the whole post you should know: the workaround described here is only necessary for ESXi 4.x. For ESXi 5.x there is an officially supported Online ACU CLI available from HP (see the update at the end of this post)!
A while ago I posted about the Offline version of HP's Array Configuration Utility (ACU). HP does not provide an Online version of this tool for VMware ESXi, so expanding a local RAID array can only be done with this Offline version which requires a downtime of the host for the whole duration of the expansion.

Somehow I did not want to accept this, and - by just trying it out - I finally discovered that you can actually run the Linux version of the ACU CLI tool (hpacucli) in ESXi! hpacucli is not the web interface version of the ACU (which is called cpqacuxe) that you may be used to from Windows systems. cpqacuxe is also available in a Linux version, but I could not get it to run in the ESXi shell.
hpacucli is surely less intuitive to use, but it offers the same functionality than the web tool and allows you to do all controller configuration and volume expansion tasks that are supported with SmartArray controllers. Before trying it be sure to look at the Configuring Arrays on HP Smart Array Controllers Reference Guide (section "Using the ACU CLI", p51 f.) to learn how to use it. The reference guide also describes another utility named hpacuscripting that is also included in this package and works with ESXi. I haven't tried that, but it looks like it is mainly meant for the initial RAID configuration on a newly deployed host.

The Linux version of hpacucli requires some script modifications to make it run in the ESXi shell. First I thought about providing a modified version of it that you could just install and run in ESXi, but then I noticed that HP's license terms do not allow to redistribute the software package. So I will provide step-by-step instructions instead for how to do the required modifications on your own using a Windows machine. Don't worry, it's easy:

1. Download the file hpacucli-8.75-12.0.noarch.rpm from HP.

2. Download 7-zip and install it.

3. Open the downloaded rpm-file in 7-zip:


4. Inside the 7-zip window navigate to the embedded directory \opt\compaq\hpacucli\bld by double-clicking on the displayed file and directory names:


5. Select all the displayed files and click on the "Extract" or "Copy" button to extract them to a new empty sub-directory (U:\$Download\hpacucli in this example):


6. Download the patch script fix4esxi.sh that I prepared and put it into the same directory.

7. Now use the vSphere Client to upload the complete directory to a datastore that is accessible by the host that you want to run the tool on. You can store the directory on a shared datastore to make the same copy accessible to multiple hosts:


8. Now open a shell as root in the ESXi host that you want to run the tool on. Change to the datastore directory that you uploaded and run the patch script like this:
. ./fix4esxi.sh


You need to run the script only once! It will patch the shell scripts that are included and restore the executable permission bits that got lost when we extracted the files in Windows.

9. Now you are ready to run the hpacucli utility in ESXi, either by running ./hpacucli after changing to the installation directory or by calling it by its full file path:


The command controller all show config will immediately show you if the utility has detected your SmartArray controller and display its configuration. With the help command you will get an overview of all available commands and how to use them. However, I strongly recommend reading the above mentioned reference guide, because it includes more detailed information and usage examples.

Now at the end ... a big fat warning and some more hints: Running hpacucli inside ESXi is (of course) not supported by HP or VMware! I can tell you that it just works for me in the configurations that I was able to test, but it could as well fail completely for you or blow up your host!

The tool was able to detect SmartArray P400 and P410i controllers with ESXi 4.1 and 5.0. On an ESXi 5.0 host with a P400 controller I was able to break a RAID1 mirror and concatenate the disks to a RAID0 volume of double size. However, ESXi would not pick up the changed disk size, so I was not able to grow the VMFS volume without rebooting the host. At least I only needed to reboot (causing a very short downtime) and did not have to take the host offline for the whole volume conversion process (which can take very long depending on hard disk sizes).
I wonder if ESXi not detecting the disk size change is due to the old cciss driver that the P400 controller is using. Other SmartArray controllers use the newer hpsa driver that might not show this issue.

Please provide feedback by commenting on this post if you manage to successfully make use of hpacucli in ESXi like described here! This will help to get me and others an overview of what works and what not.

Update (2012-04-18): In the meantime HP has officially made available hpacucli for ESXi. It is part of the HP ESXi Utilities Offline Bundle for VMware ESXi 5.0 that I reference on my HP & VMware Links page. For earlier versions of ESXi you still depend on the workaround that I described here.

Wednesday, March 7, 2012

How to subscribe to HP Software Alerts

I often complain about the hp.com web pages and how challenging it is to find the right information there. And this is a pity, because there really is quite a lot of important and useful information available here, e.g. regarding BIOS, firmware and driver updates for the HP hardware that you are using and advisories about known problems and how to avoid or fix them.
So it is good to know that you can actually subscribe to this information and have e-mails sent to you whenever there are important updates or alerts for your hardware.

Click on the Subscriber's choice for business log in page to register for and configure this service. If you already have an account with HP you can directly sign in with it, otherwise click on the "Register" button and create a new account (It's free and just requires a valid email address).
Once you are logged in you can select the products that you want to receive alerts for by clicking on the "Customize" link for Driver and Support alerts:

Subscribe to HP alerts
I recommend that you carefully think about what HP hardware components you use and want to add here. If e.g. you are using c7000 blade enclosures with BL620c G7 servers then add every single component that has its own firmware and/or drivers: the BL620s, but also their iLO boards and CNAs, the enclosure, Onboard Administrator and Virtual Connect modules (see picture). Add this list for any type of HP server model you have.

In the next steps you choose how often and in which format (HTML or text e-mail or RSS feed) you want to receive the alerts and what operating systems you are interested in.

Submit you registration and you will soon receive e-mail alerts like this one:

HP Alerts example e-mail
Highly recommended to pro-actively inform yourself about possible hardware problems (and there is a plenty of them)!

Wednesday, January 18, 2012

Hard to find HP tools: The Offline Array Configuration Utility (ACU)

If you have ever looked for a solution to a specific problem or the download page for a certain tool on www.hp.com then you probably know: Searching (and finding) something is a pain on these pages, and the more desperate you need it the longer it will take you ...
So maybe I will even make a series of "Hard to find HP tools" posts. Anyway I will start with the Offline ACU tool today.

So, what do you need this tool for? I had this challenge before and I reminded that when I came across this VMware Community forums post: Imagine you have an HP based ESXi host with VMs running on local disks attached to a Smart Array RAID Controller. You have run out of disk space and decide to add an additional hard disk to the server. Instead of creating a new (unprotected) RAID volume on this single disk you prefer to expand an existing RAID volume with it. This will give you more disk space and keep the current RAID protection level. How do you do that?
No problem, if you had Windows (or Linux) running directly on the box, because HP made available the Array Configuration Utility (ACU) for these operating systems. It will allow you to do the RAID expansion online while the OS is running. However, for ESXi this tool is not available as an online version.
This is why you need to use the Offline ACU tool. This is just a bootable CD with Linux and the Linux ACU tool on it. So, you need to schedule a downtime for the host (and the VMs running on it) and reboot with that CD to make the required changes to your RAID volumes. Not online, but better than nothing ...

You can find the download link to the current version of the HP ProLiant Offline Array Configuration Utility on my HP & VMware links page (in the General section).

Once you have successfully expanded your RAID volume (and booted into ESXi again) you just need to do the same with the VMFS datastore that resides on it. Please note that since vSphere 4.0 you can grow a VMFS datastore online, and you do not need to use VMFS extents. Choose "Increase..." from the datastore's properties menu:



Wednesday, December 21, 2011

How to do an Online Virtual Connect firmware upgrade

Okay, this is a follow-up to my previous post ... I was finally able to find out on my own how to do this. The answer is in HP's white paper "HP Virtual Connect Firmware Upgrade Steps and Procedures". This is a must read for anyone being concerned with the VC firmware upgrade process, I will try to summarize the most important points here.

You must use the Virtual Connect Support Utility (VCSU). The current version is 1.60 and is available for download here.

It helps to understand how the VCSU does the upgrade: First it uploads the new firmware to all VC modules simultaneously. This phase is absolutely uncritical, because the VC modules continue working normally during the upload. If you use the default parameters it will then activate the new firmware by rebooting the VC modules one after the other in a controlled manner - and this is the process that really impacts the network availability of your hosts and VMs!
Why? The controlled reboot takes 20 or more seconds, and - of course - the VC module will not properly forward and receive network traffic during that time. However, the blade servers, resp. their NICs that are connected to this module are not properly disconnected during that time, i.e. they do not get a link down notification! If you use the default failover detection method for your virtual switches (Link state only) the hosts will continue using the up-links to the module that is just rebooting, and this results in a loss of network connectivity.

So, how do you cope with that? One possible work around is to use Beacon probing as the failover detection method for the virtual switches. But in my opinion this is not the best and easiest choice. No, the real answer is on page 13 of the white paper:
"For the customer environments where changing Network Failover Detection options or HA settings is not possible, utilizing VCSU manual firmware activation order (-of manual) is recommended. In this case, modules will be updated but not activated and the user will need to perform manual activation by resetting (rebooting) modules via OA GUI or CLI interface. This option will eliminate potential of up to 20 sec network outage that may occur on a graceful shutdown of VC Ethernet and FlexFabric modules."
Using the manual activation order (parameters "-oe manual" and "-of manual") ensures that the VCSU will not gracefully reboot the VC modules at all. You then need to do that on your own (just manual), by resetting the VC modules through the Onboard Administrator (OA). When you do a hard reset of a VC module the connected hosts will immediately get link down notifications, just as if the module suddenly fails or loses all its own up-links because the external switch failed. You should just wait about 5 minutes for the resetted module to get fully online before you reset the second one.

If your ESX(i) hosts are properly and redundantly configured you will notice only a minimal network interruption during this process. In my test it was just a single ping drop.

Yes, that's the whole secret of doing an online VC firmware upgrade! For me only one questions remains: Why is HP making it so hard to find this information? If you search hp.com for instructions on how to do this you will find tons of useless and contradicting information on this topic, and even their own Support engineers are not able to give a quick and right answer to the question. At least, one of them sent me a copy of the white paper (he could not just provide a link to it, because he was not able to find it on the HP pages...).

Thursday, October 27, 2011

Update: ESXi 5.0 on HP G7 blades, now a Go!

About three weeks back I reported on Emulex firmware problems that prevented the use of ESXi 5.0 on HP G7 blade hardware. This was fixed now, somehow...

HP has now updated the advisory that describes the issue and published an updated firmware that fixes the VLAN handling problems with ESXi 5.0 if it is used together with the be2net driver 4.0.355.1.

Be sure that you read the release notes of the firmware! It looks like it is an emergency/workaround release that leaves many issues unresolved. A firmware version that you can really trust for production will probably be available mid-November.

Update (2012-12-09): HP and Emulex published the final version of the OneConnect firmware (4.0.360.15a) on Nov 19th. VMware's KB2007397 also lists the recommended drivers to use with this firmware for both ESXi 4.1 and 5.0.

Update (2012-03-09): HP has published yet another firmware update on March 5th. Download version 4.0.360.15b. The previous link has become invalid.

Update (2012-04-16): Please refer to my HP & VMware links page to find the download for the latest version of the firmware.

Wednesday, October 12, 2011

HP Virtual Connect profile not applied ...

When I recently rebooted one of our BL620c G7 blades with ESXi 4.1 installed on it I found that the server had suddenly lost network connectivity after the reboot.
A quick check on the console revealed that the Virtual Connect profile that was defined for that blade had not been applied to it. I realized that because the MAC addresses of the NICs had not been overwritten with the virtual addresses of the Virtual Connect profile.

I tried powering down and up the blade, re-assigning the Virtual Connect profile multiple times, all to no avail ... Then I had the idea that it might be related to the iLO-board of the blade, and - yes, indeed - after resetting the iLO3-board of the blade the Virtual Connect profile was properly applied and all was fine again.

While later looking on hp.com for some related information I stumbled over the Customer advisory c02820591 that described an issue with the Virtual Connect profile being lost upon an iLO3 reset. Not exactly the issue that I had, and the advisory also stated that this is fixed with iLO3 firmware version 1.20, and that is already installed on our iLOs. However, the advisory confirmed my assumption that the Virtual Connect profile is applied by the iLO-board.

So, if you have similar problems try resetting the iLO-board before you start pulling your hair out, or the blade out of the chassis ...

Saturday, October 1, 2011

Currently a No-Go: ESXi 5.0 on HP G7 blades

Back in May I reported on problems with ESXi 4.1 and the Emulex OneConnect CNA that is built into HP's G7 blade servers.
If you now try to install ESXi 5.0 on such a hardware you will have a strong déjà vu: The be2net driver that is available right now for ESXi 5.0 is not really functioning due to "VLAN tagging issues". HP has published an advisory on this stating that an updated driver (that should fix these issues) is "currently in the certification process" and will be made available in "Q4 2011".

Okay, I won't update our production hosts to ESXi 5.0 that soon anyway, but I just wanted to install it on some spare blades for testing and evaluation. Too bad ... waiting for a fix again ...

Update (2011-10-27):
HP has now updated the advisory and published an updated firmware that fixes the VLAN handling problems with ESXi 5.0 if it is used together with the be2net driver 4.0.355.1.
Be sure that you read the release notes of the firmware! It looks like it is an emergency/workaround release that leaves many issues unresolved. A firmware version that you can really trust for production will probably be available mid-November.

Update (2012-04-16):
In the meantime it looks like all problems have been fixed with newer firmware and driver versions. Please refer to this newer post of mine!