How to prevent HP SIM from scanning the VMotion port

If you run your ESXi hosts on HP hardware and have a look at the vmkernel.log files from time to time then you have probably already stumbled over messages like these:

vmkernel: cpu4:2741)MigrateNet: vm 2741: 1982: Accepted connection from <x.x.x.x>
vmkernel: cpu4:2741)MigrateNet: vm 2741: 2052: dataSocket 0x41002750c600 receive buffer size is 563560
vmkernel: cpu4:2741)WARNING: Migrate: 215: Invalid message type for new connection: 542393671.  Expecting message of type INIT (0).

Instead of x.x.x.x you will see an IP address that might be familiar to you: It belongs to a server that has the HP System Insight Manager (SIM) software installed for inventorying / managing / monitoring hardware devices in your network. This software is commonly used in HP shops, because it does a fairly good job with monitoring the hardware health of your servers and alerting if something is wrong, and this basic functionality is for free.

Read more »

A Daemon's VIB - Part 2 (Building a software package for VMware ESXi)

Part 2: Service configuration and startup

This is the second part of the "Daemon's VIB" series about building a software package for ESXi, using the example of the ProFTPD Offline Bundle that I recently released. In the first part I explained how to get (or create yourself) suitable binaries for ESXi. In this part I will focus on what is necessary
   a) to make the service start up automatically on system boot and
   b) to make it fully manageable via the vSphere Client.
Read more »

How to use ESXi 5 as an NTP server - OR - How to permanently add custom firewall rules?

Recently my attention was caught by a question posted to the VMware Community forums that sounds odd at first sight: Is it possible to configure ESXi 5.0 to act as a NTP server?

I wondered why should you try to do this? On the one hand it is not recommended to use ESXi for anything else than the task that it was designed for: being a hypervisor. On the other hand it is not recommended to run a VM as NTP server, because exact timekeeping can be quite a challenge in VMs as they do not own a real hardware clock timer. So, should you run a physical box just for NTP? Small shops that have reached 100% virtualization run only ESXi on their remaining physical servers. So I can understand people considering an exception and wanting to run an ESXi host as NTP server - it is a very lightweight service anyway ...

Now back to the question ..., and the answer is: Yes, it is. In fact it is very easy to do this, because once you have configured ESXi 5.0 to act as a NTP client it will also automatically act as a NTP server! The NTP daemon (/sbin/ntpd) does both at the same time, and its configuration file (/etc/ntp.conf) even allows any other machine to query it by default. There is only one hurdle: the ESXi 5.0 firewall.
By default it blocks the port for incoming NTP queries (UDP port 123). We need to create a custom firewall extension to open that port. KB2005304 explains how to do this. Basically you need to create a custom XML configuration file in the directory /etc/vmware/firewall, e.g. /etc/vmware/firewall/ntpd.xml with the following contents:

<!-- Firewall configuration information for NTP Daemon -->
<ConfigRoot>
  <service>
      <id>NTP Daemon</id>
      <rule id='0000'>
          <direction>inbound</direction>
          <protocol>udp</protocol>
          <porttype>dst</porttype>
          <port>123</port>
      </rule>
      <enabled>false</enabled>
      <required>false</required>
  </service>
</ConfigRoot>

(Take care when you copy or modify this: The XML tags are case sensitive!)

Then load the new configuration by running the following command inside a ESXi shell:
  esxcli network firewall refresh

After that you can see the custom firewall rule in the firewall properties dialog of the vSphere client:

Custom "NTP Daemon" firewall rule

 Enable the rule, and you are done ...
... until the next reboot of the host, because User defined xml firewall configurations are not persistent across ESXi host reboots. The KB article that describes this problem also includes a work-around to resolve it: Put the XML file on a shared datastore and modify the /etc/rc.local boot script to copy the file to the correct location on every reboot.

This works, but I personally consider this an ugly hack, because this modification is not inherent in the system but introduces a dependency to an external resource (the datastore). So I created a VIB file that you can effectively install on ESXi and that will permanently add the XML file to the system.
Run the following commands inside an ESXi shell to install the VIB file:

   esxcli software acceptance set --level CommunitySupported
   esxcli software vib install -v http://files.v-front.de/fwenable-ntpd-1.2.0.x86_64.vib

The first command  is needed for ESXi to accept the custom VIB, because it does not include a trusted signature file. The second command will download and install the VIB file (Note: you can also download the file with a browser, store it on a local datastore and reference the local file with the install command).

The installation will not require the host to be in maintenance mode and it will be immediately effective without the need to reboot the host! It will also automatically reload the firewall rules, so the only step left is to enable the rule in the vSphere client.

By the way, I created this VIB file with a new and improved version of my TGZ2VIB5 script that I currently work on. Once I have finished this new version and made it available here I will also post a detailed description of how I created the VIB file.

Mysterious port 903

I recently investigated what network ports are used by ESXi 4.1, because I had to compile the firewall requirements for a new deployment of ESXi hosts in a DMZ. There is a detailed source available for that in the VMware KB:

• KB1012382: TCP and UDP Ports required to access vCenter Server, ESX hosts, and other network components

And there are numerous other sources available (even nice diagrams like this one). In most cases it is obvious that their authors referred to and relied on the above mentioned official VMware KB source.

I'm usually not paranoid, but maybe I talked too much with the IT security guys (who tend to be extremely paranoid ;-)). Anyway, following the rule "Trust no one" I started looking at the network ports that are really used in our current production environment and compared them to the list in the KB article.

So I stumbled over port 903... According to the list both the vCenter server and any vSphere Client connect to an ESXi 4.1 host on that port for accessing the VM remote console. However, when I checked the network connections on the vCenter server and my Windows Desktop running the vSphere Client (with "netstat -an") I was not able to see any connection to an ESXi host's port 903, even when I opened multiple VM consoles. Instead it was obvious that port 902 is used for console connections.

This made me really curious, so I logged on to an ESXi host (in Tech support mode) and checked the open network connections there. In ESXi you use the command "esxcli network connection list" for that which produces an output that is quite similar to the netstat output (With classic ESX the netstat command is still available in the service console).

This command will also list all ports that are opened in LISTEN mode, that means there is some process waiting for connections on that port. But there was no listening process for port 903, and that means that no one and nothing would be able to connect to that port!

I opened a support request with VMware asking for clarification on the mysterious port 903 and was very curious about their answer. Of course, they quoted their own KB article first, insisted on that the port was actually used for this and that, but finally - after raising the issue to engineering - they admitted that "ESXi does not use port 903".

Also a request was made to update the KB article accordingly. So, when you read this it might already have been corrected to not include port 903 anymore, but the numerous third party documents based on KB1012382 will take some more time to be updated ...

Bottom line: Information is good. Correct information is better. Try to verify it if it is really important to you.