Illegal OpCode Red Screen of Death while booting a HP Proliant server from an USB SD card

As per Jason’s comment, with a new ILO4 update HP apparently has fixed an issue related to booting from SD cards. Whether this is the same issue is unclear though since the original KB article I linked to has not been updated.

Important note: The general symptom of such a Red Screen of Death described here is NOT specific to ESXi or booting from SD cards in general. It can happen with Windows, Linux or any other OS as well as other boot media such as normal disks/RAID arrays, if the server has a problem booting from this device (broken boot sector/partition/boot loader etc).

A couple of weeks ago I was updating a few HP Proliant DL360p Gen8 servers running ESXi on a local SD card with ESXi patches via VUM, so business as usual. Almost, because on one of the servers I ran into the following issue:
After rebooting the host, the BIOS POST completed fine and the Proliant DL360p Gen8 server should now boot the ESXi OS from it’s attached USB SD card where ESXi was installed; but instead it displayed this unsightly screen telling  something went very, very wrong:

iloillegalopcodeI reset the server several times via iLO but the issue persisted and I had no idea what exactly went bonkers here. Then I decided to boot a Linux live image, which worked fine, narrowing down the issue to the OS installation (device) itself. I thought the updates corrupted the installation but that actually wasn’t the case.
When attempting to mount the SD card USB drive from within the live Linux I noticed it was actually completely absent from the system. The USB bus was still ok, but lsusb showed no SD card reader device in the system at all!

Continue reading

Configuring and securing local ESXi users for hardware monitoring via WBEM

Besides good ol’ SNMP, the open Common Information Model (CIM) interface on an ESXi host provides a useful way of remotely monitoring the hardware health of your hosts via the Web-Based Enterprise Management (WBEM) protocol. Pretty much every major hardware management solution and agent today supports using WBEM to monitor hosts of various OSes.
Unlike SNMP (except for the painful to implement version 3), it builds on a standard HTTP(S) API, allowing secure SSL/TLS protected authentication and communication between the host and the management stations. Of course you can also use SNMP and WBEM independently at the same time too.
On ESXi, the CIM interface to is implemented through the open Small Footprint CIM Broker (SFCB) service.

sim1 sim2

Seems great, right? To manage your hosts via CIM/WBEM with for example the HP Systems Insight Management (SIM) pictured above, you just need to provide a local user on the ESXi host which SIM can use to authenticate against the host.
You can use the standard root user for example, but is that a good idea? I certainly disagree about that, even more so in environments of administrative disparity where you still have strict separation of virtualization admins and hardware admins (I agree this separation makes no sense in this day and age and causes all sorts of problems besides just this one, but this is the daily reality I’m facing).

Continue reading

HP Virtual Connect Firmware 4.01 released

After announcing it a while ago already, HP released Virtual Connect firmware 4.01 last week to the public:
The release notes can be found here.
You may also want to check out the User and Installation documentation:
HP Virtual Connect for c-Class BladeSystem Setup and Installation Guide Version 4.01 and later
HP Virtual Connect for c-Class BladeSystem Version 4.01 User Guide
HP Virtual Connect Manager Command Line Interface for c-Class BladeSystem Version 4.01 User Guide

Note that according to the most recent HP VMware Recipe document from April, the recommended VC firmware version for vSphere/ESX(i) environments is still 3.75. Continue reading

April HP ESXi bundle update fixes SmartArray warning

After the ridiculous mess HP caused with their last updates to the custom ESXi extensions back in January/Febuary, HP released new updates to the HP CIM providers a few days ago.
This update is fixing the issue that was probably responsible for all of these woes: HP SmartArray RAID Conrollers displaying a random warning message.
From the release notes:
Version: 1.4.5 (15 Apr 2013)
Version: 9.3.5 (15 Apr 2013)
Continue reading

iLO Login check script

We recently changed the iLO local account logins in favor of LDAP authentication against our AD, which is cool but raised the issue that sometimes logins seemed to work with my AD account and sometimes not, because not every system was configured for LDAP authentication properly.

Instead of checking logins on dozens of servers manually (with the nice iLO failed login delay), I took a stab at analyzing the login procedures and scripting the logins myself.
So I came up with this horrible piece of bash script doing exactly that. I checked this script with all known iLO versions 1, 2, 3 and 4, and it worked with all of them (the login procedure for versions 1/2 and 3/4 are identical). Running it requires an argument pointing to a file containing the iLO hostnames or IPs to connect to.
Here’s the script on pastebin with formatting:

Update: I have moved my scripts to GitHub and updated some of them a bit. You can find the current version of this particular script here.
Continue reading

January (or Febuary?) HP ESXi updates

Attention: [Update 16.01.2013]
HP actually pulled the updates (which were titled “February” updates) from their VIBs Depot site and purged the references from the depot metadata indexes as well. I’m not sure what’s going on but you won’t be able to apply these updates (via Update Manager) unless you downloaded them already. But even if you did, you should refrain from using these bundles at this time. Unfortunately there seems to be no way of properly removing them from Update Manager if it pulled the metadata already.

[Update 21.02.2013]
HP re-released the VIBs available at

[Update 23.02.2013]
(Thanks to milanod for the hint in the comments)
HP actually removed the re-released updates from the vibsdepot yet again?!
The updated bundles are still listed on the software/support/drivers lists for Proliant Servers though:
I’m speechless in the face of this unprecedented fail.

[Update 25.02.2013]
Uh-oh, the updates SEEM to be back at File dates are from Jan 4th and the bundles md5sums match the ones from the initial release mid-January (which this post was about) exactly. So if there really was a bug with the release, it must still be there.
Taking bets on how long it’ll take HP to offline them again.

[Update 22.04.2013]
(Thanks to Wu in the comments)
The issue with the SmartArray warning which this bundle brought us has been fixed in a recent update.

After some very minor updates back in October that did not come with release notes it’s time for another round of updates to the ESXi HP extensions and other stuff. Unfortunately, we don’t seem to be getting release notes or general infos now either.
But these updates are publicly available on already and your VMware Update Manager should have already picked them up if you set it up to use the HP VIB depot.

Since HP is so kind to not provide release notes, we can only guess about actual fixes or improvements, but we can at least check which of the VIBs contained in the offline bundles really do provide updates (spoiler: not that much).
Continue reading

Getting firmware version information on ESXi and GNU/Linux

You may not always have the convenient option to install vendor-specific hardware management agents/extension on ESXi hosts or physical servers, for example with appliance-ish OSes like the Check Point SPLAT/Gaia platform (which is just a custom RHEL descendant), or you may run into a server without these tools installed. So how can you still query firmware information on such systems directly from the command line? I will outline a couple of ways here which make it possible to obtain that information.
The example information captured here is from HP Proliant Servers (since G5), but most of it should work in similar ways with other hardware platforms too. Unless noted otherwise, the example commands here should work regardless of whether you have CIM providers or hardware management agents installed or not.

Continue reading

HP SMH error importing SIM server certificates

When trying to set the Trusted Management Servers certificate in the HP Systems Management Homepage via pasting the base64 string or getting the certificate directly from the SIM server, I was greeted by one pretty generic error message:

Error: Certificate failed validation and was not imported.

This happened on physical GNU/Linux (RHEL) systems as well as Windows Server 2008 R2 machines .
Is it impossible to properly import a SIM certificate either by pasting the base64 encoded certificate string or by letting the SMH fetch the certificate by itself from the server?

Instead I had to manually copy the .pem certificate file to the server and restart the SMH. On Windows, it goes in C:\hp\hpsmh\certs\ (if you installed the server via SmartStart) and on GNU/Linux in /opt/hp/hpsmh/certs/.
I also noticed that the displayed certificate raw data you can see under Details ( after having imported one actually cuts off a the last 2 or so lines of the base64 string, so don’t rely on copying this to create a file on your servers.

September ESXi HP updates

With the releases of vSphere 5.1, ESXi 4.1 U3 and Windows Server 2012, it’s time for hardware vendors to update their management agents again. HP did so recently and updated their bundles as follows. These bundles are supported for both, ESXi 5.0 and 5.1:

  • HP ESXi Offline Bundle for VMware ESXi 5.x (CIM providers) updated to 1.3
    Added specific HP ProLiant Gen8 servers to the server support matrix:
        HP ProLiant BL660c Gen8 server
        HP ProLiant  DL560 Gen8 server
  • HP ESXi Utilities Offline Bundle for VMware ESXi 5.x (ILO-config etc.) updated to 1.3
    Added support for the following servers:
        HP ProLiant BL660c Gen8 server
        HP ProLiant DL560 Gen8 server
  • HP NMI Sourcing Driver for VMware ESXi 5.x updated to 2.1. Note that this is not intended for HP ProLiant Gen8 servers.
    Edit 09/10/2012: Actually, the actual NMI VIB hasn’t been updated at all. It is still version  2.0.11-434156, which is exactly the same as the previous version. The only thing that changed was metadata in vendor-index.xml of the zip package to include a section to apply to version “5.1.0” too. An already installed hpnmi bundle won’t be updated because of this.
    # esxcli software sources vib list -d /vmfs/volumes/local_datastore_1/
    Name   Version        Vendor  Release Date  Acceptance Level  Status
    hpnmi  2.0.11-434156  hp      2011-07-29    PartnerSupported  Installed
  • HP Agentless Management Service Offline Bundle for VMware ESXi 5.x for the new HP ProLiant Gen8 servers updated to 9.2.0
    The following issues have been fixed:
    AMS Static data lost after clearing Active Health System Log
    iLO shows status Other for unplugged NIC port
    AMS crashes when new vSwitch is configured
    cpqSas trap not reported until query is made
    AMS Active Health does not handle iLO reset
    Added support for the following servers:
    HP ProLiant BL660c Gen8 server
    HP ProLiant DL560 Gen8 server
    Added support for the following:
    Memory DIMM status and iLO information for vCenter
    Logging to the OS event log
    cpqHoFWVer support for NIC and SAS controllers
    Added the Update number to the VMware version and build number
  • HP ProLiant Smart Array Controller Driver for VMware ESXi 5.0 updated to 5.0.0-28.0 (never really used this by the way).
    Add support for correctly displaying RAID 1(ADM) mode for logical volumes. Formerly, RAID 1(ADM) volumes were incorrectly displayed as RAID UNKNOWN.
    Remove code that limited the number of external target (array) device connections to 8 array ports. Exceeding 8 ports prevented logical volumes from registering with the driver. Improved code handling paths and targets.

Continue reading

ESXi HP Updates

HP just released a batch of new firmware for their servers and blades, Virtual Connect modules as well as updated ESXi extensions. Here’s my take on the new stuff.

HP ESXi VIBs and handling Update Manager

Updated HP Extensions and notable excerpts from the release notes:

  • The ESXi offline bundle (CIM providers) has been updated to 1.2
    Added additional support for AC Lost detection for power supplies.
    Supporting some more gen8 servers
  • The Agentless Management Service (AMS) Offline Bundle for Gen8 servers has been updated to 9.1.0
    Added network and SAS driver information reporting.
    Added performance data reporting.
    Supporting some more gen8 servers
  • The ESXi utilities bundle has been updated to 1.2
    Supporting some more gen8 servers
  • The NMI Sourcing driver has not been updated for ESXi5, but for ESXi 4.1.

If you run ESXi on HP Proliant systems, you should add the HP vibsdepot to your vCenter Update Manager repositories if haven’t done so already. But even if you did so in the past, you’ll need to add another repository for the new bundles since HP changed the way they provide bundles from their vibsdepot. Instead of just adding “” as a custom download source in UpdateManager, which would yield the most up-to-date bundles, HP now distributes mutliple repositories based on release cycles:
The following points define how to use vibsdepot under several customer scenarios:

– VUM – connect VUM to “<release date>/index.xml” to download complete update patches as well as individual patches.
– ESXCLI – use command “esxcli software vib install -d<release date>/index.xml”.

So in a nutshell, to make use of the updated bundles in VMware  Update Manager, you’ll have to add “; in UM. You can also remove or deactivate the old vibsdepot URL.
And don’t forget to update the URL once HP releases updated extensions (or HP changes this procedure all over yet again)!
[Update: You actually do not need to do that anymore if you just use This links all release versions now.]

Continue reading