Configuring and securing local ESXi users for hardware monitoring via WBEM

Besides good ol’ SNMP, the open Common Information Model (CIM) interface on an ESXi host provides a useful way of remotely monitoring the hardware health of your hosts via the Web-Based Enterprise Management (WBEM) protocol. Pretty much every major hardware management solution and agent today supports using WBEM to monitor hosts of various OSes.
Unlike SNMP (except for the painful to implement version 3), it builds on a standard HTTP(S) API, allowing secure SSL/TLS protected authentication and communication between the host and the management stations. Of course you can also use SNMP and WBEM independently at the same time too.
On ESXi, the CIM interface to is implemented through the open Small Footprint CIM Broker (SFCB) service.

sim1 sim2








Seems great, right? To manage your hosts via CIM/WBEM with for example the HP Systems Insight Management (SIM) pictured above, you just need to provide a local user on the ESXi host which SIM can use to authenticate against the host.
You can use the standard root user for example, but is that a good idea? I certainly disagree about that, even more so in environments of administrative disparity where you still have strict separation of virtualization admins and hardware admins (I agree this separation makes no sense in this day and age and causes all sorts of problems besides just this one, but this is the daily reality I’m facing).

So in the end, we’re down to creating dedicated local users on each of our ESXi hosts. Now here’s the catch:
It requires us to create the new local user and assign this user the Administrator role with all permissions on the root level of the ESXi host, just so the user can query some hardware management information through WBEM.
Bug 1: There is a local permission called “Host – CIM – CIM Interaction”, which implies that using a dedicated role for this user instead of the builtin Administrator role would work, but forget about that, it just doesn’t (related to bug 2 below).

 

 How can I check if the WBEM interface works with a user?

Even without having access to a WBEM capable management system like HP SIM, you can easily check yourself if a user is allowed to query CIM providers by connecting to the WBEM port like this:

# curl -ik 'https://myesxihost.domain:5989' --request POST --data "" --basic --user cimuser
  Enter host password for user 'cimuser':
  HTTP/1.1 200 OK
  Content-Type: application/xml; charset="utf-8"
  Content-Length: 0
  Cache-Control: no-cache
  CIMOperation: MethodResponse

As you can see the service replies with a good 200 OK message.
Wrong passwords or insufficient permissions will receive a 401 Unauthorized response:

# curl -ik 'https://myesxihost.domain:5989' --request POST --data "" --basic --user cimuser
  Enter host password for user 'cimuser':
  HTTP/1.1 401 Unauthorized
  WWW-Authenticate: Basic realm="cimom"
  Server: sfcHttpd
  Content-Length: 0

You can also reproduce these requests from the local ESXi shell directly in case you don’t have a system with curl at hand:

# openssl s_client -connect localhost:5989
 POST / HTTP/1.1
 Host: myesxihost.domain:5989
 Authorization: Basic [username:password in base64 goes here]
 Content-Length: 0

Denied login attempts are logged to /var/log/syslog.log:
2012-12-05T11:46:06Z sfcb-CIMXML-Processor[1192236]: pam_access(sfcb:auth): access denied for user `cimuser’ from `sfcb’

Access to the CIM interface via WBEM is governed through the configuration files in /etc/security/. Here we have the default /etc/security/access.conf after only creating the local cimuser account (no permissions assigned etc):

# This file is autogenerated and must not be edited.
  +:dcui:ALL
  +:root:ALL
  +:vi-admin01:ALL
  +:vpxuser:ALL
  +:vslauser:ALL
  -:vi-user01:ALL
  -:ALL:ALL

The cimuser is not included and as such treated with no rights (-:ALL:ALL). If we assign the builtin Administrator ESXi role, a new entry will be added allowing everything:
+:cimuser:ALL

Bug 2: For non-root local users, access to CIM interface is only allowed if we add permissions with the builtin Administrator role on the ESXi host. Even cloning that role and thereby retaining full permissions results in NOT being allowed to access the CIM interface. This is because any non-builtin Administrator assignment will not be reflected in /etc/security/access.conf.

So now that we know why it doesn’t work unless the user has the builtin Administrator role, we could edit the /etc/security/access.conf file manually and even control that the user only has access to the sfcb CIM service by adding the following before the -ALL rule:
+:cimuser:sfcb

This works well and allows us to assign really the most minimal permissions possible, but the problem is that this file is auto-generated at every reboot (running the auto backup scripts is not a solution because of that).

 

Can we do better than that?

Yes we can, kind of…
We can work around the problem above with the following steps:

1. Add the cimuser to the root group in /etc/group on the host:
# grep root /etc/group
root:x:0:root,cimuser

Note that local groups were officially deprecated as of ESXi 5.1 and you can’t administer them with the vSphere client anymore.

2. For good measure, also change the login shell of the cimuser from /bin/sh to /sbin/nologin, or otherwise they can connect via SSH or login through the local shell:
# grep cimuser /etc/passwd
cimuser:x:501:0:Hardware Monitoring:/:/sbin/nologin

cimuser


Now you might chime in and say “Hey, I can do that from the vSphere Client too!” And I have to retort: “Unfortunately, no you can’t”:

Bug 3: Since ESXi 5.1 the “grant shell access to user” checkbox in the user properties became useless. Even if you uncheck it, no actual change will be performed and it will automatically be checked again. By editing the /etc/passwd file for the user directly, the checkbox will correctly display an unchecked status though.








3. Finally, run the ESXi config backup scripts so the changes persist across reboots:
# backup.sh 0
Saving current state in /bootbank
Clock updated.
Time: 08:06:53   Date: 09/27/2013   UTC
# /sbin/auto-backup.sh

That’s it, done.

vsphereclient_deny

 

This approach survives reboots or updates and does not require ANY assigned permissions for the user on the local ESXi host. You only need to create the user beforehand. The user will not be able to connect to the host with the vSphere Client, APIs like Powershell, or login through SSH or the local ESXi shell:

 

In PowerCLI:
Connect-VIServer myesxihost.local
 Connect-VIServer : 27.09.2013 10:23:08    Connect-VIServer        Permission to perform this operation was denied.
 In Zeile:1 Zeichen:1
 + Connect-VIServer myesxihost.local
 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 + CategoryInfo          : NotSpecified: (:) [Connect-VIServer], NoPermission
 + FullyQualifiedErrorId : Client20_ConnectivityServiceImpl_Reconnect_Exception,VMware.VimAutomation.ViCore.Cmdlets.Commands.ConnectVIServer

Via SSH:
# ssh cimuser@myesxihost.local
 Password:
 The time and date of this login have been sent to the system logs.
VMware offers supported, powerful system administration tools.  Please
 see www.vmware.com/go/sysadmintools for details.
The ESXi Shell can be disabled by an administrative user. See the
 vSphere Security documentation for more information.
 Login disabled
 Connection to myesxihost.local closed.

However, the user will still be able to login to the DCUI. But the scope of what he could do is quite limited compared to what he could do with full admin rights on the vSphere host components before (ok, he can still reboot the host which he probably can do already anyways if he has ILO or physical access, but he can’t delete files/VMs, reconfigure stuff except the management network etc (another thing for VMware to fix: he can do that even with no permissions), obvious things that should pretty quickly raise a few eyebros of the vSphere admins in charge).
So as long as you (I hope you do so either way already) secure your out of band management network and physical access to the hosts, this should be ok. Please let me know with a comment if you’re aware of a halfway supportable method to also disable DCUI access.

 

Wrapping it up

I’m aware that fiddling with local config files on the shell isn’t the smoothest solution, but it seems VMware does not intend or is taking their sweet time again to fix the actual problems. To my knowledge, this is currently the only way to implement a least privilege approach (which, surprise surprise, is also recommended in the official VMware Security Hardening Guides) for CIM based hardware monitoring.

Note that for all of this I already filed a ticket with VMware support almost a year ago. But nothing ever came of it.
In the end I was supposed to file a “feature request” for a bunch of bugs, which I did, but good good luck with that. (Aka “Yeah thanks for it, we’re stacking it right onto our clipboard trays which coincidentally is located right next to the shredder.“)

However, it should also be noted that VMware provides another method to authenticate to the CIM service on ESXi hosts, that is through CIM tickets issues by the vCenter server. The method is described in this document but has a security related catch again:

A CIM client must authenticate before it can access data or perform operations on a VMware® ESXi™ host.The client can authenticate in one of the following ways.
– The client can authenticate directly with the CIMOM on the ESXi host by supplying a valid user name andpassword for an account that is defined on the ESXi host.
– The client can authenticate with a sessionId that the CIMOM accepts in place of the user name andpassword. The sessionId (called a “ticket”) can be obtained by invoking the AcquireCimServicesTicket() method on VMware vCenter™ Server.
VMware recommends using CIM ticket authentication for servers managed by vCenter. If the ESXi host is operating in lockdown mode, the CIMOM does not accept new authentication requests from CIM clients.
However, the CIMOM will continue to accept a valid ticket obtained from vCenter Server. The ticket must be obtained using the credentials of any user that has administrative privileges on vCenter Server.

In our case that would again breach separation of administrative roles.
I’m also not sure if this method is widely supported by management applications such as HP SIM, information about this is very scarce on the web. Please leave a reply if you happen to know more about it or have any other input on this whole topic.

Advertisements

2 thoughts on “Configuring and securing local ESXi users for hardware monitoring via WBEM

  1. Thank you. Exactly what i was looking for! Also very well written. All other articles on this topic just state “give HPSIM the root credentials”.

    Though i agree that giving administrative privileges to others (even if it claims to be monitoring “read only”) doesn’t feel good, we will probably end up with a separate user with these rights, just not the root account itself. Should VMware see these modifications during a service request for example, they have the possibility to blame that for whatever issue one might have at that point.
    i might ask our TAM about this the next time we talk.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s