ESXi host VMware Tools package update error – Broken pipe

Today when I went to install the recent VMware Tools updates on some hosts via Update Manager while they were running without maintenance mode (Tools Updates are no problem and don’t need maintenance mode/reboots), I ran into the following error:
Could not stage image profile ‘(Updated) ESXi-5.0.0-20120302001-standard’: (‘VMware_locker_tools-light_5.0.0-1.18.768111’, ‘[Errno 32] Broken pipe’)

Not really helpful this generic error, huh?
I suspected this already, and looking at the esxupdate.log of the host via SSH made pretty clear what was the causing the error:

2012-07-18T08:11:26Z esxupdate: HostImage: DEBUG:  — Stage: LockerInstaller adding [VMware_locker_tools-light_5.0.0-1.18.768111], removing []
2012-07-18T08:11:26Z esxupdate: HostImage: INFO: Attempting to download VIB tools-light
2012-07-18T08:11:26Z esxupdate: downloader: DEBUG: Downloading from http://vcenter.alpacafarm.local:9084/vum/repository/hostupdate/vmw/vib20/tools-light/VMware_locker_tools-light_5.0.0-1.18.768111.vib…
2012-07-18T08:11:30Z esxupdate: LockerInstaller: WARNING: There was an error in cleaning up product locker: [Errno 16] Device or resource busy: ‘/locker/packages/5.0.0/vmtools/windows.iso’
2012-07-18T08:11:30Z esxupdate: esxupdate: ERROR: An esxupdate error exception was caught:

Some VM obviously still has one of the tools images mounted! Just which of the ~30 running on that host is the culprit? While I was already connected to the host via SSH to check out the esxupdate.log file, I thought I might as well figure out the VM from here. So I ran the following command from the shell to dig through the VM config files:
find /vmfs/volumes/ -iname *.vmx | while read vmx; do grep -isH “/usr/lib/vmware/isoimages” $vmx ; done

Well, this filtered the list down quite a bit, but it listed a lot of innocent VMs too. That is because apparantly some tools update/installation procedures or CD (de)mounting in general doesn’t cleanup the “ide1:0.fileName = “/path/to/image/goes/here.iso” VMX directive properly. You will not see this image path in the GUI Client or elsewhere.

So let’s move on to our good ‘ol friend powercli in order to see who is to blame. The following one-liner shows us which VMs are actually actively bound to a VMware Tools image:
Get-VM | where { $_ | Get-CDDrive | where { $_.IsoPath -like “*/usr/lib/vmware*” } } | Select Name


As you can see it is actually not connected or anything, but still holding up a lock on the image. We can just set it to Client Device in the GUI to get rid of it, or if you want to do it in an automated fashion based on the previous powercli line, you may as well just execute this:
Get-VM | where { $_ | Get-CDDrive | where { $_.IsoPath -like “*/usr/lib/vmware*” } } | foreach { Get-CDDrive $_ | Set-CDDrive -NoMedia -Confirm:$false }

Et vóila, the device is now disconnected properly and I can update my tools-light packages on the host.

 

After figuring that out I went back to my SSH shell to see what I could have used to track the issue down from there. It would have had to look like this in my case:
find /vmfs/volumes/ -iname *.vmx | while read vmx; do grep -isH “clientDevice = \’false\'” $vmx ; done

Note that this kind of iterating through VMX files only works with VMX files of VMs the host you’re on is actually running. VMX files on shared storage which other hosts are accessing will generate a “Device or resource busy” warning (suppressed by via grep -s).
And of course one could have used powercli to check the raw VMX parameters too!

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s