[Script] PowerCLI – find out the host on which VMs are running when vCenter is down

Many moons ago, when I started playing with the wonderfulness that is PowerCLI, one of the first things I wrote with a particular problem in mind was a small script to quickly locate the host running our vCenter server in case anything went wrong and I lost access to vCenter directly.
So instead of trying to connect to every possible host of the cluster manually with the vSphere Client, why not just connect to all of them via PowerCLI and query them quickly?

Where’s Waldo?

This resulted in the small, simple script posted below. For this script, you can provide either a list of hosts to connect to, an alias for a cluster which member hosts you pre-populated in the script, along with one or more search strings. This search is matched against the VM names and outputs the list of found VMs with their current power state and most importantly, the host running the VM. This way you can get a VM-Host mapping of not only your vCenter VM, but other VMs as well.

I remembered this script while reading a cool article on v-front.de about various other ways to keep track on which host your vCenter VM is running.
I “polished” the old, simple code a bit but yeah, I’m still pretty horrible when it comes to scripting. Anyways, here it is in case anyone finds it useful:

Control vCenter performance counter collection and get back VM IOPS statistics

After updating from vCenter 4.1 to 5.0 quite some time ago, I noticed how my custom scripts collecting historical VM-level performance statistics suddenly logged zero-values for the IOPS counters datastore.numberReadAveraged.average and datastore.numberWriteAveraged.average of all VMs.
Turns out someone decided those statistics weren’t worth collecting in the long run anymore with 5.0  or something and simply moved them to the vCenter statistics collection level 3, which is more of a verbose troubleshooting level with lots of usually uninteresting counters.
VMware does not recommend using a statistics level greater than 2 except for short-term data collection too.

Querying for these metrics in any rollup interval yields nothing, the values are only available in realtime mode:

Get-VM vm1 | Get-StatType -Interval 1800 | ? {$_ -match "datastore.number(Read|Write)Averaged.average" }
-

Get-VM vm1 | Get-StatType -Realtime | ? {$_ -match "datastore.number(Read|Write)Averaged.average" }
datastore.numberReadAveraged.average
datastore.numberWriteAveraged.average

This naturally raised the question if it’s possible to just re-configure the counters to belong to a lower statistics level or have some more fine-grained control over this in general.

Invoke-VMScript issue: “The guest operations agent could not be contacted”

[This issue has been fixed with the VMware Tools releases of ESXi 5.0 Update 2 and ESXi 5.1 Update 1.
Note that you can run more recent versions of VMware Tools to fix this on your VMs even if you haven’t updated your hosts, this is fully supported]

The Invoke-VMScript PowerCLI function provides a cool way to execute scriptcode inside guests through the VMware tools, on Linux and Windows VMs (as long as a couple of prerequisites are met). While playing with it a while ago, I noticed that on many VMs, it wouldn’t run properly, instead just spilling the error “The guest operations agent could not be contacted”. To fix this, the VMware Tools service inside the guest had to be restarted. Not thinking too much of the oddness, I left it at that since I didn’t have a real use case for the Invoke-VMScript function.
This changed later on, when I ran into the same issue again that made me finally dig a bit deeper.

Quite a few other people reported this issue on the VMTN forums since some time, but there was no solution posted. A reference to Backup Exec initiated backup operations made me curious though – had I not fought with another problem induced by it already?

I then was able to track it down to the Reload Virtual Machine operation, which Backup Exec triggers each time before removing the snapshot of a backup too. But not only that, unlike my previous issue, this problem is fully reproducible manually without any involvement of Backup Exec: