Control vCenter performance counter collection and get back VM IOPS statistics

After updating from vCenter 4.1 to 5.0 quite some time ago, I noticed how my custom scripts collecting historical VM-level performance statistics suddenly logged zero-values for the IOPS counters datastore.numberReadAveraged.average and datastore.numberWriteAveraged.average of all VMs.ಠ_ಠ
Turns out someone decided those statistics weren’t worth collecting in the long run anymore with 5.0  or something and simply moved them to the vCenter statistics collection level 3, which is more of a verbose troubleshooting level with lots of usually uninteresting counters.
VMware does not recommend using a statistics level greater than 2 except for short-term data collection too.

Querying for these metrics in any rollup interval yields nothing, the values are only available in realtime mode:

Get-VM vm1 | Get-StatType -Interval 1800 | ? {$_ -match "datastore.number(Read|Write)Averaged.average" }
-

Get-VM vm1 | Get-StatType -Realtime | ? {$_ -match "datastore.number(Read|Write)Averaged.average" }
datastore.numberReadAveraged.average
datastore.numberWriteAveraged.average

This naturally raised the question if it’s possible to just re-configure the counters to belong to a lower statistics level or have some more fine-grained control over this in general.
Continue reading

Invoke-VMScript issue: “The guest operations agent could not be contacted”

[This issue has been fixed with the VMware Tools releases of ESXi 5.0 Update 2 and ESXi 5.1 Update 1.
Note that you can run more recent versions of VMware Tools to fix this on your VMs even if you haven’t updated your hosts, this is fully supported]

The Invoke-VMScript PowerCLI function provides a cool way to execute scriptcode inside guests through the VMware tools, on Linux and Windows VMs (as long as a couple of prerequisites are met). While playing with it a while ago, I noticed that on many VMs, it wouldn’t run properly, instead just spilling the error “The guest operations agent could not be contacted”. To fix this, the VMware Tools service inside the guest had to be restarted. Not thinking too much of the oddness, I left it at that since I didn’t have a real use case for the Invoke-VMScript function.
This changed later on, when I ran into the same issue again that made me finally dig a bit deeper.

Quite a few other people reported this issue on the VMTN forums since some time, but there was no solution posted. A reference to Backup Exec initiated backup operations made me curious though – had I not fought with another problem induced by it already?

I then was able to track it down to the Reload Virtual Machine operation, which Backup Exec triggers each time before removing the snapshot of a backup too. But not only that, unlike my previous issue, this problem is fully reproducible manually without any involvement of Backup Exec: Continue reading