After updating from vCenter 4.1 to 5.0 quite some time ago, I noticed how my custom scripts collecting historical VM-level performance statistics suddenly logged zero-values for the IOPS counters datastore.numberReadAveraged.average and datastore.numberWriteAveraged.average of all VMs.
Turns out someone decided those statistics weren’t worth collecting in the long run anymore with 5.0 or something and simply moved them to the vCenter statistics collection level 3, which is more of a verbose troubleshooting level with lots of usually uninteresting counters.
VMware does not recommend using a statistics level greater than 2 except for short-term data collection too.
Querying for these metrics in any rollup interval yields nothing, the values are only available in realtime mode:
Get-VM vm1 | Get-StatType -Interval 1800 | ? {$_ -match "datastore.number(Read|Write)Averaged.average" }
-
Get-VM vm1 | Get-StatType -Realtime | ? {$_ -match "datastore.number(Read|Write)Averaged.average" }
datastore.numberReadAveraged.average
datastore.numberWriteAveraged.average
This naturally raised the question if it’s possible to just re-configure the counters to belong to a lower statistics level or have some more fine-grained control over this in general.
Unfortunately it seemed like I was out of luck, I thought about opening a Support Request or rather “Feature Request” but did end up procrastinating and forgetting, not really expecting much out of a mere Feature Request case anyways.
But while engaging support for some other problem, I asked about this as a sidenote and the supporter immediately recalled two KB articles:
Change the collection level for Storage DRS and SIOC data counters in vSphere 5.0 by using the Level Mapping Utility
http://kb.vmware.com/kb/2014382
Change the collection level for Storage DRS and SIOC data counters in vSphere 5.0 Update 1 by using the Level Mapping Utility
http://kb.vmware.com/kb/2009532
These articles provide a small Powershell script, the “LevelMappingUtility” which allows you to manipulate the statistics level affiliation of any performance counter, or at least it seems so. Exactly what I was looking for!
First we can confirm the current statistics level of all counters and we’ll see how the “PerDeviceLevel” (per VM in my case) of the IOPS counters I’m interested in is at 3. Not so good:
Get-PxCounterLevelMapping | ? {$_.Name -match "datastore.number(Read|Write)Averaged.average" }
Name AggregateLevel PerDeviceLevel Server
---- -------------- -------------- ------
datastore.numberReadAveraged.average 1 3 vcenter.local
datastore.numberWriteAveraged.average 1 3 vcenter.local
The Set-Function to actually change statistics levels can read from a CSV-File about the counters and how they are supposed to be manipulated. The by default provided CSV-File will change the statistics level for a lot of other counters too:
#TYPE VMware.VimAutomation.PowerCliExtensions.CounterLevelMapping,,, Name,AggregateLevel,PerDeviceLevel,Server disk.numberReadAveraged.average,1,1,example.server.com disk.numberWriteAveraged.average,1,1,example.server.com virtualDisk.numberReadAveraged.average,1,1,example.server.com virtualDisk.numberWriteAveraged.average,1,1,example.server.com virtualDisk.totalReadLatency.average,1,1,example.server.com virtualDisk.totalWriteLatency.average,1,1,example.server.com datastore.numberReadAveraged.average,1,1,example.server.com datastore.numberWriteAveraged.average,1,1,example.server.com datastore.totalReadLatency.average,1,1,example.server.com datastore.totalWriteLatency.average,1,1,example.server.com datastore.datastoreIops.average,1,1,example.server.com datastore.sizeNormalizedDatastoreLatency.average,1,1,example.server.com datastore.datastoreReadIops.latest,1,1,example.server.com datastore.datastoreReadOIO.latest,1,1,example.server.com datastore.datastoreWriteIops.latest,1,1,example.server.com datastore.datastoreWriteOIO.latest,1,1,example.server.com datastore.siocActiveTimePercentage.average,1,1,example.server.com datastore.datastoreVMObservedLatency.latest,1,1,example.server.com datastore.datastoreMaxQueueDepth.latest,1,1,example.server.com disk.maxQueueDepth.average,1,1,example.server.com disk.deviceLatency.average,1,1,example.server.com
As I’m only interested in two distinct counters for now and don’t want to blow up my database too much, I simply removed all other lines and ran it with that. Besides the CSV-Import you can also run it like that:
Get-PxCounterLevelMapping | ? {$_.Name -match "datastore.number(Read|Write)Averaged.average" } | Set-PxCounterLevelMapping -PerDeviceLevel 2
Import-csv -Path D:\LevelMappingUtility\counter.csv | Set-PxCounterLevelMapping
Updating datastore.numberReadAveraged.average in server vcenter.local
Updating datastore.numberWriteAveraged.average in server vcenter.local
Get-PxCounterLevelMapping | ? {$_.Name -match "datastore.number(Read|Write)Averaged.average" }
Name AggregateLevel PerDeviceLevel Server
---- -------------- -------------- ------
datastore.numberReadAveraged.average 1 1 vcenter.local
datastore.numberWriteAveraged.average 1 1 vcenter.local
After a brief period, when the first rollup tasks on the database have finished, you should be able to access the statistics not only in real-time mode, but permanently in the respective interval slot too:
Get-VM vm1 | Get-StatType -Interval 1800 | ? {$_ -match "datastore.number(Read|Write)Averaged.average" }
datastore.numberReadAveraged.average
datastore.numberWriteAveraged.average
Finally!
You could also use that tool to remove some (for you) uninteresting counters from the lower statistics levels to keep the vCenter DB compact and tidy, but I’ll leave it at that.
Pingback: HOSTING IS LIFE!PowerCli IOPS Metrics: vCloud Org and VPS Reporting » HOSTING IS LIFE!