[Script] Network routing/failover topology change detection

A while ago I wrote a simple but useful script which I’m sharing here to detect upstream provider HSRP failover events via traceroute. It can be used for all kinds of virtual IP routing failover like VRRP, Check Point Cluster XL, actual routing protocols like BGP/OSPF or similar technologies where IP packets can be routed across multiple hops.
The script executes traceroutes to a given destination and checks whether the path is being routed over a certain hop, with the ability to send mail notifications if this is not the case.

You can get the most recent version of this script on my Github here. If you have any suggestions or improvements (which I’m sure there is plenty of room for), feel free to drop a comment or an issue or a pull-request on Github.

Continue reading

Renamed VMware Tools components and automatic installation

With the release of vSphere 6.0 and a recent update for the 5.5 VMware Tools on May 8th 2015 (9.4.12 build 2627939), VMware also changed the Windows VMware Tools installer slightly by renaming some vShield-related components:
VMware ESXi 5.5, Patch ESXi550-201505402-BG: Updates tools-light

The vShield Endpoint drivers are renamed as Guest Introspection Drivers and two of these drivers, NSX File Introspection Driver (vsepflt.sys) and NSX Network Introspection Driver (vnetflt.sys), can be installed separately now. This allows you to install the file driver without installing the network driver.

If you’ve been using custom automated installations of the VMware Tools like me then you might have to adjust the installer command for newer tools versions.

Continue reading

[Script] Extending Linux LVM partitions

Here’s a script I wrote a while a go to extend LVM partitions on Linux machines.

The script assumes that you have extended the existing underlying physical (or “virtual” if it’s a VM) storage device prior to execution. It will rescan the disks (skip with -f), resize the existing partition (basically just setting a different end sector), reboot, and run scripts to extend the actual file system after the reboot. There are other ways to extend the disk space including creating a new partition on the additional disk space, but I’ve decided against that approach in favor of a single-partition scheme for management/simplicity’s sake.

This script will work with VMs and physical servers alike. I’ve tested it with RHEL 6/7 and CentOS 6/7, but it should generally work with other Linux distributions as well.

You can get the most recent version of this script on Github here. If you have any suggestions or improvements (which I’m sure there is plenty of room for), feel free to drop a comment or an issue or a pull-request on Github.

Continue reading

[Script] Poor man’s vSphere network health check for standard vSwitches

Among many other new features, the vSphere 5.1 distributed vSwitch brought us the Network Health Check feature. The main purpose of this feature is to ensure that all ESXi hosts attached to a particular distributed vSwitch can access all VLANs of the configured port groups and with the same MTU. This is really useful in situations where you’re dealing with many VLANs and the roles of virtualization and network admin are strongly separated.

Unfortunately, like pretty much all newer networking features, the health check is only included in the distributed vSwitch which requires vSphere Enterprise+ licenses.

There are a couple other (cumbersome) options you have though:
If you’re have ESXi 5.5 you can use the pktcap-uw utility on the ESXi shell to check if your host receives frames for a specific VLAN on an uplink port:
The following example command will capture receive-side frames tagged with VLAN 100 on uplink vmnic3:
# pktcap-uw –uplink vmnic3 –dir 0 –capture UplinkRcv –vlan 100
If systems are active in this VLAN you should see a few broadcasts or multicasts already and meaning the host is able to receive frames on this NIC for this VLAN. Repeat that for every physical vmnic uplink and VLAN.

Another way to check connectivity is to create a vmkernel interface for testing on this VLAN and using vmkping. Since manually configuring a vmkernel interface for every VLAN seems like a huge PITA, I came up with a short ESXi shell script to automate that task.
Check out the script below. It uses a CSV-style list to configure vmkernel interfaces with certain VLAN and IP settings, pinging a specified IP on that network with the given payload size to account for MTU configuration. This should at least take care of initial network-side configuration errors when building a new infrastructure or adding new hosts.
This script was tested successfully on ESXi 5.5 and should work on ESXi 5.1 as well. I’m not entirely sure about 5.0 but that should be ok too. (Please leave a comment in case you can confirm/refute that).

Introducing the ghetto-vSwitchHealthCheck.sh:

Update: I have moved my scripts to GitHub and updated some of them a bit. You can find the current version of this particular script here.

Continue reading

Forefront TMG Log Export with MSDEToText.vbs messing up IPs

Logging Firewall or Web Proxy traffic on a Forefront TMG/ISA node into the local SQL Express-based database (which is the default setting) has a few advantages, like being able to query past logs through the TMG console. But sometimes it’s better to have logs stored in a plain text format as well for a 3rd party tool or your own log analysis scripts.

For this purpose, Microsoft provides the MSDEToText.vbs tool to export logs from a TMG/ISA SQL database into text files.



However, the MSDEToText script is producing some weird results for my TMG environments, namely it fails to convert the source and destination IP-addresses properly:
For example, what should be exported as “” ends up as “-63.-87.-254.-245”, with negative numbers per octet in the text log. Notice something? Yeah, subtracting each value from 255 yields us the correct IP (well, almost except for the last octet which is off by 1). This happens only for IPs that don’t have an existing computer object defined in the TMG policy.

There is obviously something wrong with the logic inside the MSDEToText VB script. Being completely clueless about VBS (I can’t even remember ever seriously coding/editing something longer than two lines), I dug into the script to see what makes it go bonkers and found the following function to be responsible:

Continue reading

[Script] PowerCLI – find out the host on which VMs are running when vCenter is down

Many moons ago, when I started playing with the wonderfulness that is PowerCLI, one of the first things I wrote with a particular problem in mind was a small script to quickly locate the host running our vCenter server in case anything went wrong and I lost access to vCenter directly.
So instead of trying to connect to every possible host of the cluster manually with the vSphere Client, why not just connect to all of them via PowerCLI and query them quickly?

Where’s Waldo?

This resulted in the small, simple script posted below. For this script, you can provide either a list of hosts to connect to, an alias for a cluster which member hosts you pre-populated in the script, along with one or more search strings. This search is matched against the VM names and outputs the list of found VMs with their current power state and most importantly, the host running the VM. This way you can get a VM-Host mapping of not only your vCenter VM, but other VMs as well.

I remembered this script while reading a cool article on v-front.de about various other ways to keep track on which host your vCenter VM is running.
I “polished” the old, simple code a bit but yeah, I’m still pretty horrible when it comes to scripting. Anyways, here it is in case anyone finds it useful:

Continue reading

iLO Login check script

We recently changed the iLO local account logins in favor of LDAP authentication against our AD, which is cool but raised the issue that sometimes logins seemed to work with my AD account and sometimes not, because not every system was configured for LDAP authentication properly.

Instead of checking logins on dozens of servers manually (with the nice iLO failed login delay), I took a stab at analyzing the login procedures and scripting the logins myself.
So I came up with this horrible piece of bash script doing exactly that. I checked this script with all known iLO versions 1, 2, 3 and 4, and it worked with all of them (the login procedure for versions 1/2 and 3/4 are identical). Running it requires an argument pointing to a file containing the iLO hostnames or IPs to connect to.
Here’s the script on pastebin with formatting: http://pastebin.com/i2Y0xSTQ:

Update: I have moved my scripts to GitHub and updated some of them a bit. You can find the current version of this particular script here.
Continue reading

Invoke-VMScript issue: “The guest operations agent could not be contacted”

[This issue has been fixed with the VMware Tools releases of ESXi 5.0 Update 2 and ESXi 5.1 Update 1.
Note that you can run more recent versions of VMware Tools to fix this on your VMs even if you haven’t updated your hosts, this is fully supported]

The Invoke-VMScript PowerCLI function provides a cool way to execute scriptcode inside guests through the VMware tools, on Linux and Windows VMs (as long as a couple of prerequisites are met). While playing with it a while ago, I noticed that on many VMs, it wouldn’t run properly, instead just spilling the error “The guest operations agent could not be contacted”. To fix this, the VMware Tools service inside the guest had to be restarted. Not thinking too much of the oddness, I left it at that since I didn’t have a real use case for the Invoke-VMScript function.
This changed later on, when I ran into the same issue again that made me finally dig a bit deeper.

Quite a few other people reported this issue on the VMTN forums since some time, but there was no solution posted. A reference to Backup Exec initiated backup operations made me curious though – had I not fought with another problem induced by it already?

I then was able to track it down to the Reload Virtual Machine operation, which Backup Exec triggers each time before removing the snapshot of a backup too. But not only that, unlike my previous issue, this problem is fully reproducible manually without any involvement of Backup Exec: Continue reading

Virtual Appliance nitpicking and vMA optimizations

Unfortunately, most virtual appliances, including those distributed by VMware are not quite well-maintained. Nitpicking incoming.

This starts with them being released in the old, ESX3.x compatible vHardware version 4. Backwards compatibility is cool and all, but I suppose they could have at least jumped on the v7 wagon with the release of vSphere5 (and HW v8).

Another thing I dislike is the frequent use of the Open VMTools. Now don’t get me wrong, I usually love open source implementations, but the problem with the Open VMTools is that you lose the ability of quickly being able to check if a guest’s tools are up-to-date on a given host.

Instead of a clear “Current” or “Out of date” message, you’re simply greeted with an ambigious “Running/3rd Party Independant” or “Unmanaged” for the VMs.
You also can’t simply upgrade the Open VMTools via the vSphere easy Install/Upgrade VMTools wizards or via Update Manager.

Virtual Appliances want to be operated in a fire&forgot approach. After swiftly deploying them, you’re supposed to just use them for what they were intended to and don’t think about anything else you might have in mind when you deploy an ordinary OS in a VM. Updating a VA also often means deploying the whole new OVF package of the current version and deleting the old VM, even for minor version upgrades. Luckily this seems to slowly improve with built-in update functions. (It’s a shame VMware never made use of the vima-update functions back in the day.)

Besides these points theres always a handful of things to make me frown or chuckle, or sometimes both:

vi-admin@vma:~> cat /proc/cmdline
root=/dev/sda3 append server= serverdir=/build/.0/repo/iso resume=/dev/sda2 splash=silent showopts quit

Um, ok VMware, whatever floats your boat at


A couple of things to make working with the vMA more convenient
Here are some things I would change on a freshly deployed vMA in order to make my life easier:

  • install GNU screen:
    yast2 -i screen
  • disable IPv6 which I don’t use in my environment:
    echo "net.ipv6.conf.all.disable_ipv6 = 1" >> /etc/sysctl.conf ; sysctl -p
  • getting a list of all registered hosts at every login/screen window so i can quickly copy hostnames for use with my vicfg-*, esxcli etc. commands:
    echo "vifp listservers | sort" >> ~/.bashrc
  • fixing keybinding problems when connecting by SSH from my urxvt host for at least forward/backward words:
    ctrl-v ctrl-leftarrow and ctrl-v ctrl-rightarrow – fixing the .inputrc with "\eOd": backward-word and "\eOc": forward-word
  • creating a tiny script that lets me quickly open resxtop sessions to my commonly-named hosts without having to change the vifptarget every time, like “./resxtop.sh 0014”:

    resxtop –server esxhost$@.localdomain –user root