Configuring the vMA as ESXi syslog server

An ESXi host has a number of important logfiles located in /var/log (or /scratch/log) for each of it’s components. Reviewing those logs the standard way can be quite cumbersome: Some logs can be viewed through the DCUI (if you like to torture yourself) or you have to enable SSH each time on each host to browse through the logs. Compared to that, having your host-local logs in one central place can be very useful for things like audits or troubleshooting. VMware also noticed that back in vSphere 4 an included the vilogger component directly into the vMA. Unfortunately vilogger was removed with vSphere5 for reasons completely beyond me.
Instead VMware included the ESXi Syslog collector for vCenter, but it’s quite basic and not really the greatest way to centralize your hosts logs, especially in larger environments. I’m also missing my bash and *nix tools to properly filter and navigate through the volume of information contained in the logs.
So as the vMA 5.x Linux OS already comes with the default syslog-ng daemon, why not take advantage of that and let your hosts log there? The steps outlined here were done with the vMA 5.0 but should work without a hitch on the 5.1 release too.

A note related to the ESXi scratch partition:
Collecting syslogs permanently is all the more important if your ESXi hosts are installed on USB or run stateless. In that case you will only find the most recent logs and those will be lost with every reboot too, making troubleshooting or support bundles possibly useless. That is because installs on USB or non-persistent storage do not have a default persistent scratch partition. This partition is used (among other things) to store older logs in a gzipped format. You can create a persistent scratch location on VMFS datastores as described in Creating a persistent scratch location for ESXi 4.x and 5.x.

First off, no matter which syslog method/daemon/application you use, you should have ESXi 5.0 Patch ESXi-5.0.0-20120704001 installed on your hosts. This patch fixes a bug with the logging stopping at seemingly random where you would need to reload the syslog daemon on each hosts to continue submitting logs.

Step 1, Adding the log disk

Having done that, you can get to work on your vMA by adding an additional disk to the VM where the logs will be stored on. You could also extend the already present disk but I prefer this way. I choose a 10GB disk but this obviously depends on the volume of logs and number of hosts in your environment (I’ll plan to post on general GNU/Linux partition extension later on).
After hot-adding the disk to the VM, rescan the SCSI bus of the OS in the usual GNU/Linux way to see the disk:

# echo "- - -" > /sys/class/scsi_host/host0/scan
 Create the partition and format it with a filesystem:
 # fdisk /dev/sdb
 n [new partition]
 p [primary partition]
 1 [default primary partition #1]
 [defaults for start/end using the whole disk]
 p [print]
 Device Boot Start End Blocks Id System
 /dev/sdb1 1 1305 10482381 83 Linux
 w [write partition table]
 # mkfs.ext3 /dev/sdb1

Create a directory where the logs will be stored, I went with /var/log/esxi-syslog/ and mount the fresh partition to this location:
# mkdir /var/log/esxi-syslog
# mount /dev/sdb1 /var/log/esxi-syslog
To automatically remount the partition at boot time, add the following line to /etc/fstab:
# echo “/dev/sdb1 /var/log/esxi-syslog ext3 acl,user_xattr,noatime 1 1” >> /etc/fstab

Step 2, configure syslog-ng.conf

Now we’re ready to actually configure the vMA syslog daemon via the configuration file /etc/syslog-ng/syslog-ng.conf . This might not be trivial if you’re not familiar with it like I was and it took me some time to get it working to the point where I was satisfied, though the config may look horrible, it works out for me. This page and this one helped me figuring most stuff out. Config hints courtesy of vGhetto guru William Lam too.
The complete config file I used can be found here.
For comparision, this is the original unchanged syslog-ng.conf.

I’ll go through a couple of important config file sections now. The only existing thing I changed and didn’t add were the global options defined at the top:

# Global options.
 #
 options { keep_hostname(yes); long_hostnames(off); use_fqdn(yes); sync(0); perm(0644); dir_perm(0755); stats(3600); };

perm(0644); dir_perm(0755) : This will make the logs readable by non-root users (vi-admin) too for convenience sake. You may want to keep the defaults here.
Besides this, none of the existing lines were changed and I only added new lines at the end of the config file:

source esxihosts { udp(ip("0.0.0.0") port(514)); }; #listen on ordinary UDP connections

You could log everything into one large file for each host (“messages”), but I split the stream of logs similar to how the they are stored on the hosts with one logfile per server and component. This requires adding a section like the following for each component:

########## Definitions for each logfile ##############
 destination vpxa_log {
  file("/var/log/esxi-syslog/$HOST/$YEAR-$MONTH/vpxa-$YEAR-$MONTH-$DAY"
  create_dirs(yes) frac-digits(3)
  template("$ISODATE $PROGRAM $MSGONLY\n")
  template_escape(no)
  );
 };
 filter vpxa_filter { match("Vpxa") and not match("verbose"); }; #don't blow up the log with verbose boring crap
 log { source(esxihosts); filter(vpxa_filter); destination(vpxa_log); };

Each such section consists of:
– a destination directive in which file to log (this will result in files like /var/log/esxi-syslog/esxihost01.local/2012-10/vpxa-2012-10-02)
– a filter directive in which I define what the log message should match or not match against
– a log directive which ties the previously defined listener to the filter and destination, writing the actual log

Especially the hostd log contains a lot of informational or frequent, possibly useless long messages, review your logs and adjust the filter if necessary:

destination hostd_log {
  file("/var/log/esxi-syslog/$HOST/$YEAR-$MONTH/hostd-$YEAR-$MONTH-$DAY"
  create_dirs(yes) frac-digits(3)
  template("$ISODATE $PROGRAM $MSGONLY\n")
  template_escape(no)
  );
 };
 filter hostd_filter { match("Hostd") and not match("Power policy is unset") and not match("Responded to service state request") and not match ("Hbrsvc") and not match ("Default resource used for \'EsxHostAdvSettings") and not match ("convert IP Address of type 0") ; }; #may want to filter verbose junk in general too | Hbrsvc = host based replication service, not even enabled/used but still crapping all over the logs
 log { source(esxihosts); filter(hostd_filter); destination(hostd_log); };

My config file linked above takes care of the vpxa, hostd, fdm, vmkernel, vmkwarning, vobd, esxupdate logs as well as all other log messages not contained in these logs. A new file will be created each day for each host in it’s separate directory:

# ll host01.local/2012-10/*2012-10-04
 -rw-r--r-- 1 root root 22K 2012-10-04 11:17 host01.local/2012-10/esxupdate-2012-10-04
 -rw-r--r-- 1 root root 526K 2012-10-05 01:57 host01.local/2012-10/fdm-2012-10-04
 -rw-r--r-- 1 root root 3.9M 2012-10-05 01:59 host01.local/2012-10/hostd-2012-10-04
 -rw-r--r-- 1 root root 75K 2012-10-05 01:54 host01.local/2012-10/other-2012-10-04
 -rw-r--r-- 1 root root 43K 2012-10-05 01:51 host01.local/2012-10/vmkernel-2012-10-04
 -rw-r--r-- 1 root root 8.8K 2012-10-04 12:04 host01.local/2012-10/vmkwarning-2012-10-04
 -rw-r--r-- 1 root root 28K 2012-10-04 12:13 host01.local/2012-10/vobd-2012-10-04
 -rw-r--r-- 1 root root 2.9M 2012-10-05 01:59 host01.local/2012-10/vpxa-2012-10-04

After editing/replacing the syslog-ng.conf file, restart the syslog daemon for the changes to take effect:
# /etc/init.d/syslog restart
Shutting down syslog services done
Starting syslog services done

Step 3, configure hosts to log to your vMA

Now the only thing left is telling your hosts to use your vMA as their syslog destination. This can be done via esxcli as described in this article or also through setting the advanced setting in thee GUI or via PowerCLI.
On the vMA, where I hope you have your hosts registered with vifastpass, execute this for each host. This will also enable a ESXi local firewall rule to allow outgoing syslog traffic:
# vifptarget -s host01.local
# esxcli –server $server system syslog config set –loghost=’udp://10.10.0.20:514′
# esxcli –server $server network firewall ruleset set –ruleset-id=syslog –enabled=true
# esxcli –server $server system syslog reload

Or just loop through your list of hosts like the following examples:

for host in "host01.local" "host02.local" "host03.local" ; do vifptarget -s $host; esxcli --server $server system syslog config set --loghost='udp://10.10.0.20:514'; esxcli --server $server network firewall ruleset set --ruleset-id=syslog --enabled=true; esxcli --server $server system syslog reload; done


vifp listservers | grep -Po "host[\w\.]+" | sort | while read host; do vifptarget -s $host; esxcli --server $server system syslog config set --loghost='udp://10.10.0.20:514'; esxcli --server $server network firewall ruleset set --ruleset-id=syslog --enabled=true; esxcli --server $server system syslog reload; done

Finally, everything should now be logged to your vMA and neatly available for fast central reviews.
Note that the logs submitted by ESXi-hosts will retain their original timestamps which are in UTC and are not converted to the vMA local timezone or anything.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s