[Script] Network routing/failover topology change detection

A while ago I wrote a simple but useful script which I’m sharing here to detect upstream provider HSRP failover events via traceroute. It can be used for all kinds of virtual IP routing failover like VRRP, Check Point Cluster XL, actual routing protocols like BGP/OSPF or similar technologies where IP packets can be routed across multiple hops.
The script executes traceroutes to a given destination and checks whether the path is being routed over a certain hop, with the ability to send mail notifications if this is not the case.

You can get the most recent version of this script on my Github here. If you have any suggestions or improvements (which I’m sure there is plenty of room for), feel free to drop a comment or an issue or a pull-request on Github.

Continue reading

[Script] Extending Linux LVM partitions

Here’s a script I wrote a while a go to extend LVM partitions on Linux machines.

The script assumes that you have extended the existing underlying physical (or “virtual” if it’s a VM) storage device prior to execution. It will rescan the disks (skip with -f), resize the existing partition (basically just setting a different end sector), reboot, and run scripts to extend the actual file system after the reboot. There are other ways to extend the disk space including creating a new partition on the additional disk space, but I’ve decided against that approach in favor of a single-partition scheme for management/simplicity’s sake.

This script will work with VMs and physical servers alike. I’ve tested it with RHEL 6/7 and CentOS 6/7, but it should generally work with other Linux distributions as well.

You can get the most recent version of this script on Github here. If you have any suggestions or improvements (which I’m sure there is plenty of room for), feel free to drop a comment or an issue or a pull-request on Github.

Continue reading

SSL POODLE Attack – What is SCSV and how does it help?

The POODLE vulnerability is currently the hot topic in the security world. Here is a nice technical overview by the Google SSL Guru. POODLE relies on SSLv3, but today nearly every server and client supports at least TLS 1.0 in addition to SSLv3, which means SSLv3 connections should (and in fact are) be rather rare. But there is still a threat because of downgrade compatibility between the protocols. With POODLE, a rather new SSL/TLS enhancement to mitigate protocol downgrade attacks has been mentioned a lot recently.

[Update 2014/12/09]
Recent developments show that the POODLE attack, which originally was only thought to be applicable CBC cipher-suites in SSLv3 connections, can in some cases be extended to TLS1+ (ALL TLS versions) connections as well. This time however, it is not due to a general protocol weakness in TLS, but because of specific flawed software implementations that just ported certain functions from their SSLv3 code over to TLS, without considering the improved CBC padding specifications of TLS.
Currently known to be affected are load balancers from F5 and A10, as well as Cisco devices , some Check point Firewall components and IBM HTTP Server.

TLS Fallback Signaling Cipher Suite Value prevents SSL/TLS protocol downgrades a man-in-the-middle can enforce when both sides actually support higher protocol versions.
For example, a Client sends a Client Hello handshake message that indicates support for TLS 1.2. Normally the server responds with its own highest protocol version in the Server Hello handshake message, and that will be the negotiated SSL/TLS version of the connection. If the server supports TLS 1.2 as well, the connection will use this protocol. If the server supports a lower version, the connection will use this instead (provided the client supports it as well of course).
Now an attacker who is able to intercept and alter traffic between the systems can screw up the handshake process and make it fail with arbitrary network errors (think: TCP RST). In this case the client often retries the connection with a lower initial SSL/TLS protocol version, say SSLv3 or TLS1, which makes it easier for the bad guy to attack the encrypted channel.

To prevent this kind of protocol downgrade attack, TLS Fallback SCSV was developed. Here’s a short overview of how it works:
Both, the client and server need to support it to make it work. TLS Fallback SCSV is used as a signaling cipher suite (TLS_FALLBACK_SCSV, value 0x5600) during the handshake. A signaling cipher suite does not provide actual encryption algorithms like other, “normal” cipher suites such as “TLS_RSA_WITH_AES_128_CBC_SHA” or “TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256or others. Instead they are used as simple protocol controls, similar to TLS extensions. A client can include the “TLS_FALLBACK_SCSV” cipher suite in his Client Hello message to tell the server, that the connection should only be established if the highest protocol version supported by the server is identical to or lower than that of what it sees in the Client Hello. So if the server supports TLS 1.2 but the Client Hello uses SSL3, TLS 1.0 or TLS 1.1, then the server should respond with an “inappropriate fallback”  TLS alert message.
The nitty gritty details of SCSV can be read in the IETF draft.
Note that TLS Fallback SCSV really only helps against protocol downgrade attacks, and does not actually prevent the POODLE attack by itself. POODLE is a vulnerability in SSL3, and SCSV makes it harder for the attacker to downgrade a connection to it in order to exploit that vulnerability.

Here is an example of a failed handshake between a TLS 1.2 server supporting SCSV and a client using TLS 1.1 supporting SCSV:downgrade

Wireshark doesn’t know about this cipher suite yet so it lists it as unknown. The TLS_EMPTY_RENEGOTIATION_INFO_SCSV” signaling cipher suite is a similar safety guard to prevent an older session renegotiation vulnerability.

In contrast, this handshake completes normally with SCSV since the client supports TLS 1.2:sslnormal

You can test whether a server supports SCSV with a recent openssl version and the -fallback_scsv option. For this to work you must tell openssl to use an older protocol version than what the server actually supports via the -ssl3, -tls1 or -tls1_1 flags. If the server supports SCSV, it will terminate the connection with a TLS alert.
Here’s an example with google.com and a Google mailserver, which both support TLS 1.2 and SCSV:

$ openssl s_client -connect google.com:443 -state -fallback_scsv -tls1_1
3072415468:error:1409443E:SSL routines:SSL3_READ_BYTES:tlsv1 alert inappropriate fallback:s3_pkt.c:1257:SSL alert number 86
3072415468:error:1409E0E5:SSL routines:SSL3_WRITE_BYTES:ssl handshake failure:s3_pkt.c:596

$ openssl s_client -connect aspmx.l.google.com.:25 -state -fallback_scsv -starttls smtp -tls1_1 
3072067308:error:1409443E:SSL routines:SSL3_READ_BYTES:tlsv1 alert inappropriate fallback:s3_pkt.c:1257:SSL alert number 86
3072067308:error:1409E0E5:SSL routines:SSL3_WRITE_BYTES:ssl handshake failure:s3_pkt.c:596:

Changing the above to use TLS 1.2 instead will result in successful connections:
$ openssl s_client -connect google.com:443 -state -fallback_scsv -tls1_2
$ openssl s_client -connect aspmx.l.google.com.:25 -state -fallback_scsv -starttls smtp -tls1_2

The SSL Labs online test will also tell you if your server supports SCSV.

If you use an Apache webserver with mod_ssl and one of the recent openssl versions, it should already support SCSV by default. The same goes for my postfix mailserver that depends on openssl and supports SCSV out of the box. I tested this on my own CentOS with openssl-1.0.1e-30.el6_5.2.i686. (Remember, you should restart the services after updating openssl.)

Currently (as of October 2014), of the 3 major browsers only Chrome supports SCSV. Firefox is supposed to add support with version 35, IE does not support it and I’m not aware of any concrete roadmap to include it.

In the meantime, you should seriously consider disabling obsolete SSLv3 on the client and server side of your networks. Firefox will disable SSLv3 by default in Firefox 34 (release November 2014) and Google plans to follow in the next months with Chrome.
SSLv3 is officially dead now.

This is what such an error looks like to the user in Firefox 34 (ESR), when the browser receives a TLS alert from the web server:

Squid and chunked transfer encoding shenanigans

A while ago I was troubleshooting an issue with a certain client software (that I don’t even know the name of) on an internal network, that was uploading images via HTTP to a remote service on the internet. The simple HTTP POST upload requests failed for some reason with a HTTP/1.0 501 Not Implemented error from our Squid proxy server, prompting me to analyze further.
Fortunately the I had a network trace from the local admins so I could easily reproduce and test the requests with curl from my Linux client.

The issue proved to be caused to the chunked Transfer-Encoding headers of the POSTs and how our Squid 3.1 proxy server is unable to process requests with this Transfer-Encoding in certain cases.

I’ve seen a few mentions of issues with chunked Transfer-Encoding and Squid on the internets but many were older, contradictory or referencing Server side responses and not client side requests like in this case.

In a nutshell, Squid 3.1 is just not fully HTTP/1.1 compatible yet and only partially supports some features like chunked encoding as stated on the project’s website:

Squid-3.2 claims HTTP/1.1 support. Squid v3.1 claims HTTP/1.1 support but only in sent requests (from Squid to servers). Earlier Squid versions do not claim HTTP/1.1 support by default because they cannot fully handle Expect:100-continue, 1xx responses, and/or chunked messages.
Both Squid-3 and Squid-2 contain at least response chunked decoding. The chunked encoding portion is available from Squid-3.2 on all traffic except CONNECT requests.

Continue reading

openssl heartbleed attack – The cryptocalypitc judgement day has arrived

The internet is now (no really, this time it is, pinky-promise) officially broken™. Or at least a large part of it that is responsible for securely handing your shopping cart and credit card information, e-mails, passwords, router configuration or whatever flows through your tubes over encrypted SSL/TLS connections.

That escalated quickly.Since yesterday there’s a big fuss on the internet over the so-called openssl heartbleed bug that was disclosed yesterday. It exploits the TLS heartbeat extensions and allows an attacker to read up to 64KiB of memory managed by openssl from the target system. This memory could contain sensitive data such as passwords, session-cookies, credit card information, or in one of the worst cases even the server’s x509 certificate private key. Since the attack is basically untraceable (no webserver logs or anything), it’s entirely possible that certificate private keys have been compromised en masse already without anyone noticing and never being able to find out.
You can find some interesting background information about the discovery and disclosure in this article.

To be on the really safe side, you have to replace your certificates, revoking the old ones, changing passwords and generally being PITA’d. Note that this has nothing to do with the openssl version a certificate was generated with, but is about the runtime application using openssl for it’s SSL/TLS traffic. Your private key can be just as compromised if you generated the certificate with Windows CA tools or gnutls and use it with an openssl-based application. Continue reading

nscd DNS caching and postfix

A few of our mail gateway servers running with postfix/policyd-weight/amavis/spamassaisin generate a lot of DNS queries to our DNS servers at times.
I’m not particularly concerned about that myself but there were some discussions about whether we should or how we could decrease the volume of DNS queries.

One suggestion for an easy and convenient solution was to just install the Name Service Cache Daemon (nscd) to cache responses locally on the mail server. They implemented this quickly on one of the servers but it didn’t really seem to work, it still generated loads of queries and the nscd statistics didn’t indicate that caching was working. Also, the statistics output of nscd -g always displayed 0% cache hit ratio and 0 cache hits on positive entries.
So they just as quickly abandoned the idea without digging into it deeper and more or less forgot about the whole plan in general, as it wasn’t like we had any real issues in the first place.
Other options discussed were setting up dedicated caching-only resolvers (onto the hosts themselves) which wouldn’t have been difficult either.

Fast forward a few months and the so called “issue” of too many DNS queries came up again recently and I decided to check nscd myself and why it supposedly wouldn’t work.

Continue reading

iLO Login check script

We recently changed the iLO local account logins in favor of LDAP authentication against our AD, which is cool but raised the issue that sometimes logins seemed to work with my AD account and sometimes not, because not every system was configured for LDAP authentication properly.

Instead of checking logins on dozens of servers manually (with the nice iLO failed login delay), I took a stab at analyzing the login procedures and scripting the logins myself.
So I came up with this horrible piece of bash script doing exactly that. I checked this script with all known iLO versions 1, 2, 3 and 4, and it worked with all of them (the login procedure for versions 1/2 and 3/4 are identical). Running it requires an argument pointing to a file containing the iLO hostnames or IPs to connect to.
Here’s the script on pastebin with formatting: http://pastebin.com/i2Y0xSTQ:

Update: I have moved my scripts to GitHub and updated some of them a bit. You can find the current version of this particular script here.
Continue reading

Getting firmware version information on ESXi and GNU/Linux

You may not always have the convenient option to install vendor-specific hardware management agents/extension on ESXi hosts or physical servers, for example with appliance-ish OSes like the Check Point SPLAT/Gaia platform (which is just a custom RHEL descendant), or you may run into a server without these tools installed. So how can you still query firmware information on such systems directly from the command line? I will outline a couple of ways here which make it possible to obtain that information.
The example information captured here is from HP Proliant Servers (since G5), but most of it should work in similar ways with other hardware platforms too. Unless noted otherwise, the example commands here should work regardless of whether you have CIM providers or hardware management agents installed or not.

Continue reading

Configuring the vMA as ESXi syslog server

An ESXi host has a number of important logfiles located in /var/log (or /scratch/log) for each of it’s components. Reviewing those logs the standard way can be quite cumbersome: Some logs can be viewed through the DCUI (if you like to torture yourself) or you have to enable SSH each time on each host to browse through the logs. Compared to that, having your host-local logs in one central place can be very useful for things like audits or troubleshooting. VMware also noticed that back in vSphere 4 an included the vilogger component directly into the vMA. Unfortunately vilogger was removed with vSphere5 for reasons completely beyond me.
Instead VMware included the ESXi Syslog collector for vCenter, but it’s quite basic and not really the greatest way to centralize your hosts logs, especially in larger environments. I’m also missing my bash and *nix tools to properly filter and navigate through the volume of information contained in the logs.
So as the vMA 5.x Linux OS already comes with the default syslog-ng daemon, why not take advantage of that and let your hosts log there? The steps outlined here were done with the vMA 5.0 but should work without a hitch on the 5.1 release too.

Continue reading

Error loading python dll when launching Anki after wine upgrade

ImageThis is at least the 2nd time it hit and took me some time to figure it out, so I might as well write it down here for future reference.

I was updating (read: breaking and re-compiling) a couple of packages on my good ‘ol Gentoo GNU/Linux Laptop, one of them being wine.
Yes, the platform of choice for running the most non-free software possible. Mr. Stallman does not approve.

I’m actually running the awesome flashcard program Anki in wine as opposed to it’s native GNU/Linux implementation. Why you may ask? Because it relies on many other bloaty-ish packages like QT which I want this system to be clean of. Instead of compiling another 350 MB of sources, running Anki in wine works quite well. I also use the unknown ion3 tiling window manager (RIP) on this system and am quite happy with it.

So anyways, after updating wine to 1.4, starting Anki in wine resulted in an ugly “Error loading python dll” error. I knew I’ve seen and fixed this before and the solution is actually quite simple:
All I needed to do was reinstalling Microsoft Visual C++ 2008 SP1 Redistributable in wine.
I have not the slightest idea how the upgrade manages to break this every time.