The internet is now (no really, this time it is, pinky-promise) officially broken™. Or at least a large part of it that is responsible for securely handing your shopping cart and credit card information, e-mails, passwords, router configuration or whatever flows through your tubes over encrypted SSL/TLS connections.
Since yesterday there’s a big fuss on the internet over the so-called openssl heartbleed bug that was disclosed yesterday. It exploits the TLS heartbeat extensions and allows an attacker to read up to 64KiB of memory managed by openssl from the target system. This memory could contain sensitive data such as passwords, session-cookies, credit card information, or in one of the worst cases even the server’s x509 certificate private key. Since the attack is basically untraceable (no webserver logs or anything), it’s entirely possible that certificate private keys have been compromised en masse already without anyone noticing and never being able to find out.
You can find some interesting background information about the discovery and disclosure in this article.
To be on the really safe side, you have to replace your certificates, revoking the old ones, changing passwords and generally being PITA’d. Note that this has nothing to do with the openssl version a certificate was generated with, but is about the runtime application using openssl for it’s SSL/TLS traffic. Your private key can be just as compromised if you generated the certificate with Windows CA tools or gnutls and use it with an openssl-based application.
The “good news”, if you can even call them that, is that at least only the newer 1.0.1 branch version of openssl is affected. In total, openssl 1.0.0 and 0.9.8 together are still more widespread than 1.0.1, so the scope is at the very least narrowed down considerably:What versions of the OpenSSL are affected?
Status of different versions:
OpenSSL 1.0.1 through 1.0.1f (inclusive) are vulnerable
OpenSSL 1.0.1g is NOT vulnerable
OpenSSL 1.0.0 branch is NOT vulnerable
OpenSSL 0.9.8 branch is NOT vulnerable
Bug was introduced to OpenSSL in December 2011 and has been out in the wild since OpenSSL release 1.0.1 on 14th of March 2012. OpenSSL 1.0.1g released on 7th of April 2014 fixes the bug.
Most Linux distros have released updated openssl packages already and I updated my Fedora20 and CentOS6 today. The fix for CentOS/RHEL6 is a backport to 1.0.1e, so no worries as long as your package is at least 1.0.1e-16.el6_5.7. Remember to also restart all services (postfix/sensmail/apache/nginx etc…) depending on openssl libraries after updating.
If we can trust these tweets, then big shots like Yahoo or ebay are already confirmed to be vulnerable. Cloudflare apparently had an edge and knew was informed about the vulnerability before it was publicly disclosed. Also this wouldn’t be such a huge (but still fairly big) issue if Perfect Forward Secrecy was is implemented properly.
At the moment there doesn’t seem to be a publicly available tool like the ssllabs test to check whether a target is vulnerable. This site provides a quick test. — So does this site. — SSLlabs now checks for heartbleed as well.
Here’s a perl script for your own testing pleasure supporting STARTTLS as well.
And here we have a heartbleed proof of concept Windows tool.
But this is probably one of the best testing scripts available currently. (Source)
Also be aware that this isn’t quite a Linux-only problem. While schannel, the native Windows cryptographic API is safe against this bug, openssl is used in a huge number of native or ported Windows applications as well.
Is vSphere ESXi affected as well?
VMware published a KB article outlining which VMware products and versions are affected.
VMware finally released a corresponding security advisory on their mailing list as well.
The first update is available for Horizon Workspace Server, while patches for other products are still pending.
VMware just released updates forESXi 5.5 and vCenter 5.5 as well, accompanied with the following KB articles to outline that there are actually two releases of each and how to replace certificates. Make sure you read them carefully:
Resolving OpenSSL Heartbleed for ESXi 5.5 – CVE-2014-0160
Resolving OpenSSL Heartbleed for vCenter Server 5.5
ESXi 5.5 seems vulnerable. I hope VMware will soon release a Security Advisory clearing things up and providing updates for this horrible issue (which isn’t their fault though).
Let’s have a look at an ESXi 5.5 GA (no U1, but doesn’t matter) host:
# vmware -vl VMware ESXi 5.5.0 build-1331820 VMware ESXi 5.5.0 GA # openssl version -a OpenSSL 1.0.1e 11 Feb 2013 built on: Tue Feb 26 16:34:26 PST 2013
Now here’s an up-to-date ESXi 5.1 U2 and an older 5.0 still U1 host:
# vmware -vl VMware ESXi 5.1.0 build-1612806 VMware ESXi 5.1.0 Update 2 # openssl version -a OpenSSL 0.9.8y 5 Feb 2013 built on: Wed Mar 20 20:44:08 PDT 2013 # vmware -vl VMware ESXi 5.0.0 build-821926 VMware ESXi 5.0.0 Update 1 ~ # openssl version -a OpenSSL 0.9.8q 2 Dec 2010 built on: Mon Mar 14 12:16:37 PDT 2011
As you can see, ESXi 5.5 runs the vulnerable openssl 1.0.1 branch. ESXi 5.1 and 5.0 on the other hand are built on the openssl 0.9.8 branch. Hence versions prior to ESXi 5.5 are unaffected.
perl check-ssl-heartbleed.pl MyESXi55host:443 ...ssl received type=22 ver=0x302 ht=0x2 size=50 ...ssl received type=22 ver=0x302 ht=0xb size=1020 ...ssl received type=22 ver=0x302 ht=0xe size=0 ...send heartbeat#1 ...ssl received type=24 ver=302 size=16384 BAD! got 16384 bytes back instead of 3 (vulnerable) perl check-ssl-heartbleed.pl MyESXi55host:5989 ...ssl received type=22 ver=0x302 ht=0x2 size=54 ...ssl received type=22 ver=0x302 ht=0xb size=1020 ...ssl received type=22 ver=0x302 ht=0xe size=0 ...send heartbeat#1 ...ssl received type=24 ver=302 size=1638
With the perl script linked above I confirmed that ESXi 5.5 can be successfully attacked with the heartbleed attack. Ouch.
Adding the -s switch to print the received output, I can see the content of some XML file that must be loaded in the hosts memory.
The problem will extend to Linux-based virtual appliances by VMware (or whatever vendor for that matter) as well. I have an older vMA 5.1 virtual appliance which is unaffected but I’m not sure with which openssl recent versions of the VCSA/vMA etc come:
# cat /etc/vma-release vMA 5.1.0 BUILD-1062361 # cat /etc/SuSE-release SUSE Linux Enterprise Server 11 (x86_64) VERSION = 11 PATCHLEVEL = 2 # openssl version -a OpenSSL 1.0.0c 2 Dec 2010
Even the Windows-based vCenter server appears to rely on openssl to some extent, at least the Inventory Service. My 5.1 U2 vCenter seems safe though:
"C:\Program Files\VMware\Infrastructure\Inventory Service\bin\openssl.exe" version -a OpenSSL 0.9.8y 5 Feb 2013 built on: Tue Feb 12 23:38:08 2013
The same can’t be said for vCenter 5.5, which comes with two different openssl binaries:
"C:\Program Files\VMware\CIS\openSSL\openssl.exe" version -a OpenSSL 1.0.1e 11 Feb 2013 built on: Tue Feb 12 19:37:08 2013 "C:\Program Files\VMware\Infrastructure\Inventory Service\bin\openssl.exe" version -a OpenSSL 0.9.8y 5 Feb 2013 built on: Tue Feb 12 23:38:08 2013
I can’t really tell what this “CIS” component is supposed to do, but I haven’t been able to reproduce the issue with any of the ports vCenter is listening on yet. I tested a few of the available heartbleed scripts against Windows-based vCenter 5.5 and 5.1 on all ports the system is listening on (including Web Client 9443, Inventory 10443, SSO 7444 etc) but they were never reported being vulnerable. I suppose this is because the actual SSL traffic is handled in the Java application’s own SSL stack instead of depending on openssl, which might only be used for certain operations such as certificate generation.
In any case, this demonstrates yet again how important it is to keep your ESXi host management network on a firewalled, non-public secure network.
Apart from the VMware vSphere components I also checked some other SSL-based applications running in our infrastructure with the scripts (this is just my own testing with the available proof of concept tools, mind you):
HP has published a support notice outlining their affected and unaffected products. In short, they acknowledge that recent versions of HP SMH, Blade OA and SUM are vulnerable.
- HP Systems Management Homepages (SMH) on physical Windows and Linux (tested with SMH v7.3.1, v18.104.22.168, v22.214.171.124, v126.96.36.199) servers are affected. (It seems like you can fix this manually (at least on Linux) by overwriting the openssl files that ship with the SMH, see comments)
HP released fixed SMH versions, see the comment by Casper42.
- HP BladeSystem Onboard Administrator Web Interface (tested with v4.11) is affected.
Fixed with HP OA 4.12/4.21.
- HP Virtual Connect Web Interface (tested with v3.70 and v4.10) is not affected.
- HP ILO interfaces (tested recent firmware versions of ILO2, ILO3 and ILO4) don’t seem vulnerable. They apparently
run an older openssl versiondon’t rely on OpenSSL and don’t support the exploited TLS Heartbeat feature to begin with. However, it appears you can DoS ILO2 interfaces with the heartbleed scripts for some reason. See the comments and The HP Community thread here. An ILO2 BETA update to prevent the DoS was posted in the thread as well.
Fixed with ILO2 Firmware 2.25.
- Check Point Products are unaffected.
- There is a huge list of other 3rd party products and services affected. I suggest you contact your vendors if they haven’t published information yet.