openssl heartbleed attack – The cryptocalypitc judgement day has arrived

The internet is now (no really, this time it is, pinky-promise) officially broken™. Or at least a large part of it that is responsible for securely handing your shopping cart and credit card information, e-mails, passwords, router configuration or whatever flows through your tubes over encrypted SSL/TLS connections.

That escalated quickly.Since yesterday there’s a big fuss on the internet over the so-called openssl heartbleed bug that was disclosed yesterday. It exploits the TLS heartbeat extensions and allows an attacker to read up to 64KiB of memory managed by openssl from the target system. This memory could contain sensitive data such as passwords, session-cookies, credit card information, or in one of the worst cases even the server’s x509 certificate private key. Since the attack is basically untraceable (no webserver logs or anything), it’s entirely possible that certificate private keys have been compromised en masse already without anyone noticing and never being able to find out.
You can find some interesting background information about the discovery and disclosure in this article.

To be on the really safe side, you have to replace your certificates, revoking the old ones, changing passwords and generally being PITA’d. Note that this has nothing to do with the openssl version a certificate was generated with, but is about the runtime application using openssl for it’s SSL/TLS traffic. Your private key can be just as compromised if you generated the certificate with Windows CA tools or gnutls and use it with an openssl-based application.

The “good news”, if you can even call them that, is that at least only the newer 1.0.1 branch version of openssl is affected. In total, openssl 1.0.0 and 0.9.8 together are still more widespread than 1.0.1, so the scope is at the very least narrowed down considerably:

What versions of the OpenSSL are affected?
Status of different versions:
    OpenSSL 1.0.1 through 1.0.1f (inclusive) are vulnerable
    OpenSSL 1.0.1g is NOT vulnerable
    OpenSSL 1.0.0 branch is NOT vulnerable
    OpenSSL 0.9.8 branch is NOT vulnerable
Bug was introduced to OpenSSL in December 2011 and has been out in the wild since OpenSSL release 1.0.1 on 14th of March 2012. OpenSSL 1.0.1g released on 7th of April 2014 fixes the bug.

Most Linux distros have released updated openssl packages already and I updated my Fedora20 and CentOS6 today. The fix for CentOS/RHEL6 is a backport to 1.0.1e, so no worries as long as your package is at least 1.0.1e-16.el6_5.7. Remember to also restart all services (postfix/sensmail/apache/nginx etc…) depending on openssl libraries after updating.

If we can trust these tweets, then big shots like Yahoo or ebay are already confirmed to be vulnerable. Cloudflare apparently had an edge and knew was informed about the vulnerability before it was publicly disclosed. Also this wouldn’t be such a huge (but still fairly big) issue if Perfect Forward Secrecy was is implemented properly.
At the moment there doesn’t seem to be a publicly available tool like the ssllabs test to check whether a target is vulnerable. This site provides a quick test.So does this site.SSLlabs now checks for heartbleed as well.
Here’s a perl script for your own testing pleasure supporting STARTTLS as well.
And here we have a heartbleed proof of concept Windows tool.
But this is probably one of the best testing scripts available currently. (Source)

Also be aware that this isn’t quite a Linux-only problem. While schannel, the native Windows cryptographic API is safe against this bug, openssl is used in a huge number of native or ported Windows applications as well.


Is vSphere ESXi affected as well?

[Update 2014/04/09]
VMware published a KB article outlining which VMware products and versions are affected.
[Update 2014/04/15]

VMware finally released a corresponding security advisory on their mailing list as well.
The first update is available for Horizon Workspace Server, while patches for other products are still pending.
[Update 2014/04/19]
VMware just released updates forESXi 5.5 and vCenter 5.5 as well, accompanied with the following KB articles to outline that there are actually two releases of each and how to replace certificates. Make sure you read them carefully:
Resolving OpenSSL Heartbleed for ESXi 5.5 – CVE-2014-0160
Resolving OpenSSL Heartbleed for vCenter Server 5.5


ESXi 5.5 seems vulnerable. I hope VMware will soon release a Security Advisory clearing things up and providing updates for this horrible issue (which isn’t their fault though).

Let’s have a look at an ESXi 5.5 GA (no U1, but doesn’t matter) host:

    # vmware -vl
    VMware ESXi 5.5.0 build-1331820
    VMware ESXi 5.5.0 GA

    # openssl version -a
    OpenSSL 1.0.1e 11 Feb 2013
    built on: Tue Feb 26 16:34:26 PST 2013

Now here’s an up-to-date ESXi 5.1 U2 and an older 5.0 still U1 host:

# vmware -vl
VMware ESXi 5.1.0 build-1612806
VMware ESXi 5.1.0 Update 2
# openssl version -a
OpenSSL 0.9.8y 5 Feb 2013
built on: Wed Mar 20 20:44:08 PDT 2013

# vmware -vl
VMware ESXi 5.0.0 build-821926
VMware ESXi 5.0.0 Update 1
~ # openssl version -a
OpenSSL 0.9.8q 2 Dec 2010
built on: Mon Mar 14 12:16:37 PDT 2011

 As you can see, ESXi 5.5 runs the vulnerable openssl 1.0.1 branch. ESXi 5.1 and 5.0 on the other hand are built on the openssl 0.9.8 branch. Hence versions prior to ESXi 5.5 are unaffected.

perl MyESXi55host:443
...ssl received type=22 ver=0x302 ht=0x2 size=50
...ssl received type=22 ver=0x302 ht=0xb size=1020
...ssl received type=22 ver=0x302 ht=0xe size=0
...send heartbeat#1
...ssl received type=24 ver=302 size=16384
BAD! got 16384 bytes back instead of 3 (vulnerable)

perl MyESXi55host:5989
...ssl received type=22 ver=0x302 ht=0x2 size=54
...ssl received type=22 ver=0x302 ht=0xb size=1020
...ssl received type=22 ver=0x302 ht=0xe size=0
...send heartbeat#1
...ssl received type=24 ver=302 size=1638

With the perl script linked above I confirmed that ESXi 5.5 can be successfully attacked with the heartbleed attack. Ouch.
Adding the -s switch to print the received output, I can see the content of some XML file that must be loaded in the hosts memory.

The problem will extend to Linux-based virtual appliances by VMware (or whatever vendor for that matter) as well. I have an older vMA 5.1 virtual appliance which is unaffected but I’m not sure with which openssl recent versions of the VCSA/vMA etc come:

# cat /etc/vma-release
vMA 5.1.0 BUILD-1062361
# cat /etc/SuSE-release
SUSE Linux Enterprise Server 11 (x86_64)

# openssl version -a
OpenSSL 1.0.0c 2 Dec 2010

Even the Windows-based vCenter server appears to rely on openssl to some extent, at least the Inventory Service. My 5.1 U2 vCenter seems safe though:

"C:\Program Files\VMware\Infrastructure\Inventory Service\bin\openssl.exe" version -a
OpenSSL 0.9.8y 5 Feb 2013
built on: Tue Feb 12 23:38:08 2013

The same can’t be said for vCenter 5.5, which comes with two different openssl binaries:

"C:\Program Files\VMware\CIS\openSSL\openssl.exe" version -a
OpenSSL 1.0.1e 11 Feb 2013
built on: Tue Feb 12 19:37:08 2013

"C:\Program Files\VMware\Infrastructure\Inventory Service\bin\openssl.exe" version -a
OpenSSL 0.9.8y 5 Feb 2013
built on: Tue Feb 12 23:38:08 2013

I can’t really tell what this “CIS” component is supposed to do, but I haven’t been able to reproduce the issue with any of the ports vCenter is listening on yet. I tested a few of the available heartbleed scripts against Windows-based vCenter 5.5 and 5.1 on all ports the system is listening on (including Web Client 9443, Inventory 10443, SSO 7444 etc) but they were never reported being vulnerable. I suppose this is because the actual SSL traffic is handled in the Java application’s own SSL stack instead of depending on openssl, which might only be used for certain operations such as certificate generation.
In any case, this demonstrates yet again how important it is to keep your ESXi host management network on a firewalled, non-public secure network.

Apart from the VMware vSphere components I also checked some other SSL-based applications running in our infrastructure with the scripts (this is just my own testing with the available proof of concept tools, mind you):

[Update 2014/04/14]
HP has published a support notice outlining their affected and unaffected products. In short, they acknowledge that recent versions of HP SMH, Blade OA and SUM are vulnerable.

  • HP Systems Management Homepages (SMH) on physical Windows and Linux (tested with SMH v7.3.1, v7.2.2.9, v7.2.1.3, v7.2.0.14) servers are affected. (It seems like you can fix this manually (at least on Linux) by overwriting the openssl files that ship with the SMH, see comments)
    HP released fixed SMH versions, see the comment by Casper42.
  • HP BladeSystem Onboard Administrator Web Interface (tested with v4.11) is affected.
    Fixed with HP OA 4.12/4.21.
  • HP Virtual Connect Web Interface (tested with v3.70 and v4.10) is not affected.
  • HP ILO interfaces (tested recent firmware versions of ILO2, ILO3 and ILO4) don’t seem vulnerable. They apparently run an older openssl version don’t rely on OpenSSL and don’t support the exploited TLS Heartbeat feature to begin with. However, it appears you can DoS ILO2 interfaces with the heartbleed scripts for some reason. See the comments and The HP Community thread here. An ILO2 BETA update to prevent the DoS was posted in the thread as well.
    Fixed with ILO2 Firmware 2.25.
  • Check Point Products are unaffected.
  • There is a huge list of other 3rd party products and services affected. I suggest you contact your vendors if they haven’t published information yet.

24 thoughts on “openssl heartbleed attack – The cryptocalypitc judgement day has arrived

  1. I’m not sure why people are belittling this issue, I tested my redhat satellite server, and in a couple of minutes had the admin credentials for the web interface giving full control over a huge number of servers… But anyway, do you have any further info on the hp system management homepage, they do not seem to have reacted to the bug atall.

    • It might depend on the SMH version, maybe only the more recent SMH 7.x is affected. I ran the perl script and another python script against a few of my servers, and they all reported being vulnerable except for one. Servers being vulnerable have the following SMH versions:

      The one host reported as ok is running v6.3.0.22. -s
      …ssl received type=22 ver=0x302 ht=0x2 size=62
      …ssl received type=22 ver=0x302 ht=0xb size=974
      …ssl received type=22 ver=0x302 ht=0xc size=327
      …ssl received type=22 ver=0x302 ht=0xe size=0
      …send heartbeat#1
      …ssl received type=24 ver=302 size=16384
      BAD! got 16384 bytes back instead of 3 (vulnerable)

      • Yeah I think anything that we have put in in the last year and a half has been on a vulnerable version. I tried copying the new OpenSSL binary into the hpsmh /opt/hp/hpsmh/bin/openssl directory and restarted, but it still flags as vulnerable so must be in the config and libraries too…

  2. I just upgraded to latest version on a system v 7.3.1-4.x86_64 and the system is vulnerable. So we copied a patched openssl binary and to the hpsmh directory and voila the server is no longer vulnerable.

  3. Do you have any source/documentation on HP ILO impact? Where does the “confirmed” come from for HP products?

    • I haven’t found a public, official statement from HP yet. I said “confirmed” because I “confirmed” this myself in my infrastructure with the named versions by running the heartbleed test scripts against these.

      Likewise, these scripts reported ILO as not vulnerable.
      It seems that ILO2-4 don’t even support the vulnerable Heartbeat TLS extension to begin with (probably still on the openssl 1.0.0 or 0.9.8 branch), so they are naturally not affected by this issue:
      # openssl s_client -connect ilo-interface:443 -tlsextdebug | grep heartbeat
      Does not show the server using the Heartbeat extension, while a server supporting it displays this in the output:
      TLS server extension “heartbeat” (id=15), len=

  4. I have not seen any vulnerable iLO myself. It is only through my own testing with the proof of concept perl and python scripts that I have seen system management homepage to be vulnerable, and then as it does not run as root it seems to not have access to any juicy memory…

    I have not seen any official announcements from HP, our account manager suggested we raise a support call, so I assume there will be a public announcement from them soon…

  5. Are printers and their web based config pages affected? Would they use open SSL? Or say a secure release printer that uses a key fob or card to release a print job?

    • If the printer’s web interface is accessible through HTTPS, it’s quite likely they are using openssl. I have no idea about physical key cards but I would guess it could be irrelevant. You should check some of the proof of concept scripts available against your printer interface. There’s also a Windows tool available here:

      But I actually doubt most printers or other embedded systems are vulnerable. The bug is present in the recent openssl branch that was released 2 years ago. Most embedded software has much, much slower release cycles, sticking to old software branches.

  6. Thow the Ilo2 seems not to be vulnerable the web interface and complete ssh access is killed after testing with the script. And this would be a DOS. So if anybody could test ilo2 and sees the same behavior, then the test itself kills the ilo-interface.

  7. Pingback: MS14-066 / CVE-2014-6321 | alpacapowered

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s