TMG “502 Proxy Error. The data is invalid” while downloading Windows 8.1/2012R2 Update KB2919355

Recently I ran into an odd issue when trying to download the KB2919355 Update bundle from Microsoft. The problem affected WSUS and local Windows Update clients that used TMG as their explicit proxy and was reproducible with browsers as well:
The TMG proxy threw 502 Proxy Error ( The data is invalid.  ) when trying to access various different hosts and URLs serving the same update file such as these:
http://au.v4.download.windowsupdate.com/d/msdownload/update/software/crup/2014/02/windows8.1-kb2919355-x64_66955196a82751d1c8d9806d321487562b159f41.psf
http://fg.v4.download.windowsupdate.com/d/msdownload/update/software/crup/2014/02/windows8.1-kb2919355-x64_66955196a82751d1c8d9806d321487562b159f41.psf
http://wsus.ds.download.windowsupdate.com/d/msdownload/update/software/crup/2014/02/windows8.1-kb2919355-x64_66955196a82751d1c8d9806d321487562b159f41.psf

Strange enough the problem did not occur in transparent proxy mode.

I first suspected the problem was related to the issue described here, but in my case neither HTTP compression nor chunked transfer-encoding are used, thus this article and its explanation do not apply.

First let’s have a look at a normal response (bypassing the Proxy/transparent mode). We can see the file is approximately 3.8GiB in size, quite big but I’ve downloaded larger files without issues. There are no strange HTTP headers or anything sent by the servers (goes for GETs as well) and everything looks just fine:

 $ curl -I 'http://wsus.ds.download.windowsupdate.com/d/msdownload/update/software/crup/2014/02/windows8.1-kb2919355-x64_66955196a82751d1c8d9806d321487562b159f41.psf'
HTTP/1.1 200 OK
Via: 1.1 TMGPROXY01
Connection: Keep-Alive
Proxy-Connection: Keep-Alive
Content-Length: 4052160113
Date: Mon, 13 Oct 2014 11:12:31 GMT
Content-Type: application/octet-stream
ETag: "0cabe2eb931cf1:0"
Server: Microsoft-IIS/7.5
Accept-Ranges: bytes
Last-Modified: Mon, 24 Feb 2014 23:36:04 GMT
X-Powered-By: ASP.NET
X-CCC: IT
X-CID: 2

Now an identical request with the TMG as explicit proxy. The error was consistently reproducible on multiple TMG Nodes with HTTP GETs and HEADs as well:

$ curl -I  'http://wsus.ds.download.windowsupdate.com/d/msdownload/update/software/crup/2014/02/windows8.1-kb2919355-x64_66955196a82751d1c8d9806d321487562b159f41.psf' \
--proxy TMGPROXY01:8080
HTTP/1.1 502 Proxy Error ( The data is invalid.  )
Via: 1.1 TMGPROXY01
Connection: close
Proxy-Connection: close
Pragma: no-cache
Cache-Control: no-cache
Content-Type: text/html
Content-Length: 4028

[…]
Error Code: 502 Proxy Error. The data is invalid. (13)
[…]

I relatively quickly found a way to bypass the issue, and that was telling TMG to not serve the content from cache via the ‘Cache-Control: no-cache’ HTTP directive (or a similar one). This would always result in a successful request:

$ curl -I  'http://wsus.ds.download.windowsupdate.com/d/msdownload/update/software/crup/2014/02/windows8.1-kb2919355-x64_66955196a82751d1c8d9806d321487562b159f41.psf' \
--proxy TMGPROXY01:80 \
--header 'Cache-Control: no-cache'
HTTP/1.1 200 OK
Via: 1.0 TMGPROXY01
Connection: Keep-Alive
Proxy-Connection: Keep-Alive
Content-Length: 4052160113
Date: Tue, 16 Sep 2014 10:33:17 GMT
Content-Type: application/octet-stream
ETag: "0cabe2eb931cf1:0"
Server: Microsoft-IIS/7.5
Accept-Ranges: bytes
Last-Modified: Mon, 24 Feb 2014 23:36:04 GMT
X-Powered-By: ASP.NET
X-CCC: AT
X-CID: 2

So we can establish that something has to be wrong with the file in the cache or something. I took a look at the TMG web cache with the cachdir tool (an auto-scrolling nightmare) and there was indeed an entry in the cache:

So I deleted this cache entry and tested again. A few HTTP HEADs via curl -I worked fine and I tried to download it with a GET, which also went fine. However, after that GET all subsequent HEADs or GETs would produce the exact same 502 Proxy Error ( The data is invalid.  )  error.I deleted the cache entry again, same behaviour. Tried the procedure with one of the other download URLs, same result as well.
TMG would continue to cache the URL in an invalid state. Every time.

In order to bypass this behavior I added a custom Cache Rule in TMG, instructing it to never cache URLs larger than 2GiB from the TMG-builtin Microsoft Update Sites domain name set (make sure you put this rule before the original builtin Microsoft Update Cache Rule):

With this workaround, I was able to always successfully download the file consistently without issues.

Since this pretty much confirmed a bug in the TMG caching mechanism I opened a ticket at Microsoft for this issue. The engineer was able to reproduce the issue and confirmed it’s caused by a bug inside TMG:
Apparently the TMG web cache is only able to handle files up to 2 GiB in size and this discrepancy resulted in the error when it tried to access the cached file.
As to why it didn’t happen with other large (cacheable) files, or why TMG still inserted the file into its cache in the first place even though the Content-Length HTTP header clearly states the file was larger, is beyond me though.

The end of the story is that Microsoft will not fix this issue, for now at least, since it’s not critical and an easy workaround is available.
It might be incorporated into a future TMG Rollup, but I can only guess at this point. TMG 2010 SP2 Rollup 5 was released just in July 2014, so it’s not too unlikely that Microsoft will release another rollup bundle before TMG reaches its official end of mainstream support in April 2015.