TMG “502 Proxy Error. The data is invalid” while downloading Windows 8.1/2012R2 Update KB2919355

Recently I ran into an odd issue when trying to download the KB2919355 Update bundle from Microsoft. The problem affected WSUS and local Windows Update clients that used TMG as their explicit proxy and was reproducible with browsers as well:
The TMG proxy threw 502 Proxy Error ( The data is invalid.  ) when trying to access various different hosts and URLs serving the same update file such as these:
http://au.v4.download.windowsupdate.com/d/msdownload/update/software/crup/2014/02/windows8.1-kb2919355-x64_66955196a82751d1c8d9806d321487562b159f41.psf
http://fg.v4.download.windowsupdate.com/d/msdownload/update/software/crup/2014/02/windows8.1-kb2919355-x64_66955196a82751d1c8d9806d321487562b159f41.psf
http://wsus.ds.download.windowsupdate.com/d/msdownload/update/software/crup/2014/02/windows8.1-kb2919355-x64_66955196a82751d1c8d9806d321487562b159f41.psf

Strange enough the problem did not occur in transparent proxy mode.

I first suspected the problem was related to the issue described here, but in my case neither HTTP compression nor chunked transfer-encoding are used, thus this article and its explanation do not apply.

First let’s have a look at a normal response (bypassing the Proxy/transparent mode). We can see the file is approximately 3.8GiB in size, quite big but I’ve downloaded larger files without issues. There are no strange HTTP headers or anything sent by the servers (goes for GETs as well) and everything looks just fine:

 $ curl -I 'http://wsus.ds.download.windowsupdate.com/d/msdownload/update/software/crup/2014/02/windows8.1-kb2919355-x64_66955196a82751d1c8d9806d321487562b159f41.psf'
HTTP/1.1 200 OK
Via: 1.1 TMGPROXY01
Connection: Keep-Alive
Proxy-Connection: Keep-Alive
Content-Length: 4052160113
Date: Mon, 13 Oct 2014 11:12:31 GMT
Content-Type: application/octet-stream
ETag: "0cabe2eb931cf1:0"
Server: Microsoft-IIS/7.5
Accept-Ranges: bytes
Last-Modified: Mon, 24 Feb 2014 23:36:04 GMT
X-Powered-By: ASP.NET
X-CCC: IT
X-CID: 2

Continue reading

Squid and chunked transfer encoding shenanigans

A while ago I was troubleshooting an issue with a certain client software (that I don’t even know the name of) on an internal network, that was uploading images via HTTP to a remote service on the internet. The simple HTTP POST upload requests failed for some reason with a HTTP/1.0 501 Not Implemented error from our Squid proxy server, prompting me to analyze further.
Fortunately the I had a network trace from the local admins so I could easily reproduce and test the requests with curl from my Linux client.

The issue proved to be caused to the chunked Transfer-Encoding headers of the POSTs and how our Squid 3.1 proxy server is unable to process requests with this Transfer-Encoding in certain cases.

I’ve seen a few mentions of issues with chunked Transfer-Encoding and Squid on the internets but many were older, contradictory or referencing Server side responses and not client side requests like in this case.

In a nutshell, Squid 3.1 is just not fully HTTP/1.1 compatible yet and only partially supports some features like chunked encoding as stated on the project’s website:

Squid-3.2 claims HTTP/1.1 support. Squid v3.1 claims HTTP/1.1 support but only in sent requests (from Squid to servers). Earlier Squid versions do not claim HTTP/1.1 support by default because they cannot fully handle Expect:100-continue, 1xx responses, and/or chunked messages.
Both Squid-3 and Squid-2 contain at least response chunked decoding. The chunked encoding portion is available from Squid-3.2 on all traffic except CONNECT requests.

Continue reading

Squid Expect: 100-continue header issues

As I mentioned on my other post about tuning Squid, this article is about the ignore_expect_100 setting and why it’s set on our Squid proxies.

We had and issue where requests for a certain site failed with an HTTP 417 error generated by Squid. You would find entries like these in the Squid access.log for the site  that wasn’t accessible:
1329478677.344      0 192.168.1.22 NONE/417 4480 POST http://somewebsite.com/webservice/data.asmx – NONE/- text/html
(Note: It’s actually an application communicating via HTTP and not manual browser-based access by human beings.)

Turned out the problem is that this application sends requests with an “Expect: 100-continue” HTTP-header and Squid doesn’t have a proper implementation of the HTTP 1.1 Expect-mechanism.

The purpose of the 100 (Continue) status (see section 10.1.1) is to allow a client that is sending a request message with a request body to determine if the origin server is willing to accept the request (based on the request headers) before the client sends the request body. In some cases, it might either be inappropriate or highly inefficient for the client to send the body if the server will reject the message without looking at the body.

In layman’s terms, this means if a client that wants to send a request (with content) to a webserver , e.g. upload a file with an HTTP-POST, it can first ask “Hey webserver, I’m about to POST you a file, here are the HTTP-Headers of my request. If that’s all right with you, send me a 100-continue response and I’ll transmit the file.” (kind of similar to an HTTP-HEAD , which is just an exchange of headers too). Now why is this useful? For example, it allows a webserver to check the Content-Type, Content-Length or any other header of the request before the client attempts to send any actual data. If the webserver doesn’t accept this MIME-type, decides the file is too large or anything else, it can simply respond with a 417 Expectation Failed and no resources would have been wasted by sending a file the web application wouldn’t process anyways.

Continue reading