Missing tiles from cached server : Content-lenght mismatch : AJP and load balancer

[MarkChilcott]

Hi Peoples,

Has anyone else seen issues with ArcGIS Server 10.2.2, specifically in relation to load balancers and cached image services that appear to be missing tiles when returned via REST?

A quick summary of our architecture.

We have two instances of ArcGIS Server 10.2.2 running.  We have a load balancer in the DMZ that is configured to communicate with ArcGIS server via our reverse proxies through AJP.

We are using cached image services for our base maps, which include an aerial photo layer, topographic base map, and scanned images of our topographic map series.

The Problem:

If we connect directly to the REST end points of our ArcGIS Servers, and view in ArcGIS JavaScript, the services work fine.  No problem.

If we connect via the Load Balancer to the REST end point, as we zoom in and pan around, tiles are missing.  If we turn Fiddler on, we obtain a Protocol Violation Report, with Content-Length mismatch errors.

We did not see this in ArcGIS Server 10.1.

Further investigation

Looking further into this, we discovered the content-length header is extremely important. If it is present and non-zero, the container assumes that the request has a body (a POST request, for example), and immediately reads a separate packet off the input stream to get that body.  There appears to be a bug fix for a security flaw in the 7.0.47 release of Tomcat specifically to do with the Content-Length.

It could be a coincidence, but ArcGIS Server 10.2.2 ships with Tomcat 7.0.47, while 10.1 ships with 7.0.27.

This issue seems to be specific to the AJP protocol.  If we swap to using HTTP, the issue seems to be fixed.  We could do this, but that is going to cause us some grief with DMZ, fire walls, existing systems etc.

We think this is due to the version of Tomcat that ships with ArcGIS Server 10.2.2.

Anyone else seen this? 

Anyone know which version of Tomcat 10.3 ships with?

References:

http://tomcat.apache.org/security-7.html#Fixed_in_Apache_Tomcat_7.0.47

https://confluence.atlassian.com/display/CONFKB/How+to+Determine+Your+Version+of+Tomcat+and+Java

http://tomcat.apache.org/connectors-doc/ajp/ajpv13a.html

http://www.wellho.net/mouth/1549_http-https-and-ajp-comparison-and-choice.html

Cheers,

Mark

[MarkChilcott]

Hi Peoples,

I installed ArcGIS Server 10.3 Pre-release, which ships with Tomcat 7.0.54.

The problem is still there.  More noticeable in Firefox.  Does not appear so much in Chrome or IE.

I shall document, report to support, and if I find anything, post to let people know the outcome.

Out of the box, ArcGIS Server uses the HTTP connector (not the AJP), so this is why people are probably not seeing this as an issue.  Our ICT staff are keen to continue using AJP (but alas they may have to change).

Cheers,

Mark