Hi Peoples,
Has anyone else seen issues with ArcGIS Server 10.2.2, specifically in relation to load balancers and cached image services that appear to be missing tiles when returned via REST?
A quick summary of our architecture.
We have two instances of ArcGIS Server 10.2.2 running. We have a load balancer in the DMZ that is configured to communicate with ArcGIS server via our reverse proxies through AJP.
We are using cached image services for our base maps, which include an aerial photo layer, topographic base map, and scanned images of our topographic map series.
The Problem:
If we connect directly to the REST end points of our ArcGIS Servers, and view in ArcGIS JavaScript, the services work fine. No problem.
If we connect via the Load Balancer to the REST end point, as we zoom in and pan around, tiles are missing. If we turn Fiddler on, we obtain a Protocol Violation Report, with Content-Length mismatch errors.
We did not see this in ArcGIS Server 10.1.
Further investigation
Looking further into this, we discovered the content-length header is extremely important. If it is present and non-zero, the container assumes that the request has a body (a POST request, for example), and immediately reads a separate packet off the input stream to get that body. There appears to be a bug fix for a security flaw in the 7.0.47 release of Tomcat specifically to do with the Content-Length.
It could be a coincidence, but ArcGIS Server 10.2.2 ships with Tomcat 7.0.47, while 10.1 ships with 7.0.27.
This issue seems to be specific to the AJP protocol. If we swap to using HTTP, the issue seems to be fixed. We could do this, but that is going to cause us some grief with DMZ, fire walls, existing systems etc.
We think this is due to the version of Tomcat that ships with ArcGIS Server 10.2.2.
Anyone else seen this?
Anyone know which version of Tomcat 10.3 ships with?
References:
http://tomcat.apache.org/security-7.html#Fixed_in_Apache_Tomcat_7.0.47
https://confluence.atlassian.com/display/CONFKB/How+to+Determine+Your+Version+of+Tomcat+and+Java
http://tomcat.apache.org/connectors-doc/ajp/ajpv13a.html
http://www.wellho.net/mouth/1549_http-https-and-ajp-comparison-and-choice.html
Cheers,
Mark
