Multiple Reverse Proxies

[StuartFletcher1]

Hi there,

We have an ArcGIS Server 10.1 instance that sits behind 2 corporate reverse proxies (Apache). Basically in this configuration:

ArcGIS Server  ---Reverse Proxy 2----   Internal Site   ----Reverse Proxy 1----    External Site

The reverse proxies we are using utilise Apache and are configured based on the info below

Reverse Proxy 2 configuration:
ProxyPass       /arcgis/rest/  http://arcgisserver_address:6080/arcgis/rest/
ProxyPassReverse /arcgis/rest/  http:// arcgisserver_address:6080/arcgis/rest/
ProxyPass       /arcgis/tokens/  http://arcgisserver_address:6080/arcgis/tokens/
ProxyPassReverse /arcgis/tokens/  http://arcgisserver_address:6080/arcgis/tokens/
ProxyPass       /arcgis/services/  http://arcgisserver_address:6080/arcgis/services/
ProxyPassReverse /arcgis/services/  http://arcgisserver_address:6080/arcgis/services/
ProxyPass       /arcgis/sdk/  http://arcgisserver_address:6080/arcgis/services/
ProxyPassReverse /arcgis/sdk/  http://arcgisserver_address:6080/arcgis/services/


Reverse Proxy 1 configuration:
ProxyPass       /  http://internalsite_address/
ProxyPassReverse  /  http://internalsite_address/

This largely works ok however some components of the ArcGIS server REST endpoint do not function correctly as it appears that some of the java libraries used by the pages appear to pickup the fact that the request has been parsed through via a reverse proxy and utilise one of the following HTTP headers X-FORWARDED_HOST or X-FORWARDED-SERVER. This works ok when there is only 1 proxy however when there are two (or more) proxies these headers contain a comma separated list of all proxies.

I.e in the case of the above example:
X-FORWARDED-HOST = externalsite_address, internalsite_address

This causes issues with many rest pages that when returned have malformed URLs as they contain multiple comma separated domains including the javascript map preview, the wmts capabilities and the links at the top of all rest pages (to name a few).

This can be fixed partially by modifying some of the JSP files used however it is not possible with some such as the the page you are redirected to after login. Fixing the JSPs involves converting the urls to just the absolute path not a fully qualified url. This is not an ideal solution and will cause maintenance issues in the future.

The best overall solution would be for ESRI to look at changing the behavior of some of the classes used including:
com.esri.discovery.app.util.DiscoveryUrlUtil
com.esri.client.app.util.ClientUrlUtil
com.esri.rf.util.RUtil


I wouldn't have thought we are the only organisations using a chain of reverse proxies. Has any one else experienced these problems and if so how have you dealt with them.

Thanks,

Stuart Fletcher

[StuartFletcher1]

I had a response from Esri in relation to this issue. It can be fixed by updating an ArcGIS parameter  called WebContextURL .

See:
http://resources.arcgis.com/en/help/server-admin-api/serverProperties.html

- Go to: http://arcgisserver_address:port/arcgis/admin/system/properties
- Click on "Update"

- In the open window put the following:
{"WebContextURL": "http://externalsite_address/arcgis"}

- Click Update.

This should also let you have different REST paths i.e /myserver/rest/services

Cheers,

Stuart Fletcher

[StuartFletcher1]

An update on the issue:

ESRI have updated their 10.1 documentation to reflect the above solution:

http://resources.arcgis.com/en/help/main/10.1/index.html#//01550000042s000000

However it doesn't look like all components of ArcGIS Server honor the WebContextURL parameter. One example is publishing a service for Mobile Data Access. ArcMobile accesses the services via SOAP and the SOAP WSDL still returns invalid soap address with both reverse proxy addresses. See fragment of the WSDL below:

<service name="Catalog">
<port name="ServiceCatalogPort" binding="e:ServiceCatalogBinding">
<soap:address location="http://developext.thelist.tas.gov.au, develop2.thelist.tas.gov.au/arcgis/services/BaseMaps/Topographic/MobileServer"/>
</port>
</service>

Regards,

Stuart Fletcher

[RaviNarayanan]

Hi Stuart,

Thank you for posting this message. We can reproduce the reported behavior with multiple reverse proxies:
1. Some of the links  in the Services Directory are incorrect.
2. rest/info resource returns both X-Forwarded-* host information in the url returned for SOAP and Token Services.
3. In wsdl  soap:address location returns both X-Forwarded-* host information in the url.

We have submitted a bug (NIM085692) and will address this in the next service pack (10.1 SP2).

Ravi

[StephanKünster]

Ravi,

is there a workaround existing to solve this problem?
Could we edit settings directly in the wsdl files of ArcGIS Server?

Cheers,
Stephan

[DavidRush]

We're experiencing the exact same problem.  Glad to hear it will be addressed in 10.1 SP2.

Can you provide any more details on how it's going to be addressed?  Will it strip down the comma-separated values from X-Forwarded-Host and use only one value (looks like using only the first value is the right thing to do, at least in my case).

I'm really pleased that AGS 10.1 is trying to be smart about returned URLs in JSON (and other) responses - it's a bit improvement over the "virtual directories" that had to be hand-built in 10.0 and earlier.  I just wish that (1) it was better documented (I discovered this behavior entirely accidentally) and (2) was just a little bit smarter about the comma-separated values.

The WebContextURL isn't a good solution for us since it appears to be server-wide, not per-service.

David

[JoanneMcGraw]

Ravi,

Similar to David Rush, I'm very interested in hearing more details regarding how the problem described in this thread will be addressed? "Will it strip down the comma-separated values from X-Forwarded-Host and use only one value (looks like using only the first value is the right thing to do, at least in my case)" (and in ours, btw) Or, is ESRI simply planning on making the WebContextURL property honoured more thoroughly? Will the fix address the issue in SOAP WSDL's URLs only (NIM085692)? Or, will it be a generic fix that will deal with any component that is currently utilizing the X-Forwarded host information and parse it accordingly when comma-delimited?

The reason I ask is because we are having a similar problem to that described in this thread (that is, routing traffic through multiple proxy servers). Our configuration is slightly more complicated in that there is one more reverse proxy in the loop than has been described. Using Stuart's original example, our "Reverse Proxy 2" routes requests to a Web Adaptor (another proxy) for load balancing multiple ArcGIS Servers rather than pointing directly to an individual ArcGIS Server as I'm guessing his example does. In our scenario, setting the WebContextURL property hasn't helped us at all...not even in the REST pages where we are unable to view our services in "ArcGIS JavaScript" (from the REST HTML page, I mean).

Although I searched the forum about this issue last week, I did not find this thread for some reason and created a new one describing my own situation more fully. If it's of any help in terms of ensuring the planned "fix" will handle multiple "multiple-proxy" scenarios, here's a link to a description of another one: forums.arcgis.com/threads/86264-quot-View-In-ArcGIS-JavaScript-quot-in-HTML-fails-when-routing-through-multiple-proxy-servers.

So, has an approach for addressing the multiple proxy routing problem been determined? Or is ESRI still ruminating on that one?

Cheers,
jtm

[PeterKovalchuk]

I think there is nothing to fix in ArcGIS Server. X-FORWARDED-HOST should have only last HOST name, not 2 hosts separated by comma. Multiple Windows IIS Reverse proxies does not have this problem at all (due to the fact that X-FORWARDED-HOST have only one, last,  HOST name).

[JoanneMcGraw]

Unfortunately, this does not seem to be the case with Apache; unless someone is aware of some way to disable this behaviour when working with proxies there?

Cheere,
jtm

[Sanderten_Arve1]

Hi, I ran into this same issue running two reverse proxies with Apache. After some digging in the Apache documentation I found the following solution:

Architecture:
client --> ApacheReverseProxy1 --> ApacheReverseProxy2 --> ArcGIS Server

When just following the reverse proxy directions you can view the "double X-Forwarded-Host" issue using two reverse proxies within a Wireshark trace at the client:

[ATTACH=CONFIG]25958[/ATTACH]

The solution is to use/load the Apache "headers_module" and unset the X-Forwarded-Host header on the ApacheReverseProxy2

So add the following line in the httpd.conf file on the ApacheReverseProxy2:
LoadModule headers_module /usr/lib/apache2-prefork/mod_headers.so

And add the following line in the virtual host file of your ApacheReverseProxy2 website YourVirtualHost.conf:
RequestHeader unset X-Forwarded-Host

(Your Apache setup might be slightly different but you get the idea)

If you now run a new Wireshark trace from the client you will see that the double host entry is removed and your AGS will work as expected.

[ATTACH=CONFIG]25959[/ATTACH]

Hope this helps
Sander