topic Re: Multiple Reverse Proxies in ArcGIS Enterprise Questions

Multiple Reverse Proxies

StuartFletcher1 — Thu, 13 Sep 2012 01:28:35 GMT

Hi there,

We have an ArcGIS Server 10.1 instance that sits behind 2 corporate reverse proxies (Apache). Basically in this configuration:

ArcGIS Server ---Reverse Proxy 2----   Internal Site   ----Reverse Proxy 1----    External Site

The reverse proxies we are using utilise Apache and are configured based on the info below

Reverse Proxy 2 configuration:
ProxyPass       /arcgis/rest/ http://arcgisserver_address:6080/arcgis/rest/
ProxyPassReverse /arcgis/rest/ http:// arcgisserver_address:6080/arcgis/rest/
ProxyPass       /arcgis/tokens/ http://arcgisserver_address:6080/arcgis/tokens/
ProxyPassReverse /arcgis/tokens/ http://arcgisserver_address:6080/arcgis/tokens/
ProxyPass       /arcgis/services/ http://arcgisserver_address:6080/arcgis/services/
ProxyPassReverse /arcgis/services/ http://arcgisserver_address:6080/arcgis/services/
ProxyPass       /arcgis/sdk/ http://arcgisserver_address:6080/arcgis/services/
ProxyPassReverse /arcgis/sdk/ http://arcgisserver_address:6080/arcgis/services/

Reverse Proxy 1 configuration:
ProxyPass       / http://internalsite_address/
ProxyPassReverse / http://internalsite_address/

This largely works ok however some components of the ArcGIS server REST endpoint do not function correctly as it appears that some of the java libraries used by the pages appear to pickup the fact that the request has been parsed through via a reverse proxy and utilise one of the following HTTP headers X-FORWARDED_HOST or X-FORWARDED-SERVER. This works ok when there is only 1 proxy however when there are two (or more) proxies these headers contain a comma separated list of all proxies.

I.e in the case of the above example:
X-FORWARDED-HOST = externalsite_address, internalsite_address

This causes issues with many rest pages that when returned have malformed URLs as they contain multiple comma separated domains including the javascript map preview, the wmts capabilities and the links at the top of all rest pages (to name a few).

This can be fixed partially by modifying some of the JSP files used however it is not possible with some such as the the page you are redirected to after login. Fixing the JSPs involves converting the urls to just the absolute path not a fully qualified url. This is not an ideal solution and will cause maintenance issues in the future.

The best overall solution would be for ESRI to look at changing the behavior of some of the classes used including:
com.esri.discovery.app.util.DiscoveryUrlUtil
com.esri.client.app.util.ClientUrlUtil
com.esri.rf.util.RUtil

I wouldn't have thought we are the only organisations using a chain of reverse proxies. Has any one else experienced these problems and if so how have you dealt with them.

Thanks,

Stuart Fletcher

Re: Multiple Reverse Proxies

StuartFletcher1 — Wed, 03 Oct 2012 03:12:52 GMT

I had a response from Esri in relation to this issue. It can be fixed by updating an ArcGIS parameter called WebContextURL .

See:
http://resources.arcgis.com/en/help/server-admin-api/serverProperties.html

- Go to: http://arcgisserver_address:port/arcgis/admin/system/properties
- Click on "Update"

- In the open window put the following:
{"WebContextURL": "http://externalsite_address/arcgis"}

- Click Update.

This should also let you have different REST paths i.e /myserver/rest/services

Cheers,

Stuart Fletcher

Re: Multiple Reverse Proxies

StuartFletcher1 — Sun, 12 Dec 2021 03:51:28 GMT

An update on the issue:

ESRI have updated their 10.1 documentation to reflect the above solution:

http://resources.arcgis.com/en/help/main/10.1/index.html#//01550000042s000000

However it doesn't look like all components of ArcGIS Server honor the WebContextURL parameter. One example is publishing a service for Mobile Data Access. ArcMobile accesses the services via SOAP and the SOAP WSDL still returns invalid soap address with both reverse proxy addresses. See fragment of the WSDL below:

<service name="Catalog">
<port name="ServiceCatalogPort" binding="e:ServiceCatalogBinding">
<soap:address location="http://developext.thelist.tas.gov.au, develop2.thelist.tas.gov.au/arcgis/services/BaseMaps/Topographic/MobileServer"/>
</port>
</service>

Regards,

Stuart Fletcher

Re: Multiple Reverse Proxies

RaviNarayanan — Tue, 16 Oct 2012 15:48:03 GMT

Hi Stuart,

Thank you for posting this message. We can reproduce the reported behavior with multiple reverse proxies:
1. Some of the links in the Services Directory are incorrect.
2. rest/info resource returns both X-Forwarded-* host information in the url returned for SOAP and Token Services.
3. In wsdl soap:address location returns both X-Forwarded-* host information in the url.

We have submitted a bug (NIM085692) and will address this in the next service pack (10.1 SP2).

Ravi

Re: Multiple Reverse Proxies

StephanKünster — Tue, 28 May 2013 10:31:14 GMT

Ravi,

is there a workaround existing to solve this problem?
Could we edit settings directly in the wsdl files of ArcGIS Server?

Cheers,
Stephan

Re: Multiple Reverse Proxies

DavidRush — Mon, 03 Jun 2013 14:02:58 GMT

We're experiencing the exact same problem. Glad to hear it will be addressed in 10.1 SP2.

Can you provide any more details on how it's going to be addressed? Will it strip down the comma-separated values from X-Forwarded-Host and use only one value (looks like using only the first value is the right thing to do, at least in my case).

I'm really pleased that AGS 10.1 is trying to be smart about returned URLs in JSON (and other) responses - it's a bit improvement over the "virtual directories" that had to be hand-built in 10.0 and earlier. I just wish that (1) it was better documented (I discovered this behavior entirely accidentally) and (2) was just a little bit smarter about the comma-separated values.

The WebContextURL isn't a good solution for us since it appears to be server-wide, not per-service.

David

Re: Multiple Reverse Proxies

JoanneMcGraw — Thu, 13 Jun 2013 18:32:18 GMT

Ravi,

Similar to David Rush, I'm very interested in hearing more details regarding how the problem described in this thread will be addressed? "Will it strip down the comma-separated values from X-Forwarded-Host and use only one value (looks like using only the first value is the right thing to do, at least in my case)" (and in ours, btw) Or, is ESRI simply planning on making the WebContextURL property honoured more thoroughly? Will the fix address the issue in SOAP WSDL's URLs only (NIM085692)? Or, will it be a generic fix that will deal with any component that is currently utilizing the X-Forwarded host information and parse it accordingly when comma-delimited?

The reason I ask is because we are having a similar problem to that described in this thread (that is, routing traffic through multiple proxy servers). Our configuration is slightly more complicated in that there is one more reverse proxy in the loop than has been described. Using Stuart's original example, our "Reverse Proxy 2" routes requests to a Web Adaptor (another proxy) for load balancing multiple ArcGIS Servers rather than pointing directly to an individual ArcGIS Server as I'm guessing his example does. In our scenario, setting the WebContextURL property hasn't helped us at all...not even in the REST pages where we are unable to view our services in "ArcGIS JavaScript" (from the REST HTML page, I mean).

Although I searched the forum about this issue last week, I did not find this thread for some reason and created a new one describing my own situation more fully. If it's of any help in terms of ensuring the planned "fix" will handle multiple "multiple-proxy" scenarios, here's a link to a description of another one: forums.arcgis.com/threads/86264-quot-View-In-ArcGIS-JavaScript-quot-in-HTML-fails-when-routing-through-multiple-proxy-servers.

So, has an approach for addressing the multiple proxy routing problem been determined? Or is ESRI still ruminating on that one?

Cheers,
jtm

Re: Multiple Reverse Proxies

PeterKovalchuk — Fri, 14 Jun 2013 13:40:19 GMT

I think there is nothing to fix in ArcGIS Server. X-FORWARDED-HOST should have only last HOST name, not 2 hosts separated by comma. Multiple Windows IIS Reverse proxies does not have this problem at all (due to the fact that X-FORWARDED-HOST have only one, last, HOST name).

Re: Multiple Reverse Proxies

JoanneMcGraw — Fri, 14 Jun 2013 15:58:24 GMT

Unfortunately, this does not seem to be the case with Apache; unless someone is aware of some way to disable this behaviour when working with proxies there?

Cheere,
jtm

Re: Multiple Reverse Proxies

Sanderten_Arve1 — Tue, 16 Jul 2013 12:55:01 GMT

Hi, I ran into this same issue running two reverse proxies with Apache. After some digging in the Apache documentation I found the following solution:

Architecture:
client --> ApacheReverseProxy1 --> ApacheReverseProxy2 --> ArcGIS Server

When just following the reverse proxy directions you can view the "double X-Forwarded-Host" issue using two reverse proxies within a Wireshark trace at the client:

[ATTACH=CONFIG]25958[/ATTACH]

The solution is to use/load the Apache "headers_module" and unset the X-Forwarded-Host header on the ApacheReverseProxy2

So add the following line in the httpd.conf file on the ApacheReverseProxy2:
LoadModule headers_module /usr/lib/apache2-prefork/mod_headers.so

And add the following line in the virtual host file of your ApacheReverseProxy2 website YourVirtualHost.conf:
RequestHeader unset X-Forwarded-Host

(Your Apache setup might be slightly different but you get the idea)

If you now run a new Wireshark trace from the client you will see that the double host entry is removed and your AGS will work as expected.

[ATTACH=CONFIG]25959[/ATTACH]

Hope this helps
Sander

Re: Multiple Reverse Proxies

JoanneMcGraw — Tue, 16 Jul 2013 14:18:09 GMT

Sander,

Thank you for your response. It's been a while since I was working on this (a month seems like forever these days) but, if I understand you properly, I tried what you are suggesting at that time and it did not work given our particular architecture. From what I remember, unset-ting the X-Forwarded-Host meant that the ArcGIS Server was creating a path something like (using your example) http://ApacheReverseProxy2/rest/services/... which is not an Apache server that the Internet has direct access to. It is forbidden. For us, the path has to be http://ApacheReverseProxy1/rest/services in that case.

When I get a moment, I will try this again to verify what I am remembering is correct ... perhaps I just made a mistake somewhere along the way before. I was trying a LOT of things at the time.

I'll let you know how it goes...
jtm

Re: Multiple Reverse Proxies

JoanneMcGraw — Tue, 16 Jul 2013 15:21:39 GMT

Okay, yes, I am remembering this correctly. I appreciate your taking the time to make the suggestion, Sander, and hope it can help others. In our environment, unfortunately, it doesn't solve the problem because we forbid direct access to ApacheReverseProxy2.

FYI, in the interim, we have received approval to add another RewriteRule on the organization's proxy server that directs the /rest/services traffic directly to the Web Adaptor host rather than through our application-specific host. They are not happy about it and they want it removed. However, there does not appear to be any other way around the problem with ArcGIS Server 10.1. We are waiting to see what the "fix" is that Ravi mentioned previously and to learn if it handles Apache's implementation to set values for the X-Forwarded-Host header key (when going through a proxy) rather than just IIS's.

Cheers,
jtm