Offline areas management packaging failed issue

[FraserHand]

Hi, 

We are running ArcGIS Enterprise 10.8 in Azure behind App GW. We have a base deployment and a federated GIS Server. Both GIS servers have contexts of /server (base deployment) and /server1 (federted GIS server).

Server1 has a dynamic map service with imagery which is cached.

When managing offline areas and attempting to create a offline map service image cache, we can see the output map cache get created in the e:\arcgisserver\directories\arcgisoutput\aerialmapimageservice directory (on server1).

However the packaging fails. We see this in the portal logs:

Item 'd4ff17894c764c4ea5c6bab170dc38d9' added from URL 'https://.../arcgis/rest/directories/arcgisoutput/../Aerial_Imagery_MapServer/_ags1282724663_ch/Map.tpk?token='.

and

Failed to add item 'd4ff17894c764c4ea5c6bab170dc38d9' from URL 'https://.../arcgis/rest/directories/arcgisoutput/.../Aerial_Imagery_MapServer/_ags1282724663_ch/Map.tpk?token=b4sE98g0vDiOCWgiftcu6F7ieOqL4KiOZhPdqxsgPZMm6whuBnpKfVt_oiBD-zlFvqOipQ9rmkb4ZxY09u6G949kc-NmNT4hSYuY5nqo8Qg3rbz4A2yMCBocARuSKaTV3XK3kMwQQ2N6rzX_XnrLVHmz-mGMxplESHZh6w5-In6-WZw28NGoTMv8jejoFSHJo7dNQiMDloJZgOUyrwVoLbkqdkkocx-L-J98UvdFmf7jG9zbhkvCKdWuwIDMT30C'. Data length is zero

Which is expected as the server context is /server1/ not /arcgis/

Following the requests going into the submitJob for the SetUpMapArea everything looks right and the outputs in the jobs directory (result etc) all have the right urls in it. So it looks like the Offline packaging tools are creating everything as expected but something somewhere is writing arcgis instead of portal.

Has anyone had a similar issue?

Jonathan Quinn‌

Trevor Hart‌

Scott Tansley‌

Thanks

Fraser

EDIT: So it looks like there is a bug in the REST API

The tpk works for the map service and is available in the output directory - it looks like the wrong url is returned by a call to the REST API.

When this URL is called

https://myurl/server1/rest/directories/arcgisoutput/service/Aerial_Imagery_MapServer/_ags1066105292_...

the API returns this:

{
folders: [ ],
files: [
{
name: "Map.tpk",
url: "https://myurl/arcgis/rest/directories/arcgisoutput/service/Aerial_Imagery_MapServer/_ags1066105292_ch/Map.tpk"
}
]
}

So the response being written seems to be ignoring the WebContextURL and defaulting to ArcGIS.

This is easily patched in the Offline Areas tools but can't be patched for collector as Collector calls that url directly.

Thanks

Bruce Harold‌ - Seen this before Bruce? I'm assuming the calls to /arcgisoutput/someservice/..MapServer/_ags<ran>_ch are handled by the API 

[JonathanQuinn]

If it's only OfflinePackaging that's a problem, then it does seem like it's something specific in that workflow. Are you using a web adaptor as well? I'm not familiar with that workflow enough to know what the traffic should look like, but it seems as though Server will tell Portal what URL to use in the item when it's creating it. Does the admin URL use "arcgis", or do they both use "server1"?

[FraserHand]

Hi Jon,

It's just not offline packaging but also Collector when trying to download a map for offline - as this also calls the https://myurl/server1/rest/directories/arcgisoutput/service/Aerial_Imagery_MapServer/_ags1066105292_...f=json url to get the tpk for the cached map service.

The deployment pattern is pretty much what Cloud Builder 10.8 (Azure) uses.

Portal web context url and private portal url are the same - pointing to external dns

Federation is external dns of both servers (hosted server and GIS server)

e.g. 

https://myurl/server/

https://myurl/server1/

Security config for the GIS servers has external DNS settings for portalUrl, privatePortalUrl. serverUrl and privatehostingserverUrl

not a /arcgis/ to be seen anywhere in the configuration of any of the 3 servers.

Web context url set correctly on both GIS servers.

Thanks!

Fraser

Jonathan Quinn‌

[JonathanQuinn]

Collector is probably looking at the body of the request, and the body contains the URL to download whatever it needs. If specific workflows are the only ones causing a problem and the rest of the workflows work as expected, then it looks like it could be a bug with those tools and utilities. I'd suggest reaching out to Support and have them test to see if they can repro.

[FraserHand]

Hi Jon,

I have found a solution to this. I kept circling back to the web adaptors as this was the only difference between a site that the offline packaging was working and the Azure site which wasn't. The App gateway sets X-Forwarded-Host as described here: Configure your portal to use a reverse proxy server—Portal for ArcGIS (10.8) | Documentation for Arc... - however when looking at the web adaptor you can see that it sets further custom headers when handling requests mainly:

X-Forwarded-Url-Base: web adaptor context

X-Forwarded-Request-Context: this appears to be the same as the web context url setting

webAdaptorID: web adaptor id

X-Forwarded-For: an external ip

I added these to my App GW and the offline packaging began working - including being able to create offline areas in Collector.  Note I could see the response from the GIS Server change from something like 

https://external url/arcgis/rest/directories/arcgisoutput/Test/Imagery_MapServer/_ags62661154_ch/Map...

to 

https://external url/design/rest/directories/arcgisoutput/Test/Imagery_MapServer/_ags62661154_ch/Map...

So it looks like the GIS Server uses the X-Forwarded-Request-Context to write the response rather than the WebContextURL system setting. It appears that if the X-Forwarded-Request-Context header isn't set then it falls back to /arcgis/ - I don't know if this is intended or not but - I had a quick look through the online documentation and couldn't find these headers documented. I naturally assumed that the GIS Server would use the WebContextURL setting for any responses (including responses in GP results or API calls)  before any custom headers as not all users use web adaptors - and in the Esri Azure cloud automation tools web adaptors aren't used - and these additional headers aren't set in the App GW configuration (at 10.8 anyway).

Bruce Harold‌ - looping you in as you were tagged in the original post

Trevor Hart‌ Scott Tansley‌

[JonathanQuinn]

I can repro this behavior; I suggest you reach out to Support to get this tested and logged. Server should only need to depend on the X-Forwarded-Host header, as documented in the help.

[Jeff_G · Accepted solution]

Support has logged the following bug for this issue:

BUG-000132479: The OfflinePackaging service output creates an incorrect URL if ArcGIS Enterprise is deployed using ArcGIS Enterprise Cloud Builder for Microsoft Azure.

Fraser Hand‌ Jonathan Quinn