webgisdr: Unable to get the S3 endpoint from the specified region ap-east-1

Question

Hi,

Environment:

* Esri ArcGIS Enterprise 10.8.0 AMI (on Windows)
* All-in-one (standalone) deployment (ap-east-1 region) using Cloudbuilder
* Portal content directory on S3 (ap-east-1 region)

TLDR: How does webgisdr map S3_REGION to the respective S3 endpoint? Is there a local file override?

We're doing the preparation work to upgrade our ArcGIS Enterprise 10.8.0 instance to 10.8.1. 
We're at the "backup your instance" step.

We created a webgisdr.properties file and ran webgisdr with --configuration which returned
   "The WebGIS DR utility completed successfully".

As a double-check, we temporarily set S3_REGION to a well-known region (us-east-1) where we don't have data, then ran webgisdr --configuration and webgisdr --export and both returned the appropriate response.
So we think our webgisdr.properties file is good.

We ran webgisdr --export (with S3_REGION = ap-east-1) but this time it reported:

"Unable to get the S3 endpoint from the specified region ap-east-1.
    Check to make sure ap-east-1 is a valid region."

We asked google which brought us to this post. Unfortunately, this post refers to a previous version and seemingly a very different configuration (we don't use proxies). Neither the webgisdr.log (at DEBUG) nor tcpview showed any connection to the hosts containing "regions.dat / endpoints.tar.gz"; we checked the data inside regions.dat and endpoints.tar.gz and did not find any references to ap-east-1.

In the webgisdr logback.xml we set logging level to debug; this line looked suspicious to us:

2021-03-17 09:12:57 DEBUG [main] com.esri.arcgis.webgis.util.WebGISUtil - The response for the url https://_____.com/server/admin/backupRestoreInfo: {"code":500,"messages":["Error while processing request. Internal error. JSON response not set."],"status":"error"}

We checked /server/admin/backupRestoreInfo with a browser and it returned a blank page; This is very first time we're running webgisdr, and we're not really sure what this means.

How does webgisdr map S3_REGION to the respective S3 endpoint? Is there a local file override?

Any thoughts? Did we miss anything?

Many thanks!

dky · Answer

Hi, quick status update ...

Support says it's a bug ... the regionsforcloudstorage.dat file apparently needs more info ... and has provided guidance to update the dat file as follows:

{
   "name": "Hong Kong",
   "id": "ap-east-1",
   "s3endpoint": "s3.ap-east-1.amazonaws.com"
},

I haven't tested this yet but will do so in a few days and update.

Thanks,
Derek

dky · Answer

Thanks @JonathanQuinn !

We added ap-east-1 to regionsforcloudstorage.dat file and placed it in <portal-install>\framework\etc but got the same error.

Hypothesis: Let's assume that webgisdr was able to resolve the S3 endpoint for ap-east-1 and perhaps the problem is somewhere further down the stack [See note 1]?

Any chance that the AWS java libraries used by webgisdr maybe needs updating?

From the S3 endpoint documentation, https://docs.aws.amazon.com/general/latest/gr/s3.html,
ap-east-1 (Hong Kong) only supports "Version 4 Signatures" (whatever this means [See note 2]). 
The other related case (link) mentioned us-west-2 (which supports both v2 and v4 signatures) ...

Any thoughts appreciated!

NOTES:

[1] I'm making this assumption because there is a lib directory in webgisdr and there are aws-java-sdk-*.jar files; i have no facts to back this up yet).

[2] I am by no means an expert with S3 and don't play one on TV.

dky · Answer

Hi, Update from our after-hours test ...

- we re-ran webgisdr --export with an updated "regionsforcloudstorage.dat" file that includes ap-east-1 and placed it into "C:\Program Files\ArcGIS\Portal\framework\etc\regionsforcloudstorage.dat"

- We got this regionsforcloudstorage.dat file from: http://arcgisstore105.s3.amazonaws.com/regionsforcloudstorage.dat,

No dice; Same error ...

Unable to get the S3 endpoint from the specified region ap-east-1.
   Check to make sure ap-east-1 is a valid region.