webgisdr: Unable to get the S3 endpoint from the specified region ap-east-1

[dky]

Hi,

Environment:

• Esri ArcGIS Enterprise 10.8.0 AMI (on Windows)
• All-in-one (standalone) deployment (ap-east-1 region) using Cloudbuilder
• Portal content directory on S3 (ap-east-1 region)

TLDR: How does webgisdr map S3_REGION to the respective S3 endpoint? Is there a local file override? 

We're doing the preparation work to upgrade our ArcGIS Enterprise 10.8.0 instance to 10.8.1. 
We're at the "backup your instance" step.

We created a webgisdr.properties file and ran webgisdr with --configuration which returned
   "The WebGIS DR utility completed successfully".

As a double-check, we temporarily set S3_REGION to a well-known region (us-east-1) where we don't have data, then ran webgisdr --configuration and webgisdr --export and both returned the appropriate response.
So we think our webgisdr.properties file is good. 

We ran webgisdr --export (with S3_REGION = ap-east-1) but this time it reported: 

   "Unable to get the S3 endpoint from the specified region ap-east-1.
    Check to make sure ap-east-1 is a valid region."

We asked google which brought us to this post. Unfortunately, this post refers to a previous version and seemingly a very different configuration (we don't use proxies). Neither the webgisdr.log (at DEBUG) nor tcpview showed any connection to the hosts containing "regions.dat / endpoints.tar.gz"; we checked the data inside regions.dat and endpoints.tar.gz and did not find any references to ap-east-1.

In the webgisdr logback.xml we set logging level to debug; this line looked suspicious to us: 

2021-03-17 09:12:57 DEBUG [main] com.esri.arcgis.webgis.util.WebGISUtil - The response for the url https://_____.com/server/admin/backupRestoreInfo: {"code":500,"messages":["Error while processing request. Internal error. JSON response not set."],"status":"error"}

We checked /server/admin/backupRestoreInfo with a browser and it returned a blank page; This is very first time we're running webgisdr, and we're not really sure what this means.

How does webgisdr map S3_REGION to the respective S3 endpoint? Is there a local file override? 

Any thoughts? Did we miss anything?

Many thanks!

[dky]

Hi, an update: 

   - re-read the post, specifically this reply ... 

   - we re-ran webgisdr --export with procmon and noticed that there was a local file operation for "C:\Program Files\ArcGIS\Portal\framework\etc\regionsforcloudstorage.dat"

   - the latest available regionsforcloudstorage.dat file is in http://arcgisstore105.s3.amazonaws.com/regionsforcloudstorage.dat, 

   - so we got that and added a new block for "ap-east-1" and threw it in "C:\Program Files\ArcGIS\Portal\framework\etc\"

   - but after a bit more reading, we're going to run webgisdr --export after business hours ... 

[dky]

Hi, Update from our after-hours test ...

  - we re-ran webgisdr --export with an updated "regionsforcloudstorage.dat" file that includes ap-east-1 and placed it into "C:\Program Files\ArcGIS\Portal\framework\etc\regionsforcloudstorage.dat" 

   - We got this regionsforcloudstorage.dat file from: http://arcgisstore105.s3.amazonaws.com/regionsforcloudstorage.dat, 

No dice; Same error ...

   Unable to get the S3 endpoint from the specified region ap-east-1.
   Check to make sure ap-east-1 is a valid region.

[JonathanQuinn]

What you're doing is the correct approach; we externalized the regions information in the event there are regions you want to use that have been newly added, updated, or not covered yet. It should work once you update the file and make sure it's under the framework\etc folder.

[dky]

Thanks @JonathanQuinn !

We added ap-east-1 to regionsforcloudstorage.dat file and placed it in <portal-install>\framework\etc but got the same error.

Hypothesis: Let's assume that webgisdr was able to resolve the S3 endpoint for ap-east-1 and perhaps the problem is somewhere further down the stack [See note 1]?

Any chance that the AWS java libraries used by webgisdr maybe needs updating?

From the S3 endpoint documentation, https://docs.aws.amazon.com/general/latest/gr/s3.html,
ap-east-1 (Hong Kong) only supports "Version 4 Signatures" (whatever this means [See note 2]). 
The other related case (link) mentioned us-west-2 (which supports both v2 and v4 signatures) ... 

Any thoughts appreciated!

NOTES:

[1] I'm making this assumption because there is a lib directory in webgisdr and there are aws-java-sdk-*.jar files; i have no facts to back this up yet). 

[2] I am by no means an expert with S3 and don't play one on TV.

[dky]

Hi, quick status update ...

Support says it's a bug ... the regionsforcloudstorage.dat file apparently needs more info ... and has provided guidance to update the dat file as follows:

{
   "name": "Hong Kong",
   "id": "ap-east-1",
   "s3endpoint": "s3.ap-east-1.amazonaws.com"
},

I haven't tested this yet but will do so in a few days and update. 

Thanks,
Derek