webgisdr not creating backups in us-west-2 region

[GirishYadav1]

Hi,

To fulfill the disaster recovery requirements. I have a deployment of ArcGIS Enterprise 10.6 on us-east-1 and us-west-2 regions. Also setup a cron job to create nightly backups of ArcGIS enterprise using webgisdr utility into the S3 buckets in respective regions. It works just fine in us-east-1 region. But in us-west-2 region it fails with following error:

==================================================

Starting the WebGIS DR utility.

==================================================

Unable to get the S3 endpoint from the specified region us-west-2.

Check to make sure us-west-2 is a valid region.

Exiting the WebGIS DR utility.

I have two S3 buckets deployed in both regions us-east-1 & us-west-2 with cross region replication. In east region ArcGIS Enterprise writes backups to us-east-1 and west region ArcGIS Enterprise deployment writes to us-west-2 region.

From ArcGIS Enterprise EC2 instances in us-west-2 region, I can access s3 buckets through aws s3 cli. 

Any thoughts?

[JonathanQuinn]

Are you using access keys or an IAMRole? Do you have a regionsforcloudstorage.dat file under <install dir>\framework\etc?

[GirishYadav1]

Using IAMRole. And IAMRole assigned to the instance has full access to S3. However, I DO NOT see "regionsforcloudstorage.dat" file under <portal install dir>/framework/etc. Following are the files inside etc folder:

root@arcgis-ent-qawest-blue:# ls /arcgis/portal/framework/etc/

actions.sh                     config-store-connection.json

agsportal.sh                   database-config.properties

arcenv                         NodeAgentExt.xml

arcgis-framework.properties    pids

arcgis-logsettings.json        portal-config.properties

arcgis-logsettings.json.rlock  portal-ha-config.properties

arcgisportal                   RestoreJre.txt

arcgisportal.service           service-logging.properties

arcgis-rmi.properties

Note: The portal content store is NOT S3. Content store is in EFS. 

[JonathanQuinn]

Ah, so you're trying to store the backup in S3, not the content? I guess that's irrelevant, it's all the same.

It's fine if you don't see it, we look at that file if we can't get the endpoint for the region you provided. Can the Portal make outbound connections to http://arcgisstore105.s3.amazonaws.com/regionsforcloudstorage.dat? If not, then you may need to download that file and move it into the framework/etc folder.

[GirishYadav1]

No Luck...

The Portal machine can not make outbound connections to http://arcgisstore105.s3.amazonaws.com/regionsforcloudstorage.dat. I downloaded the file and put inside framework/etc folder but still getting the same error. 

Also, I am using ArcGIS 10.6 therefore, downloaded http://arcgisstore106.s3.amazonaws.com/regions.dat. put it inside framework/etc. still same error. I also downloaded the http://arcgisstore106.s3.amazonaws.com/endpoints.tar.gz file and put it in the dame directory. Still no success. 

Any suggestions?

[GirishYadav1]

After further testing, I found that the Portal machine can access http://arcgisstore105.s3.amazonaws.com/regionsforcloudstorage.dat through proxy. I tested with setting up proxy in the Ec2 and able to wget this url endpoint and it worked. However, The ArcGIS webgisdr is still throwing the same error. Does webgisdr respect the proxy settings in the machine? Is there a separate proxy setting for webgisdr?

[JonathanQuinn]

We embed that file in the DR tool and Portal. Placing it there yourself is really meant for users connecting to non-default regions. I was just testing with the us-west-2 region myself and didn't see any problems, so I'm not too sure what's going on. Just to be sure, Site 2 which is in us-west-2 is attempting to save the backup to a bucket in us-west-2?

[GirishYadav1]

Yes, the site in us-west-2 region is attempting to save backup in a bucket in the same region. There is no VPC endpoint for cross region bucket access (i.e. the EC2 in us-west-2 do not have direct access to a bucket hosted in us-east-1 region. Access to the bucket in us-east-1 is only through proxy). Therefore, this url  http://arcgisstore105.s3.amazonaws.com/regionsforcloudstorage.dat (from bucket in us-east-1) is only accessible to the ArcGIS Portal machine through proxy. Is this the reason why the webgisdr utility is throwing this error? Is there a way to set proxy for webgisdr?

As a workaround, I can create a script and set it as a cron job to backup locally and upload it to the S3 bucket using aws S3 cli.