nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Portal for ArcGIS content on S3 compatible storage

NicolasGIS

Hello,

In my private cloud (Openstack), I have a private s3 compatible storage.

So far I was able to successfully use it in 2 situations with ArcGIS Enterprise:

To register it as cloud store for ArcGIS Server cache (it's really easy from the manager interface)
To store webgisdr backups (but documentation is missing: cf. https://community.esri.com/t5/high-availability-and-disaster-recovery-questions/webgisdr-s3-custom-region/m-p/1098726 )

In order to move for an HA deployment, I would like to store Portal for ArcGIS content directory, ArcGIS Server directories and config store on that storage. According to the documentation it's doable

"When ArcGIS Enterprise is deployed in the same cloud platform, you can store system directories such as the portal content directory and ArcGIS Server configuration store in the following cloud storage locations:

Amazon Simple Storage Service (S3) or an S3 compatible storage location"

https://enterprise.arcgis.com/en/cloud/latest/intro/cloud-options.htm

But I can't find how to specify my custom S3 storage. In the example, I found, it is only well known storage that are documented:

https://enterprise.arcgis.com/en/server/latest/cloud/amazon/configure-web-gis-with-shared-content-dir.htm

I tried to upload a json by specifying an S3 URL and creating a regionsforcloudstorage.dat file but I always get a 500 error:

com.esri.arcgis.portal.admin.core.PortalException:

com.esri.arcgis.portal.admin.core.PortalException: Cannot write to the mybucket S3 bucket. Please check that the bucket exists. If access keys are used to connect to the bucket, make sure they are correct. If an IAM role is used to connect to the bucket, make sure that the IAM role has write privileges to the bucket.

Any idea ? Maybe @JonathanQuinn ?

Thanks,

Nicolas

Finde mehr Beiträge mit dem Tag

Accepted answers

All comments

NicolasGIS

Just a bit more details about what I tried:

I tried to specify a JSON a bit in the format of the cloud store registration that is to say by specifying the regionEndpointUrl and defaultEndpointsProtocol just like the following:

{
"type": "cloudStore",
"provider": "Amazon",
"connectionString": {
"accessKeyId": "foo",
"secretAccessKey": "bar",
"region": "company-s3",
"credentialType": "accessKey",
"regionEndpointUrl":"s3.company.com",
"defaultEndpointsProtocol":"https"
},
"objectStore": "mybucket"
}

but got the 500 above.

I tried the same idea as it was suggested for the webgisdr that is to say creating a "regionsforcloudstorage.dat" file in "Portal\framework\etc"

{
"regions": [{
"name": "COMPANY",
"id": "company-s3",
"s3endpoint": "s3.company.com"
}]

}

and specify the following location for my portal content directory:

and got the same 500

ChristopherPawlyszyn

First, ArcGIS Server requires both DynamoDB and S3 when hosted in AWS cloud storage. DynamoDB is the primary store of config-store information while items that exceed the DynamoDB size limits are stored as pointers in the database to S3 objects. This means that mimicking this deployment style on-prem would require a DynamoDB-compatible database as well. DynamoDB Local is not supported for production deployments by AWS, so that may be a non-starter for ArcGIS Server in this deployment pattern.

I did have success with configuring a private S3-compliant object store for Portal for ArcGIS content, but there are a few additional requirements that we will work to incorporate in our documentation.

Additionally, the entries in the regionsforcloudstorage.dat are recognized from the .../Portal/framework/etc/ directory as you surmised, but require virtual-hosted style access to be available.

This article from AWS goes into the differences in virtual-hosted style and path-style routing and the eventual deprecation of the path-style access.

Amazon S3 Path Deprecation Plan – The Rest of the Story | AWS News Blog
https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/

My sample regionsforcloudstorage.dat contained the following information in both the 'regions' and 'Amazon' sections:

{
  "name": "Custom In-house",
  "id": "custom-in-house",
  "s3endpoint": "objectstore.domain.com",
  "blobStoreEndpoint": "objectstore.domain.com"
}

I confirmed my object storage was configured to allow access to portalcontent.objectstore.domain.com and DNS was configured properly for both aliases. Then my Portal for ArcGIS site creation JSON was (using a pre-created bucket named 'portalcontent'):

{
  "type": "cloudStore",
  "provider": "Amazon",
  "connectionString": {"accessKeyId":"<accessKey>","secretAccessKey": "<secretKey>","region": "custom-in-house","credentialType": "accessKey"},
  "objectStore": "portalcontent"
}

FYI: Here is the current latest AWS S3 region file: https://s3.amazonaws.com/esriresources/1091/regionsforcloudstorage.dat

NicolasGIS

Thanks @ChristopherPawlyszyn for your detailed explanations. Much appreciated.

"This means that mimicking this deployment style on-prem would require a DynamoDB-compatible database as well"

-> I had in mind using a classic file share directory for the arcgis server config-store as I. do not have a DynamoDB-compatible at my disposal (will double check though). Too bad you can't "mix" specially. knowing that it's currently not suitable for production ! Do you think this may evolve in the future to something more "flexible" ?

On the S3 storage for Portal for ArcGIS content, I am happy to read that it is doable ! Will have a look and. try the "virtual-hosted style". Will share my finding here !

Once again, many thanks !

ZeeshanKhan1

Has anyone ever solved this for the on-premises deployment? I was trying to register it for use with Object Store.
With ArcGIS 11.4, we are forced to migrate to Object Store, and the documentation available only talks about AWS, or Azure S3.

https://www.esri.com/arcgis-blog/products/arcgis-enterprise/administration/arcgis-enterprise-migrating-to-the-object-store/#block4

MMBR

@ZeeshanKhan1 - read my community.esri.com blog article below. I hope this helps. As far as I know, we cannot store the new Object Store in the Cloud, neither AWS S3, nor Azure.
As of the 11.4 release, the tile cache data store is deprecated. In a future release, it will no longer be supported.

JamesHead-Mears1

I'm looking into this at the moment too and struggling to find any current documentation on configuring the cloud regions. There is some archived documentation for 10.5 but I'm using Enterprise 11.5 and it seems to only state that a local file is required:

https://enterprise.arcgis.com/en/server/latest/deploy/windows/configure-regions-for-cloud-stores.htm

Has anyone came across any additional information on this?

VickyWang_COM

Would be good to know if any use case to use S3 bucket as Portal content in the AWS cloud. compare to fsx shared folder .... using windows server on EC2

Nimm an der Diskussion teil

Angemeldete Mitglieder können Beiträge verfassen, Updates folgen und mehr. Neu hier? Registriere ein kostenloses Konto.

Schnellzugriffe

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.