Publish a map service from a fgdb stored in S3 that is registered as cloudstore in ArcGIS Server (deployed in AWS)

[Anonymous User]

Hi all,

I have registered a S3 bucket as cloud store in my ArcGIS Server (deployed in the AWS). Very straightforward process, all worked ok.

Now I want to publish a dynamic map service that uses data stored in that S3 bucket. I have found many references in the help saying you can use S3 as data store in ArcGIS Server for shapefiles and file geodatabases but I could not find a reference that explains how to publish a service that uses data stored in S3. 

How could I do that - say from ArcGIS desktop? is that possible?

Appreciate your help. Thank you.

PS: Why would I want to do that? 1) To keep my ArcGIS Server site stateless, 2) In my organisation S3 is a very easy way to update the file data I registered in ArcGIS Server. 3) make use of all the benefits of S3, and since I'm using it as part of the ArcGIS Enterprise deployment I want to use it for other purposes.  

4) AWS EFS is possibly a better option, yes I agree but it's not yet supported as a datastore in server as far as I know. 

[1] https://enterprise.arcgis.com/en/server/10.5/cloud/amazon/strategies-for-data-transfer-to-aws.htm

[2] http://enterprise.arcgis.com/en/server/latest/publish-services/linux/registering-your-data-with-arcg...

[3] Amazon Elastic File System (Amazon EFS) – Cloud File Storage – Amazon Web Services (AWS) 

Derek Law‌ ( FYI, this is what I was talking with Simon Jackson @esri Australia )

[VinceAngelo]

Due to the high latency, I expect that S3 would not be a good fit for a dynamic map service, even if it were possible, which I don't think is the case. My only experience with S3 with ArcGIS is to serve as a repository for a map service cache, though it's an area of of active development, so you should try to use the most recent build before experimenting further.

- V

[Anonymous User]

Hi Vince, 

Yes, that was my understanding. S3 has a high latency and that's probably the reason why you might not publish a dynamic map service from it....Then, I found this video from Peter Becker (PM Imagery): AWS re:Invent 2017: How Esri Optimizes Massive Image Archives for Analytics in the C (ABD402) - YouT...  (min: 36 to 40' shows a comparison table of all the storage options in AWS) - and my confusion came when I read in the documentation that you can register a S3 bucket as cloud store with ArcGIS Server and use it to store "shapefiles and file geodatabases" - read below -

Register a cloud store

Using the Data Stores tab, you can register a cloud store that contains GIS resources and data, such as shapefiles and file geodatabases. When you register a directory, all subdirectories under the folder you specify are also registered with the server.

[2] Register your data with ArcGIS Server using Manager—Documentation | ArcGIS Enterprise 

AND

"...Amazon S3—Amazon Simple Storage Service (S3) is an Amazon service designed specifically for data storage in the cloud. This storage option has the lowest potential for data failure or loss. You can use S3 as a place for data backup, as a middle ground for data transfer between your on-premises deployment and your EBS volumes, or as the location of file-based data you register with an ArcGIS Server site on AWS...."

[1] Strategies for data transfer to Amazon Web Services—ArcGIS Enterprise on AWS | ArcGIS Enterprise 

[DerekLaw · Accepted solution]

Hi Walter,

I had to check with some folks on the Dev team and professional services to research your question. And yes, Simon did contact me about your question.

As Vince indicated, cloud stores such as S3 are supported for storing cache tiles for cached map/image services. For storing data content for dynamic map services we do not have any S3 solution. You can choose to store the shapefiles in SQL Server or any managed datastore in Amazon.

> ... and my confusion came when I read in the documentation that you can register a S3 bucket as cloud store with ArcGIS Server and use it to store "shapefiles and file geodatabases"

I was told this is a documentation issue that is going to be addressed and corrected in the ArcGIS 10.6.1 release.

Cloud stores are used for caches, raster stores, and big data file shares. Sorry for the confusion.

Hope this helps,

[Anonymous User]

Thank you, Derek. Now it's clear.

[Jeff_Demuth]

With the recent announcement of Redshift support you can create external tables in S3 and register them with Redshift spectrum.  You can use either Redshift or Redshift Serverless as a geodatabase in ArcGIS Pro/Enterprise and changes to the data in s3 will automatically be updated to users consuming the layers pointing to the redshift spectrum tables.

[xingchenc]

@DerekLaw  @VinceAngelo 

Any update on this matter? Are we still can't publish dynamic map service to S3 bucket?

[DerekLaw]

Hi @xingchenc,

Apologies for the late reply. I had to check with the Development team on your question.

> Are we still can't publish dynamic map service to S3 bucket?

Correct. Map services require that file geodatabases and shapefiles to be on a file system, there is no mechanism for them to be stored on S3. 

Hope this helps,

[xingchenc]

Thanks for the update!