AnsweredAssumed Answered

Publish a map service from a fgdb stored in S3 that is registered as cloudstore in ArcGIS Server (deployed in AWS)

Question asked by Walter.Simonazzi on Apr 25, 2018
Latest reply on May 1, 2018 by Walter.Simonazzi

Like • Show 0 Likes0
Comment • 4

Hi all,

I have registered a S3 bucket as cloud store in my ArcGIS Server (deployed in the AWS). Very straightforward process, all worked ok.

Now I want to publish a dynamic map service that uses data stored in that S3 bucket. I have found many references in the help saying you can use S3 as data store in ArcGIS Server for shapefiles and file geodatabases but I could not find a reference that explains how to publish a service that uses data stored in S3.

How could I do that - say from ArcGIS desktop? is that possible?

Appreciate your help. Thank you.

PS: Why would I want to do that? 1) To keep my ArcGIS Server site stateless, 2) In my organisation S3 is a very easy way to update the file data I registered in ArcGIS Server. 3) make use of all the benefits of S3, and since I'm using it as part of the ArcGIS Enterprise deployment I want to use it for other purposes.

4) AWS EFS is possibly a better option, yes I agree but it's not yet supported as a datastore in server as far as I know.

[1] https://enterprise.arcgis.com/en/server/10.5/cloud/amazon/strategies-for-data-transfer-to-aws.htm

[2] http://enterprise.arcgis.com/en/server/latest/publish-services/linux/registering-your-data-with-arcgis-server-using-manager.htm#ESRI_SECTION1_BB6B507EA4D049698708F98673F13DA3

[3] Amazon Elastic File System (Amazon EFS) – Cloud File Storage – Amazon Web Services (AWS)

Derek Law ( FYI, this is what I was talking with Simon Jackson @esri Australia )

dlaw-esristaff

May 1, 2018 2:32 PM

Correct Answer

Hi Walter,

I had to check with some folks on the Dev team and professional services to research your question. And yes, Simon did contact me about your question.

As Vince indicated, cloud stores such as S3 are supported for storing cache tiles for cached map/image services. For storing data content for dynamic map services we do not have any S3 solution. You can choose to store the shapefiles in SQL Server or any managed datastore in Amazon.

> ... and my confusion came when I read in the documentation that you can register a S3 bucket as cloud store with ArcGIS Server and use it to store "shapefiles and file geodatabases"

I was told this is a documentation issue that is going to be addressed and corrected in the ArcGIS 10.6.1 release.

Cloud stores are used for caches, raster stores, and big data file shares. Sorry for the confusion.

Hope this helps,

See the reply in context

No one else had this question

Outcomes

vangelo-esristaff

Apr 29, 2018 7:42 PM

Due to the high latency, I expect that S3 would not be a good fit for a dynamic map service, even if it were possible, which I don't think is the case. My only experience with S3 with ArcGIS is to serve as a repository for a map service cache, though it's an area of of active development, so you should try to use the most recent build before experimenting further.

- V

Actions

Walter.Simonazzi @ vangelo-esristaff on Apr 29, 2018 11:59 PM

Hi Vince,

Yes, that was my understanding. S3 has a high latency and that's probably the reason why you might not publish a dynamic map service from it....Then, I found this video from Peter Becker (PM Imagery): AWS re:Invent 2017: How Esri Optimizes Massive Image Archives for Analytics in the C (ABD402) - YouTube (min: 36 to 40' shows a comparison table of all the storage options in AWS) - and my confusion came when I read in the documentation that you can register a S3 bucket as cloud store with ArcGIS Server and use it to store "shapefiles and file geodatabases" - read below -

Register a cloud store

Using the Data Stores tab, you can register a cloud store that contains GIS resources and data, such as shapefiles and file geodatabases. When you register a directory, all subdirectories under the folder you specify are also registered with the server.

[2] Register your data with ArcGIS Server using Manager—Documentation | ArcGIS Enterprise

AND

"...Amazon S3—Amazon Simple Storage Service (S3) is an Amazon service designed specifically for data storage in the cloud. This storage option has the lowest potential for data failure or loss. You can use S3 as a place for data backup, as a middle ground for data transfer between your on-premises deployment and your EBS volumes, or as the location of file-based data you register with an ArcGIS Server site on AWS...."

[1] Strategies for data transfer to Amazon Web Services—ArcGIS Enterprise on AWS | ArcGIS Enterprise

Actions

dlaw-esristaff

May 1, 2018 2:32 PM

Hi Walter,

I had to check with some folks on the Dev team and professional services to research your question. And yes, Simon did contact me about your question.

> ... and my confusion came when I read in the documentation that you can register a S3 bucket as cloud store with ArcGIS Server and use it to store "shapefiles and file geodatabases"

I was told this is a documentation issue that is going to be addressed and corrected in the ArcGIS 10.6.1 release.

Cloud stores are used for caches, raster stores, and big data file shares. Sorry for the confusion.

Hope this helps,

1 person found this helpful

Actions

Walter.Simonazzi @ dlaw-esristaff on May 1, 2018 3:59 PM

Thank you, Derek. Now it's clear.

Actions

4 Replies

vangelo-esristaff Apr 29, 2018 7:42 PM
Mark Correct
Correct Answer
Due to the high latency, I expect that S3 would not be a good fit for a dynamic map service, even if it were possible, which I don't think is the case. My only experience with S3 with ArcGIS is to serve as a repository for a map service cache, though it's an area of of active development, so you should try to use the most recent build before experimenting further.

- V
Like • Show 0 Likes0
Actions
- Walter.Simonazzi @ vangelo-esristaff on Apr 29, 2018 11:59 PM
  Mark Correct
  Correct Answer
  Hi Vince,
  
  Yes, that was my understanding. S3 has a high latency and that's probably the reason why you might not publish a dynamic map service from it....Then, I found this video from Peter Becker (PM Imagery): AWS re:Invent 2017: How Esri Optimizes Massive Image Archives for Analytics in the C (ABD402) - YouTube (min: 36 to 40' shows a comparison table of all the storage options in AWS) - and my confusion came when I read in the documentation that you can register a S3 bucket as cloud store with ArcGIS Server and use it to store "shapefiles and file geodatabases" - read below -
  
  Register a cloud store
  Using the Data Stores tab, you can register a cloud store that contains GIS resources and data, such as shapefiles and file geodatabases. When you register a directory, all subdirectories under the folder you specify are also registered with the server.
  [2] Register your data with ArcGIS Server using Manager—Documentation | ArcGIS Enterprise
  AND
  
  "...Amazon S3—Amazon Simple Storage Service (S3) is an Amazon service designed specifically for data storage in the cloud. This storage option has the lowest potential for data failure or loss. You can use S3 as a place for data backup, as a middle ground for data transfer between your on-premises deployment and your EBS volumes, or as the location of file-based data you register with an ArcGIS Server site on AWS...."
  [1] Strategies for data transfer to Amazon Web Services—ArcGIS Enterprise on AWS | ArcGIS Enterprise
  Like • Show 0 Likes0
  Actions
dlaw-esristaff May 1, 2018 2:32 PM
Unmark Correct
Correct Answer
Hi Walter,

I had to check with some folks on the Dev team and professional services to research your question. And yes, Simon did contact me about your question.

As Vince indicated, cloud stores such as S3 are supported for storing cache tiles for cached map/image services. For storing data content for dynamic map services we do not have any S3 solution. You can choose to store the shapefiles in SQL Server or any managed datastore in Amazon.

> ... and my confusion came when I read in the documentation that you can register a S3 bucket as cloud store with ArcGIS Server and use it to store "shapefiles and file geodatabases"

I was told this is a documentation issue that is going to be addressed and corrected in the ArcGIS 10.6.1 release.
Cloud stores are used for caches, raster stores, and big data file shares. Sorry for the confusion.

Hope this helps,
1 person found this helpful
Like • Show 0 Likes0
Actions
- Walter.Simonazzi @ dlaw-esristaff on May 1, 2018 3:59 PM
  Mark Correct
  Correct Answer
  Thank you, Derek. Now it's clear.
  Like • Show 0 Likes0
  Actions

GeoNet

The Esri Community

Publish a map service from a fgdb stored in S3 that is registered as cloudstore in ArcGIS Server (deployed in AWS)

Outcomes

Related Content

Recommended Content