How are ArcGIS Enterprise Hosted Feature Layers Stored?

2202
13
Jump to solution
12-22-2017 11:46 AM
AndrewRudin1
Occasional Contributor

When a Pro user loads a feature class into ArcGIS Enterprise as a hosted feature layer, where is the bulk of the data actually stored?  In my example I am using the option "Copy all data" and using the Feature Layer type.

It was my impression that the data would reside in the relational data store.  However, when I test this on my sandbox ArcGIS Enterprise deployment, I see very large .sd files created under the Portal install folder at C:\arcgis\arcgisportal\content\items.  Is this .sd file the source that is actually read when users view data on a map?  And if so, then what is being stored in the data store?

Tags (1)
0 Kudos
1 Solution

Accepted Solutions
JonathanQuinn
Esri Notable Contributor

The SD file does contain the data in the map you're publishing, and the data is extracted to the ArcGIS Data Store when you publish. A copy of it is left in the content directory so if your users typically publish large amounts of data, you may want to consider referenced services and moving your data into an enterprise geodatabase or at least a location that the Server has access to.

About registering your data with ArcGIS Server—Documentation | ArcGIS Enterprise 

How/what you publish, (hosted feature layers using ArcGIS Data Store, or traditional map services using referenced data), will be based on your users familiarity with the publishing process, (although it's a bit more simple with Pro), how you store your data, what types of functionality they're interested in, your requirements for performance etc.

View solution in original post

13 Replies
MicahBabinski
Regular Contributor

Hi Andrew,

I think your relational data store contains the actual feature data, while the service definition file (.sd) contains the configurations (including pointer to the correct data source) of your service. See:

What is ArcGIS Data Store?—Portal for ArcGIS (10.5.x) | ArcGIS Enterprise

About service definition files—Documentation | ArcGIS Enterprise

Although actually as I read that second link I see there is a configuration you can set where the SD file contains the actual data you are publishing. In this situation, the data contained in the SD file gets copied to your federated/hosting server when you publish.

Micah

AndrewRudin1
Occasional Contributor

Thanks for the link.  My concern is the .SD file that persists on Portal is really big...like the size or the data that was loaded up.  Multiply that by 100's of layers and it's a lot of space used up by data that's not actively being used. 

I'm guessing the large .sd file is just an artifact of how the service and the data make their way from the Pro user to Enterprise.  Pro puts everything in the .sd file for upload to the server, then the server extracts the data and places it in the data store.  What seems to be missing is removing the data from the .sd file after.

My example was a 3GB contour layer, which is an extreme, but multiplying this for 100's of smaller hosted feature layers seems like a waste of space on the portal server. 

JonathanQuinn
Esri Notable Contributor

The SD file does contain the data in the map you're publishing, and the data is extracted to the ArcGIS Data Store when you publish. A copy of it is left in the content directory so if your users typically publish large amounts of data, you may want to consider referenced services and moving your data into an enterprise geodatabase or at least a location that the Server has access to.

About registering your data with ArcGIS Server—Documentation | ArcGIS Enterprise 

How/what you publish, (hosted feature layers using ArcGIS Data Store, or traditional map services using referenced data), will be based on your users familiarity with the publishing process, (although it's a bit more simple with Pro), how you store your data, what types of functionality they're interested in, your requirements for performance etc.

AndrewRudin1
Occasional Contributor

Thanks for the clarification.  In our existing pre-Enterprise architecture we have ~600 feature classes on an Oracle SDE referenced by our ArcGIS Desktop users and our ArcGIS Server web applications.  As we start to look at Enterprise, I've heard mumblings that migrating the SDE layers to data store hosted feature layers would increase performance and/or scalability...provided the data store server had very fast disk read/write speed.   Ultimately we can size the portal server to be bigger to handle the .SD files if need be if the hosted feature layers perform better than RDBMS registered feature layers.

JonathanQuinn
Esri Notable Contributor

You can definitely scale to tens of thousands of hosted feature services as they're lightweight whereas a couple hundred traditional SOC services will cause some problems for your Server machine.  In terms of performance, that's really dependent on the data. Hosted feature services will render the features directly, so a 3GB contour layer may not perform well especially at small scales.  Instead, a map service that's rendering an image of the same data may prove to be the better option.  So along with hard disk space, that's something else you'll need to consider.

You'll also have to manage two copies of the data.  If you have QA/QC processes on your enterprise geodatabase, every time features update you'll need to update the hosted feature services, unless you begin to update the hosted feature services directly through a web application.

You can always write a script to set as a scheduled task to find all .sd files and delete them from users content.  They allow you to republish the service directly from the already uploaded SD file or just download the SD, so if neither of those are of interest to you, you can delete them.

Not trying to discourage you from moving everything to hosted services, just giving my thoughts on the other side of the argument.

AndrewRudin1
Occasional Contributor

Thank you, Jonathan.  This explanation really helped. 

0 Kudos
AndrewRudin1
Occasional Contributor

Also you mention that I could delete the .sd files after the layer is published via a script.  I tried deleting the .sd file via the Portal user interface but it gives the error message "This item cannot be deleted until these dependent layers are deleted:" then it lists the hosted feature layer name. That lead me to believe the .sd file was used by the hosted feature layer service.  Do you think a backend script would avoid this same error message?

DavidColey
Frequent Contributor

Hi Andrew - did you try deleting the portal copy of these sd files via some sort of widows command?  If so were you successful?

0 Kudos
AndrewRudin1
Occasional Contributor

No, I have just started working with ArcGIS Enterprise for the first time in a sandbox server.  I didn't want to start hacking it on day one before I started exploring.  I've also heard that vector tile hosted layers rely on the package item for the data when users view it, so I'm leery to start deleting data from the back end.