nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

How are ArcGIS Enterprise Hosted Feature Layers Stored?

AndrewRudin1

When a Pro user loads a feature class into ArcGIS Enterprise as a hosted feature layer, where is the bulk of the data actually stored? In my example I am using the option "Copy all data" and using the Feature Layer type.

It was my impression that the data would reside in the relational data store. However, when I test this on my sandbox ArcGIS Enterprise deployment, I see very large .sd files created under the Portal install folder at C:\arcgis\arcgisportal\content\items. Is this .sd file the source that is actually read when users view data on a map? And if so, then what is being stored in the data store?

Find more posts tagged with

Accepted answers

JonathanQuinn

The SD file does contain the data in the map you're publishing, and the data is extracted to the ArcGIS Data Store when you publish. A copy of it is left in the content directory so if your users typically publish large amounts of data, you may want to consider referenced services and moving your data into an enterprise geodatabase or at least a location that the Server has access to.

About registering your data with ArcGIS Server—Documentation | ArcGIS Enterprise

How/what you publish, (hosted feature layers using ArcGIS Data Store, or traditional map services using referenced data), will be based on your users familiarity with the publishing process, (although it's a bit more simple with Pro), how you store your data, what types of functionality they're interested in, your requirements for performance etc.

All comments

dcoley

Great thank you. Yes, I certainly understand. It was more about monitoring growth, so I certainly appreciate your insight

JonathanQuinn

The database structure doesn't lend itself to viewing the data in a meaningful way, so I can't say you'll gain much by doing so, but it's under the data directory that you used to configure the Data Store, for example C:\arcgisdatastore. In there is a pgdata folder and in that, a base folder. It's then up to the database on how it arranges the data in those directories.

dcoley

Hi Jon - can you tell me where in the Relational Data Store directories the hosted layer is stored? I can see my sd file in my portal content which is fine, but I am curious to know where my data store directories the file or files are stored, if possible. I'm not trying any back-end manipulation, just trying to understand the datastore black-box a little better....

AndrewRudin1

Also you mention that I could delete the .sd files after the layer is published via a script. I tried deleting the .sd file via the Portal user interface but it gives the error message "This item cannot be deleted until these dependent layers are deleted:" then it lists the hosted feature layer name. That lead me to believe the .sd file was used by the hosted feature layer service. Do you think a backend script would avoid this same error message?

AndrewRudin1

Thank you, Jonathan. This explanation really helped.

JonathanQuinn

You can definitely scale to tens of thousands of hosted feature services as they're lightweight whereas a couple hundred traditional SOC services will cause some problems for your Server machine. In terms of performance, that's really dependent on the data. Hosted feature services will render the features directly, so a 3GB contour layer may not perform well especially at small scales. Instead, a map service that's rendering an image of the same data may prove to be the better option. So along with hard disk space, that's something else you'll need to consider.

You'll also have to manage two copies of the data. If you have QA/QC processes on your enterprise geodatabase, every time features update you'll need to update the hosted feature services, unless you begin to update the hosted feature services directly through a web application.

You can always write a script to set as a scheduled task to find all .sd files and delete them from users content. They allow you to republish the service directly from the already uploaded SD file or just download the SD, so if neither of those are of interest to you, you can delete them.

Not trying to discourage you from moving everything to hosted services, just giving my thoughts on the other side of the argument.

AndrewRudin1

Thanks for the clarification. In our existing pre-Enterprise architecture we have ~600 feature classes on an Oracle SDE referenced by our ArcGIS Desktop users and our ArcGIS Server web applications. As we start to look at Enterprise, I've heard mumblings that migrating the SDE layers to data store hosted feature layers would increase performance and/or scalability...provided the data store server had very fast disk read/write speed. Ultimately we can size the portal server to be bigger to handle the .SD files if need be if the hosted feature layers perform better than RDBMS registered feature layers.

JonathanQuinn

About registering your data with ArcGIS Server—Documentation | ArcGIS Enterprise

AndrewRudin1

Thanks for the link. My concern is the .SD file that persists on Portal is really big...like the size or the data that was loaded up. Multiply that by 100's of layers and it's a lot of space used up by data that's not actively being used.

I'm guessing the large .sd file is just an artifact of how the service and the data make their way from the Pro user to Enterprise. Pro puts everything in the .sd file for upload to the server, then the server extracts the data and places it in the data store. What seems to be missing is removing the data from the .sd file after.

My example was a 3GB contour layer, which is an extreme, but multiplying this for 100's of smaller hosted feature layers seems like a waste of space on the portal server.

MicahBabinski

Hi Andrew,

I think your relational data store contains the actual feature data, while the service definition file (.sd) contains the configurations (including pointer to the correct data source) of your service. See:

What is ArcGIS Data Store?—Portal for ArcGIS (10.5.x) | ArcGIS Enterprise

About service definition files—Documentation | ArcGIS Enterprise

Although actually as I read that second link I see there is a configuration you can set where the SD file contains the actual data you are publishing. In this situation, the data contained in the SD file gets copied to your federated/hosting server when you publish.

Micah

Join the Conversation

Signed in members can post, follow updates, and more. New here? Register a free account.

Quick Links

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.