Are there any particular best practices to follow when publishing feature services to AGOL/Portal?

Question

We are looking to deploy an Open Data Hub initiative in ArcGIS online. I want to have all of our publicly shareable layers readily available for the public to download. We have quite a few layers, so manually keeping the data up to date would have been a huge headache. I have about 30 feature classes that I am looking to publish and keep updated on a weekly basis. To get ahead of this issue, I have used this excellent post (Thanks @JonahLay!) to customize a python script that will publish and overwrite the layers from an input arcgis pro project to an AGOL service on a weekly basis.

When publishing services to AGOL, are there any best practices regarding publishing multiple feature classes in one service versus publishing them as single feature class services? I have worked with ArcGIS Online data for a long time but have never seen a concrete answer for this. Would you run into bottlenecks by publishing too many feature classes in one service? If so, is there a rule of thumb for how many? Is one way more "efficient" in terms of credit consumption? I know whatever we publish is going to consume credits, which is fine. Just not sure if one method is more efficient than another.

Would appreciate any insight that you can provide. Thank you!

jcarlson · Answer

No, not at all! We do everything through the ArcGIS Python API, and don't touch ArcPy at all. I've no doubts that process would work, but seems a bit clunky with temporary files being uploaded and deleted.

We use the module pandas to compare our source data with the hosted layer and then update only the rows and columns that have changed. Even though our parcels layer is always changing, it's about 0.5% of features that actually get edited any given day, and the vast majority will not change at all. Our script identifies those "stale" rows and passes updates through the ArcGIS Python API, modifying the features in place. So even a fairly large layer like our parcels (50,000+ features) can be updated in under a minute, and we can do so at any time of day or night, as there is never a moment when the layer is unavailable (as in an overwrite / publish process) or empty (as in truncate / append).

As far as the downloadable files, some folks do prefer an actual shapefile as opposed to accessing the service URL, so we have a separate script that exports the layers to file and shares them to the hub group.

MDB_GIS · Answer

Thank you for the response. This is very helpful!

For the most part, the layers are fairly unrelated other than the fact that they are all requested public layers. If you don't mind me asking, how do you share yours out? The Kendall County GIS hub is pretty much exactly what I am going for, so I imagine your setup would work well for us. Do you just have a script for truncate/append set up for a regular basis on certain layers?

The only layers we have the change with any degree of frequency is our zoning, parcels, centerlines, and point addresses.

I will look into the truncate/append process. I am an amateur at best when it comes to python, so it will take me some time. Is this script similar to your setup?

jcarlson · Answer

I don't know how many is too many, but I've run into publishing issues when a service was going to have over 60 layers in it (it was a weird project, don't ask!).

As far as "separate services" vs "layers in a single service", it really depends on what you're going to use them for. "Layers in a service" makes sense for some things, especially as you can make use of the $datastore Arcade variable and access other layers in the service, whether or not they are present in the map / app. It might also make sense to keep certain thematically related datasets together.

For example, we maintain a layer of voting precincts as polygons, and a layer of polling places as points. These layers are pretty much never used separately, so putting them into the same service makes sense.

On the other hand, if your layers are going to change or be republished a lot, you may want them separated. There are certain things you can do to a layer (creating hosted feature layer views, map image layers, etc) that can lock the source layer for schema edits, and you may not want that to impact half a dozen different layers.

Also, though you didn't ask about this, if you're looking to "refresh" your AGOL layers using an authoritative source regularly, I would look into ways of accomplishing this that do not involve overwriting the service. Frequent overwrites can lead to the source layer breaking sometimes. Unless your service is changing the number / types of layers in it, or making drastic changes to the schema, there's no reason to overwrite. Try a standard truncate / append process instead, using the ArcGIS Python API.