Select to view content in your preferred language

Hosted Feature Layer backed by sd not updating using python or manual update

120
4
Tuesday
ConradSchaefer__DOIT_
Regular Contributor

We have two versions (Production & Development) of the same process, with their own independent resources, that are not working. Each is a hosted feature layer that was created by and is backed by an sd file. The sd file was created through ArcPro and was initially uploaded to create the hosted services. Then we have a python process that updates the sd files weekly. In short we join non-spatial attributes with a spatial (polygon) layer and that joined result gets pushed to AGOL. We thought it was working, because all indications from the process is that it is working without exceptions/fails etc. 

I was given an updated polygon layer. The feature count has decreased and new features have also been incorporated. When I updated the feature class used in Pro, and ran the update process to affect AGOL the process worked without issue. But, when I went to AGOL to check the resulting data I see it has not updated. The update date on the sd file and hosted layer indicate they were updated but the data itself sits at the old feature count and the new polygons are not to be found. So, the data does not change but all indications locally and in AGOL are that things have updated. I have run both the Production and Development processes and each python process succeeds and each set of assets in AGOL show they have been "updated". Except that the data doesn't actually change.

I tried the manual "update" button in AGOL for the sd item and fed it the sd file manually. It acts as though it has succeeded but when I check the hosted feature layer the data has not changed. 

I have gone into both Pro projects (prod and dev) and check the feature class in the gdb and also in the only Map in the project. They are once and the same but I didn't want to make an assumption and so I checked "both". The feature class/layer contain the most recent data. So Pro has the correct setup and layer for publishing.

I then altered the python process to output a newly named sd file so that I could create a new item in AGOL as an sd file backing a feature layer. When I check the data in that new item it shows the correct (meaning newest and desired) dataset, which I want and see correctly loaded in Pro as a layer (and backed by the feature class in the project gdb).

The python process indicates it works, the AGOL items themselves show an updated date, the manual "Update" button in AGOL for the sd items appears to "work", but the data in the hosted layers does not change. This is true for both parallel sets of resources (production and dev).

Anyone else encountered this and have tips/ideas?

 

4 Replies
jcarlson
MVP Esteemed Contributor

This process is basically a deconstructed "overwrite", it sounds like?

I've had nothing but bad luck with automated overwrites. I don't have any easy suggestions, but what we've done is build a process that compares the new data with the old, identifies rows that have changed, and edits the hosted feature layer in place.

In a recent update, the ArcGIS Python API actually has a "compare" method built in for this very purpose.

https://developers.arcgis.com/python/api-reference/arcgis.features.toc.html#geoaccessor

One of the major benefits of this approach is that you don't need to join your source data into some intermediate layer, you can just work with each source separately. You can load one source into a dataframe, then query the hosted layer on those particular columns, with or without geometry. The compare method will still work!

- Josh Carlson
Kendall County GIS
JakeSkinner
Esri Esteemed Contributor

@ConradSchaefer__DOIT_ here are a few other methods you can use to keep you AGOL feature services updated:

Overwrite ArcGIS Online Feature Service using ArcGIS Pro Project

https://community.esri.com/t5/arcgis-online-documents/overwrite-arcgis-online-feature-service-using/...

Overwrite ArcGIS Online Feature Service using Truncate and Append

https://community.esri.com/t5/arcgis-online-documents/overwrite-arcgis-online-feature-service-using/...

0 Kudos
ConradSchaefer__DOIT_
Regular Contributor

@jcarlson 

The process is based off of this ESRI blog post 

https://www.esri.com/arcgis-blog/products/api-python/analytics/updating-your-hosted-feature-services... 

Basically it goes...

  • arcpy.mp.CreateWebLayerSDDraft()
  • arcpy.StageService_server()
  • agol item .update()
  • agol item .publish() with overwrite=True

Thanks for the "compare" method mention. I'll have a look.

 

0 Kudos
jcarlson
MVP Esteemed Contributor

The analogy I like is that overwriting your layer to refresh its data is like bulldozing a house and rebuilding it so that you can repaint the kitchen.

The compare-and-update process looks like:

  1. Load source to dataframe
  2. Query hosted layer to dataframe
  3. Compare
  4. Use compare's output to submit adds, edits, and deletes using arcgis.features.featureLayer.edit_features(…)

We've got a parcels layer, for instance, with 60k features. Data is sourced from at least 4 different places. Each source may only change a couple hundred parcels on any given day.

Comparing and updating means we can make the ~1000 daily edits to the features, leave the other 59k features alone, and the service never loses availability. We can run updates in the middle of the day without interrupting or affecting our end users at all. What had been a single overwrite taking several minutes is now a few separate comparisons, all of which execute in a matter of seconds.

The code is more complex, but the results are worth it.

- Josh Carlson
Kendall County GIS