nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Hosted Feature Layer backed by sd not updating using python or manual update

ConradSchaefer__DOIT_

We have two versions (Production & Development) of the same process, with their own independent resources, that are not working. Each is a hosted feature layer that was created by and is backed by an sd file. The sd file was created through ArcPro and was initially uploaded to create the hosted services. Then we have a python process that updates the sd files weekly. In short we join non-spatial attributes with a spatial (polygon) layer and that joined result gets pushed to AGOL. We thought it was working, because all indications from the process is that it is working without exceptions/fails etc.

I was given an updated polygon layer. The feature count has decreased and new features have also been incorporated. When I updated the feature class used in Pro, and ran the update process to affect AGOL the process worked without issue. But, when I went to AGOL to check the resulting data I see it has not updated. The update date on the sd file and hosted layer indicate they were updated but the data itself sits at the old feature count and the new polygons are not to be found. So, the data does not change but all indications locally and in AGOL are that things have updated. I have run both the Production and Development processes and each python process succeeds and each set of assets in AGOL show they have been "updated". Except that the data doesn't actually change.

I tried the manual "update" button in AGOL for the sd item and fed it the sd file manually. It acts as though it has succeeded but when I check the hosted feature layer the data has not changed.

I have gone into both Pro projects (prod and dev) and check the feature class in the gdb and also in the only Map in the project. They are once and the same but I didn't want to make an assumption and so I checked "both". The feature class/layer contain the most recent data. So Pro has the correct setup and layer for publishing.

I then altered the python process to output a newly named sd file so that I could create a new item in AGOL as an sd file backing a feature layer. When I check the data in that new item it shows the correct (meaning newest and desired) dataset, which I want and see correctly loaded in Pro as a layer (and backed by the feature class in the project gdb).

The python process indicates it works, the AGOL items themselves show an updated date, the manual "Update" button in AGOL for the sd items appears to "work", but the data in the hosted layers does not change. This is true for both parallel sets of resources (production and dev).

Anyone else encountered this and have tips/ideas?

Encontre mais publicações marcadas com

Content Management

Administration

Data and services

Accepted answers

ConradSchaefer__DOIT_

Follow up on this. Finally figured out what was going on. I'll provide follow up on some of my initial comments to make things clearer. First, both versions acted the same because they both had the same source problem with the code.

RECAP:
"The python process indicates it works, the AGOL items themselves show an updated date, [even] the manual "Update" button in AGOL for the sd items appears to "work", but the data in the hosted layers does not change."
The absence of errors and presence of success indicators at all stages is what made this difficult to troubleshoot.
And the hack attempt of updating the layer by updating the sd item manually didn't "work" because the sd item update is not automatically published to the layer item. You have to publish the update to affect the layer. The Python Api call to publish() on the sd item appears to take whatever data is in the sd item in AGOL and move it over. So, if you update the sd item manually in AGOL and then run the publish() function using code THEN it will move the data from the AGOL sd item to the AGOL layer item.

ANSWER:
After figuring out I could manually update the sd item in AGOL, then run publish() using Python and get the layer to update, I focused in on the update() function. This update() function is supposed to update the sd item in AGOL, and programmatically perform the same function as my manual update using the Update Data button. But, it is "sensitive". Turns out that, as the API docs indicate ( https://developers.arcgis.com/python/latest/api-reference/arcgis.gis.toc.html#arcgis.gis.Item.update ), the 'data' parameter is for an "Optional string, io.StringIO, or io.BytesIO. Either a path or URL to the data or an instance of StringIO or BytesIO objects." The essential portion relevant to my problem is that it must be a path. I had mistakingly provided the sd file name and not the path to the sd file. That was the source mistake.

But, the python process would run "successfully" using this bad data parameter value, and the sd item in AGOL and the layer item both showed date/time updates indicating they had "changed". I assumed the date/time change indicated they had been successfully operated on and should have been updated. Turns out though that the Python api update() function on the sd item was not actually working. Once I provided a path instead of a file name the process worked as you would expect. It seems that this had everything to do with

a path being required,
with all (super misleading) indications by the process and ArcGIS Online being that the update worked, and
that there is zero failure notice by the code etc to let you know that the filename is not sufficient for AGOL.

End suggestion would be that a check with raised exception be put into update() to see that a path is provided and not just a filename, or something to that effect, and also maybe for AGOL to not act as though it is updated. The esri update() code, to me, appears to check for os.path.isfile() which should fail on a str filename that is not a path but I guess it doesn't?? [I didn't do an extensive deep dive into the guts of the esri python functionality so that's not full assessment, nor am I a software developer level guy]

Anyway, the api documentation does state that it must be a path or url so really it comes down to my incorrect usage. That said, sure would be useful if update() failed if it it wasn't actually updating the AGOL sd item.

All comments

ConradSchaefer__DOIT_

a path being required,
with all (super misleading) indications by the process and ArcGIS Online being that the update worked, and
that there is zero failure notice by the code etc to let you know that the filename is not sufficient for AGOL.

ConradSchaefer__DOIT_

@jcarlson

Been checking out the compare/edit_features combo with mixed success. I'll create a new layer with a shapefile (for example) and then run the compare process using the exact same resource and the process detects that there are a ton of differences in the spatial field. I am thinking maybe the AGOL upload reduces the number of decimal places or something similar and that leads to every spatial value being "different". Need to keep exploring it. I'm hoping to find a good use case for it as it seems like a good method.

Thanks

jcarlson

The analogy I like is that overwriting your layer to refresh its data is like bulldozing a house and rebuilding it so that you can repaint the kitchen.

The compare-and-update process looks like:

Load source to dataframe
Query hosted layer to dataframe
Compare
Use compare's output to submit adds, edits, and deletes using arcgis.features.featureLayer.edit_features(…)

We've got a parcels layer, for instance, with 60k features. Data is sourced from at least 4 different places. Each source may only change a couple hundred parcels on any given day.

Comparing and updating means we can make the ~1000 daily edits to the features, leave the other 59k features alone, and the service never loses availability. We can run updates in the middle of the day without interrupting or affecting our end users at all. What had been a single overwrite taking several minutes is now a few separate comparisons, all of which execute in a matter of seconds.

The code is more complex, but the results are worth it.

ConradSchaefer__DOIT_

@jcarlson

The process is based off of this ESRI blog post

https://www.esri.com/arcgis-blog/products/api-python/analytics/updating-your-hosted-feature-services-with-arcgis-pro-and-the-arcgis-api-for-python/

Basically it goes...

arcpy.mp.CreateWebLayerSDDraft()
arcpy.StageService_server()
agol item .update()
agol item .publish() with overwrite=True

Thanks for the "compare" method mention. I'll have a look.

jskinner

@ConradSchaefer__DOIT_ here are a few other methods you can use to keep you AGOL feature services updated:

Overwrite ArcGIS Online Feature Service using ArcGIS Pro Project

https://community.esri.com/t5/arcgis-online-documents/overwrite-arcgis-online-feature-service-using/ta-p/1371248

Overwrite ArcGIS Online Feature Service using Truncate and Append

https://community.esri.com/t5/arcgis-online-documents/overwrite-arcgis-online-feature-service-using/ta-p/904457

jcarlson

This process is basically a deconstructed "overwrite", it sounds like?

I've had nothing but bad luck with automated overwrites. I don't have any easy suggestions, but what we've done is build a process that compares the new data with the old, identifies rows that have changed, and edits the hosted feature layer in place.

In a recent update, the ArcGIS Python API actually has a "compare" method built in for this very purpose.

https://developers.arcgis.com/python/api-reference/arcgis.features.toc.html#geoaccessor

One of the major benefits of this approach is that you don't need to join your source data into some intermediate layer, you can just work with each source separately. You can load one source into a dataframe, then query the hosted layer on those particular columns, with or without geometry. The compare method will still work!

Participe da Conversa

Membros conectados podem postar, seguir atualizações e mais. Novo aqui? Registre uma conta gratuita.

Links Rápidos

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.