topic Re: Hosted Feature Layer View Does Not Update When Overwriting Hosted Feature Layer in ArcGIS Online Questions

Hosted Feature Layer View Does Not Update When Overwriting Hosted Feature Layer

AmandaBeck — Wed, 04 Jan 2023 15:47:40 GMT

I just used the Overwrite Web Layer in ArcGIS Pro to update a Hosted Feature Layer in ArcGIS Online. That Hosted Feature Layer updated successfully, however, the Hosted Feature Layer View that we have in a lot of ArcGIS Online maps/apps did not update. An error message pops up saying that the layer is not available anymore.

I did this successfully back in September 2022, but it is not working right now. Does anybody know what may be happening here? Did ESRI change this capability? Any comments are appreciated.

Thanks!

Re: Hosted Feature Layer View Does Not Update When Overwriting Hosted Feature Layer

jcarlson — Wed, 04 Jan 2023 15:59:41 GMT

There are a lot of reasons why this might have happened. In general, overwriting a service to update its contents is asking for trouble. You should only do it if something has changed about the layer structure, or if there were significant changes to the schema; most everything else can be modified in-place, or by using a truncate/append process.

When you overwrote the layer, were there any changes to the order of your layers or the schema?

Re: Hosted Feature Layer View Does Not Update When Overwriting Hosted Feature Layer

AmandaBeck — Wed, 04 Jan 2023 16:06:20 GMT

Hi Josh,

The data in which I needed to overwrite was address point data - which I send to the state twice a year. My GIS department came up with this procedure of overwriting the layer from ArcGIS Pro to update the hosted feature layer which would automatically update the view layer. It worked perfectly fine when I did the last update in August. I understand the positives of appending data to a layer in ArcGIS Online, however, when we are talking about such a large layer with points that could have been moved/deleted/and added, it seems better to just overwrite the entire layer completely.

To answer your question, there were no changes made to the layer or schema before overwriting it.

Thanks for your response!

Re: Hosted Feature Layer View Does Not Update When Overwriting Hosted Feature Layer

jcarlson — Wed, 04 Jan 2023 16:16:12 GMT

Well, that eliminates any easy explanations. We used to update some of our layers this way, and would occasionally run into this very problem, especially with view layers. We'd overwrite a layer 8 times, and on the 9th, it would break.

Sometimes we could identify an error in the data that got through the publishing tool's analyze step, but more often, there was no discernible difference.

For the record, truncating and appending basically is overwriting, but only the data itself, while leaving all the deeper settings in place. It's what most people actually want to be doing when they use the built in "overwrite web layer" tool. You "zero out" the layer first, then append into it, so every record, whether new, modified, or otherwise, is reflected in the end product, and there's no risk of out-of-date data.

Re: Hosted Feature Layer View Does Not Update When Overwriting Hosted Feature Layer

AmandaBeck — Wed, 04 Jan 2023 16:23:28 GMT

Well it's nice to know that we're not the only ones that have run into this issue! I just want to verify that I am correct in saying that when updating a hosted feature layer (either overwriting or appending), the view layer should update itself automatically? From what you are explaining that has happened to you in the past, this seems to be the case as to why we would even try doing this in the first place.

Do you now only use the append data in ArcGIS Online to update hosted feature layers to show in view layers? I just want to know what a solution would be for this to not run into this problem again. Aside from recreating the view layer, there doesn't seem to be a way to fix it at this point.

To add to the confusion, when I am looking at the details page in ArcGIS Online of the View Layer, under layers where the data is, when I click on it, it shows the date I updated the data in the hosted feature layer. It's almost as if it worked partially, just not all the way if you know what I mean.

Thanks again,

Amanda Beck - Warren County GIS

Re: Hosted Feature Layer View Does Not Update When Overwriting Hosted Feature Layer

FabioLuiz2 — Wed, 04 Jan 2023 16:30:14 GMT

Hello Amanda,

I don't know how to explain this problem that also happened to me, but I managed to get around it by activating the "Assign layer IDs" option in ArcGIS Pro 3.
In previous versions I remember not needing this.

https://pro.arcgis.com/en/pro-app/latest/help/sharing/overview/assign-layer-ids.htm

(this message was translated from PT-BR)

Re: Hosted Feature Layer View Does Not Update When Overwriting Hosted Feature Layer

jcarlson — Wed, 04 Jan 2023 16:42:20 GMT

You're correct, updating the main layer should update the associated views as well. The trouble is that overwriting replaces the entire service definition of the layer, but the view layer retains its original definition, so there can be underlying differences between main layer and view layer that render the view incompatible.

We use the ArcGIS Python API to update our layers, whether they have associated views or not, if the original source of the data is elsewhere.

Here's a very simple version of the process:

from arcgis import GIS from arcgis.features import GeoAccessor # log in to portal gis = GIS('portal-url', 'user', 'password') # get feature layer by id and index fl = gis.content.get('itemid of service').layers[0] # load source data; this will vary depending on the source df = GeoAccessor.from_featureclass('path to file') # truncate existing data fl.manager.truncate() # load latest data in fl.edit_features(adds=df.spatial.to_featureset())

For larger updates, you may wish to do this in smaller batches, rather than submitting a single, massive update.

I know that there is an actual append functionality in AGOL and the Python API, but for various reasons specific to our source data, I've not been able to get that process working for our updates. But the truncate/append script runs perfectly well, so I don't see any particular reason to alter it.

We use this for lots of things, including some address data. For about 50k address points, the entire process runs in only a few minutes.

Re: Hosted Feature Layer View Does Not Update When Overwriting Hosted Feature Layer

MichaelVolz — Wed, 04 Jan 2023 18:46:44 GMT

Thanks for providing the code sample Josh, as this looks very powerful.

Re: Hosted Feature Layer View Does Not Update When Overwriting Hosted Feature Layer

RhettZufelt — Wed, 04 Jan 2023 19:59:43 GMT

I agree, thanks for the code. Seems to work well if I don't have any attachments and will get a lot of mileage out of it.

I see you can add add/update/delete attachments with this, but from what I can tell one must supply a dictionary listing all the attachements to add.

Do you know of a simple way to have this "maintain" the attachments of the features being added without somehow having to come up with the dictionary?

Thanks,

Re: Hosted Feature Layer View Does Not Update When Overwriting Hosted Feature Layer

jcarlson — Wed, 04 Jan 2023 20:19:14 GMT

Are the attachments coming from the source dataset, or are you trying to maintain things that have been attached to the destination?

I don't have an "overwrite" process that involves layers with attachments, but I do have a few layers with attachments that get updated from an external source. It's a different process, using the pandas module's compare method to identify edited rows between source and destination, then updating those features in place.

More complex to set up, but say you have a 50k feature layer and only 1000 actually get edited in the source, this would only edit those 1000 in the destination, leaving the other 49k untouched entirely. It's equivalent to just opening the attribute table and making edits, albeit much, much faster.

If you're interested, I presented on the topic at a conference last year, and you can find all the notes and Python code here: https://github.com/jdcarls2/ilgisa-2022/blob/main/hosted-copy/hosted-copy.ipynb

Re: Hosted Feature Layer View Does Not Update When Overwriting Hosted Feature Layer

RhettZufelt — Wed, 04 Jan 2023 21:53:33 GMT

Looking for a way to "Overwrite" my hosted layer with local (SDE) data.

The data is a combination of local data and hosted data, but I bring them all together in a local feature class, edit/modify, etc.

Then, I want to overwrite the hosted feature layer with the "new" data. If I don't make any schema changes, ensure layer id's dont change, etc. "most" of the time, I can overwrite the HFL with Pro, and it works just fine. However, as we all know, now and then, it will 'break' pretty much everything that is tied to it, and it all needs to be re-configured.

Trying to avoid this, and have had good success with truncating the HFL data and appending the SDE data to it. This works well, even maintains the attachments. HOWEVER, the Preserve Global IDs does NOT work with HFL. Documentation says it's a "known limitation" which past experience tells me that most likely ESRI does not plan on changing this.

Basically, trying to figure out a way to 'replace' a HFL's data without breaking any maps/popups/apps/etc. that are using it, but, need to maintain the attachments and the Global ID's, but am not able to figure out a tool/method that does both.

Interesting that I can go from HFL to SDE/MGDB with no issues, but can't go the other way.

Kind of defeats the purpose of a unique global id if it is changing on you......

Thanks again for any insight,

Re: Hosted Feature Layer View Does Not Update When Overwriting Hosted Feature Layer

JTessier — Wed, 21 Feb 2024 21:49:59 GMT

We are doing the same thing as @RhettZufelt , and get some interesting issues, have lots of retries built in to get it to work. Will ping colleagues to post some help as we figure things out.

Re: Hosted Feature Layer View Does Not Update When Overwriting Hosted Feature Layer

DanielWebb — Tue, 11 Jun 2024 19:26:42 GMT

Thanks for this code, Josh.

When I run it, it throws an error: 'Exception: The page was not displayed because the request entity is too large.'

You mentioned above about dividing it up into smaller chunks. Do you know the best way to do that?

I'm trying to update an AGOL layer from a file geodatabase.

Re: Hosted Feature Layer View Does Not Update When Overwriting Hosted Feature Layer

KateDouthat1 — Thu, 12 Dec 2024 14:56:52 GMT

Hi Everyone, I wanted to follow up on this thread because I have the same problem, but the hosted feature service is published by another organization, so I have no control over how they do their updates. Is there anyway to actually repair broken layers after an update?