idea Improve performance of Delete Rows Geoprocessing tool when source data is ArcOnline feature service in ArcGIS Pro Ideas

Improve performance of Delete Rows Geoprocessing tool when source data is ArcOnline feature service

LindsayRaabe_FPCWA — Wed, 14 Jul 2021 12:15:10 GMT

So with Pro 2.8, we got the much wanted improvements to the Append tool when appending features to a feature service in ArcOnline. This reduced upload times when appending hundreds or thousands of features from many, many mintutes/hours to mere seconds.

It would be great if the same sort of improvement could be applied to the Delete Rows tool for when you're trying to do the opposite (still painfully slow).

Re: Improve performance of Delete Rows Geoprocessing tool when source data is ArcOnline feature service

SarahAmbrose1 — Mon, 26 Jul 2021 23:59:52 GMT

Truncate Table is a nice alternative that uses faster database commands to delete rows from the table.

https://pro.arcgis.com/en/pro-app/latest/tool-reference/data-management/truncatetable.htm

But if you require the delete to only work against the selection, Truncate isn't the right tool as it cannot honor the selection. There is no way to undo or recover from the command which is how it can operate so fast compared to row-by-row delete operations.

Re: Improve performance of Delete Rows Geoprocessing tool when source data is ArcOnline feature service

LindsayRaabe_FPCWA — Tue, 27 Jul 2021 00:30:05 GMT

Thanks for the suggestion @SarahAmbrose1 . I had not previously looked at this tool. Unfortunately, reading through the documentation at the link you provided, this doesn't look like it will work for this problem;

This tool does not support a feature service layer as input.

Though, I may start implementing it in other models that are modifying Feature Classes in place of the Delete Rows tool. I'll also test to see if it honours Definition Queries and Make Feature Layer for removing only a subset of rows in a table.

Re: Improve performance of Delete Rows Geoprocessing tool when source data is ArcOnline feature service

RussellBrennan — Tue, 27 Jul 2021 19:44:25 GMT

Hi @LindsayRaabe_FPCWA

At the next (Autumn/Winter 2021) release of Pro the Truncate tool will be available to you when working with AGOL hosted services AND you are connected as a portal admin or service owner.

We are also looking to make the improvements you request for deleting features/rows in upcoming releases but this likely will not make it into the software until the release after next (approx. Spring/Summer 2022 at the earliest).

As a sort of 'hack/workaround' you can use the button to enable undo on the Delete Rows tool. This should execute very quickly because it is first done on the client (Pro). When you then you click save, this is when we will send edits to the server. Doing it this way will result in fewer edit requests to the server which should speed up the deletion.

We have some doc on this delayed editing behavior here: https://pro.arcgis.com/en/pro-app/latest/help/data/services/edit-web-feature-layers.htm but you would sort of have to read between the lines to figure this out on your and how it might benefit in these sorts of situations.

Hope this helps.

Russell

Re: Improve performance of Delete Rows Geoprocessing tool when source data is ArcOnline feature service

LindsayRaabe_FPCWA — Wed, 28 Jul 2021 00:05:55 GMT

Hi @RussellBrennan. Thanks for the info - thats welcome news. The workaround you listed is also good to know, though won't help in most cases for me we're using this in Model Builder, so no Undo/Save option. The truncate tool though should do the trick when updated (I and my Team are admins so will work fine) - though unless you can confirm for me, I'll need to test to see if it honours Make Feature Layere to only delete a subset of rows and now the whole table???

I've already swapped Delete Rows out for the Truncate Table tool in some models where all rows are the target and it's a Feature Class, but the killers are those Feature Services, so looking forward to seeing that release.

Re: Improve performance of Delete Rows Geoprocessing tool when source data is ArcOnline feature service - Status changed to: Under Consideration

RussellBrennan — Wed, 28 Jul 2021 13:05:06 GMT

Re: Improve performance of Delete Rows Geoprocessing tool when source data is ArcOnline feature service

RussellBrennan — Wed, 28 Jul 2021 13:15:39 GMT

@LindsayRaabe_FPCWA The Truncate tool operates at the data source level (geodatabase feature class or feature service layer) not at the layer level such as feature layers created in Pro maps or geoprocessing workflows. As such it will delete all rows not just a subset defined by a feature layer.

Re: Improve performance of Delete Rows Geoprocessing tool when source data is ArcOnline feature service

LindsayRaabe_FPCWA — Thu, 29 Jul 2021 00:47:20 GMT

@RussellBrennan Thanks for the extra info - saved me some trouble with testing.

Re: Improve performance of Delete Rows Geoprocessing tool when source data is ArcOnline feature service

LindsayRaabe_FPCWA — Tue, 19 Apr 2022 07:42:24 GMT

@RussellBrennan

Hi Russell. Just wondering if you have any updates on when we might see the improvements you mentioned above to the Truncate Table and Delete Rows tools?

Re: Improve performance of Delete Rows Geoprocessing tool when source data is ArcOnline feature service

LindsayRaabe_FPCWA — Wed, 22 Jun 2022 03:24:55 GMT

I've just done a quick review of the documentation for the Truncate Table tool and Delete Rows tool. Nothing has changed here so obviously didn't happen as part of the Autumn/Winter 2021 release of Pro (Truncate Table) or Spring/Summer 2022 release (Delete Rows). Hopefully just delayed and still on the way!

Re: Improve performance of Delete Rows Geoprocessing tool when source data is ArcOnline feature service

LindsayRaabe_FPCWA — Thu, 21 Jul 2022 06:36:45 GMT

Just discovered that the Truncate Table does work with feature services (for administrators only) from ArcGIS Pro 3.0.

This is a great step forward, but still waiting for the performance improvements to Delete Rows where only some rows are intended for deletion.

Re: Improve performance of Delete Rows Geoprocessing tool when source data is ArcOnline feature service

Mbizzy — Sun, 02 Oct 2022 15:21:41 GMT

Since I regularly delete hundreds or thousands of records in online feature services, one work around I have started using the rest admin page and input the object ids I wish to delete. So far I am finding this to be orders of magnitude faster: 45 minutes down to 5 in some cases.

But kudos given to Idea since the cloud is about big data.

Re: Improve performance of Delete Rows Geoprocessing tool when source data is ArcOnline feature service

LindsayRaabe_FPCWA — Mon, 03 Oct 2022 00:20:36 GMT

@Mbizzy I have used that before, but still found it would time out after a a few thousand records and I'd have to hit refresh and submit the query again, repeatedly until all the records were gone.

Re: Improve performance of Delete Rows Geoprocessing tool when source data is ArcOnline feature service - Status changed to: Implemented

RussellBrennan — Mon, 13 Feb 2023 20:48:16 GMT

@LindsayRaabe_FPCWA I am happy to report that this should be a lot faster in Pro 3.1. We are now grouping edits into fewer round trips to the feature service so it should work a lot faster especially in cases where you are deleting a lot of data. This is for both Delete Rows and Delete Features tools.

We have also made some server side changes that allow us to put more edits into each ApplyEdits request we send to the feature service (Asynchronous ApplyEdits and ApplyEdits with UploadID) . Look for these server side changes in Enterprise 11.1 and the next (summer update) to ArcGIS Online.

Re: Improve performance of Delete Rows Geoprocessing tool when source data is ArcOnline feature service

LindsayRaabe_FPCWA — Tue, 14 Feb 2023 00:14:04 GMT

@RussellBrennan Great news! I look forward to trying it out. Do you have an ETA on 3.1 release date?

Enterprise isn't something we use so won't get that benefit, but imagine many others will!

Re: Improve performance of Delete Rows Geoprocessing tool when source data is ArcOnline feature service

KoryKramer — Tue, 14 Feb 2023 00:48:20 GMT

ArcGIS Pro 3.1 is currently scheduled to be available February 23rd, 2023.

Re: Improve performance of Delete Rows Geoprocessing tool when source data is ArcOnline feature service

LindsayRaabe_FPCWA — Tue, 14 Feb 2023 00:50:31 GMT

@KoryKramer Awesome. Thanks! Been hanging out for this one as our org refused to upgrade to 3.0 which is where I've been doing all my development.

Re: Improve performance of Delete Rows Geoprocessing tool when source data is ArcOnline feature service

KoryKramer — Tue, 14 Feb 2023 00:54:05 GMT

Good to hear @LindsayRaabe_FPCWA Just remember that leapfrogging 3.0 doesn't mean you get to leapfrog breaking changes. Make sure that your organization reviews https://www.esri.com/arcgis-blog/products/arcgis-pro/announcements/get-ready-for-arcgis-pro-3-0/ and understands https://pro.arcgis.com/en/pro-app/latest/get-started/migrate-2x-to-3.htm .

Re: Improve performance of Delete Rows Geoprocessing tool when source data is ArcOnline feature service

LindsayRaabe_FPCWA — Tue, 14 Feb 2023 01:02:05 GMT

@KoryKramer Thanks. I've had a good read of those articles previously. The logic behind skipping (which I don't agree with) is to let other orgs find the bugs and have them fixed by 3.1, though as we all know, new bugs sneak in with all releases, so seems kinda pointless (especially considering I'm yet to find any bugs in 3.0 that have impacted our workflows.).

Anyways, nearly there with 3.1 so I'm happy to wait another week!

Re: Improve performance of Delete Rows Geoprocessing tool when source data is ArcOnline feature service

Mbizzy — Tue, 14 Feb 2023 13:06:18 GMT

Cant wait to try 3.1 when released!