topic model with 3 tools, including 3D spatial selection, taking long in Pro in Geoprocessing Questions

model with 3 tools, including 3D spatial selection, taking long in Pro

SamanthaGoodchild-Brown — Tue, 25 Jun 2024 06:13:38 GMT

In Pro 3.1.5 I have created a 3 step tool to classify open multipatches based on height order so I can classify ground from roof.
1. Iterate through a table of 3D lines
2. use each line to select open multipatches by 3D intersection
3. calculate field with sequential numbering (this will always classify my lowestZ as '1')
I have +200k instances to run but I'm finding it is slowing down very quickly and its only worth running about 750 at a time! I'm trying to identify if its just the nature of my animal, a bit of a bug or if theres something I can adjust to get better speed (and if so, what?).
Iterating though 742 records I get a time of 32 mins if I have just rebooted or just opened the program. In that period the seqnum goes from taking under 1 second to over 4 seconds/iteration. The number of records selectable can range from 2 to 9 with the most common quanitity being 2 (roof and floor of bld). If I run another small file of 887 records having just opened the program, it takes 55min. If I merge the 2 they take 4h55min with the seqnum duration building up to 15sec. If I keep the 2 seperate and run as a batch, it takes 4h17min. So theres a pattern.
I have tried playing with my graphics card settings (NVIDIA TX A2000 ). I have tried fiddling with indexing settings in Pro options. Both have made no difference so far.
I am processing on a laptop with i7, 32gb RAM. I've maxed my virtual memory in advanced settings - theres loads, like over a terrabyte between 3 drives. My task manager tells me I have 14 cores.

Does anyone have any insight? I'm running out of ideas and the + 200k are just with this update run.
I recall from over a decade ago that iterate holds things in memory, also I know 3D data is heavier. Is this as good as its going to get?

Re: model with 3 tools, including 3D spatial selection, taking long in Pro

DanPatterson — Tue, 25 Jun 2024 07:13:24 GMT

are you monitoring memory consumption?

are all datasets in the same coordinate system?

have you considered moving the model to a python script?

Re: model with 3 tools, including 3D spatial selection, taking long in Pro

DuncanHornby — Tue, 25 Jun 2024 15:11:03 GMT

You don't say anything about how your data is stored and where? Is your data local, network or a service being throttled over the internet? Is your data a file geodatabase or shapefiles or some transfer format like geojson? If a shapefile you need to add a spatial index, but best performance is always a local file geodatabase.

Re: model with 3 tools, including 3D spatial selection, taking long in Pro

SamanthaGoodchild-Brown — Mon, 01 Jul 2024 05:44:09 GMT

Hi Dan

Thanks for the reply.

So my data is all on the same system. The interesting thing about my memory, so far, is that is sits in the region of 30% and doesn't change much for the full duration or untill I kill it. If I try to run a second project with another model (doing the same) it goes to about 50% and then same thing.

I have not considered moving the model to a python script as its outside my comfort zone so I'm not keen to try if I'm not sure it will adress the issue. I understand that is helpful if you have a very complex model, but this is 3 steps...

Re: model with 3 tools, including 3D spatial selection, taking long in Pro

SamanthaGoodchild-Brown — Mon, 01 Jul 2024 05:55:17 GMT

Hi. Thanks for the reply.

My data is all local, SDD, in the same file gdb.

Re: model with 3 tools, including 3D spatial selection, taking long in Pro

DanPatterson — Mon, 01 Jul 2024 07:30:16 GMT

3 steps with a lot of iteration, which may be the issue. Have you considered running submodels?

Add a submodel to a model—ArcGIS Pro | Documentation

the increasing time suggests that intermediate results are being written to disk before the final result which is going to slow things down.

Re: model with 3 tools, including 3D spatial selection, taking long in Pro

SamanthaGoodchild-Brown — Mon, 01 Jul 2024 08:11:24 GMT

Hi Dan

I definitely had not thought of using a submodel. Because only one of the tools is an iterator. But Thanks to you, ... yes... maybe running the select by location and update attributes as a submodel will handle it differently....(a selection of a selection). So I've now set that up and it HAS given me markedly improved results. My processing speed for 742 records has gone from 34 min to 14.5min. More importatly, I haven't seen that horrible creep of processing duration. I'm so impressed and relieved I think I will mark this as the solution. Thank you * gazzilions.