topic Re: Performance: Determine if Select tool is stuck in a loop? in ModelBuilder Questions

Performance: Determine if Select tool is stuck in a loop?

GregMiddleton — Thu, 10 Nov 2016 19:03:15 GMT

I'm working on processing some TomTom Multinet data in a model. I've joined the Streets NW Feature Class with the GC layer Feature Class for all of the regions of California which results in a layer with approximately 3.4 million rows.

I'm then using the Select Tool from the Analysis (Extract) toolbox to convert that joined layer into an actual Feature Class for further work.

All of this is inside a File GDB (on an SSD drive).

The tool has been running for 25 hours now on a physical desktop.

I ran my model in Edit mode up to the point of this Select because the model would get stuck showing 0%.

I then ran the Select by actually selecting from the tool box with the input intermediate layer from the model to see if I can see any progress.

Using 10.3.1

How can I figure out if it's really working? and I just need to be more patient

or if it's stuck in some kind of processing loop?

Re: Performance: Determine if Select tool is stuck in a loop?

DanPatterson_Retired — Thu, 10 Nov 2016 23:22:12 GMT

did you try to scale your model by starting with a much smaller input...try 1000, then 10,000 etc. A couple of data points on a graph while monitoring system resources might help... as you run out of physical memory, things start getting swapped out to disk slowing things down even more

Re: Performance: Determine if Select tool is stuck in a loop?

GregMiddleton — Fri, 11 Nov 2016 01:43:55 GMT

Yes, the model works with smaller input.

From a machine perspective, I'm not out of physical memory.

Arcmap is not showing CPU usage and is only showing 224k of actual RAM usage. So I don't think I'm out of memory?

Re: Performance: Determine if Select tool is stuck in a loop?

DanPatterson_Retired — Fri, 11 Nov 2016 02:34:24 GMT

arc* is 32 bit hence limited to 4 gig. What was the memory useage with smaller input and how did it scale up? I would try tiling your processes if possible

Tiled processing of large datasets—Help | ArcGIS for Desktop

but it doesn't look like any of the tools you are using them employ it specifically, and you are already using a file gdb so you are good there. If it were saving to a new file, try it in chunks, it could be quite possible that may cause an issue

Re: Performance: Determine if Select tool is stuck in a loop?

DuncanHornby — Fri, 11 Nov 2016 12:33:29 GMT

It's not entirely clear what you meant by join? A spatial join or a table join? It's always good practise to upload an image of your model as it helps us understand your problem. Please get into the habit of uploading images to support your question. You can always go back and edit your question to improve it. That said and assuming you meant a table join have you applied an attribute index to the fields that you are joining by? This can significantly improve performance.

Re: Performance: Determine if Select tool is stuck in a loop?

curtvprice — Wed, 16 Nov 2016 07:15:01 GMT

Is the join indexed? If not sometimes performance can be increased 10-100x by adding an attribute index.