topic Model runs fine in ModelBuilder but hangs in tool mode in Geoprocessing Questions

Model runs fine in ModelBuilder but hangs in tool mode

ThomasDilts — Tue, 13 Nov 2018 21:24:33 GMT

I've got a model with iteration that runs fine from within ModelBuilder. In fact, I'd say that it is downright zippy taking about a minute for every 5 rows in a raster attribute table. When I run that same model as a tool using the same datasets it grinds to a halt getting hung up on iteration 3 or 4. Specifically it is the Cell Statistics tool and I'm using feedback to save the results from one iteration and then add the results from the next iteration and so on. I'm wondering if anyone has any idea what would be different enough between ModelBuilder and "tool mode" to account for such a large difference in performance. I'd hate to have to share a tool with others with the caveat that it can only be run from ModelBuilder especially when it is working so well otherwise.

Thanks

-Tom Dilts

Re: Model runs fine in ModelBuilder but hangs in tool mode

curtvprice — Thu, 15 Nov 2018 06:10:29 GMT

Just a wild guess, does your model have an output parameter that gets written on every iteration? This can really take a lot of time as the output layers are updated in the map on every iteration.

If you turn off adding outputs to display in Geoprocessing > Options that may help with your performance issue.

Re: Model runs fine in ModelBuilder but hangs in tool mode

ThomasDilts — Thu, 15 Nov 2018 15:12:16 GMT

Hi Curtis. Let me try that when I get to the office. That is a great suggestion. I've got a couple of ideas on how that problem might be remedied.

Re: Model runs fine in ModelBuilder but hangs in tool mode

ThomasDilts — Thu, 15 Nov 2018 22:16:27 GMT

Hi Curtis. I tried your suggestion. It was a really good suggestion, but it still didn't work. What I did was change the model parameters to string variables and then use those to name the output file with the feedback. Then I just removed that variable from "add to display" in ModelBuilder. At the same time I also made a few other changes. I changed the Cell Statistics tool to the Plus tool and set pyramids and raster statistics to not run by default. I suspect that the problem has to do with the iteration itself, but I still don't understand why it doesn't run in "tool mode" but does fine in "edit mode". One thing that I will try next is try to run it in ArcCatalog since there is no map display there. Another thought is that I should just convert the entire thing to a python script. I'd appreciate any more thoughts, comments, ideas, or feedback that folks might have.

Re: Model runs fine in ModelBuilder but hangs in tool mode

ThomasDilts — Thu, 15 Nov 2018 23:02:19 GMT

ArcCatalog didn't work any better. It just crashed instead of getting hung up. The freeze/crash always happens on the first time the feedback variable is used in the second iteration.

Re: Model runs fine in ModelBuilder but hangs in tool mode

DuncanHornby — Fri, 16 Nov 2018 11:24:39 GMT

Not sure if this is your issue but I will share my experience.

A few years ago I worked on a project that made extensive use of the cell statistics tool aggregating thousands of rasters. Whilst cell statistic tool would "accept" for example 3000 rasters as input (can't quite remember the actual number but in that ball park) it would grind to a halt. I over came the problem by simply doing in blocks of say 100 rasters, so for my 3000 example I ended up with 30 rasters which I then fed back into the Cell statistics tool for one final summary. This significantly reduced process time.

Re: Model runs fine in ModelBuilder but hangs in tool mode

ThomasDilts — Fri, 16 Nov 2018 16:54:35 GMT

Hi Duncan. Thanks for the heads up. I ended up dropping the Cell Statistics altogether replacing it with the Plus tool, but to no avail. I'm thinking that the next step might be to try exporting the model out to python, re-do the coding for the looping, and then see where that takes me. The good news is that it works in edit mode for what I want it to do.

Re: Model runs fine in ModelBuilder but hangs in tool mode

curtvprice — Mon, 19 Nov 2018 19:26:55 GMT

So just to be clear you did turn off add geoprocessing results to display in Geoprocessing / Options?

Re: Model runs fine in ModelBuilder but hangs in tool mode

ThomasDilts — Mon, 19 Nov 2018 19:36:27 GMT

Hi Curtis. Thanks for making that distinction. Originally I had not turned off turn off add geoprocessing results to display in Geoprocessing / Options but just had removed the layer as a parameter in the model. I've since tried running it with your suggestion of turning it off in the geoprocessing options from the geoporcessing menu in ArcMap as well, but to no avail. My next thought is to try exporting to python and re-doing the coding for the looping (does looping ever just work straight from ModelBuilder export?).

Re: Model runs fine in ModelBuilder but hangs in tool mode

curtvprice — Mon, 19 Nov 2018 19:45:11 GMT

OK, just wanted to make sure we're there for the good of the thread. I think Python is totally the way you want to go here because of the inherent issues of all these map or feature layers in memory. I think perhaps part of your issue is that you are modifying the dataset you are looping on with every iteration (the ArcCatalog crash leads me there). If you are doing this, you are likely to run into issues in Python too. In Python though it's easier to create temp datasets (or even lists or dictionaries) to store results as you go, and then have a final step to populate results.

Hope these thoughts help.

Re: Model runs fine in ModelBuilder but hangs in tool mode

ThomasDilts — Mon, 19 Nov 2018 19:50:12 GMT

Thanks for the tips. It might take me a day or two because I've got to finalize the last step of the process. I tend to start out in ModelBuilder to figure out the steps that I need to take before jumping to python (unless I know that it is going to involve a double loop).

Re: Model runs fine in ModelBuilder but hangs in tool mode

ThomasDilts — Thu, 17 Jan 2019 17:53:23 GMT

Hi everyone,

Sorry about the long delay. Converting the model to a python script did the trick. Now the tool iterates fine and works like a charm. Strangely, I still have no clue why the tool performs poorly as a model, but thank you everyone who contributed ideas. They were all helpful steps along the way. I'm expecting to a release a new geoprocessing toolbox sometime soon (Q1 of 2019) - the Lattice Connectivity Toolbox. The toolbox assesses habitat connectivity using a lattice or tessellation and a resistance kernel approach and is designed to serve as inputs into graph theoretic models. The toolbox is a refinement of the approach that I outline in a 2016 paper.

Dilts, T. E., Weisberg, P. J., Leitner, P., Matocq, M. D., Inman, R. D., Nussear, K. E., & Esque, T. C. (2016). Multiscale connectivity and graph theory highlight critical areas for conservation under climate change. Ecological Applications, 26(4), 1223-1237.