topic Re: Model performance issue in ModelBuilder Questions

Model performance issue

TimWitt2 — Thu, 06 Apr 2017 14:53:26 GMT

Hey everybody,

I am working on a Model in which you can use your current street centerline file and transform it into a NENA standard street centerline file. This includes having a lot of drop down boxes and check boxes. It seems the more drop down boxes and checkboxes I use, the more my model takes to respond.

I was wondering if anyone has any suggestions to enhance the performance?

Thanks,

Tim

Re: Model performance issue

DuncanHornby — Mon, 10 Apr 2017 14:25:58 GMT

I've had a look at your model, it's quite big! You have hit the limitation of model builder and it is creaking under a complex interface. When you change anything a whole load of validation is run behind the scenes which you have no control over.

There is really only 1 option here and that is to convert the entire model into python code, I suspect it will also run faster and you can organize it to be much more efficient.

With python you have two options:

Develop it as a python script and then wire up an interface to it. This would be the easiest approach but could potentially lead to the same slow down.
Develop it as a python addin toolbox and define a new interface for it. An obvious one is as you have a lot of repetition for similar like inputs a value table would be a better choice. Your user would complete the table then it is just ONE parameter that needs validating not the dozen you have now.

If you are not comfortable in programming in python then there are many developers out there who could do this for you but obviously for a fee.

Re: Model performance issue

TimWitt2 — Mon, 10 Apr 2017 15:28:27 GMT

Duncan,

great post! I have dabbled in Python Addins before and you are right this would be the best way to go. I just need to figure out how to in cooperate all the dropdown boxes that I need.

Tim

Re: Model performance issue

ChrisDonohue__GISP — Mon, 10 Apr 2017 15:55:44 GMT

Tim:

That's a pretty intense model you have there. I can see why it is bogging down - Modelbuilder unfortunately is not known for speedy processes.

When you do get it worked out, some of us might bribe you for copy of the model/code. Like when the NENA standard finally becomes a mandatory requirement, me and this guy: jborgion

Hope the coding goes well,

Chris Donohue, GISP

Re: Model performance issue

JoeBorgione — Mon, 10 Apr 2017 16:15:09 GMT

Here's a thought; if NENA is going to force it's standards upon PSAPs, the very least they can do is provide a little coin to help defray the development costs...

Re: Model performance issue

TimWitt2 — Mon, 10 Apr 2017 17:42:57 GMT

Of course I would share it!

For our data I have the street model finished already, but I thought it might be a good idea to create this customized model, so other agencies don't need to go through all the hassle.

Re: Model performance issue

JoeBorgione — Mon, 10 Apr 2017 18:06:46 GMT

You are a good man Tim Witt....

Re: Model performance issue

DuncanHornby — Tue, 11 Apr 2017 09:04:55 GMT

The obvious place to be sharing it would be http://codesharing.arcgis.com/ rather than say attached to a reply in this thread.

Re: Model performance issue

TimWitt2 — Tue, 11 Apr 2017 17:56:00 GMT

Agreed, that is a good website.

Thanks again for the python addin suggestion, it is so much faster.

This is how it will look like:

The only issue I need to solve is how to manipulate the layout because I will need 50!!!! combo boxes.

Re: Model performance issue

DuncanHornby — Wed, 12 Apr 2017 08:42:05 GMT

I have got my terminology mixed up, it is a python toolbox not a python addin I was thinking about, my apologies. The image above looks like a toolbar to me? Python Addins are more for interacting with the map window where you want to create a geo-processing tool.

So one can have python scripts, python addins or python toolboxes. I have developed more complex interfaces with using python toolboxes. I suggest you look at this document it shows you how to construct value tables.

Re: Model performance issue

TimWitt2 — Wed, 12 Apr 2017 15:08:34 GMT

Again great information! Your code was really helpful!!!

I think this is what you meant.

Once they are done filling it out it will run through the "Add field" and "Calculate field" code.

Re: Model performance issue

DuncanHornby — Wed, 12 Apr 2017 15:17:57 GMT

Yeah that was what I was envisaging. Don't know and not tried it but if your inputs always have the same NENA field names you could try populating the first column and then it is up to the user to match the field name in "your field"?

Your code would then read from the value table the rows with values and process accordingly or you take the approach of starting with an empty table and they add just the fields that need processing? What ever works best for your logic.

Either way the interface will be much more responsive.

Re: Model performance issue

TimWitt2 — Wed, 12 Apr 2017 15:22:02 GMT

As you mentioned in your first paragraph, the Nena fields will be set fields. I will populate the first column with the Nena field names and the user will match their street field, with the corresponding Nena field name.

Now my next challenge will be how I access the values that the user chose and match them up with my calculate field code.

Re: Model performance issue

curtvprice — Mon, 17 Apr 2017 08:17:22 GMT

The slowness we see with big models with lots of tools and parameters is caused by the need for ModelBuilder to validate all the parameters for all the tools. This validation is very helpful when putting together a workflow but when many tools and parameters (including those in nested models) are there, it requires a LOT of validation, and, well, that takes time.

I disagree that it is all or nothing. A middle way I have found when I am running into parameter validation problems or very slow validation is to take some of the workflow and convert it to a Python function inside the Calculate Value tool. This hides that workflow from ModelBuilder's slow (and sometimes buggy) parameter validation.

If the tool has a extensive, complex input like the ones Tim is working with, I'll use model builder to test to get the parameters I want, then export model to script to get the python code written for me. Then I can use that code, either in Calculate Value or in a script tool.

However, from where this thread went it is obvious Tim does need a .pyt for this job so he can do the value table thing. But he figured out a lot of his workflow using ModelBuilder!

Re: Model performance issue

TimWitt2 — Mon, 17 Apr 2017 12:55:12 GMT

Great post Curtis! You are right, without model builder I wouldn't have known how to structure my python code. Not to mention, I didn't have to write many lines of code by exporting the model to python. Now I can just copy paste it into my python toolbox.