Model Builder + Batch Creator do not work together

3683
6
01-08-2013 07:23 AM
kevinbrown1
New Contributor
I love using model builder.  I also love the batch creator.  However, they do not seem to be compatible.  When I open a custom model in the batch processor, the following seems to happen:

Each analysis function is completed for ALL runs before the next step is begun for ANY run.  For example if my model uses "plus" for two rasters, and then reclassifies the result, "plus" is completed for each run before moving on to "reclassify."  The result is that any intermediary file that has a temporary, auto-generated name is written-over 50 times before the next step.  This means that regardless of the unique inputs used for each run in the batch creator, the model ends up only carrying ONE version of each intermediary file into each next step.  Egad.

Is there a way to specify that the batch creator wait until a single run COMPLETES before initializing the next one?

Another consequence of this problem is that for a complex model which needs to be run 100+ times on different input, the batch creator seems to insist on loading ALL inputs to memory before initializing any analysis (which could be 300 GB - so yea, not going to work).  Is there a way to just load run 1, analyze run 1, save, load run 2, analyze run 2, save?  Would make my work 90% easier.
0 Kudos
6 Replies
kevinbrown1
New Contributor
bump................
0 Kudos
curtvprice
MVP Esteemed Contributor
Batch is an older functionality that does not work with ModelBuilder.

I recommend instead creating a new model with an iterator and embedding your model inside it.

This has the added benefit that you can use ModelBuilder to name the outputs using model variables, instead of having to create and validate each output dataset name.
0 Kudos
kevinbrown1
New Contributor
Batch is an older functionality that does not work with ModelBuilder.

I recommend instead creating a new model with an iterator and embedding your model inside it.

This has the added benefit that you can use ModelBuilder to name the outputs using model variables, instead of having to create and validate each output dataset name.


Model builder absolutely *does* work in batch creator in 10.1, BUT only if each intermediary file is set as a "parameter."  Could you please elaborate what you mean by "does not work?"

Also, I thought you could only have one iterator per model.  I'm not just iterating down a single list of input.  It's more like:

in1.tif + in2.tif = out1
in3.tif + in4.tif = out2

That alone would need two iterators.  I'm not sure if that kind of iteration is supported, how to even approach it.
0 Kudos
curtvprice
MVP Esteemed Contributor
Model builder absolutely *does* work in batch creator in 10.1, BUT only if each intermediary file is set as a "parameter."  Could you please elaborate what you mean by "does not work?"


I've never tested batch with a model that all the intermediate datasets set up as parameters; I stand corrected. When I've tried to batch a model (@ 10.0), my experience has been that the first tool in the chain runs for batch iterations, then the second, etc. Maybe your approach would get around that -- but it sure sounds cumbersome.


Also, I thought you could only have one iterator per model.  I'm not just iterating down a single list of input.  It's more like:

in1.tif + in2.tif = out1
in3.tif + in4.tif = out2
That alone would need two iterators.  I'm not sure if that kind of iteration is supported, how to even approach it.


This is true. However, if there is a pattern to your file names, you could use Calculate Value to generate the paths you need and use a for iterator. In your example:

path1 = "in{}.tif".format(%n% * 2 + 1)
path2 = "in{}.tif".format(%n% * 2 + 2)
0 Kudos
kevinbrown1
New Contributor
Thanks.  It's a real shame that custom models don't interface better with batch.  I still can't understand why from a design standpoint ESRI would allow this limitation to exist.  It would be a simple matter of not beginning the batch's next row until the previous is complete and saved as output.  It's so easy to set up really huge, complicated, lists of parameters in excel and just copy-paste into batch. Oh well.
0 Kudos
curtvprice
MVP Esteemed Contributor
This is something that has bugged me for a while.

I've attached my test model, which runs two Calculate Value tools in a chain.

If I give the inputs

batch row 1: 10 11
batch row 2: 20 21

The model tool chain runs in "parallel", not in sequence:

Messages
Executing: Model2 10 11
Start Time: Tue Jan 15 12:37:02 2013
Executing (Model 1): Model1 10 11
Start Time: Tue Jan 15 12:37:02 2013
Executing (Calculate Value): CalculateValue 10 # Variant
Start Time: Tue Jan 15 12:37:02 2013
Value = 10
Executing (Calculate Value): CalculateValue 20 # Variant
Value = 20
Succeeded at Tue Jan 15 12:37:02 2013 (Elapsed Time: 0.00 seconds)
Executing (Calculate Value (2)): CalculateValue 11 # Variant
Start Time: Tue Jan 15 12:37:02 2013
Value = 11
Executing (Calculate Value (2)): CalculateValue 21 # Variant
Value = 21
Succeeded at Tue Jan 15 12:37:02 2013 (Elapsed Time: 0.00 seconds)
Succeeded at Tue Jan 15 12:37:02 2013 (Elapsed Time: 0.00 seconds)
Succeeded at Tue Jan 15 12:37:02 2013 (Elapsed Time: 0.00 seconds)


I tried nesting the model in another model and batching that; I get the same results.

The only workaround I can think of is to wrap the model in a script tool that calls the model. This does batch the model in proper order:

# script tool that wraps model
import sys
import os
import arcpy
Here = os.path.dirname(sys.argv[0])
arcpy.ImportToolbox(os.path.join(Here,"ModelBatchTest.tbx"),"mbt")
p1 = arcpy.GetParameterAsText(0)
p2 = arcpy.GetParameterAsText(1)
arcpy.Model1_mbt(p1,p2)
arcpy.AddMessage(arcpy.GetMessages(0))


Results of "batching" script tool:
Executing: Script 10 11
Start Time: Tue Jan 15 12:56:45 2013
Running script Script...
Executing: Model1 10 11
Start Time: Tue Jan 15 12:56:46 2013
Executing (Calculate Value): CalculateValue 10 # Variant
Start Time: Tue Jan 15 12:56:46 2013
Value = 10
Succeeded at Tue Jan 15 12:56:46 2013 (Elapsed Time: 0.00 seconds)
Executing (Calculate Value (2)): CalculateValue 11 # Variant
Start Time: Tue Jan 15 12:56:46 2013
Value = 11
Succeeded at Tue Jan 15 12:56:46 2013 (Elapsed Time: 0.00 seconds)
Succeeded at Tue Jan 15 12:56:46 2013 (Elapsed Time: 0.00 seconds)
Completed script Script...
Executing: Script 20 21
Running script Script...
Executing: Model1 20 21
Start Time: Tue Jan 15 12:56:46 2013
Executing (Calculate Value): CalculateValue 20 # Variant
Start Time: Tue Jan 15 12:56:46 2013
Value = 20
Succeeded at Tue Jan 15 12:56:46 2013 (Elapsed Time: 0.00 seconds)
Executing (Calculate Value (2)): CalculateValue 21 # Variant
Start Time: Tue Jan 15 12:56:46 2013
Value = 21
Succeeded at Tue Jan 15 12:56:46 2013 (Elapsed Time: 0.00 seconds)
Succeeded at Tue Jan 15 12:56:46 2013 (Elapsed Time: 0.00 seconds)
Completed script Script...
Succeeded at Tue Jan 15 12:56:46 2013 (Elapsed Time: 1.00 seconds)
0 Kudos