Collecting Moran's I output values

1939
5
Jump to solution
03-20-2019 03:13 PM
JonathanFleming
New Contributor III

I am running an iterator in model builder that ultimately takes the output from each iteration (a point feature class) and runs the spatial auto-correlation tool (global Moran's I).  I have about 450 output feature classes, and what I would like to have is the feature class name, index value, z-score, and p-value from each iteration (of Moran's I) added as a new record in a table.  These values are reported in the .HTML report and the model run summary window, but I can't figure out how to extract them and add them to a table. 

If I were doing this with a statistical model in R, I could simply call the model output values (index, p-value, etc.).  But Ideally I would like to do this in model builder.  I'm not opposed to writing a python script, but before I go down that road I'm wondering if there's a more direct way to do it with existing geoprocessing or model builder tools.

As an additional note, I would also like to get the default calculated bandwith from the Kernel density tool output to a table similar to the Moran's I values.

Any help is greatly appreciated.

Running ArcGIS Pro 2.3

0 Kudos
1 Solution

Accepted Solutions
curtvprice
MVP Esteemed Contributor

Cool! So glad this worked nicely for you!

Sorry about the over-writing, my bad. The open() should use the option 'a' not 'w'.  

https://docs.python.org/2.7/library/functions.html#open

In Python the file doesn't have to exist to an 'a' write, but if you don't hard-code the pathname you need to have the file existing already. The way this is normally handled with ArcGIS tools is the tool takes parameters for an existing folder and a (new) file name (look at the Create Feature Class tool for an example) so the parameter validation will work.

I also found that chr(10) works better for newline. (12 was just wrong!).  I tried os.linesep() but that is getting written as two newlines for some reason. (Again can't use \n because it doesn't work inside Calculate Value (or Calculate Field) code blocks, I think this has something with ArcGIS's geoprocessing tool framework.)

>>> os.linesep
'\r\n'
>>> ord(os.linesep[0]), ord(os.linesep[1])
(13, 10)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Code sample corrected up-thread.

One more thing, I usually connect all variables used in the Calculate Value function as preconditions to the Calculate Value tool, just to document what's going on. This has the side benefit of ensuring your Calculate Value will not execute until the the Moran's I tool completes and its outputs are updated!

View solution in original post

5 Replies
DanPatterson_Retired
MVP Emeritus

fake example with fake results so you get the idea how to collect in a loop

moran_result = [1,2,3,"report"]  # fake results
number_results = []
html_results = []
for i in range(3):
    index, z_score, p_val, report = moran_result
    number_results.append([index, z_score, p_val])
    html_results.append(report)
args = [number_results, html_results]
print("number results\n{}\nhtml reports\n{}".format(*args))
number results

[[1, 2, 3], [1, 2, 3], [1, 2, 3]]
html reports
['report', 'report', 'report']
0 Kudos
curtvprice
MVP Esteemed Contributor

This Model Builder workflow is done using the Collect Values tool, with the output set up as an output parameter for the iterating model. Then from the model that calls your iterating model (I call this a "driver" model), you pass the output from your iterating model to the Merge tool in the driver model. The trickiest part here is to use your iterating model variables to generate paths that are unique for each iteration.

Examples of using Model Only tools in ModelBuilder—Help | ArcGIS Desktop 

But, this would require you dump the Moran's i outputs (which are kicked out as derived variables from the Global Moran's I tool, so they show up in Model Builder as the tool's outputs) to a table, and then merge all the tables in the driver model. Complicated!

If you know a little Python, you could do this a different way by writing a little python function to write a row of data to a file, even a text file (with open()), which can be run on each iteration using the Calculate Value tool, each iteration appending a row of data to the output file.  In your situation this may be a simpler approach, especially since you know some Python.  Here's an example of what I'm talking about. The things in %% are model elements. the "r" is needed to protect backslashes in paths. UPDATE   code corrected

# Calculate Value Expression
write_results("%Index%","%Zscore%", "%PValue%", r"%Report File%", r"%Input features%", r"%Output file%")
# Code Block
def write_results(index, zscore, pvalue, reportfile, infeatures, outfile):
    rec = ",".join([infeatures, index, zscore, pvalue, reportfile])
    with open(outfile, "a") as f:
        f.write(rec + chr(10)) # backslash n does not work inside calculate value
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

As for the Kernel Density default calculated "Silverman's rule of thumb" radius value, this is not exposed as a derived output so my suggestion to capture this would be to copy the messages from the model run and parse them with a python script or your favorite text file editor's macro capability. Sometimes that's the only way... At least since you are using model builder, so these GP Messages can be easily captured and pasted into a text editor.

JonathanFleming
New Contributor III

Thanks, this is very helpful.  I think it's almost there, but with my limited python skills I can't seem to get the last part going.  The line of data that gets written to the output file keeps getting overwritten with every iteration; so I end up with a text file with only 1 line of data.  This may be part of the issue you alluded to in the first part of your reply, or an issue with not being able to use \n for a new line.  I'm also a bit curious about the reason for adding the up arrow character at the end...?  I included an image of the model, calculate value code, and output below.  This is actually a part of a much larger model, but I've broken it down into multiple pieces to use the iterators (the test gdb I'm using only has 3 point feature classes instead of >4000).  Thanks a ton!Model, calculation, and output

curtvprice
MVP Esteemed Contributor

Cool! So glad this worked nicely for you!

Sorry about the over-writing, my bad. The open() should use the option 'a' not 'w'.  

https://docs.python.org/2.7/library/functions.html#open

In Python the file doesn't have to exist to an 'a' write, but if you don't hard-code the pathname you need to have the file existing already. The way this is normally handled with ArcGIS tools is the tool takes parameters for an existing folder and a (new) file name (look at the Create Feature Class tool for an example) so the parameter validation will work.

I also found that chr(10) works better for newline. (12 was just wrong!).  I tried os.linesep() but that is getting written as two newlines for some reason. (Again can't use \n because it doesn't work inside Calculate Value (or Calculate Field) code blocks, I think this has something with ArcGIS's geoprocessing tool framework.)

>>> os.linesep
'\r\n'
>>> ord(os.linesep[0]), ord(os.linesep[1])
(13, 10)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Code sample corrected up-thread.

One more thing, I usually connect all variables used in the Calculate Value function as preconditions to the Calculate Value tool, just to document what's going on. This has the side benefit of ensuring your Calculate Value will not execute until the the Moran's I tool completes and its outputs are updated!

JonathanFleming
New Contributor III

This is working beautifully.  Thanks so much.

I haven't made the whole process very elegant yet, but you've helped me get exactly what I need for now.  I'm using a slightly cumbersome method to get the new lines to work.  Instead of the chr(10) newline, I just added a semi-colon to separate the results.  Then I open the text file with all the output values in word and replace all the semi-colons with a new line (^l).  That gets saved and opened in excel to create the .csv.  It isn't pretty, but it's fine for the current project I'm working on.

Thanks again!