Why does this script take so long to run?

3839
11
Jump to solution
01-17-2013 05:02 AM
Zeke
by
Regular Contributor III
The script below works in that it does what it's supposed to: creates a single Case feature from selected parcels. But it takes anywhere from 1.5 - 2 minutes to run, which seems a long time for a relatively simple operation. I could do it faster manually, although the people it's intended for probably couldn't  🙂

Any ideas on why it's so slow, an/or what could be done to speed it up? Thanks.

import arcpy, sys, os from arcpy import env env.overwriteOutput = True  temp_lyr_location = r"\Cases_Geodatabase.gdb\temp_parcel_lyr" parcel = r"Parcel"  try:     lyr = os.getcwd() + temp_lyr_location     mxd = arcpy.mapping.MapDocument("CURRENT") except Exception as e:     arcpy.AddError('ERROR initializing: /n' + e.message)  #-------------------------------------------------------------------------------  def MakeCaseFeature():     '''  Dissolve selected parcel features into one temporary lyr file.          Append it to the Cases feature class.          Delete the temporary lyr file. '''      clear = "CLEAR_SELECTION"     target = "Cases"     schemaType = "NO_TEST"     fld = "CENTRACT"    # no data is stored in this field     selectType = "HAVE_THEIR_CENTER_IN"      try:         # make temporary parcel lyr file from selected parcels, dissolving them         # into one feature         arcpy.Dissolve_management(parcel, lyr, fld)          # add case feature in temporary layer to Cases         arcpy.Append_management(lyr, target, schemaType, "", "")          # select new case feature         arcpy.SelectLayerByLocation_management(target, selectType, lyr)          # delete temporary layer         arcpy.Delete_management(lyr)          # clear selection on parcels         arcpy.SelectLayerByAttribute_management(parcel, clear)      except Exception as e:         arcpy.AddError("ERROR in MakeCaseFeature: \n" + e.message)  #-------------------------------------------------------------------------------  def main():     ''' Check how many parcels are selected. Count returns all parcels if none are         selected, or only selected ones if there are any.     '''     try:         count = int(arcpy.GetCount_management(parcel).getOutput(0))         arcpy.AddMessage(str(count) + " parcels selected")          # make sure parcels are selected before running rest of script, otherwise exit         if count >= 0 and count <= 10:             MakeCaseFeature()       # create the case feature             arcpy.RefreshActiveView()             arcpy.RefreshTOC()         else:             arcpy.AddError("No features selected! \n Please select at least one parcel feature. \n")             arcpy.AddError("Quitting the Create Case tool \n")      except Exception as e:         arcpy.AddError('ERROR in main: /n' + e.message)  #-------------------------------------------------------------------------------  if __name__ == '__main__':     main() 
Tags (2)
0 Kudos
1 Solution

Accepted Solutions
JamesCrandall
MVP Frequent Contributor
Chris:
Thanks for the tip on in-memory. I used that, not much time difference, though.


Is this correct?

env.workspace = "IN-MEMORY"


I thought it should be

env.workspace = "IN_MEMORY"



You might want to go back and make sure the in_memory space is correctly implemented (ie, copy your temporary FC's there and run the dissolve on those copied/in_memory FC's).  My experience has been huge performance improvements.

View solution in original post

0 Kudos
11 Replies
MathewCoyle
Frequent Contributor
I would scatter some time print outs to see where the slow down(s) are occurring.
import time

t0 = time.clock()
# some tool/code block
arcpy.AddMessage("Elapsed time: {0} seconds".format(int(time.clock() - t0)))


The refresh active view can take a while depending on the complexity of your map document as well.
0 Kudos
JamesCrandall
MVP Frequent Contributor
Instead of creating temporary data in a specific folder location, do all of your temp data and processing of that temp data within the IN_MEMORY workspace.  You should see significant performance gains.
0 Kudos
T__WayneWhitley
Frequent Contributor
Curious, what is the return time calling your function, MakeCaseFeature() ?
Also, I would make sure layers aren't unnecessarily being added to your display...may be some wasted overhead there.  In that case, it may be faster to 'copy' layers in-memory and operate on them there (so that you're minimizing updating anything in the view or TOC), I think what jamesfreddyc was saying.
0 Kudos
Zeke
by
Regular Contributor III
I've attached a png of the processing times in the results window. Dissolve takes the most time, averaging 50 seconds. I've also modified the script below, to use in memory workspace, although I'm not sure if this is correct; never used it before and don't find much in Help files about using it.

I'm only displaying three layers on my test runs, Parcels (~28k features), Case (0 features to start) and city boundary. Not all parcels are displayed, 28K is the total number of parcels; the map is typically zoomed in to areas of less than 100 parcels. Thanks, really appreciate the help.

import arcpy, sys, time #, os
from arcpy import env
env.overwriteOutput = True
env.workspace = "IN-MEMORY"

#temp_lyr_location = r"\Cases_Geodatabase.gdb\temp_parcel_lyr"
parcel = r"Parcel"

try:
    #lyr = os.getcwd() + temp_lyr_location
    lyr = r"temp_parcel_lyr"
    mxd = arcpy.mapping.MapDocument("CURRENT")
except Exception as e:
    arcpy.AddError('ERROR initializing: /n' + e.message)

#-------------------------------------------------------------------------------

def MakeCaseFeature():
    '''  Dissolve selected parcel features into one temporary lyr file.
         Append it to the Cases feature class.
         Delete the temporary lyr file. '''

    clear = "CLEAR_SELECTION"
    target = "Cases"
    schemaType = "NO_TEST"
    fld = "CENTRACT"    # no data is stored in this field
    selectType = "HAVE_THEIR_CENTER_IN"

    try:
        # make temporary parcel lyr file from selected parcels, dissolving them
        # into one feature
        t0 = time.clock()
        arcpy.Dissolve_management(parcel, lyr, fld)
        arcpy.AddMessage("Elapsed time dissolving: {0} seconds".format(int(time.clock() - t0)))

        # add case feature in temporary layer to Cases
        t0 = time.clock()
        arcpy.Append_management(lyr, target, schemaType, "", "")
        arcpy.AddMessage("Elapsed time appending: {0} seconds".format(int(time.clock() - t0)))

        # select new case feature
        t0 = time.clock()
        arcpy.SelectLayerByLocation_management(target, selectType, lyr)
        arcpy.AddMessage("Elapsed timeselecting case layer: {0} seconds".format(int(time.clock() - t0)))

        # delete temporary layer
        #arcpy.Delete_management(lyr)

        # clear selection on parcels
        t0 = time.clock()
        arcpy.SelectLayerByAttribute_management(parcel, clear)
        arcpy.AddMessage("Elapsed time clearing parcel selection: {0} seconds".format(int(time.clock() - t0)))

    except Exception as e:
        arcpy.AddError("ERROR in MakeCaseFeature: \n" + e.message)

#-------------------------------------------------------------------------------

def main():
    ''' Check how many parcels are selected. Count returns all parcels if none are
        selected, or only selected ones if there are any.
    '''
    try:
        t0 = time.clock()
        count = int(arcpy.GetCount_management(parcel).getOutput(0))
        arcpy.AddMessage(str(count) + " parcels selected")
        arcpy.AddMessage("Elapsed time counting parcels: {0} seconds".format(int(time.clock() - t0)))

        # make sure parcels are selected before running rest of script, otherwise exit
        if count >= 0 and count <= 10:
            MakeCaseFeature()       # create the case feature
            t0 = time.clock()
            arcpy.RefreshActiveView()
            arcpy.RefreshTOC()
            arcpy.AddMessage("Elapsed time refreshing view: {0} seconds".format(int(time.clock() - t0)))
        else:
            arcpy.AddError("No features selected! \n Please select at least one parcel feature. \n")
            arcpy.AddError("Quitting the Create Case tool \n")

    except Exception as e:
        arcpy.AddError('ERROR in main: /n' + e.message)
0 Kudos
MathewCoyle
Frequent Contributor
You want this
env.workspace = "in_memory"


Where is your source data located? If they are stored on a network it may account for slower than expected times. ~2 minutes for your operations really isn't that slow though.
0 Kudos
T__WayneWhitley
Frequent Contributor
Thought I'd chime in a little further.  Yes, you are right, there isn't much in the help files, only a little more added at 10.1 here, funny it's in the ModelBuilder section (see the small subtopic, Using in_memory in Python):

Using in-memory workspace
Desktop » Geoprocessing » ModelBuilder
Using in_memory in Python
http://resources.arcgis.com/en/help/main/10.1/index.html#//002w0000005s000000


in_memory is great when it works correctly - careful not to run out of memory, 'Delete_management' frees it up...

However, for further interest, see this blog entry by Kevin (for when you upgrade):

The new ScratchGDB and ScratchFolder environments
by Kevin Hibma on October 19, 2012
http://blogs.esri.com/esri/arcgis/2012/10/19/the-new-scratchgdb-and-scratchfolder-environments/
0 Kudos
ChrisSnyder
Regular Contributor III
Question/comment for recurvata: Why are you dissolving the selected parcels? Why not just use them directly (the feature layer with the selected parcels) to run your SelectByLocation?
0 Kudos
Zeke
by
Regular Contributor III
Chris:
A typical project where the user would run this tool is a rezoning request on one or more parcels. For their purposes (display and further analysis), they want one case feature covering all the parcels they've selected. Dissolve takes the temporary layer that consists of n number of parcels and makes them into one feature. Similar to the Merge tool in the Editor drop-down. The gp Merge tool would create a new feature class, which I don't want. The Select By Location is just an additional feature so that they don't have to manually select it once created. It's not really necessary. I can accomplish similar results manually myself much more quickly, but the end users don't really know Arc, so I'm trying to make it untrained monkey easier for them. But if there's a better method, please let me know. I'm not an ArcPy expert! Thanks.

Wayne:
Thanks for the info, that will come in handy, I'm sure.

Matt:
The data is stored on a network, but I've never noticed this amount of processing time before. It does work, so maybe they'll just have to accept it. I doubt they have a handle on how much time it should take anyway (maybe I don't either, eh?). Thanks for the tip on in-memory. I used that, not much time difference, though.
0 Kudos
JamesCrandall
MVP Frequent Contributor
Chris:
Thanks for the tip on in-memory. I used that, not much time difference, though.


Is this correct?

env.workspace = "IN-MEMORY"


I thought it should be

env.workspace = "IN_MEMORY"



You might want to go back and make sure the in_memory space is correctly implemented (ie, copy your temporary FC's there and run the dissolve on those copied/in_memory FC's).  My experience has been huge performance improvements.
0 Kudos