topic Why does this script take so long to run? in Python Questions

Why does this script take so long to run?

Zeke — Thu, 17 Jan 2013 13:02:24 GMT

The script below works in that it does what it's supposed to: creates a single Case feature from selected parcels. But it takes anywhere from 1.5 - 2 minutes to run, which seems a long time for a relatively simple operation. I could do it faster manually, although the people it's intended for probably couldn't 🙂

Any ideas on why it's so slow, an/or what could be done to speed it up? Thanks.

import arcpy, sys, os from arcpy import env env.overwriteOutput = True  temp_lyr_location = r"\Cases_Geodatabase.gdb\temp_parcel_lyr" parcel = r"Parcel"  try:     lyr = os.getcwd() + temp_lyr_location     mxd = arcpy.mapping.MapDocument("CURRENT") except Exception as e:     arcpy.AddError('ERROR initializing: /n' + e.message)  #-------------------------------------------------------------------------------  def MakeCaseFeature():     '''  Dissolve selected parcel features into one temporary lyr file.          Append it to the Cases feature class.          Delete the temporary lyr file. '''      clear = "CLEAR_SELECTION"     target = "Cases"     schemaType = "NO_TEST"     fld = "CENTRACT"    # no data is stored in this field     selectType = "HAVE_THEIR_CENTER_IN"      try:         # make temporary parcel lyr file from selected parcels, dissolving them         # into one feature         arcpy.Dissolve_management(parcel, lyr, fld)          # add case feature in temporary layer to Cases         arcpy.Append_management(lyr, target, schemaType, "", "")          # select new case feature         arcpy.SelectLayerByLocation_management(target, selectType, lyr)          # delete temporary layer         arcpy.Delete_management(lyr)          # clear selection on parcels         arcpy.SelectLayerByAttribute_management(parcel, clear)      except Exception as e:         arcpy.AddError("ERROR in MakeCaseFeature: \n" + e.message)  #-------------------------------------------------------------------------------  def main():     ''' Check how many parcels are selected. Count returns all parcels if none are         selected, or only selected ones if there are any.     '''     try:         count = int(arcpy.GetCount_management(parcel).getOutput(0))         arcpy.AddMessage(str(count) + " parcels selected")          # make sure parcels are selected before running rest of script, otherwise exit         if count >= 0 and count <= 10:             MakeCaseFeature()       # create the case feature             arcpy.RefreshActiveView()             arcpy.RefreshTOC()         else:             arcpy.AddError("No features selected! \n Please select at least one parcel feature. \n")             arcpy.AddError("Quitting the Create Case tool \n")      except Exception as e:         arcpy.AddError('ERROR in main: /n' + e.message)  #-------------------------------------------------------------------------------  if __name__ == '__main__':     main()

Re: Why does this script take so long to run?

MathewCoyle — Sat, 11 Dec 2021 10:21:03 GMT

I would scatter some time print outs to see where the slow down(s) are occurring.

import time

t0 = time.clock()
# some tool/code block
arcpy.AddMessage("Elapsed time: {0} seconds".format(int(time.clock() - t0)))

The refresh active view can take a while depending on the complexity of your map document as well.

Re: Why does this script take so long to run?

JamesCrandall — Thu, 17 Jan 2013 13:41:47 GMT

Instead of creating temporary data in a specific folder location, do all of your temp data and processing of that temp data within the IN_MEMORY workspace. You should see significant performance gains.

Re: Why does this script take so long to run?

T__WayneWhitley — Thu, 17 Jan 2013 14:23:13 GMT

Curious, what is the return time calling your function, MakeCaseFeature() ?
Also, I would make sure layers aren't unnecessarily being added to your display...may be some wasted overhead there. In that case, it may be faster to 'copy' layers in-memory and operate on them there (so that you're minimizing updating anything in the view or TOC), I think what jamesfreddyc was saying.

Re: Why does this script take so long to run?

Zeke — Sat, 11 Dec 2021 10:21:06 GMT

I've attached a png of the processing times in the results window. Dissolve takes the most time, averaging 50 seconds. I've also modified the script below, to use in memory workspace, although I'm not sure if this is correct; never used it before and don't find much in Help files about using it.

I'm only displaying three layers on my test runs, Parcels (~28k features), Case (0 features to start) and city boundary. Not all parcels are displayed, 28K is the total number of parcels; the map is typically zoomed in to areas of less than 100 parcels. Thanks, really appreciate the help.

import arcpy, sys, time #, os
from arcpy import env
env.overwriteOutput = True
env.workspace = "IN-MEMORY"

#temp_lyr_location = r"\Cases_Geodatabase.gdb\temp_parcel_lyr"
parcel = r"Parcel"

try:
    #lyr = os.getcwd() + temp_lyr_location
    lyr = r"temp_parcel_lyr"
    mxd = arcpy.mapping.MapDocument("CURRENT")
except Exception as e:
    arcpy.AddError('ERROR initializing: /n' + e.message)

#-------------------------------------------------------------------------------

def MakeCaseFeature():
    '''  Dissolve selected parcel features into one temporary lyr file.
         Append it to the Cases feature class.
         Delete the temporary lyr file. '''

    clear = "CLEAR_SELECTION"
    target = "Cases"
    schemaType = "NO_TEST"
    fld = "CENTRACT"    # no data is stored in this field
    selectType = "HAVE_THEIR_CENTER_IN"

    try:
        # make temporary parcel lyr file from selected parcels, dissolving them
        # into one feature
        t0 = time.clock()
        arcpy.Dissolve_management(parcel, lyr, fld)
        arcpy.AddMessage("Elapsed time dissolving: {0} seconds".format(int(time.clock() - t0)))

        # add case feature in temporary layer to Cases
        t0 = time.clock()
        arcpy.Append_management(lyr, target, schemaType, "", "")
        arcpy.AddMessage("Elapsed time appending: {0} seconds".format(int(time.clock() - t0)))

        # select new case feature
        t0 = time.clock()
        arcpy.SelectLayerByLocation_management(target, selectType, lyr)
        arcpy.AddMessage("Elapsed timeselecting case layer: {0} seconds".format(int(time.clock() - t0)))

        # delete temporary layer
        #arcpy.Delete_management(lyr)

        # clear selection on parcels
        t0 = time.clock()
        arcpy.SelectLayerByAttribute_management(parcel, clear)
        arcpy.AddMessage("Elapsed time clearing parcel selection: {0} seconds".format(int(time.clock() - t0)))

    except Exception as e:
        arcpy.AddError("ERROR in MakeCaseFeature: \n" + e.message)

#-------------------------------------------------------------------------------

def main():
    ''' Check how many parcels are selected. Count returns all parcels if none are
        selected, or only selected ones if there are any.
    '''
    try:
        t0 = time.clock()
        count = int(arcpy.GetCount_management(parcel).getOutput(0))
        arcpy.AddMessage(str(count) + " parcels selected")
        arcpy.AddMessage("Elapsed time counting parcels: {0} seconds".format(int(time.clock() - t0)))

        # make sure parcels are selected before running rest of script, otherwise exit
        if count >= 0 and count <= 10:
            MakeCaseFeature()       # create the case feature
            t0 = time.clock()
            arcpy.RefreshActiveView()
            arcpy.RefreshTOC()
            arcpy.AddMessage("Elapsed time refreshing view: {0} seconds".format(int(time.clock() - t0)))
        else:
            arcpy.AddError("No features selected! \n Please select at least one parcel feature. \n")
            arcpy.AddError("Quitting the Create Case tool \n")

    except Exception as e:
        arcpy.AddError('ERROR in main: /n' + e.message)

Re: Why does this script take so long to run?

MathewCoyle — Thu, 17 Jan 2013 15:31:33 GMT

You want this

env.workspace = "in_memory"

Where is your source data located? If they are stored on a network it may account for slower than expected times. ~2 minutes for your operations really isn't that slow though.

Re: Why does this script take so long to run?

T__WayneWhitley — Thu, 17 Jan 2013 16:18:46 GMT

Thought I'd chime in a little further. Yes, you are right, there isn't much in the help files, only a little more added at 10.1 here, funny it's in the ModelBuilder section (see the small subtopic, Using in_memory in Python):

Using in-memory workspace
Desktop » Geoprocessing » ModelBuilder
Using in_memory in Python
http://resources.arcgis.com/en/help/main/10.1/index.html#//002w0000005s000000

in_memory is great when it works correctly - careful not to run out of memory, 'Delete_management' frees it up...

However, for further interest, see this blog entry by Kevin (for when you upgrade):

The new ScratchGDB and ScratchFolder environments
by Kevin Hibma on October 19, 2012
http://blogs.esri.com/esri/arcgis/2012/10/19/the-new-scratchgdb-and-scratchfolder-environments/

Re: Why does this script take so long to run?

ChrisSnyder — Thu, 17 Jan 2013 16:28:34 GMT

Question/comment for recurvata: Why are you dissolving the selected parcels? Why not just use them directly (the feature layer with the selected parcels) to run your SelectByLocation?

Re: Why does this script take so long to run?

Zeke — Thu, 17 Jan 2013 16:54:36 GMT

Chris:
A typical project where the user would run this tool is a rezoning request on one or more parcels. For their purposes (display and further analysis), they want one case feature covering all the parcels they've selected. Dissolve takes the temporary layer that consists of n number of parcels and makes them into one feature. Similar to the Merge tool in the Editor drop-down. The gp Merge tool would create a new feature class, which I don't want. The Select By Location is just an additional feature so that they don't have to manually select it once created. It's not really necessary. I can accomplish similar results manually myself much more quickly, but the end users don't really know Arc, so I'm trying to make it untrained monkey easier for them. But if there's a better method, please let me know. I'm not an ArcPy expert! Thanks.

Wayne:
Thanks for the info, that will come in handy, I'm sure.

Matt:
The data is stored on a network, but I've never noticed this amount of processing time before. It does work, so maybe they'll just have to accept it. I doubt they have a handle on how much time it should take anyway (maybe I don't either, eh?). Thanks for the tip on in-memory. I used that, not much time difference, though.

Re: Why does this script take so long to run?

JamesCrandall — Thu, 17 Jan 2013 17:39:29 GMT

Chris:
Thanks for the tip on in-memory. I used that, not much time difference, though.

Is this correct?

env.workspace = "IN-MEMORY"

I thought it should be

env.workspace = "IN_MEMORY"

You might want to go back and make sure the in_memory space is correctly implemented (ie, copy your temporary FC's there and run the dissolve on those copied/in_memory FC's). My experience has been huge performance improvements.

Re: Why does this script take so long to run?

Zeke — Thu, 17 Jan 2013 18:33:30 GMT

James,

Duh! 😮 You are right. Fixing that gave me a 33 second run time. Much better, thank you, and everyone else as well. Very much appreciated. Nice to learn something new.

Re: Why does this script take so long to run?

JamesCrandall — Sat, 11 Dec 2021 10:21:10 GMT

Great to hear!

Perhaps it isn't that relevant to this instance, but Wayne's comments certainly hold true.

"...in_memory is great when it works correctly - careful not to run out of memory, 'Delete_management' frees it up..."

It's a double-edged sword thing -- too much reliance on it and it can cut ya. One of the things that I have done to help mitigate any kwirks/wierdness with the in_memory is to make darn sure nothing is in it when it shouldn't be or when I am finished with it. This simply def() can be called right before and after you process 'stuff' and helps to be sure it is all cleaned up (sorry, it's Arcgisscripting 9.3 because I am still waiting to migrate to 10.1).

(this is just for featureclasses but you can alter it to include other types like tables and such)

def ClearINMEM():
   ## clear out the IN_MEMORY workspace of any featureclasses
   
   try:
     gp.Workspace = "IN_MEMORY"
     fcs = gp.ListFeatureClasses()
     
     ### for each FeatClass in the list of fcs's, delete it.
     for f in fcs:
        gp.Delete_management(f)
        gp.AddMessage("deleted: " + f)
   except:
     gp.AddMessage("The following error(s) occured attempting to clear in_memory space " + gp.GetMessages())
     return