topic Re: Help improving tool in Python Questions

Help improving tool

TonyAlmeida — Tue, 25 Feb 2014 13:58:44 GMT

I have this script that i have put into a tool but it runs super slow. it take about 4 minutes to create a point and populate.
The tool does not give me an error it's just super slow, I believe it is taking all the the testParcelsAdmit and putting them into memory instead of just the on that pertains to the point. I've been struggling with this too script for a while so if anyone could PLEASE help me improve this tool i would gratefully appreciate it.

here is the script i am working with.

import arcpy
import pythonaddins
import os
from arcpy import env

class Add_points(object):
    """Implementation for AddPoints_addin.Add_points (Tool)"""
    def __init__(self):
        self.enabled = True
        self.cursor = 3 # Can set to "Line", "Circle" or "Rectangle" for interactive shape drawing and to activate the onLine/Polygon/Circle event sinks.
    def onMouseDownMap(self, x, y, button, shift):

        fc = "TonyTwoWay.DBO.TT"
        workspace = r"C:\Users\talmeida\AppData\Roaming\ESRI\Desktop10.1\ArcCatalog\Connection to dsd15_sqlexpress.sde"
        arcpy.env.overwriteOutput = True
        #arcpy.ChangeVersion_management('TonyTwoWay.DBO.TT','TRANSACTIONAL','dbo.DEFAULT', "")
        # Start an edit session. Must provide the worksapce.
        edit = arcpy.da.Editor(workspace)

        # Edit session is started without an undo/redo stack for versioned data
        #  (for second argument, use False for unversioned data)
        edit.startEditing(True)

        # Start an edit operation
        edit.startOperation()

        CC_list = []
        with arcpy.da.SearchCursor(fc, ["AddressID"]) as cursor:
            for row in cursor:
                try:
                    if "CC" in row[0]:
                        CC_list.append(int(row[0].strip("CC")))   
                except TypeError:
                    pass        
        del cursor

        CC_list.sort()
        AddressID = CC_list[-1] + 1
        AddressID = 'CC' + str(AddressID)

        row_values = [(x, y, (x, y), AddressID)]
        cursor = arcpy.da.InsertCursor(fc, ["X_Coord", "Y_Coord", "SHAPE@XY", "ADDRESSID"])

        for row in row_values:
            cursor.insertRow(row)
        del cursor

                 # Stop the edit operation.
        edit.stopOperation()

        # Stop the edit session and save the changes
        edit.stopEditing(True)        

        arcpy.RefreshActiveView()
#####################################################
        fcTarget = "TonyTwoWay.DBO.TT"
        fcJoin = "testParcelsAdmit"
        add_fields = ["ACCOUNT","SiteNum","OwnerName","SiteAddres","SiteNumSfx","SiteStreet","predir","StreetType","SubName"]

        # fix args
        if not isinstance(add_fields, list):
            # from script tool
            add_fields = add_fields.split(';')

        # do not need this scratch file
        fcOutput = r'in_memory\temp_join'
        arcpy.SpatialJoin_analysis(fcJoin, fcTarget,fcOutput, 'JOIN_ONE_TO_MANY')


        # grab oid field from points
        oid_t = arcpy.Describe(fcTarget).OIDFieldName

        # init rowW and rowR
        curR = arcpy.SearchCursor(fcOutput)
        join_dict = dict([(r.JOIN_FID,[r.getValue(f) for f in add_fields]) for r in curR])
        del curR

        # Now update the new target
        curW = arcpy.UpdateCursor(fcTarget)
        for row in curW:
            t_oid = row.getValue(oid_t)
            if t_oid in join_dict:
                for f in add_fields:
                    row.setValue(f, join_dict[t_oid][add_fields.index(f)])
            curW.updateRow(row)
        del row, curW
        arcpy.AddMessage('Updated all records sucussefully')
        
        pass

Re: Help improving tool

DuncanHornby — Tue, 25 Feb 2014 14:10:08 GMT

Tony,

One comment I can make is that you start with the new faster da.cursor then swap back to the slower old style cursor after the ####... line

I would recommend as a first stab at improving the speed to change your cursor types.

Duncan

Re: Help improving tool

JamesCrandall — Sat, 11 Dec 2021 18:15:48 GMT

When confronted with performance issues, I like to implement timers along the way at various processing points to get a good idea of where bottlenecks are occuring. You do get fancy with it, or just put in print statements strategically and run a stopwatch to see where things slow down.

Use the in_memory workspace where possible: typically I see 1/2 the time to process things there compared to disk reads/writes.

Here's a time class you can use. Put it at the very top of your script:


class Timer:    
    def __enter__(self):
        self.start = time.clock()
        return self

    def __exit__(self, *args):
        self.end = time.clock()
        self.interval = self.end - self.start

Usage:


   t = Timer()
   with t:
        CC_list.sort()
        AddressID = CC_list[-1] + 1
        AddressID = 'CC' + str(AddressID)

        row_values = [(x, y, (x, y), AddressID)]
        cursor = arcpy.da.InsertCursor(fc, ["X_Coord", "Y_Coord", "SHAPE@XY", "ADDRESSID"])

        for row in row_values:
            cursor.insertRow(row)
        del cursor

                 # Stop the edit operation.
        edit.stopOperation()

        # Stop the edit session and save the changes
        edit.stopEditing(True)  

   timetest = completed in %.02f secs." % (t.interval)
   print timetest

Re: Help improving tool

TonyAlmeida — Sat, 11 Dec 2021 18:15:51 GMT

Thank you both for your reply. i have changed both search cursor to arcpy.da.searchCursor.

I am getting an error in the timer.

  File "C:\Users\talmeida\AppData\Local\ESRI\Desktop10.1\AssemblyCache\{BB229966-100D-4482-A3FE-B298BD262597}\AddPoints_addin.py", line 65
    timetest = completed in %.02f secs." % (t.interval)
                            ^
SyntaxError: invalid syntax

current code

import arcpy
import pythonaddins
import os
from arcpy import env

class Timer:    
            def __enter__(self):
                self.start = time.clock()
                return self

            def __exit__(self, *args):
                self.end = time.clock()
                self.interval = self.end - self.start

class Add_points(object):
    """Implementation for AddPoints_addin.Add_points (Tool)"""
    def __init__(self):
        self.enabled = True
        self.cursor = 3 # Can set to "Line", "Circle" or "Rectangle" for interactive shape drawing and to activate the onLine/Polygon/Circle event sinks.
    def onMouseDownMap(self, x, y, button, shift):

        fc = "TonyTwoWay.DBO.TT"
        workspace = r"C:\Users\talmeida\AppData\Roaming\ESRI\Desktop10.1\ArcCatalog\Connection to dsd15_sqlexpress.sde"
        arcpy.env.overwriteOutput = True
        #arcpy.ChangeVersion_management('TonyTwoWay.DBO.TT','TRANSACTIONAL','dbo.DEFAULT', "")
        # Start an edit session. Must provide the worksapce.
        edit = arcpy.da.Editor(workspace)

        # Edit session is started without an undo/redo stack for versioned data
        #  (for second argument, use False for unversioned data)
        edit.startEditing(True)

        # Start an edit operation
        edit.startOperation()

        CC_list = []
        with arcpy.da.SearchCursor(fc, ["AddressID"]) as cursor:
            for row in cursor:
                try:
                    if "CC" in row[0]:
                        CC_list.append(int(row[0].strip("CC")))   
                except TypeError:
                    pass        
        del cursor

        t = Timer()
        with t:
             CC_list.sort()
             AddressID = CC_list[-1] + 1
             AddressID = 'CC' + str(AddressID)

             row_values = [(x, y, (x, y), AddressID)]
             cursor = arcpy.da.InsertCursor(fc, ["X_Coord", "Y_Coord", "SHAPE@XY", "ADDRESSID"])

             for row in row_values:
                 cursor.insertRow(row)
             del cursor

             # Stop the edit operation.
             edit.stopOperation()

             # Stop the edit session and save the changes
             edit.stopEditing(True)  

        timetest = completed in %.02f secs." % (t.interval)
        print timetest        

        arcpy.RefreshActiveView()
#####################################################
        fcTarget = "TonyTwoWay.DBO.TT"
        fcJoin = "testParcelsAdmit"
        add_fields = ["ACCOUNT","SiteNum","OwnerName","SiteAddres","SiteNumSfx","SiteStreet","predir","StreetType","SubName"]

        # fix args
        if not isinstance(add_fields, list):
            # from script tool
            add_fields = add_fields.split(';')

        # do not need this scratch file
        fcOutput = r'in_memory\temp_join'
        arcpy.SpatialJoin_analysis(fcJoin, fcTarget,fcOutput, 'JOIN_ONE_TO_MANY')


        # grab oid field from points
        oid_t = arcpy.Describe(fcTarget).OIDFieldName

        # init rowW and rowR
        curR = arcpy.da.SearchCursor(fcOutput)
        join_dict = dict([(r.JOIN_FID,[r.getValue(f) for f in add_fields]) for r in curR])
        del curR

        # Now update the new target
        curW = arcpy.UpdateCursor(fcTarget)
        for row in curW:
            t_oid = row.getValue(oid_t)
            if t_oid in join_dict:
                for f in add_fields:
                    row.setValue(f, join_dict[t_oid][add_fields.index(f)])
            curW.updateRow(row)
        del row, curW
        arcpy.AddMessage('Updated all records sucussefully')
        
        pass

Changing just the SEarch Cursor to arcpy.da.SearchCursor
I get this error,

Traceback (most recent call last):
  File "C:\Users\talmeida\AppData\Local\ESRI\Desktop10.1\AssemblyCache\{BB229966-100D-4482-A3FE-B298BD262597}\AddPoints_addin.py", line 74, in onMouseDownMap
    curR = arcpy.da.SearchCursor(fcOutput)
TypeError: Required argument 'field_names' (pos 2) not found

Just Search Cursor change to arcpy.da.SearchCursor

import arcpy
import pythonaddins
import os
from arcpy import env

class Add_points(object):
    """Implementation for AddPoints_addin.Add_points (Tool)"""
    def __init__(self):
        self.enabled = True
        self.cursor = 3 # Can set to "Line", "Circle" or "Rectangle" for interactive shape drawing and to activate the onLine/Polygon/Circle event sinks.
    def onMouseDownMap(self, x, y, button, shift):

        fc = "TonyTwoWay.DBO.TT"
        workspace = r"C:\Users\talmeida\AppData\Roaming\ESRI\Desktop10.1\ArcCatalog\Connection to dsd15_sqlexpress.sde"
        arcpy.env.overwriteOutput = True
        #arcpy.ChangeVersion_management('TonyTwoWay.DBO.TT','TRANSACTIONAL','dbo.DEFAULT', "")
        # Start an edit session. Must provide the worksapce.
        edit = arcpy.da.Editor(workspace)

        # Edit session is started without an undo/redo stack for versioned data
        #  (for second argument, use False for unversioned data)
        edit.startEditing(True)

        # Start an edit operation
        edit.startOperation()

        CC_list = []
        with arcpy.da.SearchCursor(fc, ["AddressID"]) as cursor:
            for row in cursor:
                try:
                    if "CC" in row[0]:
                        CC_list.append(int(row[0].strip("CC")))   
                except TypeError:
                    pass        
        del cursor

        CC_list.sort()
        AddressID = CC_list[-1] + 1
        AddressID = 'CC' + str(AddressID)

        row_values = [(x, y, (x, y), AddressID)]
        cursor = arcpy.da.InsertCursor(fc, ["X_Coord", "Y_Coord", "SHAPE@XY", "ADDRESSID"])

        for row in row_values:
            cursor.insertRow(row)
        del cursor

                 # Stop the edit operation.
        edit.stopOperation()

        # Stop the edit session and save the changes
        edit.stopEditing(True)        

        arcpy.RefreshActiveView()
#####################################################
        fcTarget = "TonyTwoWay.DBO.TT"
        fcJoin = "testParcelsAdmit"
        add_fields = ["ACCOUNT","SiteNum","OwnerName","SiteAddres","SiteNumSfx","SiteStreet","predir","StreetType","SubName"]

        # fix args
        if not isinstance(add_fields, list):
            # from script tool
            add_fields = add_fields.split(';')

        # do not need this scratch file
        fcOutput = r'in_memory\temp_join'
        arcpy.SpatialJoin_analysis(fcJoin, fcTarget,fcOutput, 'JOIN_ONE_TO_MANY')


        # grab oid field from points
        oid_t = arcpy.Describe(fcTarget).OIDFieldName

        # init rowW and rowR
        curR = arcpy.da.SearchCursor(fcOutput)
        join_dict = dict([(r.JOIN_FID,[r.getValue(f) for f in add_fields]) for r in curR])
        del curR

        # Now update the new target
        curW = arcpy.UpdateCursor(fcTarget)
        for row in curW:
            t_oid = row.getValue(oid_t)
            if t_oid in join_dict:
                for f in add_fields:
                    row.setValue(f, join_dict[t_oid][add_fields.index(f)])
            curW.updateRow(row)
        del row, curW
        arcpy.AddMessage('Updated all records sucussefully')
        
        pass

Re: Help improving tool

MathewCoyle — Tue, 25 Feb 2014 15:41:02 GMT

For the second the da cursors have a different parameter format. Review the ArcGIS documentation of cursors.

http://resources.arcgis.com/en/help/main/10.2/index.html#//018w00000011000000

Re: Help improving tool

GerryGabrisch — Tue, 25 Feb 2014 16:10:26 GMT

I have had good luck speeding up tools with 64-bit background geoprocessing.
http://resources.arcgis.com/en/help/main/10.1/index.html#//002100000040000000

Re: Help improving tool

TonyAlmeida — Sat, 11 Dec 2021 18:15:54 GMT

mzcoyle i did import the time model but i got the following error,

  File "C:\Users\talmeida\AppData\Local\ESRI\Desktop10.1\AssemblyCache\{BB229966-100D-4482-A3FE-B298BD262597}\AddPoints_addin.py", line 65
    timetest = completed in %.02f secs." % (t.interval)
                            ^
SyntaxError: invalid syntax

I am not understand how to emplement the second arcpy.da.searchCursor with the spatial join. Sorry python newbie here.

gabrisch i will look into 64-bit background geoprocessing.

Re: Help improving tool

JamesCrandall — Sat, 11 Dec 2021 18:15:56 GMT

mzcoyle i did import the time model but i got the following error,
  File "C:\Users\talmeida\AppData\Local\ESRI\Desktop10.1\AssemblyCache\{BB229966-100D-4482-A3FE-B298BD262597}\AddPoints_addin.py", line 65
    timetest = completed in %.02f secs." % (t.interval)
                            ^
SyntaxError: invalid syntax
I am not understand how to emplement the second arcpy.da.searchCursor with the spatial join. Sorry python newbie here.

gabrisch i will look into 64-bit background geoprocessing.

You are missing the quote at the begining:

timetest = "completed in %.02f secs." % (t.interval)

Re: Help improving tool

TonyAlmeida — Sat, 11 Dec 2021 18:15:59 GMT

jamesfreddyc after adding the quote i ran and the script.
I get no error but now the points are not being populated anymore...?

Python window

>>> 
completed in 0.05 secs.
>>> 
completed in 0.03 secs.
>>> 
completed in 0.07 secs.
>>> 
completed in 0.04 secs.
>>> 
completed in 0.08 secs.

Re: Help improving tool

JamesCrandall — Sat, 11 Dec 2021 18:16:01 GMT

jamesfreddyc after adding the quote i ran and the script.
I get no error but now the points are not being populated anymore...?

Python window
>>> 
completed in 0.05 secs.
>>> 
completed in 0.03 secs.
>>> 
completed in 0.07 secs.
>>> 
completed in 0.04 secs.
>>> 
completed in 0.08 secs.

If you believe the Timer code is causing this, then remove it. It's just a suggestion to determine the processing time for things, and it seems to work for you now, but I am not sure how it could cause other code to no longer work.

Re: Help improving tool

TonyAlmeida — Tue, 25 Feb 2014 19:26:22 GMT

jamesfreddyc thank you for your input, i can see how the time model would be very helpful.

If i can just get this to work. being a newbie and trying to learn python...