Search Cursor in Python

[ModernElectric]

I have a Python script that starts with a Search Cursor that goes through a File Geodatabase and looks for a unique number and produces a graph and prints it out to a .PDF. The Script works out just fine. BUT - each time I run it - it seems to take longer and longer. The first few times - it produces about 4 .PDFs each minute. Now - when I run it - it goes slower and slower - about 1 .PDF every 2 minutes.

Does the Search Cursor produce "Temp" files somewhere that needs to be cleared out? How can I speed this up?

[MathewCoyle]

What version of ArcGIS are you using? Can you post the code you are running? Search Cursors are quite light and shouldn't be creating any additional files. Are you sure it isn't some other part of your code that is causing the issue?

[ModernElectric]

# Name: Electric Meter Demand Graphs - Usage.py
# Description: Python script to take a single TWACs Meter and create a Usage Graphs per Feeder
# Date Created: Tuesday, September 17th, 2013 @ 1:20 PM
# Date Modified: Tuesday, September 17th, 2013 @ 1:20 PM
# Author: C. Wafstet #94 - Modern Electric Water Company (Spokane, WA)

import arcpy
import arcpy
import logging
from arcpy import env
env.overwriteOutput = 1
env.workspace = r"C:\MEWCo GIS System\Electric System\MEWCo_Electric_Model-LOCAL.gdb"

table = "SERVICE_METERS_DEMAND_F1FEEDER_ABC"

list = []

# if run in ArcGIS Desktop, show messages, also print to log
def log(method, msg):
    print msg
    method(msg)

logging.basicConfig(
        level=logging.INFO,
        format='%(asctime)s %(levelname)-s %(message)s',
        datefmt='%a, %d %b %Y %H:%M:%S',
        filename='MakeGraph.log',
        filemode='w'
        )

log(logging.info, "Creation of Graphs Started")

rows = arcpy.SearchCursor(table)
for row in rows:
    list.append(row.TWACs_NUMBER)

del row, rows

# Remove duplicates from list
list = dict.fromkeys(list)
list = list.keys()

for n in list:
    arcpy.TableSelect_analysis(table, r"in_memory\table_sel", "TWACs_NUMBER = " + str(n))

    # Get TWACs_Number value
    for row in arcpy.SearchCursor(r"in_memory\table_sel"):
        METER_NUMBER = row.getValue("METER_NUMBER")

    out_graph_name = n
    out_graph_pdf = r"C:\MEWCo GIS System\Electric Graphs\Electric Meters\Usage Graphs\F-1 Feeder" + "\\" + str(n)[:-2] + "-Usage" + ".pdf"
    input_template = r"C:\MEWCo GIS System\Electric Graphs\GIS Graph Temps\ELECTRIC METER DEMAND - KWH USAGE.grf"
    input_data = r"in_memory\table_sel"

# Create the Graph
    graph = arcpy.Graph()

# Add a Vertical Bar series to the graph - KWH Usage
    graph.addSeriesBarVertical(input_data, "KWH_USAGE")

# Specify the title of the Left Axis
    graph.graphAxis[0].title = "KWH_USAGE"

# Specify the title of the Bottom Axis
    graph.graphAxis[2].title = "BILLING_PERIOD"

# Specify the title of the Graph
    graph.graphPropsGeneral.title = "KWH Usage by Electric TWACs Number: " + METER_NUMBER

# Output a graph, which is created in-memory
    arcpy.MakeGraph_management(input_template, graph, out_graph_name)

# Save the graph as an .PDF
    arcpy.SaveGraph_management(out_graph_name, out_graph_pdf, "MAINTAIN_ASPECT_RATIO", 600, 375)

log(logging.info, "Creation of Graphs Complete")
    

The code works just fine.... I worked on it for awhile to make it work the way I want it too. It just seems like - each time I run it - it takes longer and longer.

[MathewCoyle]

I would find it more likely your log file is the issue. Try disabling logging and running it a few times to see if you get the same slow downs. Also, are you closing the instance of python this script is running in between executions? The in_memory workspace won't clear properly until you terminate the instance it was executed in or you clear it explicitly.

[ModernElectric]

I would find it more likely your log file is the issue. Try disabling logging and running it a few times to see if you get the same slow downs. Also, are you closing the instance of python this script is running in between executions? The in_memory workspace won't clear properly until you terminate the instance it was executed in or you clear it explicitly.

I will see what I does when I remove the logging from the script. Forgive me, but the second part you are talking about... what do you mean

[MathewCoyle]

This table "in_memory\table_sel" you create will persist until you close the instance it was created in.

[ModernElectric]

This table "in_memory\table_sel" you create will persist until you close the instance it was created in.

And how do I do that?

[ChrisSnyder]

Not sure about the reason for the slow down, but some ideas to make it faster:

1. Just run the search cursor on the on-disk table directly. Copy the table to in_memory most likely is just adding overhead processing time.


2. In you are running v10.1+, use the data access cursors. They are about 20-30x faster than the old cursors you are using.

Like Matt was saying, make sure you are restarting Python (or whatever .exe you are using) between runs of this script as to clear the RAM and remove locks.

[ChrisSnyder]

Also, instead of creating a list of the"TWACs_NUMBERs", and running a seperate cursor for each one, just run a single cursor and store the lists in a dictionary, where each twacNo is a key and each key points to a list of meterNos. In v10.1 this would look like:

twacDict = {}
searchRows = arcpy.da.SearchCursor(table, ["TWACs_NUMBER","METER_NUMBER"])
for searchRow in searchRows:
   twacNo, meterNo = searchRow
   if twacNo in twacDict:
      twacDict[twacNo].append(meterNo)
   else:
      twacDict[twacNo] = [meterNo]
del searchRow, searchRows

twackList = twacDict.keys()
twackList.sort()
for twacNo in twackList:
   meterList = twacDict[twacNo]
   #make a graph here, since you now have a hook to the twacNo and it's list of meterNos

[ModernElectric]

Chris-

Thank you for the past (2) posts. I will look at both ideas and see what I can come up with. Guess I should have said I am working on ArcMap 10.2

Also, instead of creating a list of each "TWACs_NUMBER", and running a seperate cursor for each one, just run a single cursor and store the lists in a dictionary, where each twacNo is a key and each key points to a list of meterNos. In v10.1 this would look like:

twacDict = {}
searchRows = arcpy.da.SearchCursor(table, ["TWACs_NUMBER","METER_NUMBER"])
for searchRow in searchRows:
   twacNo, meterNo = searchRow
   if twacNo in twacDict:
      twacDict[twacNo].append(meterNo)
   else:
      twacDict[twacNo] = [meterNo]
del searchRow, searchRows

twackList = twacDict.keys()
twackList.sort()
for twacNo in twackList:
   meterList = twacDict[twacNo]
   #make a graph here, since you now have a hook to the twacNo and it's list of meterNos