in-process .da searchcursor question

Discussion created by csny490 on Nov 20, 2012
Latest reply on Dec 3, 2012 by csny490
I have a script that uses the arcpy.da.SearchCursor method. I am using v10.1 SP1. The script basically runs the SelectByLocation tool in a recurisive loop and identifies "spatial clusters" - that is groups of features that are all less than a certain distance from each other. I originally have this script running as an "in-process" toolbox based script. However, when run on extreemly large datasets, it exhibits some sort of memory leak (I notice the ArcMap screen flashes - I assume everytime the SelectByLocation/SelectByAttribute tool fires off), which ultimately causes ArcMap to issue an "out of memory" error at ~3.4 GB of RAM usage (ESRI gets lots of thanks for the now-standard large address aware in v10.1 BTW).

So, I noticed that if I run the script as a stand alone process via PythonWin, the memory usage never becomes and issue (and in fact stays < 300MB), and generally the script finishes much faster... Which is not what I would expect since I though running a script "in-process" was always the way to go. So I think, "Hey, I'll just make the script tool run "out-of-process" (via the toolbox), and memory problem solved!". Unfortunatly, when I do this, I find that the .da.SearchCursor can't read the featurelayer.

Anyone know:

  1. Why is the "in-process" method of running the script via ArcMap exhibiting memory leak behavior while running the script as "stand alone" via Python/PythonWin so much more efficient?

  2. Why does the arcpy.da.SearchCursor blow up when it tries to read my feture layer when the script is run "out-of-process" via ArcToolbox in ArcMap". The pertinent error message is:

Tue Nov 20 15:31:47 2012 - 
Tue Nov 20 15:31:47 2012 - Python Traceback Info:   File "C:\csny490\clusterId_v3.py", line 92, in <module>
    preSelectionOidSet = set([r[0] for r in arcpy.da.SearchCursor("fl1", ["OID@"])])

Tue Nov 20 15:31:47 2012 - Python Error Info: <type 'exceptions.RuntimeError'>: cannot open 'fl1'