How to avoid ArcMap crash due to GDI Objects > 10000 in python script?

[KimFisher]

Hello,

I've got an arcpy script that essentially iterates over a polygon feature class and for each polygon extracts a raster, users RasterToNumPyArray to create a NumPy array, then does a few things with the values and writes to a postgres db. Everything works fine for small numbers of features.

Despite using del to destroy variables at the end of each loop (following ArcGIS help at http://resources.arcgis.com/en/help/main/10.1/index.html#//002z00000028000000), this script generates large numbers of GDI objects (which I view in Task Manager), probably because of RasterToNumPyArray, and generally exceeds the default Windows maximum of 10000 after 2-300 polygons. This causes ArcMap to crash entirely.

To deal with this, I changed the GDI object quota on my machine, but this is obviously not a longterm solution:
http://msdn.microsoft.com/en-us/library/ms724291(v=vs.85).aspx

I'm using ArcGIS 10.1 for Desktop SP1; there was a related bug fixed in a 10.0 service pack:
http://support.esri.com/en/bugs/nimbus/TklNMDYzNjU2

My question is: what is the preferred strategy for avoiding GDI Object overruns (ie memory leaks), other than using del to explicitly destroy variables (which doesn't seem to work), with particular reference to RasterToNumPyArray? Has anybody else run into this?

Best,
Kim

[KimOllivier]

Short of waiting for a bugfix here are some strategies that people have used:

Encapsulate the tool in a python function using local variables in the hope that they are removed on exit from the function

Use a python system call to start a new thread so that the objects are destroyed on exit. You have to pass parameters, temporary featureclass references and other complications.

You might also try installing the 10.1 SP 1 Python64x which can use Python 64 bit for geoprocessing tools running in background.
However if it is a memory leak then it would just postpone the crash.

Any process that opens a cursor to interate over features using another geoprocessing tool is a poor workflow design for tools that are designed to process the whole dataset as one entity. Could you not redesign the flow to process the featureclass in one pass?

Perhaps post your workflow for help in doing this.

[KimFisher]

Thanks for the reply. I thought of option #2 but the task didn't warrant the effort. Your observation about process is a good one but in my current special case it's unavoidable.

I was about to try your first suggestion but then option #3 reminded me that all along I hadn't been using my 64-bit geoprocessing, because it only works on background geoprocessing, and that running things in foreground (which I was doing for easier debugging from the message window) might very well be the problem. So I ran in background, and voila, no more GDI objects created. Thanks!