Python Program Crashes with Larger Dataset

Question

I developed a Python program for development of files necessary for my Network Dataset.  I did my testing on a sample dataset (a small fraction of my actual dataset, same schema, etc.) to make sure everything is working as it is supposed to.  When I move to utilizing my full dataset with over 92,000 line segments, the program crashes after about 5 to 7 minutes and goes to my finally statement where my cursors are being deleted.  If I debug my program using printed arcpy.AddMessage statements, the stopping points have correct syntax and seem to be fine.  As I mentioned, when I use my smaller dataset, everything runs smoothly.  Anyone had this issue or know of what I could do to avoid it?

MichaelAlires · Answer

It had to do with the cursors I was creating with SearchCursor, UpdateCursor and InsertCursor.  So it wasn't specifically a tool.  With this script/tool I was developing, I was trying to prepare some of my speed data collected from GPS units that I had attached to street segments before actually generating the network dataset.

ChrisSnyder · Answer

ArcGIS/arcpy does suffer from memory leak issues, but let me assure you that it is MUCH better than it used to be. Instead of rebooting the machine, all you need to do to clear up RAM consumption is to exit and then restart the application that is calling on arcpy (ArcMap, PythonWin, Eclipse, etc). There are many tricks to handle memory issues... My favorite is using Python to launch a seperate "worker" python.exe process that will then exit upon completion (thus freeing up the memory it was using). Search this forum for "subprocess", "os.spawnv", "multiprocessing", or "parallel python".

You never said what specific tool is causing the issue (i.e. Solve_na)... Care to elaborate?

MichaelAlires · Answer

*Update*
Should anyone search this in the future, here is what I found:  I was deleting my cursors with a finally statement at the very end of my python script.  The reason for this was to make sure that the cursors got deleted regardless of what happened.  This meant that the memory was maintaining a place for all of that information and was causing the program to crash out.  I was creating my own "memory leak".  Once I set things up to delete the cursors after they were done being used, the program became much more stable.  I had to place all of the while statements using my cursors into individual try/except statements.  However, I did notice that the program still crashed if I ran it multiple times without a reboot of my computer which makes me think there is some type of memory leak happening within ArcPy.  Once I did the reboot, all problems went away until I ran it multiple times again.  I haven't installed any of the 10 service packs (as I am waiting for 3 to be soon released) so hopefully that will be resolved once I update.