Python, arcpy and memory use

[GrantHerbert]

I have a script which searches subdirectories for mxds and reports on them. Nothing new, except that I am having trouble with memory management.

There seems to be a memory leak, although I have tried hard to del variables as I go, and separate the MXD parsing to a separate function in the script so that they are all local variables although I hold a reference to the logging and reporting text files as globals.

This has been working well, until I hit a certain MXD in my test folders (I just grabbed a handful of folders off our network drives, so they are real MXDs). This MXD  is 2MB in size and has nothing wrong with it that I can see, although it  has a number of broken links it opens fine in ArcMap, and I can access it fine from the python window using "current", but when I access it with arcpy from my script, memory use jumps up 300-400MB for a moment (peaking at 600MB+). This usually crashes the process even though I have 16GB of RAM. The python process is 32bit.

The MXD in question has a broken events layer at the top of the TOC, but I can access it OK from PyScripter (with the same memory jump) if I do so directly and before accessing any other MXDs.  If I open the MXD and use Arcpy in the python window I have no trouble accessing it either. If I have already processed an MXD and am using around 300MB of memory then the spike often kills the python.exe process. Making a copy of the MXD with the events layer deleted seems to work OK in my script, even after processing with other MXDs, suggesting that the events layer itself may be the problem, but I am at a loss what to do about it. I cannot tell if there is a problem until I access it and I can't seem to catch it when there is (it skips my try block and windows reports that python has stopped working).

Any ideas of what I can try?

[WengNg]

I'm kind of late into this discussion but I notice there isn't a solution yet for this error. I have a Python script running arcpy.da.walk trying to iterate through data in a workspace and write them into a CSV file.  The script always crashes without specifying the error even with error trapping. I manage to identify a potential cause from the Windows log pointing to CADEngine.dll as a culprit.

Tried running it on 64bit Python and no luck as well. Was wondering if there is a workaround this issue ?

[DanPatterson_Retired]

You are still using python 2.7 I presume.... It would be worthwhile trying the workflow using python 3.X, since memory leaks etc associated with python have been addressed in the various releases in the last 8 years.  Any persistent ones would be Arc* associated.  Upgrade to Pro to get a fairly recent version of python (is 3.4) and see if that works or improves things.

[WengNg]

Hi Dan,

I switched to 3.4 with a minor code change and it worked, it is no longer crashing for no reason. I think most people still do their coding on python 2.7 since not everyone is on Pro, wonder if ESRI will support the newer Python releases for Arcpy on Desktop. 

It does require some syntax and code modifications .

Thanks for your help on this !

[DanPatterson_Retired]

I have loads of python related links on my blog.

Just for an FYI  Python 3.7 alpha is starting, 3.6 is in final beta  and I have been programming in 3.5 on my iPad (could a iPhone) for over a year now...

To get you caught up...

/blogs/dan_patterson/2016/04/12/coming-to-python-preparation-and-anticipation 

/blogs/dan_patterson/2016/05/09/the-links 

/blogs/dan_patterson/2016/07/17/anaconda-spyder-and-arcgis-pro 

[Dan_Joyce_OE]

Using a bit of "Topic Necromancy" for anyone else who encounters this issue.

I have a very simple script that walks through a directory structure and lists data frame properties to a text file for every MXD it encounters (total of around 40 MXDs).  This is a similar concept to that posted by WengNg back in 2016 in this topic.

Running the script under 32-bit ArcGIS Desktop on a Windows processing box it would crash after about 30 MXDs and I could see a couple of ArcMap.exe instances hanging around in the Task Manager.  I had tried the approaches mentioned by others in this topic to mitigate the issue.

In the end I installed the 64-bit Desktop background GP and tried running the same script in that environment.  I watched the python.exe process in the Task Manager as saw the memory usage jump to an average of  1GB each time the script access a MXD and then drop down again once it had finished with it.  It peaked at around 1.5GB.  Pretty extreme considering I was only pulling out the data frame name, spatial reference and transformation.  The script ran through all MXDs and took over 30mins to complete, but it did complete without crashing.

Perhaps there is a difference in the way that the 32-bit and 64-bit instances handle MXDs and the memory cleanup afterwards?  Perhaps 64-bit's ability to access more memory is the key?  Either way, running my script using the 64-bit instance was the key for me.