Problems with memory cleanup and computation time

Question

I recently wrote a toolbox for arcgis pro that to calculate pullution distributions. I had a few hundred points in my example, then calculated the distribution of contaminants near each point, stored it as a raster, and then merged those rasters into the final result using the mosaic function. But I'm running into two problems.

The first problem is that I find that when I run the code in pycharm with arcpy called, the code calculates significantly faster than under arcgis pro. May I ask what is the reason for this? And as I run my code under arcgis pro over and over, the code takes more and more time to run. My guess is that it's because the code stores some intermediate shapefile variables in memory. But I manually added the statement to remove the intermediate shapefile variables at the end of the code and it still causes the code to slow down. Can I avoid this problem? I know I can avoid this problem by restarting arcgis pro, but are there any other solutions?

Another problem is that after running the code very many times in arcgis pro, it is possible to get an error saying that the mosaic function did not execute properly (ERROR 999999: Something unexpected caused the tool to fail. Contact Esri Technical Support (http://esriurl.com/support) to Report a Bug, and refer to the error help for potential solutions or workarounds. Failed to execute (Mosaic).). May I ask what is the reason for this? When I restart arcgis pro, my code still executes fine.

WeiMao · Answer

Thank you. I have over 1000 lines of code and it does not seem convinient to put so much code here. We are going to open source that code on github when we are done.

I manually added the code to delete the file in memory at the end of the code. So far I've run it 30 times in a row with no errors reported. But I can clearly feel that running the code under arcgis pro is roughly twice as fast as I can run it in pycharm.

WeiMao · Answer

Thank you.

Hayden Welch · Answer

Would you mind sharing some of your code? If you're writing out intermediate features to a database in ArcPro instead of building them in memory that can cause issues with locks and ownership while Pro is open.

A workaround it to create a datastore in a tempfile (import tempfile) and do your intermediate operations there to keep them away from any data that might be locked by the running arc instance.

Another thing is to use the arcpy.da.Editor object to handle any edit sessions that you use in your code. This will allow you to safely check that the features you're creating/editing are not open by another running process.

I've also noticed that the namespace in a running Pro instance can get really polluted as you keep running functions and especially on the terminal will slow down.

You can use the builtin Process Viewer (under the Help tab) to see what processes are being handled by ArcPro. Also check your memory usage using Windows Task Manager to see if you are overutilizing your system ram, which will cause operations to run in swap/the hard drive.

Seeing as how you're using raster data, when you run in Pro, if that data is already loaded and you reload it into memory for the processing, you now have 2 instances of that data being stored in memory (one for display/cache, the other for the processing tool). This will quickly double your memory usage