topic Re: Cleaning up temporary layers created by python toolbox script in Python Questions

Cleaning up temporary layers created by python toolbox script

JohnJackson — Mon, 24 Mar 2014 14:24:13 GMT

ArcMap 10.2.1 / Python 2.7
I have several tools in a python toolbox. When I run through the tools in succession, temporary layers created in previously run tools show up as parameters in later tools. These temporary layers are created with arcpy.MakeFeatureLayer_management and the corresponding feature classes are in the scratch workspace. The layers show up in other tools for parameters where parameterType = GPFeatureLayer.

For example: Tool_A creates temporary layers temp1 and temp2 and then adds "Final Layer" to the TOC. If I run Tool_B which takes a layer parameter, I would expect "Final Layer" to show up in the list but NOT layers temp1 & temp2. The behavior I am seeing however is that all three layers show up as choices.

I am setting arcpy.env.addOutputsToMap = False at the beginning of each tool script and the temporary layers do not show up in the TOC as expected. I also delete all layers from scratch at the end of each tool script. Any help would be greatly appreciated.

Re: Cleaning up temporary layers created by python toolbox script

AdamCox1 — Mon, 24 Mar 2014 14:53:19 GMT

Maybe try arcpy.RefreshTOC() and arcpy.RefreshActiveView() after you delete the temporary layers?

Re: Cleaning up temporary layers created by python toolbox script

JohnJackson — Mon, 24 Mar 2014 15:14:56 GMT

Thanks Adam. I am actually calling both functions when I add the final layer to the map.

Re: Cleaning up temporary layers created by python toolbox script

AdamCox1 — Mon, 24 Mar 2014 15:33:52 GMT

Hmm, it may help to add those operations to the initializeParameters() method in the ToolValidator class for each tool, so that the parameters are definitely referencing an up-to-date mxd. Otherwise storing those temp feature classes in "in_memory\" instead of a scratch gdb may help too.

Re: Cleaning up temporary layers created by python toolbox script

FilipKrál — Mon, 24 Mar 2014 22:36:20 GMT

Hi,
Strange. I thought in-memory objects like layers and in_memory feature classes are deleted when a script finishes, but anyway...

Make sure that by the end of each script you delete the temporary feature classes (the actual data) as well as the temporary layers (in-memory objects pointing to the data).

I would love to hear how you guys do this and if there is good practice that really works. So far in my scripts I have been doing something like this (pseudo-code):

import arcpy # start of script  try:     todlt = [] # list to store everything I want to delete in the end.          # ... body of the script, whatever ...      my_buffer = arcpy.Buffer_analysis(...).getOutput(0)     todlt.append(my_buffer) # if my_buffer is a temporary feature class          my_buffer_lyr = arcpy.MakeFeatureLayer_management(my_buffer, 'lyrname', ...).getOutput(0)     todlt.append(my_buffer_lyr) # my_buffer_lyr is a temporary feature layer      # ... rest of the body of script ...      except arcpy.ExecuteError as e:     # handle ... except Exception as e:     # handle ... finally:     # cleanup     if 'todlt' in dir():         for dl in todlt:             try:                  arcpy.Delete_management(dl)             except:                 arcpy.AddWarning('Could not delete temporary data ' + str(dl))

By the way, I often use functions from my arcapi repository you may want to check out: https://github.com/NERC-CEH/arcapi
Specifically, function dlt() comes handy for the cleanup:

if 'todlt' in dir():     for dl in todlt:         ap.dlt(dl)

and function tstamp is useful to generate temporary layer names:

my_buffer_lyr = arcpy.MakeFeatureLayer_management(my_buffer, "lr" + ap.tstamp(tf = "%H%M%S") , ...).getOutput(0)

Filip.

Re: Cleaning up temporary layers created by python toolbox script

JohnJackson — Fri, 10 Dec 2021 21:43:12 GMT

Thank you Filip! After looking at your code and re-reading the documentation on Internal Layers http://resources.arcgis.com/en/help/main/10.2/index.html#//00210000000m000000, specifically:

When you create a layer or table view using geoprocessing tools, the new layer or table view is stored in an internal layer list, which is a different list from the ArcMap table of contents. This means that geoprocessing actually keeps two lists of layers and table views:

The list of layers in the ArcMap table of contents

The internal list of layers created by geoprocessing tools

I can see what I was doing wrong. I thought I was deleting the temporary layers by getting a list of layers for the data frame and deleting from that list:

def clearLayers():
    mxd = arcpy.mapping.MapDocument('CURRENT')
    for df in arcpy.mapping.ListDataFrames(mxd):
        for lyr in arcpy.mapping.ListLayers(mxd, "", df):
            arcpy.mapping.RemoveLayer(df, lyr) 
    del mxd

This was not clearing the internal layer list (layers created by other geoprocessing tools) that the python tools use in part to generate parameter lists. Having arcpy.env.addOutputsToMap = False had no effect on the internal layer list either (which it shouldn't).

Re: Cleaning up temporary layers created by python toolbox script

curtvprice — Mon, 15 Sep 2014 17:25:00 GMT

This is worth discussing further. The help is not explicit about this and it's too bad -- if you don't clean up your layers, you leave joins and file locks polluting your session -- especially with script tools launched in-process from ArcGIS Desktop.

Here's the approach that I have been using.

from datetime import datetime
dtag = datetime.now().strftime("%Y%m%d%H%M%S") # '20140915110231'
lyrName = "lyr" + dtag
# set up temp variables
tmp_layer, tmp_table = [None] * 2
try:
    # ... create tmp_table name
    tmp_table = arcpy.CreateScratchName("", dtag, "table")
    arcpy.CreateTable_management(os.path.dirname(tmp_table),
                                os.path.basename(tmp_table))
    tmp_layer = arcpy.MakeFeatureLayer(tmp_table, lyrName)
    # ...
except:
    pass # normal error handling here
finally:
    for f in [tmp_layer, tmp_table]:
        if f:
            try:
                arcpy.Delete_management(f)
            except:
                pass

Note - we don't need to clean up lyrName because it's just a simple string object that will be deleted when the python script ends. The variable tmp_layer is assigned a result object returned from MakeFeatureLayer, but when you use it as input to a tool (like Delete_management) the tool is only passed the result object's string representation, which is just the string (above, "lyr201409015110231", or, below, "lyrname") so we don't need to unpack it with result.getOutput(0).

Here are some details, shown from the Python prompt (in ArcMap). Gotta love it.

>>> lyr = arcpy.MakeFeatureLayer_management(r"temp.shp", "lyrname")
>>> type(lyr)
<class 'arcpy.arcobjects.arcobjects.Result'>
>>> lyr
<Result 'lyrname'>
>>> str(lyr)
'lyrname'
>>> lyr.getOutput(0)
<map layer u'lyrname'>
>>> arcpy.Delete_management(lyr)
<Result 'true'>
>>> print(arcpy.GetMessages())
Executing: Delete lyrname #
Start Time: Mon Sep 15 11:32:05 2014
Succeeded at Mon Sep 15 11:32:05 2014 (Elapsed Time: 0.00 seconds)

Re: Cleaning up temporary layers created by python toolbox script

JamesCrandall — Fri, 10 Dec 2021 21:43:15 GMT

I just call this def() before/after any in_memory processing. Seems legit.


def clearINMEM():


  arcpy.env.workspace = r"IN_MEMORY"

  fcs = arcpy.ListFeatureClasses()
  tabs = arcpy.ListTables()

  ### for each FeatClass in the list of fcs's, delete it.
  for f in fcs:
    arcpy.Delete_management(f)
    arcpy.AddMessage("deleted: " + f)
  ### for each TableClass in the list of tab's, delete it.
  for t in tabs:
    arcpy.Delete_management(t)
    arcpy.AddMessage("deleted: " + t)

Re: Cleaning up temporary layers created by python toolbox script

curtvprice — Mon, 15 Sep 2014 18:32:44 GMT

James, I believe we are discussing layers, not datasets (in_memory, on disk, or in an RDBMS). I realize I confused the thread a bit by including a table in my code sample. If you delete a feature class that is referenced by a open layer in memory, the Delete_management will not throw an error but the dataset will persist until the python session (not the script) is over. At least that's my experience.

For this reason, you'll see in my finally block that I listed layer first, then the dataset in my delete list.

Re: Cleaning up temporary layers created by python toolbox script

JamesCrandall — Mon, 15 Sep 2014 18:54:18 GMT

Ah. Your Delete_management of FeatureLayers works for me too.

Re: Cleaning up temporary layers created by python toolbox script

FilipKrál — Mon, 15 Sep 2014 19:25:51 GMT

Aha, that's a good point about the locks, Curtics. So it's best to delete the layer as soon as you are done doing what you needed the layer (or table view) for.

Re: Cleaning up temporary layers created by python toolbox script

AdamCox1 — Mon, 15 Sep 2014 19:32:08 GMT

This is all great info, and I especially like the delete everything in memory function. I thought I'd throw my own two cents in also... I began to deal with deleting feature classes like this a while ago by creating my own little python module. It's super useful to have a some functions that you can call anytime from anywhere, because I need to do this "delete if exists" thing all the time (plus a lot of other stock operations that I'm tired of retyping).

Just make a folder in your python site-packages directory (something like C:\Python2x\Lib\site-packages, or, more likely, C:\Python2x\ArcGIS10.x\Lib\site-packages) called mypack, and then in that folder create a text file called __init__.py. You can define all the functions you want in there, and to access them from any console or script just say "from mymod import functionname".

For what we're talking about here, I just have this in the __init__.py file:

import arcpy

def TakeOutTrash(dataset):

if arcpy.Exists(dataset):

arcpy.management.Delete(dataset)

so in the console, making a feature layer could look like:

from mypack import TakeOutTrash

fl = "fl"

TakeOutTrash(fl) # this is important if you are iterating or running the script many times

arcpy.management.MakeFeatureLayer(r"path",fl)

##do some stuff with feature layer

TakeOutTrash(fl)

That function could be modified to accept a list of datasets too, I just haven't done that yet.

Re: Cleaning up temporary layers created by python toolbox script

AdamCox1 — Mon, 15 Sep 2014 19:42:12 GMT

Do you copy and paste this code into the console, or run it from somewhere else? See my post below; it'll be a good thing for me to add to my python package...

Re: Cleaning up temporary layers created by python toolbox script

JamesCrandall — Mon, 15 Sep 2014 20:19:50 GMT

Adam,

I include that cleanup def() in just about all of my .py source files to any of the Geoprocess tools I build that utilize in_memory space. But like Curtis said, it's a little off topic to this thread.