Clearing Arc/Arcpy workspace locks

[StacyRendall1]

Particularly with the release of ArcGIS 10.1 I have had a great deal of trouble with workspaces being locked within complex codes that are creating, reading from and writing to numerous feature classes and workspaces. These locks cause the code to fail intermittently at different places, although occasionally making it through without a problem.

I have previously posted on this with a solution that sometimes helps reduce the problem, but is unable to eliminate it completely. That is using arcpy.Exists() and arcpy.Compact() in sequence upon the workspace. This combination of tools, when used in ArcGIS 10.0, seemed adequate, however it is no longer sufficient in 10.1 to prevent issues 100% of the time.

In response to this I have developed a new method for clearing tricky locks which, for my purposes, works 100% of the time. It is a small function that can be added to the start of any arcpy script, and it requires the psutil Python library, which can be acquired here: http://code.google.com/p/psutil/. When using multiprocessing the code seems to work fine with the Parallel Python library, but will hang indefinitely if you are using the Multiprocessing library due to (I guess) the method in which child processes are spawned. However, the Parallel Python library has other issues, for example, I have had trouble with it when getting it to run checked out extensions such as Network Analyst.

The following codeblock shows the necessary imports, the function block and an example of the usage.

import arcpy

import os
import psutil


def clearWSLocks(inputWS):
 '''Attempts to clear ArcGIS/Arcpy locks on a workspace.

 Two methods:
  1: if ANOTHER process (i.e. ArcCatalog) has the workspace open, that process is terminated
  2: if THIS process has the workspace open, it attempts to clear locks using arcpy.Exists, arcpy.Compact and arcpy.Exists in sequence

 Notes:
  1: does not work well with Python Multiprocessing
  2: this will kill ArcMap or ArcCatalog if they are accessing the worspace, so SAVE YOUR WORK

 Required imports: os, psutil
 '''

 # get process ID for this process (treated differently)
 thisPID = os.getpid()

 # normalise path
 _inputWS = os.path.normpath(inputWS)

 # get list of currently running Arc/Python processes
 p_List = []
 ps = psutil.process_iter()
 for p in ps:
  if ('Arc' in p.name) or ('python' in p.name):
   p_List.append(p.pid)

 # iterate through processes
 for pid in p_List:
  p = psutil.Process(pid)

  # if any have the workspace open
  if any(_inputWS in pth for pth in [fl.path for fl in p.get_open_files()]):
   print '      !!! Workspace open: %s' % _inputWS

   # terminate if it is another process
   if pid != thisPID:
    print '      !!! Terminating process: %s' % p.name
    p.terminate()
   else:
    print '      !!! This process has workspace open...'

 # if this process has workspace open, keep trying while it is open...
 while any(_inputWS in pth for pth in [fl.path for fl in psutil.Process(thisPID).get_open_files()]):
  print '    !!! Trying Exists, Compact, Exists to clear locks: %s' % all([arcpy.Exists(_inputWS), arcpy.Compact_management(_inputWS), arcpy.Exists(_inputWS)])

 return True


#######################################
## clearWSLocks usage example #########
#######################################

# define paths and names
workspace = 'C:\\Temp\\test.gdb'
tableName = 'testTable'

# create table
arcpy.CreateTable_management(workspace, tableName)

# usage of clearWSLocks
clearWSLocks(workspace)

I have also attached the code as a separate module (clearWSLocks.py), which can be used in the following fashion:

import arcpy
import clearWSLocks

# define paths and names
workspace = 'C:\\Temp\\test.gdb'
tableName = 'testTable'

# create table
arcpy.CreateTable_management(workspace, tableName)

# usage of clearWSLocks
clearWSLocks.clear(workspace)

[KimOllivier]

psutil has changed the function names in my download. p.name is now a function p.name() and

p.get_open_files() is now p.open_files()

documentation

This will work for processes on your machine, but I don't know how to close files open across a network.

This could happen if someone goes home with ArcMap open and their screen saver on. Anyone with ideas?

[RebeccaStrauch__GISP]

Scott,

Working with the suggestions from Kim Ollivier​ I was able to get the program to run without error, although I'm not sure it did anything in my case.  I did add a print statement so you can see the PiD and process names it is processing.  For me, it didn't close my ArcMap or get rid of my locks, but I don't have anymore time to debug tonight.  I'll include the code as I have it now.

import arcpy  
import os  
import psutil  

def clearWSLocks(inputWS):  
  '''Attempts to clear ArcGIS/Arcpy locks on a workspace.  
  Two methods:  
   1: if ANOTHER process (i.e. ArcCatalog) has the workspace open, that process is terminated  
   2: if THIS process has the workspace open, it attempts to clear locks using arcpy.Exists, arcpy.Compact and arcpy.Exists in sequence  
  Notes:  
   1: does not work well with Python Multiprocessing  
   2: this will kill ArcMap or ArcCatalog if they are accessing the worspace, so SAVE YOUR WORK  
  Required imports: os, psutil  
  '''
  # get process ID for this process (treated differently)
  thisPID = os.getpid()
  # normalise path
  _inputWS = os.path.normpath(inputWS)
  # get list of currently running Arc/Python processes  
  p_List = []
  ps = psutil.process_iter()
  for p in ps:
    if ('Arc' in p.name()) or ('python' in p.name()):
      print("{0}, {1}".format(p.pid, p.name()))
      p_List.append(p.pid)

  # iterate through processes  
  for pid in p_List:
    p = psutil.Process(pid)
    # if any have the workspace open
    if any(_inputWS in pth for pth in [fl.path for fl in p.open_files()]):
      print '      !!! Workspace open: %s' % _inputWS
      # terminate if it is another process 
      if pid != thisPID:
        print '      !!! Terminating process: %s' % p.name()
        p.terminate()  
      else:  
        print '      !!! This process has workspace open...'  

  # if this process has workspace open, keep trying while it is open...  
  while any(_inputWS in pth for pth in [fl.path for fl in psutil.Process(thisPID).open_files()]):  
    print '    !!! Trying Exists, Compact, Exists to clear locks: %s' % all([arcpy.Exists(_inputWS), arcpy.Compact_management(_inputWS), arcpy.Exists(_inputWS)])  
  return True  

#######################################  
## clearWSLocks usage example #########  
#######################################  
# define paths and names  
workspace =  r"C:\temp\test.gdb" 
tableName = 'testTable'  
# create table  
arcpy.CreateTable_management(workspace, tableName)  
# usage of clearWSLocks  
clearWSLocks(workspace)  

[PaulDavidson1]

I recently came across a post or a blog that had a method (I think based in arcpy) for clearing locks.  But I cannot find the link or the sample code.

I bring this up because my memory (faulty at best) tells me that it was using an arcpy function specifically for the task.  Hence it was almost a one liner.

Ring any bells for anyone?

BTW - we've been using Stacy's clearWSLocks since it's early days when it was just a few lines of code.

Works great and is a staple component of our nightly python scripts.

Thanks Stacy!

[KimOllivier]

You might be thinking of the tool in ArcCatalog for removing locks on multiuser enterprise databases such as SqlServer or Oracle. This is a different (and much easier problem) because the database is managing the locks, not Windoze. But there is no python scripting tool, it is just a GUI in ArcCatalog.

[PaulDavidson1]

Thanks Kim...

That might be what I'm thinking of, except I thought I remembered seeing Python code.

I also thought I saved the code as a sample to somewhere safe for later review.

Obviously, it was too safe of a location.

[BruceHarold]

arcpy.RefreshCatalog() ?

Kim used to be a big fan 🙂

[KimOllivier]

Until it was removed at 10.x! I see it has returned to update the in-memory reference after a system copy so maybe it is just a null-op if the paths have not changed? It doesn't mention closing any open table locks so my assumption is that it doesn't.

It is certainly helpful to keep the Catalog up to date with the windows file system when using file geodatabases, which may lag due to various remote services, buffers, file lock indicators (which are just a temporary file) cleanup not kicking in fast enough during running a script.

Bruce can you help us there?

[BruceHarold]

RefreshCatalog() doesn't have any lock handling functionality 😞

[AndréVinko]

As you mentioned correctly, it does not close any 'Arc' process(es). In my environment the "if any(..." expression returns never true because neither the personal gdb (*.mdb) nor its lock file are listed by p.open_files() object as being accessed. I did no try with *.gdb.

I assume, when I take a look at Microsoft's Access JetEngine, that not the "ArcMAP.exe" process itself is locking the file but some other handles will take this in charge. Unfortunately I did not find a way to jump over to such a process and get the needed identifying information.

A second aspect to consider may be the warning statement in "https://pythonhosted.org/psutil/#psutil.Process.open_files" paragraph about Windows API liability. Well, this one seems not to be solvable quite soon despite taken efforts.

Either way no solution shows up for the moment here. Very frustrating...

Any other ideas from around the world? 😉

[LukeCatania]

I am using arcpy.gp.CreateSQLiteDatabase which must have some lock file somewhere whether on disk or memory as once the script ends, the file cannot be deleted or written to unless I exit ArcMap.  If I run the tool to overwrite the file, it won't because it says the file is in use.  I am trying to write to the DB it creates and this does not work unless i exit ArMap and then run the tool to write to the DB. 

I tried this script in hopes it would fix my issue but I get a message:

Traceback (most recent call last):

  File "D:\SAGE Development\ToolBox\raster2gpkg-master\raster2gpkg-master\GeoPackageToolbox.tbx#CreateSQLiteDatabase.py", line 90, in <module>

  File "D:\SAGE Development\ToolBox\raster2gpkg-master\raster2gpkg-master\GeoPackageToolbox.tbx#CreateSQLiteDatabase.py", line 85, in create_sqlite_database

  File "D:\SAGE Development\ToolBox\raster2gpkg-master\raster2gpkg-master\GeoPackageToolbox.tbx#CreateSQLiteDatabase.py", line 39, in clear

TypeError: argument of type 'instancemethod' is not iterable

It seems to have issue with the if statement in

       if ('Arc' in p.name) or ('python' in p.name):