topic Re: Clearing Arc/Arcpy workspace locks in Python Questions

Clearing Arc/Arcpy workspace locks

StacyRendall1 — Sat, 11 Dec 2021 06:57:01 GMT

Particularly with the release of ArcGIS 10.1 I have had a great deal of trouble with workspaces being locked within complex codes that are creating, reading from and writing to numerous feature classes and workspaces. These locks cause the code to fail intermittently at different places, although occasionally making it through without a problem.

I have previously posted on this with a solution that sometimes helps reduce the problem, but is unable to eliminate it completely. That is using arcpy.Exists() and arcpy.Compact() in sequence upon the workspace. This combination of tools, when used in ArcGIS 10.0, seemed adequate, however it is no longer sufficient in 10.1 to prevent issues 100% of the time.

In response to this I have developed a new method for clearing tricky locks which, for my purposes, works 100% of the time. It is a small function that can be added to the start of any arcpy script, and it requires the psutil Python library, which can be acquired here: http://code.google.com/p/psutil/. When using multiprocessing the code seems to work fine with the Parallel Python library, but will hang indefinitely if you are using the Multiprocessing library due to (I guess) the method in which child processes are spawned. However, the Parallel Python library has other issues, for example, I have had trouble with it when getting it to run checked out extensions such as Network Analyst.

The following codeblock shows the necessary imports, the function block and an example of the usage.

import arcpy

import os
import psutil


def clearWSLocks(inputWS):
 '''Attempts to clear ArcGIS/Arcpy locks on a workspace.

 Two methods:
  1: if ANOTHER process (i.e. ArcCatalog) has the workspace open, that process is terminated
  2: if THIS process has the workspace open, it attempts to clear locks using arcpy.Exists, arcpy.Compact and arcpy.Exists in sequence

 Notes:
  1: does not work well with Python Multiprocessing
  2: this will kill ArcMap or ArcCatalog if they are accessing the worspace, so SAVE YOUR WORK

 Required imports: os, psutil
 '''

 # get process ID for this process (treated differently)
 thisPID = os.getpid()

 # normalise path
 _inputWS = os.path.normpath(inputWS)

 # get list of currently running Arc/Python processes
 p_List = []
 ps = psutil.process_iter()
 for p in ps:
  if ('Arc' in p.name) or ('python' in p.name):
   p_List.append(p.pid)

 # iterate through processes
 for pid in p_List:
  p = psutil.Process(pid)

  # if any have the workspace open
  if any(_inputWS in pth for pth in [fl.path for fl in p.get_open_files()]):
   print '      !!! Workspace open: %s' % _inputWS

   # terminate if it is another process
   if pid != thisPID:
    print '      !!! Terminating process: %s' % p.name
    p.terminate()
   else:
    print '      !!! This process has workspace open...'

 # if this process has workspace open, keep trying while it is open...
 while any(_inputWS in pth for pth in [fl.path for fl in psutil.Process(thisPID).get_open_files()]):
  print '    !!! Trying Exists, Compact, Exists to clear locks: %s' % all([arcpy.Exists(_inputWS), arcpy.Compact_management(_inputWS), arcpy.Exists(_inputWS)])

 return True


#######################################
## clearWSLocks usage example #########
#######################################

# define paths and names
workspace = 'C:\\Temp\\test.gdb'
tableName = 'testTable'

# create table
arcpy.CreateTable_management(workspace, tableName)

# usage of clearWSLocks
clearWSLocks(workspace)

I have also attached the code as a separate module (clearWSLocks.py), which can be used in the following fashion:

import arcpy
import clearWSLocks

# define paths and names
workspace = 'C:\\Temp\\test.gdb'
tableName = 'testTable'

# create table
arcpy.CreateTable_management(workspace, tableName)

# usage of clearWSLocks
clearWSLocks.clear(workspace)

Re: Clearing Arc/Arcpy workspace locks

JamesCrandall — Tue, 20 Nov 2012 11:29:23 GMT

Hi there Stacy,

I'm mainly replying here so that I am subscribed to this thread, but also uncover possible alternatives (for me) as other comments come in as I am new to Python development. Plus I will likely need to implement the code example you've provided!

I've actually had a run-in with the locking issue, but was able to mitigate (actually, eliminate) the problem by reading/writing intermediate data to the IN_MEMORY space, then move that processed data to it's final location(s) in their relevent workspaces and repositories (PGDB and FDGB's).

I am wondering, is there something in your application architecture that is limiting your ability to utilize this IN_MEMORY space to read/write intermediate data?

Is the IN_MEMORY space inadequate for some reason (eg, the data is simply too large)?

The multiprocessing requirements do not allow for the use of the IN_MEMORY space?

Anyway -- great topic and useful code.

Take Care,
j

Re: Clearing Arc/Arcpy workspace locks

KimOllivier — Thu, 22 Nov 2012 08:35:11 GMT

I have simple scripts that sometimes report a file lock and when I run them again immediately, just run normally.
It is very hard to make some processes give up the file locks.

At 10.1 it has been suggested in the technical sessions to use the 'with' construct which is guaranteed to close cursors instead of relying on garbage collection after a del cursor.

Any other hints and experiences?

Re: Clearing Arc/Arcpy workspace locks

StacyRendall1 — Sun, 25 Nov 2012 18:46:14 GMT

Thanks Kim - for a while I required my code to run in both 10.0 and 10.1 at the same time, so I couldn't use the with statement. However, I still had issues with locks after creating feature classes and adding fields, etc., which do not have a with statement...

They really should implement something to let the program definitely close the file (like a .close() function), such as discussed by Sean Gillies in his excellent blog here. I guess the issue occurs because files are just represented by paths, not file objects as they are in GDAL/OGR.

Re: Clearing Arc/Arcpy workspace locks

StacyRendall1 — Sun, 25 Nov 2012 18:48:37 GMT

Thanks James,

Is the IN_MEMORY space inadequate for some reason (eg, the data is simply too large)?

I have seen a few people talk about IN_MEMORY, but I have never seen any help files or code examples using it, so I have never tried it out! Sounds like something I should look into!

Cheers,
Stacy

Re: Clearing Arc/Arcpy workspace locks

JamesCrandall — Sat, 11 Dec 2021 06:57:03 GMT

Thanks James,

I have seen a few people talk about IN_MEMORY, but I have never seen any help files or code examples using it, so I have never tried it out! Sounds like something I should look into!

Cheers,
Stacy

In addition to relief from the locking issue I was having, the in_memory space performs much better. However, I do see that threre are limitations to it as well and it can be a double-edged sword as you will then be upping the RAM usage (I think). But the idea is to do the processing there, then write out the final output to where you need the result(s) to be.

It is very easy to implement, just set your workspace to it and process as normal. Here is an example of a def() I use before I start any processing to "clear" out the IN_MEMORY sapce (although, I am not so sure this is even required. But because I am still unsure how garbage collection fully works when tools are being run from ArcCatalog in succession -- so I just clear it out anyway).

I am implementing arcgisscripting(9.3) for ArcGIS 9.3.1 here


def ClearINMEM():
   ## clear out the IN_MEMORY workspace of any featureclasses
   
   try:
     gp.Workspace = "IN_MEMORY"
     
     ### clear in_memory of featureClasses
     fcs = gp.ListFeatureClasses()
     ### for each FeatClass in the list of fcs's, delete it.
     for f in fcs:
        gp.Delete_management(f)
        gp.AddMessage("deleted feature class: " + f)
        


     ### clear in_memory of tables
     tables = gp.ListTables()
     for tbl in tables:
        gp.Delete_management(tbl)
        gp.AddMessage("deleted table: " + tbl)
        
        
     gp.AddMessage("in_memory space cleared")
     
   except:
     gp.AddMessage("The following error(s) occured attempting to ClearINMEM " + gp.GetMessages())
     return

Re: Clearing Arc/Arcpy workspace locks

PaulDavidson1 — Fri, 06 Mar 2015 00:23:54 GMT

Late to the party but....

Stacy, thanks for the post.

I've inherited some python code that makes use of your 10.0 ClearWSLocks so very helpful to find this "newer" version since we're still running servers on 10.1

Just to add a contribution, the IN_MEMORY is very useful, IF you have sufficient RAM on the processing device.

I wrote a script to pickup a file geodB of roads for a large county (almost 1200 Sq Miles) and dump part of it out into an Excel file. Using IN_MEMORY really sped it up.

Re: Clearing Arc/Arcpy workspace locks

ScottLinton — Wed, 19 Aug 2015 01:25:04 GMT

Hi Stacy,

I am trying to update a feature class by running a python script after hours.

Unfortunately I am having trouble with file locks caused by other users.

I have tried your clearWSLocks.py, but it runs with this error:

So it is complaining about this section of code:

Do you have any idea what could be causing this?

I am using ArcGIS 10.2... not sure if that makes a difference or not.

Thanks for your help!

Re: Clearing Arc/Arcpy workspace locks

RebeccaStrauch__GISP — Wed, 19 Aug 2015 02:46:22 GMT

Scott, did you download psutil (or do you have it already?) Looks like you can get it

https://pypi.python.org/pypi/psutil/#downloads

If that isn't imported, then is will have an issue with line 26 in your code above.

Re: Clearing Arc/Arcpy workspace locks

ScottLinton — Wed, 19 Aug 2015 02:54:09 GMT

Hi there,

Yes I have installed psutil... any other ideas?

Thanks.

Re: Clearing Arc/Arcpy workspace locks

KimOllivier — Wed, 19 Aug 2015 05:12:24 GMT

psutil has changed the function names in my download. p.name is now a function p.name() and

p.get_open_files() is now p.open_files()

documentation

This will work for processes on your machine, but I don't know how to close files open across a network.

This could happen if someone goes home with ArcMap open and their screen saver on. Anyone with ideas?

Re: Clearing Arc/Arcpy workspace locks

RebeccaStrauch__GISP — Sat, 11 Dec 2021 06:57:06 GMT

Scott,

Working with the suggestions from Kim Ollivier I was able to get the program to run without error, although I'm not sure it did anything in my case. I did add a print statement so you can see the PiD and process names it is processing. For me, it didn't close my ArcMap or get rid of my locks, but I don't have anymore time to debug tonight. I'll include the code as I have it now.

import arcpy  
import os  
import psutil  

def clearWSLocks(inputWS):  
  '''Attempts to clear ArcGIS/Arcpy locks on a workspace.  
  Two methods:  
   1: if ANOTHER process (i.e. ArcCatalog) has the workspace open, that process is terminated  
   2: if THIS process has the workspace open, it attempts to clear locks using arcpy.Exists, arcpy.Compact and arcpy.Exists in sequence  
  Notes:  
   1: does not work well with Python Multiprocessing  
   2: this will kill ArcMap or ArcCatalog if they are accessing the worspace, so SAVE YOUR WORK  
  Required imports: os, psutil  
  '''
  # get process ID for this process (treated differently)
  thisPID = os.getpid()
  # normalise path
  _inputWS = os.path.normpath(inputWS)
  # get list of currently running Arc/Python processes  
  p_List = []
  ps = psutil.process_iter()
  for p in ps:
    if ('Arc' in p.name()) or ('python' in p.name()):
      print("{0}, {1}".format(p.pid, p.name()))
      p_List.append(p.pid)

  # iterate through processes  
  for pid in p_List:
    p = psutil.Process(pid)
    # if any have the workspace open
    if any(_inputWS in pth for pth in [fl.path for fl in p.open_files()]):
      print '      !!! Workspace open: %s' % _inputWS
      # terminate if it is another process 
      if pid != thisPID:
        print '      !!! Terminating process: %s' % p.name()
        p.terminate()  
      else:  
        print '      !!! This process has workspace open...'  

  # if this process has workspace open, keep trying while it is open...  
  while any(_inputWS in pth for pth in [fl.path for fl in psutil.Process(thisPID).open_files()]):  
    print '    !!! Trying Exists, Compact, Exists to clear locks: %s' % all([arcpy.Exists(_inputWS), arcpy.Compact_management(_inputWS), arcpy.Exists(_inputWS)])  
  return True  

#######################################  
## clearWSLocks usage example #########  
#######################################  
# define paths and names  
workspace =  r"C:\temp\test.gdb" 
tableName = 'testTable'  
# create table  
arcpy.CreateTable_management(workspace, tableName)  
# usage of clearWSLocks  
clearWSLocks(workspace)

Re: Clearing Arc/Arcpy workspace locks

PaulDavidson1 — Thu, 20 Aug 2015 00:12:36 GMT

I recently came across a post or a blog that had a method (I think based in arcpy) for clearing locks. But I cannot find the link or the sample code.

I bring this up because my memory (faulty at best) tells me that it was using an arcpy function specifically for the task. Hence it was almost a one liner.

Ring any bells for anyone?

BTW - we've been using Stacy's clearWSLocks since it's early days when it was just a few lines of code.

Works great and is a staple component of our nightly python scripts.

Thanks Stacy!

Re: Clearing Arc/Arcpy workspace locks

KimOllivier — Thu, 20 Aug 2015 22:03:37 GMT

You might be thinking of the tool in ArcCatalog for removing locks on multiuser enterprise databases such as SqlServer or Oracle. This is a different (and much easier problem) because the database is managing the locks, not Windoze. But there is no python scripting tool, it is just a GUI in ArcCatalog.

Re: Clearing Arc/Arcpy workspace locks

PaulDavidson1 — Thu, 20 Aug 2015 22:53:30 GMT

Thanks Kim...

That might be what I'm thinking of, except I thought I remembered seeing Python code.

I also thought I saved the code as a sample to somewhere safe for later review.

Obviously, it was too safe of a location.

Re: Clearing Arc/Arcpy workspace locks

BruceHarold — Thu, 20 Aug 2015 23:08:39 GMT

arcpy.RefreshCatalog() ?

Kim used to be a big fan 🙂

Re: Clearing Arc/Arcpy workspace locks

KimOllivier — Thu, 20 Aug 2015 23:32:00 GMT

Until it was removed at 10.x! I see it has returned to update the in-memory reference after a system copy so maybe it is just a null-op if the paths have not changed? It doesn't mention closing any open table locks so my assumption is that it doesn't.

It is certainly helpful to keep the Catalog up to date with the windows file system when using file geodatabases, which may lag due to various remote services, buffers, file lock indicators (which are just a temporary file) cleanup not kicking in fast enough during running a script.

Bruce can you help us there?

Re: Clearing Arc/Arcpy workspace locks

BruceHarold — Fri, 21 Aug 2015 16:17:19 GMT

RefreshCatalog() doesn't have any lock handling functionality 😞

Re: Clearing Arc/Arcpy workspace locks

AndréVinko — Tue, 26 Jan 2016 15:12:20 GMT

As you mentioned correctly, it does not close any 'Arc' process(es). In my environment the "if any(..." expression returns never true because neither the personal gdb (*.mdb) nor its lock file are listed by p.open_files() object as being accessed. I did no try with *.gdb.

I assume, when I take a look at Microsoft's Access JetEngine, that not the "ArcMAP.exe" process itself is locking the file but some other handles will take this in charge. Unfortunately I did not find a way to jump over to such a process and get the needed identifying information.

A second aspect to consider may be the warning statement in "https://pythonhosted.org/psutil/#psutil.Process.open_files" paragraph about Windows API liability. Well, this one seems not to be solvable quite soon despite taken efforts.

Either way no solution shows up for the moment here. Very frustrating...

Any other ideas from around the world? 😉

Re: Clearing Arc/Arcpy workspace locks

LukeCatania — Fri, 25 Mar 2016 01:24:40 GMT

I am using arcpy.gp.CreateSQLiteDatabase which must have some lock file somewhere whether on disk or memory as once the script ends, the file cannot be deleted or written to unless I exit ArcMap. If I run the tool to overwrite the file, it won't because it says the file is in use. I am trying to write to the DB it creates and this does not work unless i exit ArMap and then run the tool to write to the DB.

I tried this script in hopes it would fix my issue but I get a message:

Traceback (most recent call last):

File "D:\SAGE Development\ToolBox\raster2gpkg-master\raster2gpkg-master\GeoPackageToolbox.tbx#CreateSQLiteDatabase.py", line 90, in <module>

File "D:\SAGE Development\ToolBox\raster2gpkg-master\raster2gpkg-master\GeoPackageToolbox.tbx#CreateSQLiteDatabase.py", line 85, in create_sqlite_database

File "D:\SAGE Development\ToolBox\raster2gpkg-master\raster2gpkg-master\GeoPackageToolbox.tbx#CreateSQLiteDatabase.py", line 39, in clear

TypeError: argument of type 'instancemethod' is not iterable

It seems to have issue with the if statement in

if ('Arc' in p.name) or ('python' in p.name):