topic Re: Locks left on shapefiles when using SelectLayerBy tools in Python Questions

Locks left on shapefiles when using SelectLayerBy tools

JimDetwiler1 — Mon, 30 Mar 2020 21:07:29 GMT

I recently switched from Desktop to Pro and have run into a problem with locks being left on shapefiles when I use SelectLayerByAttribute or SelectLayerByLocation. Both the Desktop and Pro documentation show the same stand-alone script for both tools, though the Pro version of the script is a bit different.

DESKTOP:

env.workspace = "c:/data/mexico.gdb"

# Make a layer from the feature class

arcpy.MakeFeatureLayer_management("cities", "lyr")

# Select all cities which overlap the chihuahua polygon

arcpy.SelectLayerByLocation_management("lyr", "intersect", "chihuahua", 0, "new_selection")

PRO:

arcpy.env.workspace = 'c:/data/mexico.gdb'

# Select all cities which overlap the chihuahua polygon

chihuahua_cities = arcpy.SelectLayerByLocation_management('cities', 'INTERSECT', 'chihuahua', 0, 'NEW_SELECTION')

The Desktop sample shows using MakeFeatureLayer to create a feature layer out of the cities feature class, then passing that as the first argument to the SelectLayerByLocation tool. The Pro sample doesn't bother to create a feature layer first; it simply plugs in the feature class ('cities'). The Pro sample also shows storing a returned object in a variable (chihuahua_cities).

When I've used these tools on shapefiles in Desktop, I've always used MakeFeatureLayer as shown in the sample. It places a lock on the shapefile, but I can remove the lock by using Delete when I'm done (in this case, I'd do arcpy.Delete_management("lyr")).

When I try to follow the syntax shown in the Pro help (storing the returned object in a variable), I also get a lock placed on the shapefile, but I haven't found a way to remove the lock aside from closing my IDE. I've tried:

arcpy.Delete_management(chihuahua_cities)

arcpy.Delete_management(chihuahua_cities[0])

But neither actually removes the lock. I can continue to code it the way that's shown in the Desktop help, but was wondering if there was a fix to the problem that follows the syntax shown in the Pro help.

Thanks!

Re: Locks left on shapefiles when using SelectLayerBy tools

DanPatterson_Retired — Mon, 30 Mar 2020 21:19:00 GMT

If you are running a script and not doing the process line-by-line, then you shouldn't have issues with locks.

You might want to try deleting the python variables if you have them.

Most IDE's have a 'delete variable names' and a restart kernel (like Spyder) so you don't need a complete shutdown of the IDE (although it provides a suitable coffee break time)

Re: Locks left on shapefiles when using SelectLayerBy tools

JimDetwiler1 — Tue, 31 Mar 2020 00:34:54 GMT

Thanks for the quick reply. I had been running line by line, but just tried running it normally and the locks are still there. I had also already tried del variable, but that doesn't remove the lock either. And I discovered the Restart option in Spyder because of this issue. 🙂

Re: Locks left on shapefiles when using SelectLayerBy tools

DanPatterson_Retired — Tue, 31 Mar 2020 01:19:29 GMT

hmmmm

Select Layer By Location—Data Management toolbox | Documentation

Determines how the selection will be applied to the input and how to combine it with an existing selection. Note that there is no option here to clear an existing selection. To clear a selection, use the CLEAR_SELECTION option on the Select Layer By Attribute tool.

Since you are doing it line by line, why not whip in a

Select Layer By Attribute—Data Management toolbox | Documentation

with the above clearing the selection to see if that does anything

I assume that you have Pro open when you are doing this, which may also be an issue.

Re: Locks left on shapefiles when using SelectLayerBy tools

JimDetwiler1 — Tue, 31 Mar 2020 02:09:39 GMT

No effect after adding SelectLayerByAttribute with CLEAR_SELECTION. Yes, I've had Pro open. I just tried again with Pro closed, but no difference.

The documentation says the tools return a Result object, so I've also tried using getOutput(0) and using Delete on that object. And I've tried doing setSelectionSet([]). No luck.

Re: Locks left on shapefiles when using SelectLayerBy tools

DanPatterson_Retired — Tue, 31 Mar 2020 02:12:42 GMT

Turn it into a script, run it. Python won't lock it, and if the lock is there it is a Pro issue.

Also, dump the shapefile and try it with a featureclass in a gdb

Re: Locks left on shapefiles when using SelectLayerBy tools

JimDetwiler1 — Tue, 31 Mar 2020 03:14:18 GMT

Sorry if I wasn't clear. I am running a script in Spyder. I've tried stepping through it with the debugger and I've tried just running it normally with the same result.

And what I'm talking about happening is a .lock file appears in my Windows File Explorer as soon as I run SelectLayerByLocation or SelectLayerByAttribute.

I just put my data into a file GDB and ran the script on it with the same result. When I look inside the .gdb folder, a .lock file appears after running SelectLayerByLocation. If I try using Delete on the object returned by the tool, the .lock file persists.

However, if I run these tools the way they're demonstrated in the Desktop help -- creating a layer using MakeFeatureLayer, using the layer as an input to the tool, then running Delete on the layer when I'm done with it -- the lock file immediately disappears.

Maybe you could try it and see if you get the same behavior?
import arcpy

lyr= arcpy.SelectLayerByAttribute_management(yourFC, 'NEW_SELECTION', yourWhereClause)

arcpy.Delete_management(lyr)

Re: Locks left on shapefiles when using SelectLayerBy tools

DanPatterson_Retired — Sun, 12 Dec 2021 07:42:30 GMT

import arcpy

f = r"C:\Git_Dan\npgeom\Project_npg\npgeom.gdb\Polygons2"

res = arcpy.SelectLayerByAttribute_management(f, "NEW_SELECTION", '"OBJECTID" > 1')

res[0]
<arcpy._mp.Layer at 0x18f06bc35f8>

res[1]
'2'

res.getInput(0)
'C:\\Git_Dan\\npgeom\\Project_npg\\npgeom.gdb\\Polygons2'

res.getInput(1)
'NEW_SELECTION'

res.getOutput(0)
<arcpy._mp.Layer at 0x18f06bc5fd0>

res.cancel()

del res

‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Lock remained until the kernel was restarted.

Project not opened, lock file had a size of 0 bytes.

Delete_management did squat.

del res killed it in python and wasn't available after line 24.

So script or no script.... locks remain until the calling/using thread was terminated.. which sort of makes sense to me, just in case you might want to use the selection.

clear_selection, cleared it, but produces another result (getting a headache) and the lock remained.

Restarting the kernel killed the process and the lock went away (and I got coffee).

Now what I didn't do was try Copy_management to save the result to disk to see if that freed the lock.

Re: Locks left on shapefiles when using SelectLayerBy tools

DanPatterson_Retired — Tue, 31 Mar 2020 07:01:45 GMT

FAQ: Why am I getting schema locks on shapefiles?

effectively, the observations are as expected since they don't persist on process close

Re: Locks left on shapefiles when using SelectLayerBy tools

JimDetwiler1 — Tue, 31 Mar 2020 14:30:56 GMT

Thanks again for your help, Dan. For background, I teach an introductory Python scripting class and have always operated under the assumption that scripts shouldn't leave locks in place when they've finished. I see that after running my script I can open Pro and edit a shapefile that had a lock created on it by my script, so I guess it's a "benign" one. Still, it would be nice if I didn't have to try to explain that locks are sometimes a problem, sometimes not.

Re: Locks left on shapefiles when using SelectLayerBy tools

DanPatterson_Retired — Tue, 31 Mar 2020 16:02:28 GMT

well I just finished 30+ years of teaching... so enjoy!

I think the file size of 0 bytes is important and that i could repeat the selection that it is benign. The help files contain information on when a lock is an issue and for the most part, it is the "sharesies thing" with multiple people potentially using the same data. I also think python might be done with "arc stuff" when you delete it and it is garbage collected... but that doesn't mean that arc is done with python until you kill the python thread that is running then there is nothing left to hand on to.

Re: Locks left on shapefiles when using SelectLayerBy tools

MichaelWallace3 — Tue, 21 Jun 2022 15:22:20 GMT

@JimDetwiler1 thanks for posting this issue report and @DanPatterson_Retired for confirming the behavior. For reference I created a script tool to iteratively do select by location so that I could attribute features that were connected together. This script works fine in the python command line in pro, but run as a script tool it will run once and then I can't select any other features afterwards. Resolution so far has been to shutdown and restart Pro. I don't know if someone could look at this issue again and maybe create a suitable resolution to kill the lock on the selected object? If not at least thank you for explaining all of this.

Re: Locks left on shapefiles when using SelectLayerBy tools

Adrian — Tue, 28 Mar 2023 12:22:17 GMT

Left behind locks are a big issue within arcpy and we struggle with that almost every day. There are several arcpy functions which are not cleaning up lock files, respectively the resources are only unlocked after the Python process terminates. This is cumbersome, not best practise and results in many problems. For example serially running geo processing tools can stumble across lock files from previous steps. Moving or deleting the processed data afterwards is not possible with locks.

A few months ago, we tried to explain the general problem to Esri support with a ticket and a simple use case:

arcpy.Exists(path/to/a/gdb)

shutil.rmtree(path/to/a/gdb)

The code fails because of locks (PermissionError), which is not expected.

The answer from Esri support showed us, that the message of this general problem has not been received: "When this script is run it fails on "shutil.rmtree(path/to/a/gdb) [...] It would not be considered an Esri product in what we are responsible for. [...] When the python process is closed the lock goes away."

So we remain stuck with our workaround to use the multiprocessing library of Python to run affected arcpy functions in separate processes. Because these separate processes terminate, the lock files are cleared and the main process can go on without any problems.