python toolbox doesn't release locks after running tool

[SarahGe]

I have a python toolbox tool that downloads multidimensional rasters via API calls. The tool has a checkbox to allow the user to merge the downloaded rasters. If this box is checked, the tool merges those rasters using the merge multidimensional rasters tool. If I then try to run the tool again, intending to download and overwrite the individual rasters again, the raster download fails because the existing rasters are "being  used by another process". The other process is the merge operation  which is long finished at that point. If the merge checkbox remains unchecked, I can re-run the tool as many times as I like without a problem. None of the input or output rasters are in geodatabases

The only thing that seems to release the lock is closing and opening ArcGIS Pro. I have tried the following without any luck:

• changing the scratch workspace
• adding arcpy.env.overwriteOutput = True to my script 
• putting the merge operation in another script that gets loaded in the main python toolbox
• executing the merge inside a "with" statement: with arcpy.EnvManager(workspace=xyzxy)
• clearing the cache, refreshing the map, saving the project
  

def execute(self, parameters, messages):
	"""The source code of the tool."""

	paramdict={}
	for x in range(0,len(parameters)):
		p=parameters[x]
		paramdict[p.name]=p
		
	inputlist=[
	"--url" , paramdict["baseUrl"].valueAsText,
	"--orders",paramdict["ordersToDownload"].valueAsText,
	"--runs" , paramdict["orderRuns"].valueAsText,
	"--apikey" , paramdict["apikey"].valueAsText,
	"--location" , paramdict["baseFolder"].valueAsText,
	"--verbose"]

	message_queue = queue.Queue()
	files=[]
	folders=[]
# download files using api call
	cda_download(inputlist,message_queue,module=arcpy, files=files, folders=folders)


	if paramdict["addToMap"].value==True:
		messages.addMessage("Merging rasters and adding to map")
		arcpy.AddMessage(folders)
		arcpy.AddMessage(files)
		outfile = paramdict["outRas"].valueAsText
		files_to_merge=files
		try:
			arcpy.md.MergeMultidimensionalRasters(files_to_merge,outfile)
		except Exception as ex:
			messages.addMessage("Something went wrong merging\n")
			messages.addMessage(f"input files: {files_to_merge} \noutput file: {outfile}\n")
			messages.addMessage(ex)


	arcpy.AddMessage("\n===========\nFor Info. Messages produced during downloading:\n")
	print("\n===========\nFor Info. Messages produced during downloading:\n")
	while not message_queue.empty():
		msgq=message_queue.get()
		arcpy.AddMessage(msgq)
		print(msgq)

any thoughts on how to resolve this?

[BrandonMcAlister]

@SarahGe 

Try using arcpy.da.Editor, I am assuming the lock is created because the tool is "editing" the raster layer. 

Your code would look something like this if you have the rasters saved in a gdb, should be the same for an sde the file path would look different though

wksp = arcpy.env.workspace = "File path to where rasters are saved, assuming its an sde or gdb"

edit = arcpy.da.Editor(wksp) #sets edit environment to gdb or sde

edit.startEditing(False, False) #starts editing

edit.startOperation() #starts edit opertion

"This is where the merge tool would go"

edit.stopOperation() #stops edit operation

edit.stopEditing(True) # saves edits

https://pro.arcgis.com/en/pro-app/latest/arcpy/data-access/editor.htm

[SarahGe]

Hi Brandon, just gave that a go but it's still not working. None of the files are in geodatabases, or sdes they are just in folders. I suppose I could try to copy the rasters temporarily somewhere else and then try to delete them after merging.

[BrandonMcAlister]

@SarahGe 

Hey Sarah,

Well since they're not in a Geodatabase or SDE, then I think the python script is creating a variable that is creating the lock on those rasters. To remove the lock you would have to delete the variable for those features using del (variable name).

A good practice that I struggle to remember is to delete variables when they are done being used. in your sample script above the files and files to merge variables.

On line 36 and 37 adding this might work

del files
del files_to_merge

Since those variables appear to be the same files_to_merge is redundant and parsing files would work in the same manner.

I hope this works 🤞

[SarahGe]

many thanks for your suggestions Brandon, it's much appreciated. I tried that, but that didn't work either. I have since also discovered that the lock also occurs when I use the "merge multidimensional rasters" as a stand-alone toolbox tool (ie not using a python script) and have logged a support ticket on myEsri. Will report back here if a solution is found, in case anyone else has this problem.

[SarahGe]

After having been in touch with ESRI support, this has now been logged as a bug. A workaround is to copy and/or rename the files before running the merge while making sure that the new names are not the same as in a previous run. Then delete the copied data after arcgis pro is closed and the locks are released.

[SarahGe · Accepted solution]

And a further update. The problem occurred for me when merging .grib2 files. The locks release fine when .crf datasets are merged. So I use "subset multidimensional raster" to copy the grib2 files to an intermediate data folder and merge them from there. The locks on the crf files are released as soon as merge is complete - so it's possible to re-run the tool now.
Note - I use "subset multidimensional raster" rather than "copy raster" as using "copy raster" somehow changes the way the .grib files are treated in ArcGIS Pro when added to a project: not all raster properties are visible in the layer properties dialog.