grid files not deleting

[MarcusBeck]

Greetings,

I've been using Python to batch process several thousand grid files.  My code is written to produce temporary files along the way, which are meant to be deleted after each loop iteration. 

Some of the temporary files get tied up in a process with PyScripter and therefore can't be deleted.  The files are left over '.adf' files (not sure what these are for) in the grid folders, and running a remove command produces the following error message.

>>> os.remove("C:/Projects/g2tmpc1/w001001.adf")

Traceback (most recent call last):
  File "<interactive input>", line 1, in <module>
WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'C:/Projects/GIS data/OBIA nearshore/DNR fish lakes/classifications/g2tmpc1/w001001.adf'

Attempting to delete the file in Windows explorer indicates that 'the action can't be completed because the file is open in PyScripter Python IDE'. 

Is there someway to kill python processes so I can delete these files?  Some web sleuthing revealed that the file can be opened and closed manually in Python before removal, but this fails to work.  Any help is greatly appreciated!

-Marcus

[ChrisSnyder]

Try using the arcpy.Delete_managment() tool rather than os.remove.

[MarcusBeck]

Yes, sorry I forgot to mention the reason I am using os.remove is that the gp.delete_management() tool (I am using ArcMap v9.3) fails in this case.  I needed a workaround outside of the geoprocessor.

[ChrisSnyder]

get tied up in a process with PyScripter

What process(es) are you performing? Could it be that you are running a cursor or something on the grid attribute tables and forgetting to delete the cursor objects (and therefore the locks are persisting)?

It would help if you posted your code.

Normally, the grids should delete just fine. I have many a script that loops, does all sorts of fancy stuff, and then cleans up the temp processing grids upon each iteration. There must be something specific you are doing (or not doing) that prevents deleteing the grids.

[MarcusBeck]

What process(es) are you performing? Could it be that you are running a cursor or something on the grid attribute tables and forgetting to delete the cursor objects (and therefore the locks are persisting)?

It would help if you posted your code.

Normally, the grids should delete just fine. I have many a script that loops, does all sorts of fancy stuff, and then cleans up the temp processing grids upon each iteration. There must be something specific you are doing (or not doing) that prevents deleteing the grids.

The code probably raises more questions than answers...  Basically I had to write a function to handle the different outputs based on what happens after running the rest of the code (not shown, it's long and very cryptic).  The temporary files that are created either end in "_final" or "_temp" for vector files, or "tmp" for grid files.  The arguments are provided to specify how many files of each are to be deleted.  For example, delete_temps(2,2,2) would delete "DOW_temp1.shp", "DOW_final1.shp" and "g1tmp" for the grid file.

def delete_temps(arg1,arg2,arg3):
    gp.Delete_management("g1tmp.img")
    temps, temps2 = range(1,arg1), [rasts(y) for y in range(1,arg2)]
    [gp.Delete_management(DOW + "_temp" + str(y) + ".shp") for y in temps]
    if arg1 > 4:
        [gp.Delete_management(y) for y in (DOW + "_segtemp1.img", DOW + "_segtemp2.shp")]
    if arg3 != 0:
        temps3 = range(1,arg3)
        [gp.Delete_management(DOW + "_final" + str(y) + ".shp") for y in temps3]
    try:
        [gp.Delete_management(y) for y in temps2]
    except:
        for y in temps2:
            for x in glob.glob(gp.Workspace + "\\g" + str(y) + "*"):
                try: os.remove(x)
                except: shutil.rmtree(x)
        tables = gp.ListTables("g*","INFO")
        table = tables.next()
        while table:
            gp.Delete_management(table)
            table = tables.next()

I don't have any cursors running so that isn't the problem.  Interestingly (?), the constituent files for the  grid files are deleted, except for the path folders that contain two files with ".adf" extensions, which are what Python/Pyscripter (v2.5.2, v2.4.1, respectively) claims to be running, thus preventing their deletion.  The problem only occurs on, for example, 1 out of every twenty grid files making it hard to diagnose.  I've tried replacing the original grid files that are giving me problems but no such solution. 

I think the only solution might be a forced 'kill' of the Python process that is using these files.  Is there a way to identify the process and then manually stop it?  For example, listing all the processes, identify which ones are using the ".adf" files, then kill it.

[ChrisSnyder]

I don't see anything in that code that would cause a lock - it must be happening in the criptic part.

Do you ever use the MakeRasterLayer tool? If so, make sure you delete the raster layer using the Delete_managment tool.

Once I had a heck of a time deleteing a FGDB that was created in a script (when the script was done, I wanted to delete the FGDB - simple, right?). Turned out that, for some random reason, this one time, the MakeFeatureLayer tool somehow put a lock on the FC that that it was basing the layer off of. Once I deleted the feature layer, I could then delete the FGDB.

Look through your code and see what variables/ESRI objects refer  tothe grid, and delete them. This could be things like the describe object, result objects, cursors,feature layesr, table views, list objects, etc.

[ChrisSnyder]

Only other idea is to create a controler script to call seperate subscripts that:

1. Run your process
2. Delete the temp results

Use os.spawnv or subprocess to lauch the subprocess scripts.

[MarcusBeck]

One thing I haven't tried yet is deleting feature layers... the rest of the code creates anywhere between 3-9 feature layers depending on how the output behaves.  I never bothered to delete these since they are deleted automatically when the session is over (right?), but they might be causing the lock since they are used for creating some of the shapefiles and grid rasters.

Thanks for the suggestions!

[MaartenHofman1]

I found similar problems with deleting grids...

1. Sometimes when I run my script through the PythonWin debugger, it gives no error for any of the grids I'm deleting, and yet the grids are still in the folder by the end of the script. Why would that be?

2. There is one exception: for the bolded line in the code underneath it gives an error (that the file is locked by another application). After running the script, what remains of that raster is a folder with two .adf files. The error is preventing my script to run smoothly as a tool in ArcDesktop...
(btw. I'm converting the script from ArcGIS 9.3 to 10, and I have imported 'arcpy' as 'gp' to make it easier to synchronize the two versions)

    try:
        gp.CheckOutExtension("Spatial")
        if Ecotype == "Droge Vleermuis (Gewone dwergvleermuis)" or Ecotype == "Natte Vleermuis (Watervleermuis)":
            reclasslist = [0, MDVcd, 1]
        else:
            reclasslist = [0, MDAcd / 2, 1]
        reclassifyRange = RemapRange([reclasslist])
        outDispBuffBinary = Reclassify(DispBuff, "Value", reclassifyRange, "DATA")
        outDispBuffBinary.save(DispBuffBinary)
    except:
        print gp.GetMessages()
    # 6.3 Region group binary
    try:
        gp.CheckOutExtension("Spatial")
        outDispBuffRG = RegionGroup(DispBuffBinary, "EIGHT", "WITHIN", "NO_LINK", "")
        outDispBuffRG.save(DispBuffRG)
    except:
        print gp.GetMessages()
    # 6.4 Convert RegionGroup to shapefile
    try:
        gp.RasterToPolygon_conversion(DispBuffRG, DispBuffVector, "SIMPLIFY")
    except:
        print gp.GetMessages()
    # 6.5 Delete stepping stone files
    try:
        gp.Delete_management(DispBuffBinary)
        gp.Delete_management(DispBuffRG)
    except:
        print gp.GetMessages()

The file that is being deleted has only just been made and was saved, wouldn't that remove the LOCK files?

3. After finishing the script, I check in the workfolder and find a series of LOCK files still in there. Is that how it should be?

I haven't tried to make a function for deleting temporary files, which I will try. For now, I'm just trying to understand why deleting grids seems to be so randomly successful..

[curtvprice]

One thing I haven't tried yet is deleting feature layers... the rest of the code creates anywhere between 3-9 feature layers depending on how the output behaves.  I never bothered to delete these since they are deleted automatically when the session is over (right?), but they might be causing the lock since they are used for creating some of the shapefiles and grid rasters.

I have seen the behavior Maarten has - no error message, but the datasets are not removed.

I have found I do have to delete feature and raster layers with gp.Delete() before deleting their associated data sets to release os file locks. In 9.3, table views cannot be deleted, but fortunately they don't seem to hold a lock (but if there are relates, make sure they are dropped before you try to delete the dataset(s) in the table view or its relates).

In 10.0 I've discovered you can (and need to) explicitly delete table views with gp.Delete().

Here's my method of managing temp files. If the delete is unsuccessful I just ignore any problems. If I have created a temp workspace and can't delete it, I add a check to see if it exists to warn the user that the temp workspace is still there.

# note layer and views come BEFORE the data sets in the list
lyrFeat,lyrRast,tvTable,tmpFeat,tmpRast = [None] * 5
...
try:
  # work goes here
except:
  ...
finally:
  try:
     del Row, Rows # any cursors deleted here
  except:
    pass
  # the following is list copied and pasted from above
  for f in lyrFeat,lyrRast,tvTable,tmpFeat,tmpRast:
    try:
       if f: gp.Delete(f)
    except:
      pass