topic Re: Script runs fine, but results come slower and slower in Python Questions

Script runs fine, but results come slower and slower

ArnaudTemme — Sun, 05 Aug 2012 01:36:37 GMT

Dear colleagues,

The script below seems to work fine, but slows down considerably and gradually after a few hundred datasets have been added to the geodatabase. The speed comes back when the script is restarted. This makes me believe that the problem is unrelated to the fact that the workspace that I use is on an external drive. The dataset and geodatabase are also too large to move to my internal harddisk so that I could test this. The memory that the process related to the script uses, does not increase as more datasets have been imported (visible through task manager).
Questions: why does this happen and how to avoid it?

import arcpy
from arcpy import env
import random

# Check out any necessary licenses
arcpy.CheckOutExtension("spatial")
arcpy.CheckOutExtension("3D")


# Set environment settings
env.workspace = "G:/My_workspace"

fullset = arcpy.ListDatasets("*", "ALL")
fcCount = len(fullset)
print fcCount

# randomize list so that any proportion of the full workload 
# already delivers a random sample
random.shuffle(fullset)

for tiff in fullset:
 try: 
  arcpy.RasterToGeodatabase_conversion(tiff, "big.gdb")
  arcpy.Delete_management(tiff, "")
  fcCount -= 1
  print fcCount
 except:
  fcCount -= 1
  print "failed at" + fcCount

Re: Script runs fine, but results come slower and slower

ArnaudTemme — Mon, 06 Aug 2012 13:50:42 GMT

After all, this may be due to the cache of the external disk filling up after a few (hundred) datasets. After that, real disk-writing occurs, which is slower. Restarting the script would then mean that the cache gets cleared before work restarts and fills it again.

Re: Script runs fine, but results come slower and slower

curtvprice — Mon, 06 Aug 2012 23:07:03 GMT

After all, this may be due to the cache of the external disk filling up after a few (hundred) datasets. After that, real disk-writing occurs, which is slower. Restarting the script would then mean that the cache gets cleared before work restarts and fills it again.

When you delete an object from a geodatabase, it is not truly deleted, just flagged for deletion. The way to make it really go away and no longer take up disk space is to Compact (not compress) the geodatabase.