topic Re: Code gets slower and slower in Python Questions

Code gets slower and slower

CorinneFrey — Wed, 18 Jul 2012 08:09:12 GMT

Hi

I'm new to phyton and need to convert a bunch of dababase files to shapefiles. I have a batch processing code which is working fine. However, after a while it gets incredibly slow and takes more than an hour for something it took only 10 minutes at the start. I guess that it's somehow a memory - related problem.

Anybody an idea how to clear memory or otherwise fasten the program?

Here is the code:

import arcpy
from arcpy import env
import os

# Set the workspace for the ListFiles function
env.workspace = "g:/CAPAS_Data/N50/"
# Use the ListFile function to return a list
ListDatabase = arcpy.ListFiles("*.mdb")
# print ListDatabase
for x in ListDatabase: 
     env.workspace = "g:/CAPAS_Data/N50/"+x
     
     ListDatasets = arcpy.ListDatasets()

     for y in ListDatasets: 
          env.workspace = "g:/CAPAS_Data/N50/"+x+"/"+y
          ListClasses = arcpy.ListFeatureClasses()

          for z in ListClasses:
               inFeature = 
               inFeature_str = str(inFeature)
               outLocation= "g:/CAPAS_Data/shapefiles/"+inFeature_str[3:-2]+"/"

               if not os.path.exists(outLocation):
                    os.makedirs(outLocation)
                    
               arcpy.FeatureClassToShapefile_conversion(inFeature, outLocation)

Thanks for any help!
Corinne

Re: Code gets slower and slower

AndrewChapkowski — Wed, 18 Jul 2012 12:17:30 GMT

How are you running the code? What IDE are you using? PythonWin for example will not clear out items from memory when you run the code.

In addition, you can delete objects you are no longer using, for example: del ListDatabase, ListDatasets at the end of your code.

You might want to consider putting calling your code from a toolbox script, so a new python instance spins up every time you run the script.

Other factors include:
- hard drive speed
- network speed
- RAM in machine
- CPU speed
- Size of feature classes copied

Nothing really jumps out at me besides not managing your objects.
How does this function compare to arcpy.CopyFeatures()?

Hope this helps.

Re: Code gets slower and slower

AustinDavis — Wed, 18 Jul 2012 12:34:16 GMT

Try using "while" loops as well. These are often faster than "for" loops.

Also try using direct OS calls like os.listdir().

Re: Code gets slower and slower

BruceNielsen — Wed, 18 Jul 2012 13:09:42 GMT

Since you're accessing personal geodatabases, you will want to keep your eye on the size of the MDB files, and run a Compact and Repair in Access if they start growing too large.

Re: Code gets slower and slower

MathewCoyle — Wed, 18 Jul 2012 14:00:53 GMT

Since you're accessing personal geodatabases, you will want to keep your eye on the size of the MDB files, and run a Compact and Repair in Access if they start growing too large.

Whenever I am dealing with PGDB I throw a Compact in at the end of the script to clean it up again. I remember it solved some issues for me at one time and I do it just out of habit now.

Also, I'd recommend putting everything in a function to help deal with persisting variables.

Re: Code gets slower and slower

CorinneFrey — Thu, 19 Jul 2012 05:29:57 GMT

Thanks for the answers.

I'm running Phyton from the ArcGIS installation.

I got the access files from somebody else and I don't have influence on their size.

I'll try to improve the code now, thanks for you help!

Cheers,
Corinne