Select to view content in your preferred language

Personal GeoDatabase exceeds capacity and crashes, how to solve?

1033
2
08-09-2010 08:26 AM
DuncanHornby
MVP Notable Contributor
All,

Below is a python script I developed for extracting out elevation information for a point dataset. Basically it gets a list of ID's from a table (the barrier ID table) then for each ID is selects 2 points from a point Featureclass, it then snaps them to the flow accumulation grid, extracts the values and writes the min and max value to an output table.

The code runs fine but eventually crashes after 3 or 4 hours. I then have to compact the personal geodatabase and restart the code from the point it crashed. What I've noticed is that the personal geodatabase just keeps on getting bigger until it hits the 2GB limit (hence the code failing). So my question to all you python gurus is how do I change my code to stop this annoying behaviour? I tried putting a "compact" in every 500th ID but it comes up with an error.

'''
Description: Extracts the minimum and maximum DEM values for each barriers
             upstream and downstream limit
Constraints: Designed to work only in the 9.3 ArcGIS system.
Author: Duncan Hornby
Email: ddh@geodata.soton.ac.uk
Created: 6/8/2010
'''

# Import system modules
import sys, arcgisscripting

# Create the Geoprocessor object & set any environment settings
gp = arcgisscripting.create(9.3)
gp.SetProduct("ArcView")
gp.CheckOutExtension("Spatial")
gp.Workspace = "c:\\temp\\"
gp.cellSize = "50"
gp.OverWriteOutput = 1

# Set any objects and variables
tempRaster = "snapIntPnt"
tempRaster2 = "ExtDEM"
tempLayer = "tempPointLayer.shp"
pgd = "G:\LiveData\UC1168_Sediment_WQ0128\Vector\pGDB93_UC1168_Output.mdb"

# Get parameters
BarrierIDTable = gp.GetParameterAsText(0)
IntersectionPointLayer = gp.GetParameterAsText(1)
FlowAccum = gp.GetParameterAsText(2)
DEM = gp.GetParameterAsText(3)
Output = gp.GetParameterAsText(4)
gp.SnapRaster = FlowAccum

# Create a cursor over the barrier ID table
rows = gp.SearchCursor(BarrierIDTable,"","","OBSTRUCTID","")
row = rows.next()

# Iterate through the rows in the cursor
while row:

    # Get barrier ID
    bID = row.GetValue("OBSTRUCTID")

##    # Check to see if we should compact the geodatabase at every 500th
##    if bID%500 == 0:
##        gp.AddWarning("Compacting GeoDatabase...")
##        gp.Compact_management(pgd)

    gp.AddMessage("Processing Barrier: " + str(bID))

    # Select all the intersection points that belong to this barrier
    gp.SetProgressorLabel("Selecting intersection points...")
    gp.SelectLayerByAttribute_management(IntersectionPointLayer,"NEW_SELECTION","[OBSTRUCTID] = " + str(bID) + " AND [UseAsSlopeLimit] = 'Y'")

    # Convert selection to a layer so we can get an extent
    gp.SetProgressorLabel("Getting extent of selection...")
    gp.CopyFeatures_management(IntersectionPointLayer,tempLayer)
    obj = gp.Describe(tempLayer)
    gp.Extent = obj.Extent
    del obj # Destroy object
    
    # Snap selected points to flow accumulation grid
    gp.SetProgressorLabel("Snapping selected points to flow accumulation grid...")
    gp.SnapPourPoint_sa(IntersectionPointLayer,FlowAccum,tempRaster,"25","OBSTRUCTID")

    # Now extract the DEM values from the pre-prepared raster layer
    gp.SetProgressorLabel("Extracting DEM cells...")
    gp.ExtractByMask_sa(DEM,tempRaster,tempRaster2)

    # Extract the properties from this raster
    gp.SetProgressorLabel("Reading raster properties...")
    resultObj = gp.GetRasterProperties_management(tempRaster2,"MINIMUM")
    zMin = resultObj.GetOutput(0)    
    resultObj = gp.GetRasterProperties_management(tempRaster2,"MAXIMUM")
    zMax = resultObj.GetOutput(0)
    gp.AddMessage("Elevation - Min:" + str(zMin) + " , Max: " + str(zMax))
    del resultObj # Destroy object
    
    # Write data to output table
    gp.SetProgressorLabel("Updating output table...")
    InsertRows = gp.InsertCursor(Output) # Create insert cursor for table
    aNewRow = InsertRows.NewRow()
    aNewRow.SetValue("OBSTRUCTID",bID)
    aNewRow.SetValue("MinEle",zMin)
    aNewRow.SetValue("MaxEle",zMax)
    InsertRows.InsertRow(aNewRow)
    InsertRows.Reset()    
    del aNewRow # Destroy object
    del InsertRows # Destroy object
    
    # Get next Barrier ID    
    row = rows.next()

# Return extension license
gp.CheckInExtension("Spatial")
0 Kudos
2 Replies
ChrisSnyder
Honored Contributor
Use a shapefile. Contrary to popular belief, the ESRI formats in order of (in general) best to worst performance:

1. Shapefile
2. FGDB
3. PGDB

Also, if you are attempting to store a raster in a geodatabase and are wanting to do actual analysis, stop right now! Put it in Grid format, and I guarantee you better performance.
0 Kudos
DuncanHornby
MVP Notable Contributor
Hi Chris,

Thanks for the top tip, I will transfer my point layer to a shapefile to see if I can fix this problem.

The more I delve into Python the more I find my self begging ESRI not to drop VBA at least that works without quirky problems.

I've not been writing the rasters to a Geodatabase, I've never been a fan of that ;).

Duncan
0 Kudos