Raster Index Script Tool

RyanFortier · ‎06-05-2015

Hello,

I am still somewhat raw with python, although I am learning and trying to develop some tools. What I do need help with is a script tool which creates an index of rasters. It was created by someone who no longer works here anymore. He was fairly advanced with python and he created a wide variety of script tools. The tool in question does almost exactly what I need, however I would like it to avoid certain file extensions (.OVR, .TOC). I am just unsure of where to place a statement in the code to have these file types ignored.

Hopefully someone can help out.

Here is the code

#------------------------------------------------------------------------------
# Description:
#   Generate footprints for all raster products within a directory. Can
#   optionally perform a recursive traverse to find all rasters within the
#   tree.
#
#   Produces a single shapefile output with attributes to identify each raster
#   dataset found.
#
# ArcGIS Version: 10.0
# Python Version: 2.6
#------------------------------------------------------------------------------

import arcpy
import datetime
import os
import sys

def message(msg, severity=0):
"""Notify the user of some event with a message.

    Arcpy does not print messages to command line, and print statements
    do not appear within the Arc runtime. This method can act as a global
    notification system for both environments.
    """
    # print a message to the command line
    print msg

    # notify the user within the arc environment
    # splits a message on \n, making a notification for each line in a
    # multiline string
    try:
        for string in msg.split('\n'):
            # add message under appropriate severity
            if severity == 0:
                arcpy.AddMessage(string)
            elif severity == 1:
                arcpy.AddWarning(string)
            elif severity == 2:
                arcpy.AddError(string)
    except:
        # ignore errors
        pass

def listRasters(rasterTypes = "ALL"):
"""Generate a list of rasters found within a workspace."""

# keep a list of all files for all types
foundFiles = []

    if rasterTypes == "ALL":
        # just gather all file types
        foundFiles = arcpy.ListRasters("*", rasterTypes)
    else:
        # arcpy will not let us specify multiple file types to search for, so
        # we replicate that ability by appending the file lists for each
        # specified file type.
        fileTypes = rasterTypes.split(';')
        for fileType in fileTypes:
            foundFiles.extend(arcpy.ListRasters("*", fileType))

# return all discovered files
return foundFiles

def processRasterList(shapefile, rasterList):
    """Process a list of rasters into coordinates and attributes."""
    # make sure there is some data to be processed
    # if no data is found, notify the user and move on
    if not len(rasterList):
        message("No rasters found.")
        return

    # notify the user work is starting
    message("Processing %d rasters." % len(rasterList))
    arcpy.SetProgressor("step",
                        "Processing rasters...",
                        0,
                        len(rasterList))

    # keep track of if we find any projected data
    projDataFound = False
    for rasterStr in rasterList:
        # inform the user which raster is being processed
        arcpy.SetProgressorLabel("Processing %s" % rasterStr)

        # Wrap the whole thing in a try block to catch possible problems
        # Problems are ignored, and the user is notified that a raster was
        # not processed.
        try:
            # make a raster object
            raster = arcpy.Raster(rasterStr)
        except:
            message("Processing raster failed: %s" % rasterStr, 1)
            message(arcpy.GetMessages(2), 1)
            # abondon this raster and move on
            continue

        # notify the user if projected data is found (prefer geographic)
        if raster.spatialReference.type == "Unknown":
            projDataFound = True

# generate an array to hold each corner of the raster
coords = arcpy.Array()

        # add each corner of the raster to the array
        coords.add(raster.extent.lowerLeft)
        coords.add(raster.extent.upperLeft)
        coords.add(raster.extent.upperRight)
        coords.add(raster.extent.lowerRight)
        # Polygons require the start coordinate to match the end coordinate
        coords.add(raster.extent.lowerLeft)

# make a polygon from the points array
rasterPolygon = arcpy.Polygon(coords, raster.spatialReference)

        try:
            # use an insert cursor to create new records for this raster
            fcRows = arcpy.InsertCursor(shapefile, )
            # make a new row
            row = fcRows.newRow()
            # set all the attributes
            row.shape = rasterPolygon
            row.path = raster.path
            row.filename = raster.name
            row.type = raster.format
            row.process_d = str(datetime.date.today())
            row.num_rows = raster.height
            row.num_cols = raster.width
            row.xmin = raster.extent.XMin
            row.xmax = raster.extent.XMax
            row.ymin = raster.extent.YMin
            row.ymax = raster.extent.YMax
            row.Projection = raster.spatialReference.name
            row.datum = raster.spatialReference.spheroidName
            row.epsg_code = raster.spatialReference.factoryCode
            row.num_bands = raster.bandCount
            row.pixelw = raster.meanCellWidth
            row.pixelh = raster.meanCellHeight
            row.bit_depth = int(raster.pixelType[1:])
            row.comptype = raster.compressionType

            # insert the row into the featureclass
            fcRows.insertRow(row)

            # free up some memory
            del row
            del fcRows
            del coords

        except:
            message("Cataloging raster failed: %s" % rasterStr, 1)
            message(arcpy.GetMessages(2), 1)

# update the progress bar position
arcpy.SetProgressorPosition()

    # notify the user if any unprojected data was found
    if projDataFound:
        message("Unprojected data found. Check outputs.", 1)
    # reset the progress bar to normal
    arcpy.SetProgressorLabel("Looking for rasters...")
    arcpy.ResetProgressor()

def create_featureclass(fcPath, inMemory=False):
    """Create a shapefile to store all the data that is collected from a raster
    list. Also adds all fields required for metadata about each raster.

    """
    # split the path into its component parts
    (dirName, fileName) = os.path.split(fcPath)
    # make all outputs WGS84 Geographic
    spatialReference = os.path.join(arcpy.GetInstallInfo()["InstallDir"],
                             r"Coordinate Systems\Geographic Coordinate Systems\World\WGS 1984.prj")
    # get a path to the template shapefile
    tmplShp = os.path.abspath(os.path.join(os.path.dirname(sys.argv[0]),
                           '..',
                           'data',
                           'raster_index_tmpl.shp'))
    # check if creating feature in memory
    if inMemory == "true":
        dirName = "in_memory"
        fileName = "memoryindex"

    # create the featureclass
    arcpy.CreateFeatureclass_management(dirName,
                                        fileName,
                                        "POLYGON",
                                        tmplShp,
                                        "DISABLED",
                                        "DISABLED",
                                        spatialReference)

return os.path.join(dirName, fileName)

## fields come from the template shapefile, this is just for reference
##    # add appropriate fields to the featureclass
##    arcpy.AddField_management(fcPath, 'Path', 'TEXT', 254)
##    arcpy.AddField_management(fcPath, 'Filename', 'TEXT', 254)
##    arcpy.AddField_management(fcPath, 'Type', 'TEXT', 16)
##    arcpy.AddField_management(fcPath, 'Process_D', 'DATE')
##    arcpy.AddField_management(fcPath, 'NUM_Rows', 'LONG')
##    arcpy.AddField_management(fcPath, 'NUM_Cols', 'LONG')
##    arcpy.AddField_management(fcPath, 'XMin', 'FLOAT', 15, 10)
##    arcpy.AddField_management(fcPath, 'XMax', 'FLOAT', 15, 10)
##    arcpy.AddField_management(fcPath, 'YMin', 'FLOAT', 15, 10)
##    arcpy.AddField_management(fcPath, 'YMax', 'FLOAT', 15, 10)
##    arcpy.AddField_management(fcPath, 'Projection', 'TEXT', 50)
##    arcpy.AddField_management(fcPath, 'Datum', 'TEXT', 50)
##    arcpy.AddField_management(fcPath, 'EPSG_Code', 'TEXT', 6)
##    arcpy.AddField_management(fcPath, 'NUM_Bands', 'SHORT')
##    arcpy.AddField_management(fcPath, 'PixelW', 'FLOAT', 15, 10)
##    arcpy.AddField_management(fcPath, 'PixelH', 'FLOAT', 15, 10)
##    arcpy.AddField_management(fcPath, 'BIT_DEPTH', 'SHORT')
##    arcpy.AddField_management(fcPath, 'Compression', 'TEXT', 20)

def index_rasters(inputWorkspace, outputFeatureClass, processRecursive,
                  inMemoryProcessing, fileTypes = ""):
    """Process all rasters within a given directory."""
    # notify if using in memory processing
    if inMemoryProcessing == "true":
        message("Using in memory processing")

    # support the ability to index multiple directories at the same time, either
    # as recursive or not.
    inputWorkspaces = inputWorkspace.split(';')
    for inWS in inputWorkspaces:
        # arc wraps directories in quotes for fun
        inWS = inWS.replace("'", "")
        # set the workspace to be the base directory from the user
        arcpy.env.workspace = inWS

        # if the user doesn't want to process only certain files, set "ALL"
        if fileTypes == "":
            fileTypes = "ALL"

        try:
            # create a featureclass to hold all raster metadata
            if len(inputWorkspaces) == 1 or not arcpy.Exists(outputFeatureClass):
                featureData = create_featureclass(outputFeatureClass,
                                                  inMemoryProcessing)

            # Determine if recursive. No need to walk if not doing a recursive
            # list.
            if processRecursive == 'false':
                message("Gathering raster list from %s" % arcpy.env.workspace)
                rasterList = listRasters(fileTypes)
                processRasterList(featureData, rasterList)
            else:
                message("Recursive processing started.")
                for root, dirs, files in os.walk(inWS):
                    # have to do this once for the top level folder
                    arcpy.env.workspace = root
                    message("Gathering raster list from %s" % arcpy.env.workspace)
                    rasterList = listRasters(fileTypes)
                    processRasterList(featureData, rasterList)
##                    for subdir in dirs:
##                        arcpy.env.workspace = os.path.join(root, subdir)
##                        message("Gathering raster list from %s" % arcpy.env.workspace)
##                        rasterList = listRasters(fileTypes)
##                        processRasterList(outputFeatureClass, rasterList)

            if inMemoryProcessing == "true":
                # if processing was done in memory, write items to disk
                arcpy.Merge_management(featureData, outputFeatureClass)
                # delete the in memory feature class
                arcpy.Delete_management(featureData)

        except arcpy.ExecuteError:
            print arcpy.GetMessages(2)
            arcpy.AddError(arcpy.GetMessages(2))
        except Exception as e:
            print e.args[0]
            arcpy.AddError(e.args[0])

# Test how the script is running. This allows it to run from multiple
# environments, or to be included as a module into another script without
# destroying environments.
if __name__ == '__main__':
    # gather all arguments that have been passed in
    argv = tuple(arcpy.GetParameterAsText(i)
                 for i in xrange(arcpy.GetArgumentCount()))
    index_rasters(*argv)

IanMurray · ‎06-05-2015

Hi Ryan,

def listRasters(rasterTypes = "ALL"):
    """Generate a list of rasters found within a workspace."""
    # keep a list of all files for all types
    foundFiles = []
    if rasterTypes == "ALL":
        # just gather all file types
        foundFiles = arcpy.ListRasters("*", rasterTypes)
    else:
        # arcpy will not let us specify multiple file types to search for, so
        # we replicate that ability by appending the file lists for each
        # specified file type.
        fileTypes = rasterTypes.split(';')
        for fileType in fileTypes:
            foundFiles.extend(arcpy.ListRasters("*", fileType))
    # return all discovered files
    return foundFiles

The file type is specified fairly early on, in line 1 of the code I posted, which is close to the beginning of the code you pasted in. Currently it is set to all, but it can be changed to many file types see this help.

For help posting code blocks with formatting in the future see this help, Posting Code blocks in the new GeoNet

Probably the easier thing to do is modify the variable foundfiles, which is a list with the file name of each raster that will be analyzed. A simple for loop checking for the extensions you are wanting to remove and taking them out of the list should fix your problem before that function closes would do the trick.

JakeSkinner · ‎06-05-2015

Hi Ryan,

Here is an example of what Ian mentioned doing. Lines 20-22 will remove the rasters that have extensions of .OVR or .TOC.

def listRasters(rasterTypes = "ALL"):
    """Generate a list of rasters found within a workspace."""

    # keep a list of all files for all types
    foundFiles = []

    if rasterTypes == "ALL":
        # just gather all file types
        foundFiles = arcpy.ListRasters("*", rasterTypes)

    else:
        # arcpy will not let us specify multiple file types to search for, so
        # we replicate that ability by appending the file lists for each
        # specified file type.
        fileTypes = rasterTypes.split(';')
        for fileType in fileTypes:
            foundFiles.extend(arcpy.ListRasters("*", fileType))

    # return all discovered files
    for raster in foundFiles:
      if '.OVR' or '.TOC' in raster:
        foundFiles.remove(raster)

    return foundFiles