Raster To Geodatabase Automation

[TonyWu]

Hello all,

I currently have over 4000 .tif raster data that have to be loaded into a SDE Raster Catalog in ArcGIS 9.3.1.  I had been manually uploading 1800 in batches using the Raster to Geodatabase (multiple) tools.  I want to look into python script for the automation of the process. 

Details of the data: All the .tifs are in 1 folder, 18mbs each, 8bits and JPEG compression (the purpose is for visual purpose so dont have to perform and image analysis)

I am very new to python script so if anyone of you can point me to the right direction.

I have found this old Batch Raster to GDB python script and modified to my specifics...

## Script Name: raster2gdb.py
## Description: Imports a raster into a geodatabase.
#-------------------------------------------------------------------------------
# function usage
#-------------------------------------------------------------------------------
def usage():
print " USAGE: raster2gdb.py <raster> <geodatabase>"
sys.exit(1)
#-------------------------------------------------------------------------------
# function bailout
#-------------------------------------------------------------------------------
def bailout(errMsg):
print " ERROR: "+errMsg
sys.exit(0)
#===============================================================================
# main routine
#===============================================================================
import sys, string, os, arcgisscripting

gp = arcgisscripting.create()

gp.AddToolbox("C:/Program Files (x86)/ArcGIS/ArcToolbox/Toolboxes/Conversion Tools.tbx")

# first argument is the input raster
try:
InRaster = "G:\\Teranet2013\\Simcoe_2012_tiles\\final\\ORTH175820495802012FBS.tif"
except:
usage()

# second argument is the geodatabase to import the raster into
try:
OutGDB = "Database Connections\\Connection to PSGISTEST01-Raster.sde\\Raster.PSGIS.SimcoeRegion"
except:
usage()

# does raster exist?
if not gp.exists(InRaster):
bailout("Raster "+InRaster+" does not exist")
# does geodatabase exist?
if not gp.exists(OutGDB):
bailout("Geodatabase "+OutGDB+" does not exist")

gp.pyramid = "PYRAMIDS -1 BILINEAR"
#gp.pyramid = "PYRAMIDS -1 CUBIC"
gp.rasterStatistics = "NONE"
gp.compression = "LZ77"
gp.tilesize = "128 128"

# start reporting
import time
me=os.path.splitext(os.path.basename(sys.argv[0]))[0]
print me,"started at: " + str(time.asctime())
timestamp=time.clock()

try:
# Process: RasterToGeodatabase_conversion
print "Loading " + InRaster + " to " + OutGDB
gp.RasterToGeodatabase_conversion(InRaster,OutGDB,"#")
print "Finished loading " + InRaster
except:
# If an error occurred print the message to the screen
print gp.GetMessages()

# report time
delta=time.clock()-timestamp
timestamp=time.clock()
print me,"successfully finished: ",str(time.asctime())
print "Total time: ",str(delta / 60)," minutes\n"

this script runs good but only process 1 specific file at a time. I was wondering if anyone can give me some help in making a loop to process the 4000+ files in the one folder.

I have look into loop and this is what I try to in corporate in my script to run all the .tifs files in that one folder.
# For loop iterates through every file in input folder
for found_file in os.listdir(G:\Teranet2013\Simcoe_2012_tiles\final):
    # Searches for .tif files
    if found_file.endswith(".tif"):

print "Converting: "+foundfile
InRaster = os.path.join(input_folder, found_file)

So if anyone can give me some guidance in automating batch raster to geodatabase that would be much appreciated!

Thanks,
Tony

[curtvprice]

I currently have over 4000 .tif raster data that have to be loaded into a SDE Raster Catalog in ArcGIS 9.3.1.  I had been manually uploading 1800 in batches using the Raster to Geodatabase (multiple) tools.

Have you considered creating a file-based raster catalog instead? If you do this you can access all tifs as one seamless raster, and if you set it up as "unmanaged", you will not have to convert them -- they will be accessed as they are by reference. (You are working in 9.3.1; in 10.x you should use mosaic datasets, which are similar but better.)

ArcGIS 9.3 Help: Creating raster catalogs in the geodatabase

The Workspace To Raster Catalog tool will batch-load a whole folder of .tif files into your raster catalog.

If what you truly want to make a single ArcSDE raster dataset, it is fairly straightforward to convert the entire raster catalog to a geodatabase raster -- a lot easier than trying to individually load thousands of rasters.

[TonyWu]

The Workspace To Raster Catalog tool will batch-load a whole folder of .tif files into your raster catalog.

If what you truly want to make a single ArcSDE raster dataset, it is fairly straightforward to convert the entire raster catalog to a geodatabase raster -- a lot easier than trying to individually load thousands of rasters.

Can you describe more on the "The Workspace to Raster Catalog Tool"?  Because I can't seem to find the function where I can load the whole folder of .tif into the raster catalog.

When I right click the raster catalog and click "load", A new window "Raster To Geodatabase (multiple)" pop up.  Is this what you mean?

Thanks.

[curtvprice · Accepted solution]

When I right click the raster catalog and click "load", A new window "Raster To Geodatabase (multiple)" pop up.  Is this what you mean?

No, that method you've been using loads the rasters one by one into separate raster datasets.

If I'm understanding right, you have a bunch of map tiles that you want to view together as a single raster layer. The tool I'm talking about is in the ArcToolbox and is not available on the right-click menu.

I recommend reading up on it in the 9.3 help:

First read that article, so you understand raster catalogs. (I think "unmanaged" is what you want):

'>Creating raster catalogs in a geodatabase

And here are some tools to look consider using:

• http://webhelp.esri.com/arcgisdesktop/9.3/index.cfm?id=1905&pid=1903&topicname=Create%20Raster%20Cat...


• '>Create Raster Catalog (Data Management)


• Workspace To Raster Catalog (Data Management)


• This sample tool is designed to create your catalog and load it up in a single step. This may be your best option:

Workspace to New Raster Catalog (Samples)

[TonyWu]

This sample tool is designed to create your catalog and load it up in a single step. This may be your best option:

Workspace to New Raster Catalog (Samples)

I got this command line to run.  But the .tifs failed to load into the SDE Raster Catalog.  Any ideas?

Thank you for your help so far! i really appreciate it!.

[curtvprice]

I got this command line to run.  But the .tifs failed to load into the SDE Raster Catalog. 

Without error messages, I wouldn't have a clue.

Note that Workspace to New Raster catalog assumes the catalog does not exist yet.  If you're running into trouble, I recommend going back to the help and following the instructions carefully so you are certain you are making the right flavor of raster catalog for your needs. You also may want to test with a small number of rasters loading to a file geodatabase.

I am not an expert on the ArcSDE side, but I have heard that a good approach is to get an unmanaged raster catalog up and working with a file geodatabase and your tif files. Once that's working, then you can convert the whole package to a true ArcSDE raster dataset (that process can take days, but in the end you have a very high-performance image service).

There are a lot of tweaks on how this is setup so I recommend piloting a lot with small subsets before you load gigabytes of data into ArcSDE.  This is not a simple point and click deal, there is research, database design, testing, and tuning involved to get what you need.

You may have noticed: I moved this thread over to the Imagery and Raster Data forum in the hope that some experts in this area may chime in.

[TonyWu]

Thank you for your help!!

"Workspace To Raster Catalog" and "Workspace to New Raster Catalog" seems like the right direction for me.  I will look into it and test it out and see what works.  I'll update this thread once I figure out the way.

Thanks again.