Arcpy: How do I copy FGDBR from multiple file geodatabases into one file geodatabase?

IngoWolff · ‎06-17-2020

Hello,

I have several thousands of file geodatabases with different FGDBR Datasets I want to identify and copy into a central file geodatabase. Unfortunately my ArcPy skills are not so advanced to get the nested loops to run from one file geodatabase to the next, selecting and copying the FGDBR into my targed file geodatabase.

I started with this code snipped, but it does not find any of my selected FGDBRs. What am I missing?

Any help is greatly appreciated!

import arcpy
from arcpy import env
import os
arcpy.env.overwriteOutput = True
arcpy.env.parallelProcessingFactor = "75%"
#
#
#--------------------
#Setting up first directories for source and destination for iterative raster data loop no 1
dir = r"F:\Daten_Wolff\Topographische_Gefaehrdungsanalyse\Test_Source"
arcpy.env.workspace = dir
folder001 = r"F:\Daten_Wolff\Topographische_Gefaehrdungsanalyse\BlueSpot_050.gdb"+ "\\"
gdbList = arcpy.ListWorkspaces('*','FileGDB')
for gdb in gdbList:
 arcpy.env.workspace = gdb #--change working directory to each GDB in list
 datasetList = arcpy.ListDatasets('*','Raster') #--make a list of all (if any) rastere datasets that exist in current GDB
 for dataset in datasetList:
 arcpy.env.workspace = dataset #--change working directory to each dataset (if any) in list
 for Raster in arcpy.ListRasters('*050*'):
 arcpy.env.workspace = dataset
 for Raster in arcpy.ListRasters('*050*'):
 print("processing raster: %s" %os.path.join("env.workspace",Raster))
 outRasterdataset = folder001 + str(Raster.rsplit("_",0)[0])
 arcpy.CopyRaster_management (Raster, outRasterdataset, "DEFAULTS", "", "-999", "NONE", "NONE" , "32_BIT_FLOAT", "NONE", "NONE" )‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

DanPatterson · ‎06-18-2020

Ingo Wolff See

/blogs/dan_patterson/2016/08/14/script-formatting

so your indentation can be checked and line numbers provided

... sort of retired...

IngoWolff · ‎06-18-2020

Cheers for that advice, I changed my snipped accordingly

DanPatterson · ‎06-18-2020

Ingo Wolff‌ your indentation is still inconsistent.

I would suggest you have a look at arcpy.Walk to simplify your path formation

Walk—Data Access module | Documentation

... sort of retired...

IngoWolff · ‎06-18-2020

Hi Dan, cheers for your quick answer!

Regarding my indentation, I now follow this example arcgis 10.0 - Listing feature classes in multiple geodatabases in folder using ArcPy? - Geographic I...

to go throgh my file geodatabases and datasets.

My script does look like this:

import arcpy
from arcpy import env
import os

arcpy.env.overwriteOutput = True

arcpy.env.parallelProcessingFactor = "75%"
#
#
#--------------------
#Setting up first directories for source and destination for iterative raster data loop no 1
dir = r"F:\Daten_Wolff\Topographische_Gefaehrdungsanalyse\Test_Source"
arcpy.env.workspace = dir
folder = r"F:\Daten_Wolff\Topographische_Gefaehrdungsanalyse\BlueSpot_050.gdb"+ "\\"
gdbList = arcpy.ListWorkspaces('*','FileGDB')
for gdb in gdbList:
    arcpy.env.workspace = gdb               #--change working directory to each GDB in list
    datasetList = arcpy.ListDatasets('*','Raster')     #--make a list of all (if any) rastere datasets that exist in current GDB
    for dataset in datasetList:
        arcpy.env.workspace = dataset   #--change working directory to each dataset (if any) in list
    for Raster in arcpy.ListRasters():
        arcpy.env.workspace = dataset
        for Raster in arcpy.ListRasters('*050*'):
            print("processing raster: %s" %os.path.join("env.workspace",Raster))
            outRasterdataset = folder  + str(Raster.rsplit("_",0)[0])
            arcpy.CopyRaster_management (Raster, outRasterdataset, "DEFAULTS", "", "-999", "NONE", "NONE" , "32_BIT_FLOAT", "NONE", "NONE" )
    
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

There is no error message, but no result either after this script has run through...

Thank you for your patience with me newbie!

Ingo

IngoWolff · ‎06-18-2020

Ok, after some iterations on my script I found out, that the

gdbList = arcpy.ListWorkspaces('*','FileGDB')
for gdb in gdbList:
    arcpy.env.workspace = gdb               #--change working directory to each GDB in list
    datasetList = arcpy.ListDatasets('*','Raster')     #--make a list of all (if any) rastere datasets that exist in current GDB
    for dataset in datasetList:
        arcpy.env.workspace = dataset ‍‍‍‍‍‍

part is useless because I just want the FGDBR within each file geodatabase...

I shortened my loops accordingly:

import arcpy
from arcpy import env
import os

arcpy.env.overwriteOutput = True

arcpy.env.parallelProcessingFactor = "75%"
#
#
#--------------------
#Setting up first directories for source and destination for iterative raster data loop no 1
dir = r"F:\Daten_Wolff\Topographische_Gefaehrdungsanalyse\Test_Source"
arcpy.env.workspace = dir
gdbList = arcpy.ListWorkspaces('*','FileGDB')
for gdb in gdbList:
    print("processing Workspace: %s" %os.path.join("env.workspace",gdb))
    arcpy.env.workspace = gdb               #--change working directory to each GDB in list
    folder = r"F:\Daten_Wolff\Topographische_Gefaehrdungsanalyse\BlueSpot_50.gdb"+ "\\"
    RasterList = arcpy.ListRasters('*_050*')
    for Raster in RasterList:
        print("processing raster: %s" %os.path.join("env.workspace",Raster))
        outRasterdataset = folder  + str(Raster.rsplit("_",0)[0])
        arcpy.CopyRaster_management (Raster, outRasterdataset, "DEFAULTS", "", "-999", "NONE", "NONE" , "32_BIT_FLOAT", "NONE", "NONE" )
                
        ‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

So technically my script should go through each file geodatabase and then check für FGDBRs ending with *_050 in the filename and copy them into my target file geodatabase named "folder".

Sadly I get this error message now:

Python 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 11:27:44) [MSC v.1900 64 bit (AMD64)] on win32
Type "copyright", "credits" or "license()" for more information.
>>> 
= RESTART: F:/Daten_Wolff/Scripte/Zentrale_Scripte/Topostar_Copy_raster_4.py =
processing Workspace: F:\Daten_Wolff\Topographische_Gefaehrdungsanalyse\Test_Source\T2718199.gdb
processing raster: env.workspace\bluespot_T2718199_050
Traceback (most recent call last):
  File "F:/Daten_Wolff/Scripte/Zentrale_Scripte/Topostar_Copy_raster_4.py", line 24, in <module>
    arcpy.CopyRaster_management (Raster, outRasterdataset, "DEFAULTS", "", "-999", "NONE", "NONE" , "32_BIT_FLOAT", "NONE", "NONE" )
  File "C:\Program Files\ESRI\ArcGISPro\Resources\ArcPy\arcpy\management.py", line 15289, in CopyRaster
    raise e
  File "C:\Program Files\ESRI\ArcGISPro\Resources\ArcPy\arcpy\management.py", line 15286, in CopyRaster
    retval = convertArcObjectToPythonObject(gp.CopyRaster_management(*gp_fixargs((in_raster, out_rasterdataset, config_keyword, background_value, nodata_value, onebit_to_eightbit, colormap_to_RGB, pixel_type, scale_pixel_value, RGB_to_Colormap, format, transform), True)))
  File "C:\Program Files\ESRI\ArcGISPro\Resources\ArcPy\arcpy\geoprocessing\_base.py", line 498, in <lambda>
    return lambda *args: val(*gp_fixargs(args, True))
arcgisscripting.ExecuteError: ERROR 999999: Something unexpected caused the tool to fail. Contact Esri Technical Support (http://esriurl.com/support) to Report a Bug, and refer to the error help for potential solutions or workarounds.
The table was not found. [VAT_bluespot_T2718199_050]
No spatial reference exists.
The table was not found. [bluespot_T2718199_050]
Failed to execute (CopyRaster).

>>> ‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

So, where is my mistake? Any advice greatly appreciated!

Cheers

Ingo

JoshuaBixby · ‎06-18-2020

As Dan suggests, try using Walk—Help | Documentation. The ArcPy DA Walk function emulates Python's built-in os.walk function. What I would add for a suggestion is, don't rely on setting the workspace for tools to find your data sets, it will inevitably cause issues. Use 10.1. os.path — Common pathname manipulations — Python 2.7.18 documentation to create the full path to your data sets, and then pass that full path to the tools.

IngoWolff · ‎06-19-2020

Hi,

ok, I tried to use walk instead of the more complicated list-approach.

It now seems to go more exactly through the directories, however, I cannot get my rasters within the gdbs selected and copied into a new gdb.

import arcpy
import os
workspace = r"F:\Daten_Wolff\Topographische_Gefaehrdungsanalyse\Test_Source"
folder = r"F:\Daten_Wolff\Topographische_Gefaehrdungsanalyse\BlueSpot_50.gdb" + "\\"
rasters = []
for dirpath, dirnames, filenames in arcpy.da.Walk(workspace, topdown=True, datatype="RasterDataset"):
    for filename in filenames:
        #print("processing Workspace: %s" %os.path.join(dirpath,filename))
        if "_050" in filename:
            print("processing Workspace: %s" %os.path.join(dirpath,filename))
            outRasterdataset = folder  + str(filename.rsplit(".",0)[0])
            arcpy.CopyRaster_management (filename, outRasterdataset, "DEFAULTS", "", "-999", "NONE", "NONE" , "32_BIT_FLOAT", "NONE", "NONE" )

‍‍‍‍‍‍‍‍‍‍‍‍‍

I only get following error message:

Python 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 11:27:44) [MSC v.1900 64 bit (AMD64)] on win32
Type "copyright", "credits" or "license()" for more information.
>>> 
 RESTART: F:\Daten_Wolff\Scripte\Zentrale_Scripte\Topostar_Copy_raster_5_walk.py 
processing Workspace: F:\Daten_Wolff\Topographische_Gefaehrdungsanalyse\Test_Source\T2718199.gdb\bluespot_T2718199_050
Traceback (most recent call last):
  File "F:\Daten_Wolff\Scripte\Zentrale_Scripte\Topostar_Copy_raster_5_walk.py", line 12, in <module>
    arcpy.CopyRaster_management (filename, outRasterdataset, "DEFAULTS", "", "-999", "NONE", "NONE" , "32_BIT_FLOAT", "NONE", "NONE" )
  File "C:\Program Files\ESRI\ArcGISPro\Resources\ArcPy\arcpy\management.py", line 15289, in CopyRaster
    raise e
  File "C:\Program Files\ESRI\ArcGISPro\Resources\ArcPy\arcpy\management.py", line 15286, in CopyRaster
    retval = convertArcObjectToPythonObject(gp.CopyRaster_management(*gp_fixargs((in_raster, out_rasterdataset, config_keyword, background_value, nodata_value, onebit_to_eightbit, colormap_to_RGB, pixel_type, scale_pixel_value, RGB_to_Colormap, format, transform), True)))
  File "C:\Program Files\ESRI\ArcGISPro\Resources\ArcPy\arcpy\geoprocessing\_base.py", line 498, in <lambda>
    return lambda *args: val(*gp_fixargs(args, True))
arcgisscripting.ExecuteError: Failed to execute. Parameters are not valid.
ERROR 000732: Input Raster: Dataset bluespot_T2718199_050 does not exist or is not supported
Failed to execute (CopyRaster).

>>> ‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

So it seems, that my raster data format FileGeoDatabaseRaster (FGDBR) is not supported? At least it finds the right raster I wanted...

What am I missing?

Many thanks!

Ingo

DanPatterson · ‎06-19-2020

Raster file formats—ArcGIS Pro | Documentation

does it go by another name? I am familiar with esri grids and other raster formats, but I work with locally stored data so that may limit my exposure to other raster formats

... sort of retired...

IngoWolff · ‎06-19-2020

What exactly do you mean with another name? Type of raster or just a different part of filename for the selection?

As far as I found out, FGDBR Raster are not mentioned within the list of types in Walk—Help | Dokumentation ...
Therefore I'm afraid, "Walk" is not really the thing I could use.
When I use the List approach it fails to write the rasters into the targed gdb instead of failing of reading the selected rasters...quite an hard nut to crush