topic Re: Arcpy: How do I copy FGDBR from multiple file geodatabases into one file geodatabase? in Python Questions

Arcpy: How do I copy FGDBR from multiple file geodatabases into one file geodatabase?

IngoWolff — Sat, 11 Dec 2021 20:06:28 GMT

Hello,

I have several thousands of file geodatabases with different FGDBR Datasets I want to identify and copy into a central file geodatabase. Unfortunately my ArcPy skills are not so advanced to get the nested loops to run from one file geodatabase to the next, selecting and copying the FGDBR into my targed file geodatabase.

I started with this code snipped, but it does not find any of my selected FGDBRs. What am I missing?

Any help is greatly appreciated!

import arcpy
from arcpy import env
import os
arcpy.env.overwriteOutput = True
arcpy.env.parallelProcessingFactor = "75%"
#
#
#--------------------
#Setting up first directories for source and destination for iterative raster data loop no 1
dir = r"F:\Daten_Wolff\Topographische_Gefaehrdungsanalyse\Test_Source"
arcpy.env.workspace = dir
folder001 = r"F:\Daten_Wolff\Topographische_Gefaehrdungsanalyse\BlueSpot_050.gdb"+ "\\"
gdbList = arcpy.ListWorkspaces('*','FileGDB')
for gdb in gdbList:
 arcpy.env.workspace = gdb #--change working directory to each GDB in list
 datasetList = arcpy.ListDatasets('*','Raster') #--make a list of all (if any) rastere datasets that exist in current GDB
 for dataset in datasetList:
 arcpy.env.workspace = dataset #--change working directory to each dataset (if any) in list
 for Raster in arcpy.ListRasters('*050*'):
 arcpy.env.workspace = dataset
 for Raster in arcpy.ListRasters('*050*'):
 print("processing raster: %s" %os.path.join("env.workspace",Raster))
 outRasterdataset = folder001 + str(Raster.rsplit("_",0)[0])
 arcpy.CopyRaster_management (Raster, outRasterdataset, "DEFAULTS", "", "-999", "NONE", "NONE" , "32_BIT_FLOAT", "NONE", "NONE" )‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Arcpy: How do I copy FGDBR from multiple file geodatabases into one file geodatabase?

DanPatterson — Thu, 18 Jun 2020 08:45:34 GMT

Ingo Wolff See

/blogs/dan_patterson/2016/08/14/script-formatting

so your indentation can be checked and line numbers provided

Re: Arcpy: How do I copy FGDBR from multiple file geodatabases into one file geodatabase?

IngoWolff — Thu, 18 Jun 2020 09:01:42 GMT

Cheers for that advice, I changed my snipped accordingly

Re: Arcpy: How do I copy FGDBR from multiple file geodatabases into one file geodatabase?

DanPatterson — Thu, 18 Jun 2020 09:08:40 GMT

Ingo Wolff‌ your indentation is still inconsistent.

I would suggest you have a look at arcpy.Walk to simplify your path formation

Walk—Data Access module | Documentation

Re: Arcpy: How do I copy FGDBR from multiple file geodatabases into one file geodatabase?

IngoWolff — Sat, 11 Dec 2021 20:06:31 GMT

Hi Dan, cheers for your quick answer!

Regarding my indentation, I now follow this example arcgis 10.0 - Listing feature classes in multiple geodatabases in folder using ArcPy? - Geographic Information Systems S…

to go throgh my file geodatabases and datasets.

My script does look like this:

import arcpy
from arcpy import env
import os

arcpy.env.overwriteOutput = True

arcpy.env.parallelProcessingFactor = "75%"
#
#
#--------------------
#Setting up first directories for source and destination for iterative raster data loop no 1
dir = r"F:\Daten_Wolff\Topographische_Gefaehrdungsanalyse\Test_Source"
arcpy.env.workspace = dir
folder = r"F:\Daten_Wolff\Topographische_Gefaehrdungsanalyse\BlueSpot_050.gdb"+ "\\"
gdbList = arcpy.ListWorkspaces('*','FileGDB')
for gdb in gdbList:
    arcpy.env.workspace = gdb               #--change working directory to each GDB in list
    datasetList = arcpy.ListDatasets('*','Raster')     #--make a list of all (if any) rastere datasets that exist in current GDB
    for dataset in datasetList:
        arcpy.env.workspace = dataset   #--change working directory to each dataset (if any) in list
    for Raster in arcpy.ListRasters():
        arcpy.env.workspace = dataset
        for Raster in arcpy.ListRasters('*050*'):
            print("processing raster: %s" %os.path.join("env.workspace",Raster))
            outRasterdataset = folder  + str(Raster.rsplit("_",0)[0])
            arcpy.CopyRaster_management (Raster, outRasterdataset, "DEFAULTS", "", "-999", "NONE", "NONE" , "32_BIT_FLOAT", "NONE", "NONE" )
    
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

There is no error message, but no result either after this script has run through...

Thank you for your patience with me newbie!

Ingo

Re: Arcpy: How do I copy FGDBR from multiple file geodatabases into one file geodatabase?

IngoWolff — Sat, 11 Dec 2021 20:06:34 GMT

Ok, after some iterations on my script I found out, that the

gdbList = arcpy.ListWorkspaces('*','FileGDB')
for gdb in gdbList:
    arcpy.env.workspace = gdb               #--change working directory to each GDB in list
    datasetList = arcpy.ListDatasets('*','Raster')     #--make a list of all (if any) rastere datasets that exist in current GDB
    for dataset in datasetList:
        arcpy.env.workspace = dataset ‍‍‍‍‍‍

part is useless because I just want the FGDBR within each file geodatabase...

I shortened my loops accordingly:

import arcpy
from arcpy import env
import os

arcpy.env.overwriteOutput = True

arcpy.env.parallelProcessingFactor = "75%"
#
#
#--------------------
#Setting up first directories for source and destination for iterative raster data loop no 1
dir = r"F:\Daten_Wolff\Topographische_Gefaehrdungsanalyse\Test_Source"
arcpy.env.workspace = dir
gdbList = arcpy.ListWorkspaces('*','FileGDB')
for gdb in gdbList:
    print("processing Workspace: %s" %os.path.join("env.workspace",gdb))
    arcpy.env.workspace = gdb               #--change working directory to each GDB in list
    folder = r"F:\Daten_Wolff\Topographische_Gefaehrdungsanalyse\BlueSpot_50.gdb"+ "\\"
    RasterList = arcpy.ListRasters('*_050*')
    for Raster in RasterList:
        print("processing raster: %s" %os.path.join("env.workspace",Raster))
        outRasterdataset = folder  + str(Raster.rsplit("_",0)[0])
        arcpy.CopyRaster_management (Raster, outRasterdataset, "DEFAULTS", "", "-999", "NONE", "NONE" , "32_BIT_FLOAT", "NONE", "NONE" )
                
        ‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

So technically my script should go through each file geodatabase and then check für FGDBRs ending with *_050 in the filename and copy them into my target file geodatabase named "folder".

Sadly I get this error message now:

Python 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 11:27:44) [MSC v.1900 64 bit (AMD64)] on win32
Type "copyright", "credits" or "license()" for more information.
>>> 
= RESTART: F:/Daten_Wolff/Scripte/Zentrale_Scripte/Topostar_Copy_raster_4.py =
processing Workspace: F:\Daten_Wolff\Topographische_Gefaehrdungsanalyse\Test_Source\T2718199.gdb
processing raster: env.workspace\bluespot_T2718199_050
Traceback (most recent call last):
  File "F:/Daten_Wolff/Scripte/Zentrale_Scripte/Topostar_Copy_raster_4.py", line 24, in <module>
    arcpy.CopyRaster_management (Raster, outRasterdataset, "DEFAULTS", "", "-999", "NONE", "NONE" , "32_BIT_FLOAT", "NONE", "NONE" )
  File "C:\Program Files\ESRI\ArcGISPro\Resources\ArcPy\arcpy\management.py", line 15289, in CopyRaster
    raise e
  File "C:\Program Files\ESRI\ArcGISPro\Resources\ArcPy\arcpy\management.py", line 15286, in CopyRaster
    retval = convertArcObjectToPythonObject(gp.CopyRaster_management(*gp_fixargs((in_raster, out_rasterdataset, config_keyword, background_value, nodata_value, onebit_to_eightbit, colormap_to_RGB, pixel_type, scale_pixel_value, RGB_to_Colormap, format, transform), True)))
  File "C:\Program Files\ESRI\ArcGISPro\Resources\ArcPy\arcpy\geoprocessing\_base.py", line 498, in <lambda>
    return lambda *args: val(*gp_fixargs(args, True))
arcgisscripting.ExecuteError: ERROR 999999: Something unexpected caused the tool to fail. Contact Esri Technical Support (http://esriurl.com/support) to Report a Bug, and refer to the error help for potential solutions or workarounds.
The table was not found. [VAT_bluespot_T2718199_050]
No spatial reference exists.
The table was not found. [bluespot_T2718199_050]
Failed to execute (CopyRaster).

>>> ‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

So, where is my mistake? Any advice greatly appreciated!

Cheers

Ingo

Re: Arcpy: How do I copy FGDBR from multiple file geodatabases into one file geodatabase?

JoshuaBixby — Thu, 18 Jun 2020 14:43:07 GMT

As Dan suggests, try using Walk—Help | Documentation. The ArcPy DA Walk function emulates Python's built-in os.walk function. What I would add for a suggestion is, don't rely on setting the workspace for tools to find your data sets, it will inevitably cause issues. Use 10.1. os.path — Common pathname manipulations — Python 2.7.18 documentation to create the full path to your data sets, and then pass that full path to the tools.

Re: Arcpy: How do I copy FGDBR from multiple file geodatabases into one file geodatabase?

IngoWolff — Sat, 11 Dec 2021 20:06:37 GMT

Hi,

ok, I tried to use walk instead of the more complicated list-approach.

It now seems to go more exactly through the directories, however, I cannot get my rasters within the gdbs selected and copied into a new gdb.

import arcpy
import os
workspace = r"F:\Daten_Wolff\Topographische_Gefaehrdungsanalyse\Test_Source"
folder = r"F:\Daten_Wolff\Topographische_Gefaehrdungsanalyse\BlueSpot_50.gdb" + "\\"
rasters = []
for dirpath, dirnames, filenames in arcpy.da.Walk(workspace, topdown=True, datatype="RasterDataset"):
    for filename in filenames:
        #print("processing Workspace: %s" %os.path.join(dirpath,filename))
        if "_050" in filename:
            print("processing Workspace: %s" %os.path.join(dirpath,filename))
            outRasterdataset = folder  + str(filename.rsplit(".",0)[0])
            arcpy.CopyRaster_management (filename, outRasterdataset, "DEFAULTS", "", "-999", "NONE", "NONE" , "32_BIT_FLOAT", "NONE", "NONE" )

‍‍‍‍‍‍‍‍‍‍‍‍‍

I only get following error message:

Python 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 11:27:44) [MSC v.1900 64 bit (AMD64)] on win32
Type "copyright", "credits" or "license()" for more information.
>>> 
 RESTART: F:\Daten_Wolff\Scripte\Zentrale_Scripte\Topostar_Copy_raster_5_walk.py 
processing Workspace: F:\Daten_Wolff\Topographische_Gefaehrdungsanalyse\Test_Source\T2718199.gdb\bluespot_T2718199_050
Traceback (most recent call last):
  File "F:\Daten_Wolff\Scripte\Zentrale_Scripte\Topostar_Copy_raster_5_walk.py", line 12, in <module>
    arcpy.CopyRaster_management (filename, outRasterdataset, "DEFAULTS", "", "-999", "NONE", "NONE" , "32_BIT_FLOAT", "NONE", "NONE" )
  File "C:\Program Files\ESRI\ArcGISPro\Resources\ArcPy\arcpy\management.py", line 15289, in CopyRaster
    raise e
  File "C:\Program Files\ESRI\ArcGISPro\Resources\ArcPy\arcpy\management.py", line 15286, in CopyRaster
    retval = convertArcObjectToPythonObject(gp.CopyRaster_management(*gp_fixargs((in_raster, out_rasterdataset, config_keyword, background_value, nodata_value, onebit_to_eightbit, colormap_to_RGB, pixel_type, scale_pixel_value, RGB_to_Colormap, format, transform), True)))
  File "C:\Program Files\ESRI\ArcGISPro\Resources\ArcPy\arcpy\geoprocessing\_base.py", line 498, in <lambda>
    return lambda *args: val(*gp_fixargs(args, True))
arcgisscripting.ExecuteError: Failed to execute. Parameters are not valid.
ERROR 000732: Input Raster: Dataset bluespot_T2718199_050 does not exist or is not supported
Failed to execute (CopyRaster).

>>> ‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

So it seems, that my raster data format FileGeoDatabaseRaster (FGDBR) is not supported? At least it finds the right raster I wanted...

What am I missing?

Many thanks!

Ingo

Re: Arcpy: How do I copy FGDBR from multiple file geodatabases into one file geodatabase?

DanPatterson — Fri, 19 Jun 2020 10:24:47 GMT

Raster file formats—ArcGIS Pro | Documentation

does it go by another name? I am familiar with esri grids and other raster formats, but I work with locally stored data so that may limit my exposure to other raster formats

Re: Arcpy: How do I copy FGDBR from multiple file geodatabases into one file geodatabase?

IngoWolff — Fri, 19 Jun 2020 11:11:22 GMT

What exactly do you mean with another name? Type of raster or just a different part of filename for the selection?

As far as I found out, FGDBR Raster are not mentioned within the list of types in Walk—Help | Dokumentation ...
Therefore I'm afraid, "Walk" is not really the thing I could use.
When I use the List approach it fails to write the rasters into the targed gdb instead of failing of reading the selected rasters...quite an hard nut to crush

Re: Arcpy: How do I copy FGDBR from multiple file geodatabases into one file geodatabase?

DanPatterson — Fri, 19 Jun 2020 13:18:22 GMT

walk = arcpy.da.Walk(workspace, topdown=True, datatype="RasterDataset")

So that doesn't work then? (from the help)

I am assuming that it is a file geodatabase raster dataset

Re: Arcpy: How do I copy FGDBR from multiple file geodatabases into one file geodatabase?

IngoWolff — Mon, 22 Jun 2020 12:02:20 GMT

Exactly, the rasters are file geodatabase raster datasets (fgdbr)...

I manage to get all fgdbrs properly displayed with "print", but with the list approach I can't write the fgdbrs into the correct targed directory and withe the walk-approach I can't read the fgdbrs properly ...

Would be lovely to get some official help from ‌ as the error code suggests.

Cheers

Ingo

Re: Arcpy: How do I copy FGDBR from multiple file geodatabases into one file geodatabase?

JoshuaBixby — Mon, 22 Jun 2020 14:13:50 GMT

Can you elaborate on what isn't working with reading raster datasets with ArcPy Walk?

Re: Arcpy: How do I copy FGDBR from multiple file geodatabases into one file geodatabase?

DanPatterson — Mon, 22 Jun 2020 14:56:35 GMT

You have to file a case with Tech Support.

They normally don't troll around here looking for problems to solve

Re: Arcpy: How do I copy FGDBR from multiple file geodatabases into one file geodatabase?

IngoWolff — Sat, 11 Dec 2021 20:06:39 GMT

Hi!

I got this error-message:

arcgisscripting.ExecuteError: Failed to execute. Parameters are not valid.
ERROR 000732: Input Raster: Dataset bluespot_T2718199_050 does not exist or is not supported
Failed to execute (CopyRaster).

I think it is due to the fgdbr Format. But this is rather strange, as it is an ESRI dataformat which should easily be readable in arcpy...

Re: Arcpy: How do I copy FGDBR from multiple file geodatabases into one file geodatabase?

IngoWolff — Sat, 11 Dec 2021 20:06:42 GMT

Ok, it appears I found a solution!

Just run everything with ArcPy 2.7.xx instead of ArcPy3.6.xx... for the first time it occured to me, that ArcPy 3.6.xx cannot solve a task ArcPy 2.7.xx can solve.

Here the working script, tested on a good dozen of gdbs:

import arcpy
from arcpy import env
import os

arcpy.env.overwriteOutput = True

arcpy.env.parallelProcessingFactor = "75%"
#
#
#--------------------
#Setting up first directories for source and destination for iterative raster data loop no 1
dir = r"F:\Daten_Wolff\Topographische_Gefaehrdungsanalyse\Test_Source"
arcpy.env.workspace = dir
gdbList = arcpy.ListWorkspaces('*','FileGDB')
for gdb in gdbList:
    arcpy.env.workspace = gdb               #--change working directory to each GDB in list
    folder = r"F:\Daten_Wolff\Topographische_Gefaehrdungsanalyse\BlueSpot_50.gdb"+ "\\"
    RasterList = arcpy.ListRasters('*_100')
    for Raster in RasterList:
        print("processing raster: %s" %os.path.join("env.workspace",Raster))
        outRasterdataset = folder + arcpy.Describe(Raster).baseName
        arcpy.CopyRaster_management (Raster, outRasterdataset, "DEFAULTS", "", "-999", "NONE", "NONE" , "32_BIT_FLOAT", "NONE", "NONE" )
                
        

‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

This might explain, why the error message beforehand told me to go to Esri Support. Possibly a bug in ArcGISPro.

However, many thanks for your kind replies!

Ingo

Re: Arcpy: How do I copy FGDBR from multiple file geodatabases into one file geodatabase?

JoshuaBixby — Tue, 23 Jun 2020 15:02:54 GMT

How are you building the path to the data set before passing to Copy Raster? By the looks of it, you are passing only the data set name and not the data set full path. When using ArcPy Walk, just like when using Python Walk, you need to use os.path.join to get the full path from the root directory and file name.

Re: Arcpy: How do I copy FGDBR from multiple file geodatabases into one file geodatabase?

IngoWolff — Sat, 11 Dec 2021 20:06:45 GMT

Hi Joshua,

the scripts jumps first into the workspace directory with

dir = r"F:\Daten_Wolff\Topographische_Gefaehrdungsanalyse\Test_Source"
arcpy.env.workspace = dir‍‍

Then, inside the workspace directory, the list of gdbs is called by

gdbList = arcpy.ListWorkspaces('*','FileGDB')
for gdb in gdbList:
    arcpy.env.workspace = gdb               #--change working directory to each GDB in list‍‍‍

Now it goes from gdb to gdb.

For each gdb, I then apply the classical list of Raster loop, not befor telling the target directory as "folder":

folder = r"F:\Daten_Wolff\Topographische_Gefaehrdungsanalyse\BlueSpot_50.gdb"+ "\\"
    RasterList = arcpy.ListRasters('*_050')
    for Raster in RasterList:
        print("processing raster: %s" %os.path.join("env.workspace",Raster))
        outRasterdataset = folder + arcpy.Describe(Raster).baseName
        arcpy.CopyRaster_management (Raster, outRasterdataset, "DEFAULTS", "", "-999", "NONE", "NONE" , "32_BIT_FLOAT", "NONE", "NONE" )‍‍‍‍‍‍‍‍‍‍‍‍

This works in ArcPy 2.7.14 quite nicely so far.

I have exatly 5107 gdbs to be searched für fgdbr Raster with the name suffix '*_050' or '*_100' and so on. Not in all of thes gdbs are thes Raster with that exact suffices, but the script shecks them all! So far, from my 5107 gdbs the script copied nicely 4936 fgdbrs with the suffix '*_050' into my targed gdb. I directly added the same loops for more suffices, and its working calm and steadily on my VM.

All the best!

Ingo