topic Re: Combining File GDBs with different schema in Python Questions

Combining File GDBs with different schema

ZOURMBAZOA_CINQ — Mon, 28 Dec 2015 12:38:33 GMT

Hi all,

I started to develop in Python for GIS since three months and It's really great! I have to combine three versions of File GDBs into one, but the main problem is that the schema has advanced from GDB1 to GDB2: some fields name have changed and in addition, there is a raster field type column in the tables. I could have simply load the two precedent versions into even a dictionary than using arcpy.da module inserting them in the final version, using corresponding columns and and filling empty columns with null, but there is no way to download and upload raster field. I found a way to download, but not for uploading it. Can somebody help?

Re: Combining File GDBs with different schema

DarrenWiens2 — Mon, 28 Dec 2015 18:20:30 GMT

I'm not sure about the raster field type issue, but for handling different schemas through arcpy, look into the FieldMap and FieldMappings objects (in particular, see the examples on these pages to see how to use with the Merge tool).

Re: Combining File GDBs with different schema

ZOURMBAZOA_CINQ — Tue, 29 Dec 2015 06:00:45 GMT

Hi,

Thank You for the info; it's a good beginning. Let me try some stuffs and will return soon.

Once more, Thanks!

Re: Combining File GDBs with different schema

ZOURMBAZOA_CINQ — Wed, 30 Dec 2015 11:45:52 GMT

Hi Darren,

I advanced with your help but as you said, I found difficulties with my raster field and also shape field. Seem like it's not supported.

I was thinking about a loop through the arcpy.da with a searchcursor on the input table and an updatecursor on the output table, there, i could manage shape fields, but there is not a way to upload a raster to a raster field type with arcpy.

There is one way to handle BLOB fields but I'm I not loosing something by migrating from raster to BLOB type?

Re: Combining File GDBs with different schema

JamesCrandall — Wed, 30 Dec 2015 13:44:59 GMT

It says it right in the documentation on the arcpy.da.UpdateCursor ArcGIS Help 10.1

Raster fields are not supported.

Is there any reason why you are not using Attachments? The other alternative I can come up with is to store paths to images on a file server, but that's going to require a lot of code to manage things and seems dangerous to lose the image-to-feature relationships if you manage this entirely outside of the database.

Re: Combining File GDBs with different schema

JamesCrandall — Sat, 11 Dec 2021 13:08:15 GMT

This workflow might satisfy your requirement, although I'd further research the continued use of the Raster column and perhaps determine if the Attachments model is a more appropriate option for the final repository.

Your first task is to retrieve the contents of the Raster column from the source Feature Class. If you will ultimately want to populate the destination with Attachments then you will need to establish a unique value for each feature and name the output image with that value. Here I have an "Imgs" column representing the Raster column and "DBKEY" column that is simple a text type that has 5 character values unique to each feature.

def writeRasFldContents():
    #set the env workspace, otherwise it will write results to C:\users... dir
    arcpy.env.workspace = r'H:\saveFolder'
    arcpy.env.overwriteOutput = True
    #specify an output folder to write out the Raster column images
    output_path = r'H:\saveFolder'
    os.chdir(output_path)
    
    #input FeatureClass with the Raster column you wish to write out to a blob field
    fc = r'H:\Documents\ArcGIS\Default.gdb\XY01TEST'
    
    #write the images from the raster column, skipping any nulls or else it will error
    with arcpy.da.SearchCursor(fc, ("Imgs", "DBKEY"), skip_nulls=True) as cursor:
        for row in cursor:
            filename = "{0}.tif".format(row[1])
            row[0].save(filename)
            print filename
            del row
            del filename

Now that we have a file folder of output images in a folder and each relates back to each feature, you can now go thru the process to setup Attachments on the SDE feature class (just use the Geoprocessing tools to perform this).

if you simply want to add a blob field onto the source feature class and populate it with the Raster column, these two additional def's will accomplish that:

def writeBlobs():

    fc = r'H:\Documents\ArcGIS\Default.gdb\XY01TEST'
    input_path = r'H:\saveFolder'
    full_file_paths = get_filepaths(input_path)
    for f in full_file_paths:
        if f.endswith(".tif"):
            imgname = os.path.basename(f)
            dbkey = os.path.splitext(imgname)[0]
            expr = "DBKEY= '" + dbkey + "'"
            print expr
            myfile = open(f, 'rb').read()
            #Imgs2 is the new blob column to populate the imgs
            with arcpy.da.UpdateCursor(fc, ['Imgs2'], expr) as ucur:
                for urow in ucur:
                    urow[0] = myfile
                    ucur.updateRow(urow)
                    
def get_filepaths(directory):
    
    file_paths = []  # List which will store all of the full filepaths.
    # Walk the tree.
    for root, directories, files in os.walk(directory):
        for filename in files:
            # Join the two strings in order to form the full filepath.
            filepath = os.path.join(root, filename)
            file_paths.append(filepath)  # Add it to the list.
    return file_paths

Finally if you want to read the blobs to validate:

def readBlobs():
    #to check if the blob column was updated with the content of the raster column
    fc = r'H:\Documents\ArcGIS\Default.gdb\XY01TEST'
    with arcpy.da.SearchCursor(fc, ['Imgs2', 'DBKEY'], skip_nulls=True) as cursor:
        for row in cursor:
            binaryRep = row[0]
            fileName = row[1]
            print fileName
            #save the content of the raster field to a new folder
            open(r'H:\saveFolder2' + os.sep + fileName, 'wb').write(binaryRep.tobytes())
            del row
            del binaryRep
            del fileName

Re: Combining File GDBs with different schema

ZOURMBAZOA_CINQ — Wed, 30 Dec 2015 17:10:50 GMT

Hi James,

Thank you for your answer. The fact is that there are already File geodatabases with data inside. so i have to deal with it. The other reason for the output geodatabase to not use attachment is that further, the data should be inserted in an enterprise geodatabase (SQL Server spatially enabled). And as it uses and manages BLOB, then it's better for us to gather that information right now ready for the upload as ArcMap directly upload that format in Enterprise Geodatabases.

That's the why!

But once more Thank You!

Re: Combining File GDBs with different schema

JamesCrandall — Wed, 30 Dec 2015 18:27:59 GMT

The fact is that there are already File geodatabases with data inside

It makes perfect sense to employ the Attachments model on File Geodatabases, it's going to allow you to have many related attachments to any 1 feature. To replicate this with your own method will require some significant investment in development.

The other reason for the output geodatabase to not use attachment is that further, the data should be inserted in an enterprise geodatabase (SQL Server spatially enabled).

Do you mean ArcSDE registered within the SQL Server? If so, then again, the Attachment model is perfectly well suited for this environment. I'm not so sure why you think it isn't. I could see a storage limitations might be a valid concern, but that's unrelated to the fact that setting up Attachments on a SDE instance in SQL Server can and does perform well.