topic Re: Append Tool and Field Mapping ... Help? Examples? in Python Questions

Append Tool and Field Mapping ... Help? Examples?

Brownschuh — Thu, 03 Nov 2016 15:19:28 GMT

Having a bit of trouble with the field mapping component of the append tool; I cannot seem to get it to work properly as I am left with a feature class that is populated with all the features, but the attribute table remains blank.

Background info: I have two feature classes in an enterprise geodatabase with address point data. One is the authoritative source edited by our address coordinator and the other is used as to host a map service. The data is essentially the same, but the schemas are slightly different; half of the fields are the exactly the same, but others have different names. My workflow has been to update the 'map service' feature class on a daily basis by deleting all the records and loading the data manually, selecting the NO_TEST option, and manually mapping the fields where the names are different ... but this is where I run into trouble with my script.

I've ran the append process manually using the geoprocessing tool and copied the python snippet of the field mapping component and am left with an output attribute table that has all null values. Ideally I'd like to move away from this style of coding as I am aware it can be rather clunky/lengthy ... and it just does not work (even though it throws no errors).

I'd like to do 'real' field mapping, but so far the examples I've found have a bit more complicated or focus mainly on the Merge tool. I'd like to see a nice, simple Append tool example if possible.

Also, how does field mapping work if some fields are the same and others are different? I haven't run across a good example that deals with this problem.

Here are some of the sources I've looked at so far:

Field mapping and Python scripting | ArcGIS Blog

Mapping input fields to output fields—ArcPy Get Started | ArcGIS for Desktop

Re: Append Tool and Field Mapping ... Help? Examples?

MitchHolley1 — Thu, 03 Nov 2016 15:24:41 GMT

I would try to build a model in ModelBuilder first, then export it to a python script.

Re: Append Tool and Field Mapping ... Help? Examples?

Brownschuh — Thu, 03 Nov 2016 15:27:14 GMT

Ah yes, I should have noted that I did attempt this but ended up with the same result ... an attribute table with all null values.

Re: Append Tool and Field Mapping ... Help? Examples?

MitchHolley1 — Thu, 03 Nov 2016 15:42:35 GMT

In my minimal experience with using the Append tool in Python, there really is no simple way to do it. I've given up on field mappings, because it's so clunky and confusing. Sorry, but I can't help you anymore

Re: Append Tool and Field Mapping ... Help? Examples?

LukeWebb — Sat, 11 Dec 2021 19:13:43 GMT

import arcpy

append_layer = r'C:\a_layer.shp'
target_layer = r'C:\target_layer.shp'


#This is like defining an empty grid of fields you see when you run it manually in the toolbox
fieldmappings = arcpy.FieldMappings()

#Add the target datasets fields to the field map table  (Like when you manually choose a tagert layer in the toolbox)
fieldmappings.addTable(target_layer)

#Add the append datasets fields to the field map table  (Like when you manually choose a tagert layer in the toolbox)
fieldmappings.addTable(append_layer)

#At this point, you have a grid like when you run it manually saved in your field mappings.




#####Lets map a field that have different names!


#Find which "Index" the field has as we cant refer to them by name when editing the data only index
field_to_map_index = fieldmappings.findFieldMapIndex("Field_name_in_target")  #Field name that exists in the target layer but not append data source!

#Grab "A copy" of the field map object for this particular field
field_to_map = fieldmappings.getFieldMap(field_to_map_index)

#Update its data source to add the input from the the append layer
field_to_map.addInputField(append_layer, "Field_name_in_append")


#####Lets update the master field map using this updated copy of a field

fieldmappings.replaceFieldMap(field_to_map_index, field_to_map)


#Create a list of append datasets and run the the tool
inData = [append_layer]
arcpy.Append_management(inData, target_layer, field_mappings=fieldmappings)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

I havent tested this code, but I think it should work

Re: Append Tool and Field Mapping ... Help? Examples?

Brownschuh — Thu, 03 Nov 2016 18:17:24 GMT

Excellent, tried this code and it worked for my test script. However, I was only able to get it to work for one instance of a mismatched field ("TaxPIN" vs "Permanent_Identifier"). Not sure how to tweak the code to map the other mismatched fields; is this done findFieldMapIndex component?

I should also note that all the matching fields populated just fine; pretty happy about that.

Re: Append Tool and Field Mapping ... Help? Examples?

LukeWebb — Sat, 11 Dec 2021 19:13:46 GMT

__author__ = 'WebbL'

import arcpy

append_layer = r'C:\a_layer.shp'
target_layer = r'C:\target_layer.shp'


#This object looks like the empty grid of fields 
#you see when you first open the append tool in the toolbox
fieldmappings = arcpy.FieldMappings()

#Like when you manually choose a layer in the toolbox and it adds the fields to grid
fieldmappings.addTable(target_layer)
fieldmappings.addTable(append_layer)

#####Lets map fields that have different names!
list_of_fields_we_will_map = []
#Lets chuck some tuples into the list we made
list_of_fields_we_will_map.append(('TaxPIN', 'Permanent_Identifier'))
list_of_fields_we_will_map.append(('SwissBankNo', 'RetirementFundAccNo'))
list_of_fields_we_will_map.append(('Balance', 'Amount'))

for field_map in list_of_fields_we_will_map:
    #Find the fields index by name. e.g 'TaxPin'
    field_to_map_index = fieldmappings.findFieldMapIndex(field_map[0])  
    #Grab "A copy" of the current field map object for this particular field
    field_to_map = fieldmappings.getFieldMap(field_to_map_index)
    #Update its data source to add the input from the the append layer
    field_to_map.addInputField(append_layer, field_map[1])
    #We edited a copy, update our data grid object with it
    fieldmappings.replaceFieldMap(field_to_map_index, field_to_map)

#Create a list of append datasets and run the the tool
inData = [append_layer]
arcpy.Append_management(inData, target_layer, field_mappings=fieldmappings)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Here you go

Re: Append Tool and Field Mapping ... Help? Examples?

LukeWebb — Sat, 11 Dec 2021 19:13:49 GMT

My understanding of field mappings is you create an empty fieldmap.

fieldmappings = arcpy.FieldMappings()

Then you popupulate it using tables like we did, or alternatively you could do it totally manually as the example on the help page shows. There they create an output field mapping that has 2 fields. It looks pretty complex but heres a simple, 1 field output map!

input_file1 = 'data.gdb/Trees'
input_file2 = 'Plants.shp'

######You could loop this bit
field_map = arcpy.FieldMap()
field_name_input_file1 = "Tree_Type"
field_name_input_file2 = "Tree_Species"
field_map.addInputField(input_file1, field_name_input_file1)
field_map.addInputField(input_file2, field_name_input_file2)
field_map.name = 'Tree_Classification'
fieldmappings.addFieldMap(field_map)

With this in mind, our output fieldmap we have created, actually contains all of the 'target' and all of the 'append' layers fields, but as you are running append you dont see all the unmatched 'append' fields in the output. If you ran the same field map through a merge, you would end up with all the append layers fields as well.

Re: Append Tool and Field Mapping ... Help? Examples?

Brownschuh — Sun, 12 Dec 2021 16:27:37 GMT

After a bit of trial and error, here's the approach I took:

   field_to_map_index1 = fieldmappings.findFieldMapIndex("Permanent_Identifier")
   field_to_map_index2 = fieldmappings.findFieldMapIndex("DateAdded")

   field_to_map1 = fieldmappings.getFieldMap(field_to_map_index1)
   field_to_map2 = fieldmappings.getFieldMap(field_to_map_index2)

   field_to_map1.addInputField(Address_Points, "TaxPIN")
   field_to_map2.addInputField(Address_Points, "created_date")

Looks like it's similar to what you posted above.

"With this in mind, our output fieldmap we have created, actually contains all of the 'target' and all of the 'append' layers fields, but as you are running append you dont see all the unmatched 'append' fields in the output. If you ran the same field map through a merge, you would end up with all the append layers fields as well."

/\ This is good to know, appreciate the clarification. And thank you for the help!!

Re: Append Tool and Field Mapping ... Help? Examples?

deleted-user-zpcJw1u0IXiO — Tue, 19 Dec 2017 19:01:44 GMT

Thank you, Luke, for this. It works great, but for anyone who gets an error on the last line, try changing it to:

arcpy.Append_management(inData, target_layer, "NO_TEST", fieldmappings)

Re: Append Tool and Field Mapping ... Help? Examples?

ChrisHolmes — Wed, 25 Jul 2018 04:27:54 GMT

Bringing this thread back to life.

Is it possible to accomplish the following using the field mappings. I would like values for two of the fields in the target dataset to come from parameters entered by the user when the script is run? If so, does anyone have a snippet of code that would show how to work the parameter values into the fields mappings.

Thanks,

Chris

Re: Append Tool and Field Mapping ... Help? Examples?

NeilAyres — Wed, 25 Jul 2018 07:47:57 GMT

Field maps are about the definitions of the columns, names, alias, type etc and automatically point the content of one column to another.

If you want to alter content you will have to use a field calculator or a da.UpdateCursor

Re: Append Tool and Field Mapping ... Help? Examples?

ChrisHolmes — Wed, 25 Jul 2018 14:49:48 GMT

Thanks for clearing that up Neil. I was working on a script with an update cursor to use after the append tool but what I’m noticing is that after the append tool finishes the target shape is not selected so I’ll have to figure out either a way to select it or a way to select it using a search cursor prior to doing the update. Thanks!

Re: Append Tool and Field Mapping ... Help? Examples?

NeilAyres — Fri, 27 Jul 2018 11:18:47 GMT

Or just run through the UpdateCursor and update only where your search column equals the value you are looking for.

Re: Append Tool and Field Mapping ... Help? Examples?

ModernElectric — Wed, 01 Aug 2018 19:27:40 GMT

This is a great script that initial testing has been successful. However, I am having another problem.

A little background first... I have a WORKING GDB on my local drive that is used to run analysis and edits. We have a separate PRODUCTION GDB that is on our network that is used to "run" the .PMF on ArcReader. I successfully built an ArcGIS Model to Truncate and Append data from the WORKING GDB to the PRODUCTION GDB without requiring ArcReader users from exiting out. It has been my goal to build a group of Python scripts to do this automatically with the help Windows Task Scheduler. Here is the problem. I have a few Feature Class(s) that has attachment(s) and the only way I have found the ArcGIS Model to work successfully is by opening the .MXD for the PRODUCTION GDB and run the ArcGIS Model. All of the attachments append/copy over without a problem. Now, doing this with the Field Mapping Python script isn't working properly. Any advice to duplicate the process I have been doing, but with the Field Mapping script with attachments?

Thanks

Again, GREAT script!!

Re: Append Tool and Field Mapping ... Help? Examples?

ShawnRoberts1 — Sat, 11 Dec 2021 19:13:51 GMT

I found the syntax in the examples very confusing as well! The above example provided by Luke got my code off the ground, so I figured I'd post my little solution. Maybe it will be of help to someone as well.

Essentially I just call this function before I run the tool requiring the field map (append,merge ect).The dictionary is contains the "in"( and "out values", then it just iterates through each item in the dictionary and assigns the field map values.

Only note I'm using python 3.X so .iteritems() isn't available.

def fieldMapping(fc_in,fc_out):
    try:
        # value in the 'in' feature class, value in the feature class to be appended to
        dico = {'Assessed_Value': 'ARN', 'Total_Value': 'TotalValue'}

        fieldmappings = arcpy.FieldMappings()
        fieldmappings.addTable(fc_in)
        fieldmappings.addTable(fc_out)

        for input,output in dico.items():
            field_to_map_index = fieldmappings.findFieldMapIndex(output)
            field_to_map = fieldmappings.getFieldMap(field_to_map_index)
            field_to_map.addInputField(fc_in,input)
            fieldmappings.replaceFieldMap(field_to_map_index, field_to_map)

        return fieldmappings

    except:
        logging.exception('Generic error raised during fieldmap creation')
        raise

Re: Append Tool and Field Mapping ... Help? Examples?

TonyAlmeida — Sat, 11 Dec 2021 19:13:54 GMT

Here is what i use, i found this in a thread in this forum but i can't find the link. I had a hard time understanding field mapping until i cam across this code. Since i can't find the thread i can't explain all the syntax. Maybe someone can shine in and explain the syntax please?

UG1A = "C:/Temp/Scratchworkspace.gdb/lyr1"
IRRG = "D:/GIS Folder/lyr2"

def Layers2(FZon1,UG1A,IRRG):
    FieldMapString = "" \
                
                + """PIN "PIN" true true false 13 Text 0 0 ,First,#,""" + UG1A + """,Pin,-1,-1;"""\
                + """ACRES "ACRES" true true false 4 Double 0 0  ,First,#,""" + UG1A + """,ACRES,-1,-1;"""\
                + """Instrument "Instrument" true true false 10 Text 0 0 ,First,#,""" + UG1A + """,Instrument,-1,-1;"""\
                + """SiteAddres "SiteAddres" true true false 106 Text 0 0 ,First,#,""" + UG1A + """,SiteAddres,-1,-1;"""\
                + """SiteCity "SiteCity" true true false 32 Text 0 0 ,First,#,""" + UG1A+ """,SiteCity,-1,-1;"""\
                + """SiteZip "SiteZip" true true false 10 Text 0 0 ,First,#,""" + UG1A + """,SiteZip,-1,-1;"""\
                + """UG_NAME "UG_NAME" true false false 20 Text 0 0 ,First,#,"""+ UG1A + """,UG_NAME,-1,-1;"""\
                + """IRRIG_DIST "IRRIG_DIST" true false false 60 Text 0 0 ,First,#,"""+ IRRG + """,NAME,-1,-1;"""\


    fieldmappings = arcpy.FieldMappings()
    fieldmappings.loadFromString(FieldMapString)
    return fieldmappings

def main(args=None):
        if args is None:
                args = sys.argv
                
arcpy.SpatialJoin_analysis(UG1A, IRRG, "C:\Temp\Scratchworkspace.gdb\lyr3",
                           "JOIN_ONE_TO_ONE", "KEEP_ALL", Layers2(FZon1,UG1A,IRRG), "INTERSECT")‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Append Tool and Field Mapping ... Help? Examples?

AngelaBlanco — Sun, 03 Mar 2019 15:52:11 GMT

I tried to create a pyhton script using FieldMapping following the examples on https://pro.arcgis.com/en/pro-app/arcpy/classes/fieldmappings.htm.

When I use exportToString () for the FieldMapping in the python console I got different syntax than from Python Snippet

Python console

" FREQUENCY " true true false 4 Long 0 0 ,First,#,D:\Processed_Data\03_Geodatabase\07_20170226_Afternoon.gdb\Afternoon_20170226_Average,FREQUENCY,-1,-1,

Python Snippet

FREQUENCY "FREQUENCY" true true false 4 Long 0 0 ,First,#, Afternoon_20170226_Average,-1,-1;

I don’t understand why one must set properties of the output name. https://preply.com/en/question/can-anyone-give-me-some-examples-please-of-the-correct-way-to-use-usted-34777

Can anyone give more examples, please?

Re: Append Tool and Field Mapping ... Help? Examples?

DylanHarwell — Sat, 11 Dec 2021 19:13:57 GMT

I just ran into a situation where I needed to figure out field mapping with Python, and my situation is slightly different than the examples posted here so I thought I'd share my solution.

The Problem: We have a web map that has what we call 'compiled layers'. These are layers built from features joined to SQL Server views, which can be incredibly slow and impractical on a web map. I built a python module to reuse in scripts that run every night and perform the join, delete and append to target feature classes in our database. Originally these were set up with models, but now we are going to create a lot more and that is the reason behind scripting this. Whenever these models were first created, the target feature class only contained a subset of fields from the joined tables - so field mapping came into play.

The examples above assume that you know what fields you are mapping, or that you already have a field map string. The method I came up with is generic enough to create a field map with no knowledge of the append fields or target fields. And since we are an organization, some trickery is involved to match the fields created after the join since they aren't just a field name but are something like <DB name>.DBO.<feature name>.

For info on what the EmailService module is and how to get email notifications when scripts fail, check this out.

#-------------------------------------------------------------------------------
# Name:        CompiledLayers.py
#
# Author:      Dylan Harwell - GIS Tech - City of GJ
#
# Created:     25/04/2019
# Copyright:   (c) dylanh 2019
#
#-------------------------------------------------------------------------------
import arcpy
arcpy.env.overwriteOutput = True


class CompiledLayers(object):
    """Custom class to create compiled layers in lieu of joined layers on
    web maps."""
    def __init__(self):
        self.ignore = ['OBJECTID', 'Shape', 'Shape.STArea()',
                       'Shape.STLength()', 'STArea()', 'STLength()',
                       'GlobalID', 'ESRI_OID', 'SHAPE' ]

    def createCompiledLayer(self, join_fc, in_field, join_table,
                            join_field, join_type, target_fc):
        """Makes feature layer, joins to sql server table, sets up fieldmapping,
        deletes and appends target feature class in gjgisweb.sde."""
        arcpy.MakeFeatureLayer_management(join_fc, 'temp_lyr')
        arcpy.AddJoin_management('temp_lyr', in_field, join_table, join_field,
                                 join_type)

        target_fields = [f for f in arcpy.ListFields(target_fc)
                         if f.name not in self.ignore]
        append_fields = [f for f in arcpy.ListFields('temp_lyr')
                         if f.name.split('.')[-1] not in self.ignore]

        fields_to_map = []
        for a in append_fields:
            for t in target_fields:
                if a.name.split('.')[-1].lower() == t.name.lower():
                    fields_to_map.append((a.name, t.name))

        fieldmapping = arcpy.FieldMappings()
        for fields in fields_to_map:
            field_map = arcpy.FieldMap()
            field_map.addInputField('temp_lyr', fields[0])
            field_map.addInputField(target_fc, fields[1])
            fieldmapping.addFieldMap(field_map)

        arcpy.DeleteFeatures_management(target_fc)
        arcpy.Append_management('temp_lyr', target_fc, 'NO_TEST', fieldmapping)


def main():
    pass
if __name__ == '__main__':
    main()


################### Example of using this module in a script ########################
import CompiledLayers as CL
import EmailService as ES

def getTime():
    """Function to fetch current date and time."""
    date_time = time.strftime("%Y%m%d") + "_" + time.strftime("%H%M%S")
    return date_time

# Set path to log file
log = 'T:...\\Log\\CompileNorthstarTrashParcels' + getTime() + '.txt'

# Set variables
join_fc    = 'T:...\\dbConnections\\gjgisprod.sde\\gjgisprod.DBO.Parcel'
in_field   = 'PARCEL_NUM'
join_table = 'T:...\\dbConnections\\ns.sde\\Northstar64_Live.dbo.gj_trashdata'
join_field = 'PARCEL'
join_type  = 'KEEP_COMMON'
target_fc  = 'T:...\\dbConnections\\gjgisweb.sde\\gjgisweb.DBO.CompiledLayers\\gjgisweb.DBO.northstarTrashParcels'

with open(log, 'w') as log_file:
    try:
        log_file.write('Calling CompiledLayers module at ' + getTime() + '\n')
        compiler = CL.CompiledLayers()
        compiler.createCompiledLayer(join_fc, in_field, join_table, join_field,
                                     join_type, target_fc)
        log_file.write('Succeeded creating compiled layer at ' + getTime())

    except Exception as e:
        log_file.write('Failed at ' + getTime() + '\n')
        log_file.write(e.message)

        # Send email to server admin if script fails
        service = ES.EmailService()
        subject = 'Server Script Failure!'
        body = 'CompiledNorthstarTrashParcels.py failed'
        recip = 'jacksont@gjcity.org'

        service.configMessage(subject, body)
        service.sendEmail(recip)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Append Tool and Field Mapping ... Help? Examples?

AllenDailey1 — Thu, 17 Sep 2020 19:57:31 GMT

I have a similar situation; I used Model Builder to export a python file. I am wondering what do the "true true false" etc. in the field map syntax mean or represent?

Did you ever find out?

Thanks!

Allen