topic Append and field mapping with Python in Python Questions

Append and field mapping with Python

Matt_Trebesch — Sat, 11 Dec 2021 11:02:19 GMT

I am learning to append and use fieldmap, flieldmappings to get my fields squared away.

basically I'm appending data into a "shell" featureclass that is empty. There's only a few fields I'm appending and trying to match.

I'm using this post as a reference:

Append Tool and Field Mapping ... Help? Examples?

I'm able to add ONE field change with this script:

(append and target layers are defined earlier in the script as well as importing the arpy library

# field map and append

append_layer = outputclip
target_layer = out_name


#This is like defining an empty grid of fields you see when you run it manually in the toolbox
fieldmappings = arcpy.FieldMappings()

#Add the target datasets fields to the field map table  (Like when you manually choose a tagert layer in the toolbox)
fieldmappings.addTable(target_layer)

#Add the append datasets fields to the field map table  (Like when you manually choose a tagert layer in the toolbox)
fieldmappings.addTable(append_layer)

#At this point, you have a grid like when you run it manually saved in your field mappings.




#####Lets map a field that have different names!


#Find which "Index" the field has as we cant refer to them by name when editing the data only index
field_to_map_index = fieldmappings.findFieldMapIndex("PropertyAddress")  #Field name that exists in the target layer but not append data source!

#Grab "A copy" of the field map object for this particular field
field_to_map = fieldmappings.getFieldMap(field_to_map_index)

#Update its data source to add the input from the the append layer
field_to_map.addInputField(append_layer, "Addy_Concat")



#####Lets update the master field map using this updated copy of a field

fieldmappings.replaceFieldMap(field_to_map_index, field_to_map)


#Create a list of append datasets and run the the tool
arcpy.Append_management(append_layer, target_layer, "NO_TEST", fieldmappings, "1")‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

So, this works for me. I end up appending the "Addy_Concat" field into the "PropertyAddress" field. Great.

But I have 3 other fields to do this with. So I used this:

# field matching for append
# Name the Append and Target Layer (target layer is named in the ArcGIS Pro Map)
append_layer = outputclip
target_layer = out_name

fieldmappings = arcpy.FieldMappings()

# Like when you manually choose a layer in the toolbox and it adds the fields to grid
fieldmappings.addTable(target_layer)
fieldmappings.addTable(append_layer)

# Lets map fields that have different names!
list_of_fields_we_will_map = []

# list_of_fields_we_will_map.append(( append layer field name, target layer field name)
list_of_fields_we_will_map.append(('Addy_Concat', 'PropertyAddress'))
#list_of_fields_we_will_map.append(('CMUNITY', 'PropertyCity'))
#list_of_fields_we_will_map.append(('ZipCode', 'PropertyZip'))

print(*list_of_fields_we_will_map, sep = ", ")


for field_map in list_of_fields_we_will_map:
    # Find the fields index by name.
    field_to_map_index = fieldmappings.findFieldMapIndex(field_map[0])
    # Grab "A copy" of the current field map object for this particular field
    field_to_map = fieldmappings.getFieldMap(field_to_map_index)
    # Update its data source to add the input from the the append layer
    field_to_map.addInputField(append_layer, field_map[1])
    # We edited a copy, update our data grid object with it
    fieldmappings.replaceFieldMap(field_to_map_index, field_to_map)

# Create a list of append datasets and run the the tool - the "1" at the end is for the subtype.
inData = [append_layer]
arcpy.Append_management(inData, target_layer, "NO_TEST", fieldmappings, "1")‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

I keep on getting this error:

Traceback (most recent call last):
File "<string>", line 119, in <module>
File "c:\program files\arcgis\pro\Resources\arcpy\arcpy\arcobjects\arcobjects.py", line 666, in addInputField
return convertArcObjectToPythonObject(self._arc_object.AddInputField(*gp_fixargs(args)))
RuntimeError: FieldMap: Error in adding input field to field map

Specifically, it's this line (119) that's causing issues:

field_to_map.addInputField(append_layer, field_map[1])‍

If you noticed, I remarked out my multi fields to let the list just take the one field match combo that worked in the simpler, non-list script.

Still throws this error.

I'm confused how this list loops works. Not sure how it's figuring out position 0,1 with the case like

('Addy_Concat', 'PropertyAddress')

Any ideas what I'm doing wrong here? Or what I can do to try accomplish what I want to do?

There must be an easier way to field match when appending.

Re: Append and field mapping with Python

RandyBurton — Sat, 11 Dec 2021 11:02:22 GMT

RuntimeError: FieldMap: Error in adding input field to field map

I suspect that the error is because the "append_layer" does not have a field named 'PropertyAddress' (the value of field_map[1]). It appears that the "addInputField" method is doing some sort of checking.

Since you are fieldmapping for the append tool, I would suggest that you use the target layer's mapping and use 'removeInputField' and replace its input fields with the appropriate input fields from the append layer. My suggested code (tested in desktop 10.5's Python window) is:

# function used for printing debug info
def tempMap(fm): # function takes a field map and returns it as a string
    tfms = arcpy.FieldMappings() # temporary field mappings object
    tfms.addFieldMap(fm) # add the field map
    return tfms.exportToString() # reurn string version

appendFC = 'fm_append'
targetFC = 'fm_target'

fmsAppend = arcpy.FieldMappings()
fmsAppend.addTable(appendFC)
fmsTarget = arcpy.FieldMappings()
fmsTarget.addTable(targetFC)

match = { 'PropertyAddress' : 'Addy_Concat',
          'PropertyCity' : 'CMUNITY',
          'PropertyZip' : 'ZipCode' }

for x in range(0, fmsTarget.fieldCount):
    fmTarget = fmsTarget.getFieldMap(x)
    # print(tempMap(fmTarget))

    if fmTarget.outputField.name in match.keys():
        # look for the paired name
        matchField = match[fmTarget.outputField.name]
    else:
        # look for a matching name
        matchField = fmTarget.outputField.name

    # try:
    fmAppend = fmsAppend.getFieldMap(fmsAppend.findFieldMapIndex(matchField))
    # print(tempMap(fmAppend))

    # code does not check if field types match; add here if required
    if fmAppend.outputField.type == fmTarget.outputField.type:
        print("OK:  Append field {} matches target field's type.".format(matchField))
    else:
        print("ERROR:  Append field {} does not match target field's type.".format(matchField))
        # abort code goes here
        sys.exit(1)
        
    # update input field 
    fmTarget.removeInputField(0)
    fmTarget.addInputField(appendFC,matchField)

    if fmAppend.outputField.length > fmTarget.outputField.length:
        print('WARNING:  Append field {} is too long; making adjustments.'.format(matchField))

        fmTarget.setStartTextPosition(0,0)
        fmTarget.setEndTextPosition(0,fmTarget.outputField.length-1)
        # print(tempMap(fmTarget))
   
    # replace updated field map
    fmsTarget.replaceFieldMap(x, fmTarget)
    
    # except:
    # no matching field code - perhaps to remove field from mappings
    # print('\n')

# print(fmsTarget.exportToString())

print("Appending.")
arcpy.Append_management(inputs=appendFC, # feature data to be added
                        target=targetFC, # feature where data will go
                        schema_type="NO_TEST",
                        field_mapping=fmsTarget, subtype="")

print("Done.")‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

I've commented out some print statements that were used for debugging. I assumed that the number of fields would match; but the situation may actually require a try/except block to remove unmatched fields.

Hope this helps.

Re: Append and field mapping with Python

RandyBurton — Sat, 11 Dec 2021 11:02:25 GMT

As an alternative to field mapping (and for future reference), look at Xander Bakker ‌'s code near the bottom of this thread: FieldMap and FieldMappings arrgh....

But since you are learning field mappings (as am I) here's some code that may assist in that process:

# field mapping test
import arcpy

def tempMap(fm): # function takes a field map and returns it as a string
    tfms = arcpy.FieldMappings() # temporary field mappings object
    tfms.addFieldMap(fm) # add the field map
    return tfms.exportToString() # reurn string version

# what is in a field map:
# name "aliasName" editable isNullable required length type scale precision ,mergeRule,"joinDelimiter",inputTableName,inputFieldName,startTextPosition,endTextPosition;

# create a test feature
arcpy.CreateFeatureclass_management(out_path = 'in_memory', out_name = 'test_feature', geometry_type = "POINT",
                                    template = "#", has_m = "DISABLED", has_z = "DISABLED",
                                    spatial_reference = arcpy.SpatialReference("WGS 1984 Web Mercator (auxiliary sphere)"))

# add some fields
arcpy.AddField_management(in_table = r'in_memory\test_feature', field_name = 'stringField',
                          field_type = "STRING", field_precision = "#", field_scale = "#",
                          field_length = 20, field_alias = "String Alias", field_is_nullable = "NULLABLE",
                          field_is_required = "NON_REQUIRED", field_domain = "#")

arcpy.AddField_management(in_table = r'in_memory\test_feature', field_name = 'doubleField',
                          field_type = "Double", field_precision = "#", field_scale = "#",
                          field_length = 0, field_alias = "Double Alias", field_is_nullable = "NULLABLE",
                          field_is_required = "NON_REQUIRED", field_domain = "#")

# create a field mapping object and display as text
fms = arcpy.FieldMappings()
fms.addTable(r'in_memory\test_feature')
print(fms.exportToString().replace(';',';\n'))
'''
stringField "String Alias" true true false 20 Text 0 0 ,First,#,in_memory\test_feature,stringField,-1,-1;
doubleField "Double Alias" true true false 0 Double 0 0 ,First,#,in_memory\test_feature,doubleField,-1,-1
'''

# examine stringField using a field map object
fm = fms.getFieldMap(fms.findFieldMapIndex('stringField'))
print(fm.outputField.name, fm.outputField.aliasName, fm.outputField.length, fm.outputField.type)
# (u'stringField', u'String Alias', 20, u'String')

print(tempMap(fm))
# stringField "String Alias" true true false 20 Text 0 0 ,First,#,in_memory\test_feature,stringField,-1,-1

# change some things
outfield = fm.outputField
outfield.name = 'hello'
outfield.aliasName = 'welcome'
outfield.length = '10'
fm.outputField = outfield
# see what we get
print(tempMap(fm))
# hello "welcome" true true false 10 Text 0 0 ,First,#,in_memory\test_feature,stringField,-1,-1

print("Toggle editable, is nullable, and required")

outfield.editable = not outfield.editable
fm.outputField = outfield
print(tempMap(fm))

outfield.isNullable = not outfield.isNullable
fm.outputField = outfield
print(tempMap(fm))

outfield.required  = not outfield.required 
fm.outputField = outfield
print(tempMap(fm))
'''
Toggle editable, is nullable, and required
hello "welcome" false true false 10 Text 0 0 ,First,#,in_memory\test_feature,stringField,-1,-1
hello "welcome" false false false 10 Text 0 0 ,First,#,in_memory\test_feature,stringField,-1,-1
hello "welcome" false false true 10 Text 0 0 ,First,#,in_memory\test_feature,stringField,-1,-1
'''

# change scale and precision for numeric field
# since this is just an experiment, ignore the fact that it is using a text field
outfield.scale = 5
outfield.precision = 6
fm.outputField = outfield
print(tempMap(fm))
# hello "welcome" false false true 10 Text 5 6 ,First,#,in_memory\test_feature,stringField,-1,-1

# examine merge rule and join delimiter
print(fm.mergeRule, fm.joinDelimiter)
# (u'First', u'')

# change merge and delimiter, check results
fm.mergeRule = 'Join'
fm.joinDelimiter = ';'
print(tempMap(fm))
# hello "welcome" false false true 10 Text 5 6 ,Join,";",in_memory\test_feature,stringField,-1,-1

print(fm.getStartTextPosition(0), fm.getEndTextPosition(0)) # only one item in fm, so index is 0
# (-1, -1)
fm.setStartTextPosition(0,1) # index, position
fm.setEndTextPosition(0,8)
# check changes
print(tempMap(fm))
# hello "welcome" false false true 10 Text 5 6 ,Join,";",in_memory\test_feature,stringField,1,8

# input table and field
print(fm.getInputTableName(0), fm.getInputFieldName(0)) # index
# (u'in_memory\\test_feature', u'stringField')

fms.replaceFieldMap(0, fm) # index, field map
print(fms.exportToString())
'''
hello "welcome" false false true 10 Text 5 6 ,Join,";",in_memory\test_feature,stringField,1,8;
doubleField "Double Alias" true true false 0 Double 0 0 ,First,#,in_memory\test_feature,doubleField,-1,-1
'''

# delete test feature from memory
arcpy.Delete_management(r'in_memory\test_feature')‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Code was developed using desktop 10.5's Python window. It shows how to access and modify some parts of the fieldmappings object. The Python window will provide some clues as to the properties and methods being used. Also, check the appropriate documentation.