topic Re: Model Builder not Appending Data Correctly ?! in ModelBuilder Questions

Model Builder not Appending Data Correctly ?!

timdunlevie — Tue, 10 Feb 2015 08:55:51 GMT

Hi guys,

I'm using ArcGIS 10.2...and I'm getting some frustrating results using Model Builder to append data to a Feature Class within a FGDB.

I have 6 datasets (a mix of shapefiles and FC in a FGDB)

The steps I perform in the model include:

1. add fields to these datasets

2. calculate fields

3. append to a "master" FC in a FGDB using schema type "NO_TEST" and field map the various fields from the input datasets to the Target.

for some reason the data is not appending correctly. The field mapping looks correct.

I've deleted features from the Target FC (empty FC) and tried appending over again, yet the same (wrong) fields map to the target fc.

is it holding this data in memory ???

Is MB always so buggy ?

many thanks,

Tim

Re: Model Builder not Appending Data Correctly ?!

PaulLohr — Tue, 17 Mar 2015 19:30:58 GMT

If you are still working on this, you might want to look for a different approach. Maybe try Copy Features then append that result into your master feature class. Maybe try Make Feature Layer then append. Things simply don't work right in modelbuilder and this is how I have been able to make things work.

Re: Model Builder not Appending Data Correctly ?!

timdunlevie — Tue, 17 Mar 2015 23:48:09 GMT

Hi Paul,

Thanks for the suggestion….I will give it a go.

Seems strange that Model Builder is very buggy.

Tim

Re: Model Builder not Appending Data Correctly ?!

DarrenWiens2 — Wed, 18 Mar 2015 00:36:51 GMT

Does it work as expected outside of ModelBuilder?

Re: Model Builder not Appending Data Correctly ?!

timdunlevie — Wed, 18 Mar 2015 00:41:33 GMT

Yep – tested this.

All I am doing is adding 6 shapefiles/FCs to MB.

Adding a few fields to each.

Updating the metadata…

Then appending to a blank FC….field map to the relevant fields.

I was just curious to see if anyone else have had a similar experience.

Thanks

Re: Model Builder not Appending Data Correctly ?!

DarrenWiens2 — Wed, 18 Mar 2015 00:46:26 GMT

It looks like plenty of other users are experiencing the same thing with field mappings in ModelBuilder. Working alternative seems to be to implement in Python, which may or may not be a significant time commitment for you.

Re: Model Builder not Appending Data Correctly ?!

PaulLohr — Wed, 18 Mar 2015 17:34:21 GMT

Tim,

I am also frustrated by the bugs in Modelbuilder. It has been around long enough that a lot of these problems could be corrected by now. It is rare that I build a Modelbuilder process and it simply works without research time and a call to tech support.

Re: Model Builder not Appending Data Correctly ?!

timdunlevie — Thu, 19 Mar 2015 00:46:00 GMT

Thanks Paul.

I might revert to a python script instead…as I know this works !!!

Re: Model Builder not Appending Data Correctly ?!

curtvprice — Sat, 02 May 2015 02:44:50 GMT

Field mappings are a little problematic in ModelBuilder because they are set up so they can be user-edited within the MB edit mode. If you change the inputs, those field maps don't always update unless the model is re-validated, and event then if you've done edits to the field map it tries to keep it around.

I have worked around this by using code to create field maps at runtime inside the Calculate Value tool. Not an easy or elegant solution.

Re: Model Builder not Appending Data Correctly ?!

timdunlevie — Mon, 04 May 2015 00:25:45 GMT

Thanks mate.

In the end I converted to a python script instead….MB was doing my head in.

Tim

Re: Model Builder not Appending Data Correctly ?!

JoeBryant2 — Wed, 05 Apr 2017 19:29:44 GMT

Curtis, I am having this same Field Mapping issue. Python is not an option for me currently because of the time commitment as Darren suggested.

Do you have a custom Calculate Value script that I could insert in my model to get the field mapping to update each time it runs?

Re: Model Builder not Appending Data Correctly ?!

curtvprice — Thu, 06 Apr 2017 02:08:05 GMT

Can you describe exactly what this field map is supposed to do?

The "poor man's" way of doing this is to create a template dataset with the fields you want, and then for the inputs add and calculate fields with the appropriate names. If you do this, and provide an empty field map as a parameter so one will be created at runtime, the Append tool with NO_TEST will just ignore the fields that do not match.

Re: Model Builder not Appending Data Correctly ?!

JoeBryant2 — Thu, 06 Apr 2017 15:47:50 GMT

Sure: The model I am running is joining customer account information tables the I have exported from our billing database to Parcel Points we get from the County, based on APN. I am using these points as "placeholders" for water meters (wServiceConnections) that will have their locations updated later using GPS. Most of our meters have already been located and created; these meters are the leftovers we have not been able to collect yet that need to be represented in the feature class for accurate hydraulic modelling.

We recently migrated to the Local Government Information Model, so the field names in our billing system don't exactly match the fields names for wServiceConnections in the LGIM, except for the extra fields from the billing database I have added to the schema. But, because these fields are the result of a JOIN, their names don't exactly match (outputs from my Joins create the field name with the source table name as a prefix). So the field mapping feature of the Append Tool could save me a lot of time as I go through each meter zone in our District and create the meters that have not been collected yet.

I had thought about the "Poor Man's" way; it will significantly increase the size of the model, which is already kind of large. If I have to, that's the way I will go.

PS: I got your field map parameter suggestion in a separate post - I will try that first. Thanks!

"failure-using-model-builder-iteration-to-append-many-shapefiles"

Re: Model Builder not Appending Data Correctly ?!

curtvprice — Thu, 06 Apr 2017 18:41:06 GMT

From what I'm reading from your description, it sounds like you will need to create a field map in a python function to do what you want. But do that you will need logic to create a map of old names to new name, for example will prefix_table_name_SERVED always map to NUMBER_SERVED no matter what the prefix table name?

Re: Model Builder not Appending Data Correctly ?!

JoeBryant2 — Thu, 06 Apr 2017 19:01:25 GMT

Yes. For example: FeatureClassName_JoinedTableName_Full_Addr always field maps to ADDRESS in wServiceConnections. As I run the model on different zones in our district, the FeatureClassName changes (I create a selected subset of Parcel Points in a local file geodatabase), and JoinedTableName changes as I export separate tables of customer meter information from the billing system for each zone. But "..._Full_Addr" is always the name at the end of the field that will map to ADDRESS.

I am attempting your Field Map parameter suggestion, but it is currently adding a lot of manual work to each run of the model, particularly because there are a lot of fields and I can't see the full field name in the migration mapping dialog - I have to expand the window and scroll through the list for each selection.

The Poor Man's approach is looking more attractive - rename all the fields once in the model and Append using TEST. I had just hoped that Model Builder was up to the task of what seems like a very common workflow: join a table to a collections of points, select out features that have already been created, and load the new features into the master feature class, while create tables for the unmatched rows and points left over.

Re: Model Builder not Appending Data Correctly ?!

curtvprice — Sun, 12 Dec 2021 09:37:18 GMT

Here's a shot at it. No guarantees, but this is a way to use python to create field maps on the fly. You'd use the Calculate Value expression: make_maps(r"%Input features%"), paste the code below into the code block parameter, and set the output data type to Field Map so you could connect it as a parameter to the Append tool. You'd also have to hard code your from-name and to to-names in the code block into the dictionary I called renames.

Your mileage may vary, but this an approach to consider. You may be stuck with the "poor man method", but that would sure be a lot extra work.

def FieldNameMap(tbl, maps):
    """Easy field mappings generator

    arguments

      tbl - input feature class, table, or table view
      maps - field names map list (';'-delimited string of field name pairs)

    example

      import arcpy
      Maps = "Shape_Area AREA;BID BID2;AREASQMI AREAMI2"
      Mapper = FieldNameMap("temp.dbf", Maps)
      # to debug:
      print Mapper.exportToString().replace(";","\n")
      # Output will only have fields: AREA, BID2, AREAMI2
      # (plus ObjectID and Shape fields, if a feature class)
      arcpy.Merge_management("temp.dbf", "temp2.dbf", Mapper)

    author

    Curtis Price, U.S. Geological Survey, cprice@usgs.gov
    Not reviewed/approved use at your own risk
    """

    field_mappings = arcpy.FieldMappings()
    mapList = maps.split(';')

    for rec in mapList:
        fromName,toName = rec.split()
        # create a new field map
        field_map = arcpy.FieldMap()
        # populate it and add to field_mappings
        try:
            field_map.addInputField(tbl, fromName)
            field = field_map.outputField
            field.name = toName
            field_map.outputField = field
            field_mappings.addFieldMap(field_map)
        except:
            raise Exception(
                "FieldNameMap: Cannot not map fields ({}) in {}".format(rec,tbl))
    return field_mappings


# set up dictionary of from and to names
# first entry is db name, second is output name
renames = {"DRAINS":"DRAINS", 
           "PERSONS":"PEOPLE"}
def make_maps(tbl):
    fnames = [f.name.upper() for f in arcpy.ListFields(tbl)]
    # create field pairs list
    maps = [] 
    for f in fnames:
        for r in renames:
            if f.find(r) > -1: 
                maps.append("{} {}".format(f,renames[f]))
                break # done for this field
    maps = ";".join(maps)
    return FieldNameMap(tbl, maps).exportToString  # pass back to MB as string
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Model Builder not Appending Data Correctly ?!

JoeBryant2 — Fri, 07 Apr 2017 21:11:08 GMT

Curtis, thanks. That's awesome that you were able to mock that up and share it with me and everyone who may stumble across this post.

I am going to return to this script the next time I need to model an Append and I have a little more time to get familiar with Python.

I was able to get the Poor Man's version setup (20 Alter Field operations plus some field calculating) and it works with the Append Tool when you set the field map as a parameter and leave it blank as you mentioned. I'm about to put it through it's paces, so we'll see if it holds up.

Re: Model Builder not Appending Data Correctly ?!

CaryLancaster — Thu, 02 Aug 2018 23:11:55 GMT

100% agree with you Paul. it is 2018 and still the bugs are there.

Re: Model Builder not Appending Data Correctly ?!

curtvprice — Sun, 05 Aug 2018 04:53:41 GMT

I think it's important not to ask too much more of Model Builder than the environment is designed to support. If your workflow is complex (as in this example, where field mappings must be modified on each run based on the inputs), you may have to go to Python scripting. Even then, prototyping the process in ModelBuilder can save you a lot of time if the workflow and tools are complicated.