Failure using model builder iteration to append many shapefiles

[JoeCook3]

I have a very simple model set up with the goal of mining many shapefiles from nested folders and appending them into one shapefile. I am using the Iterate Feature Classes Iterator (recursive) to parse through the folders and feed the shapefiles into the Append tool. There are some columns in some of the shapefiles that are not used/do not match up so I'm using the "no test" option in Append to auto match the columns that can be matched. Using this setup I am getting all the shapefiles appended onto my target shapefile and the points plot on the map in the correct location but all the appended values are zero.

If I run the Append tool by itself using the same parameters for a handful of shapefiles it works fine. There is something I am missing between setting up the iterator and transferring the field values into the target shapefile. Is there a different iterator I should be using? I have tried iterate files and a couple other methods but nothing will let me run the append tool successfully other than Iterate Feature Classes.

I plan on using model builder for many upcoming tasks so I'd like to put in the time to understand how it works. Any help is appreciated.

[DuncanHornby]

Are you saying when you use the append tool by itself then all the attributes are passed over rather than getting set to zero? May be it's just coincident that you have selected a bunch of shape files that have the same fields and it works?

[JoeCook3]

When I use the append tool without the iterator fields are populated correctly even though the shapefiles have a different assortment of columns.

Add the iteration which mines the shapefiles from many folders and feeds them into the append tool and I get rows appended with zero values all the way across. The shapefile all the others are appended into displays the points correctly in space but without any of the attributes for each point.

I'll see about posting a screenshot of the model and the parameters I'm using if that helps. I could also post a couple of the shapefiles with different columns if anyone thinks that would help.

Thx,
Joe

[JoeCook3]

I've done a bit more experimenting with mixed results. Using Feature Class Iteration with a folder containing only one subfolder with a shapefile in it successfully appends to my target shapefile using the model I've built. Attributes are included where they should be. Once I add a second shapefile to the mix (withing the same subfolder or another) it is appended but with no information in the attribute table. The point is displayed in the target shapefile but with no information.

The wildcard and recursive processes within the iteration appear to be working just fine, but I lose something or have set something set up wrong with the append tool. See the screenshot below for an example of the simple model. If you see something fishy or superfluous please let me know. I am not using inline substitution and that "test_append.shp (2)" bubble appears when I attach the iterator to the append tool, I'm not 100% on what is going on there.

121708mkm is the folder I am pointing the iteration to, there is a subfolder within that contains the shapefiles I want to append to "Test_append.shp"

Point_ge.shp is the name of the shapefile pulled from the folder. The other shapefiles I want to append are also named Point_ge.shp but are located in other folders, could the same name be causing my issue?

EDIT: The name does not appear to be the issue, it's every shapefile appended after the first one that appends with zero values. Something gets messed up after the first append

[JoeCook3]

Sorry to keep replying to my own thread, but if I run the same model with the "Test" option in the Append tool selected everything works. This is good but I will be unable to append older shapefiles that use a different arrangement of columns using modelbuilder (this results in zero values for all appends after the first). It only works if the columns match perfectly. I'm sure someone out there can figure out how to make it work the way I initially wanted but at least I can get some of my shapefiles merged this way.

Thx,
Joe

[DuncanHornby]

Joe,

Try using the Collect value tool. In your model attach the point_ge.shp output of the iterator to a collect value tool. In model builder go to to menu option insert > model only tools > collect values.  This as the name suggests collects the values generated by the iterator which can then feed into the append tool.

Duncan

[JoeCook3]

Thanks for the suggestion Duncan, the end result is the same, appended rows with zero values. It knows the shape is a point and the coordinates get transferred over, just not the other info. The model works when the schemas are the same (TEST append option) so I'm thinking something funky is happening with the NO TEST option. I have to use the No TEST option with these shapefiles because they are an older format with a different assortment of coumns than the newer format (8 of the columns are the same, those are the ones I want). The field mapping appears to me mapping the old columns to the new correctly but nothing is transferred. The data type for each mapped column is the same (float, text, etc).

~Joe

PS - If I run the same model with the same structure, but for just one folder at a time (thereby defeating the point of using model builder) it works fine even with the NO TEST option. It's feeding the Append tool with the iterator that messes everything up.

[JoeCook3]

Here's an update for anyone else who might stumble across this thread.

The model with recursive feature class iteration and Append succeeds if all the columns in the file (.shp in my case) match and you use the TEST option in the Append tool

If you use the NO TEST option the model appends the appropriate number of rows but fills the cells with zero values.

If you just run one file through the iterator it appends it properly and populates the fields. If there is more than one file in the folder the first one is appended properly while all the following get zeros.

Something about Feature Class Iteration and the Append tool using the NO TEST option do not play nicely together.

[ShaunConway]

I am having this same issue. I am using the Iterate Values tool rather than the Iterate Feature Classes tool, but with the same outcome as described above. If I set the Schema Type parameter to "No Test", values from the first iteration of the tool will be appended with correct attribute values. All subsequent iterations will append the appropriate number of records, but with blank attribute values.

[JoeBryant2]

I also am seeing this behavior when using the Append tool at the end of a Model Builder sequence. The first time I run the model, the attributes are transferred to the fields mapped using NO_TEST. But when I try to run the model a second time on different inputs, only the spatial information is appended.

I am not using an iterator, but instead have parameters assigned to the inputs to manually select the table I have created for joining to a specific area within our district each time I run the tool.

At first I thought this may have something to do with appending from a local file geodatabase to a versioned enterprise geodatabase. So I created an empty local feature class in the same file geodatabase as my tables and intermediate feature classes, using the schema from the destination feature class in SDE. However, the append tool still behaves the same way - the first run is fine, but subsequent runs only append the spatial attributes without giving any indication of errors or warnings in the results.