nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Merge option: remove non-essential fields; always create duplicate fields

IlkaIllers1

Very often, I will merge a large number of datasets. Since many come from a similar source, they will often have the same fields: "Comment", "Notes", "Name", "Number", "ID", etc. The problem is that these fields have the same name, but not the same properties - ranging from different lengths of text to completely different data types. Merging them results in Error 001156: Failed on input OID <value>; could not write value '<value>' to output field <value>. This means manually checking which fields caused the problem and either removing them, changing the names or increasing the text length. This can be a lot of manual processing.

It is even more problematic when run through Python, without a try/except with arcpy.ExecuteWarning you wouldn't even realise that there is a problem.

I would love if there were two options in merge:

1. Remove all non-essential fields (this would delete all fields apart from ObjectID, Shape, Shape_Length and Shape_Area)

2. Always create duplicate fields: instead of trying to stuff fields from different feature classes into the same output field, this would automatically create a second field, so that there would be Comment_1, Comment_2, Comment_3, etc. Essential fields like ObjectID, Shape, Shape_Length and Shape_Area would obviously need to be excluded from this.

Find more posts tagged with

Geoprocessing

Status: Open

Comments

ChristopherCounsell

This exists under field mapping. You can remove all fields or amend the output field.

https://pro.arcgis.com/en/pro-app/latest/tool-reference/data-management/merge.htm

Use the field map to reconcile schema differences and match attribute fields between multiple datasets. By default, the output includes all fields from the input datasets.
Use the field map to add, delete, rename, and reorder fields, as well as change other field properties.
The field map can also be used to combine values from two or more input fields into a single output field.

I don't see a use case for wanting to keep the same data but in different fields. Renders the merge kind of pointless. Better to manage your data correctly and update the field types as applicable. There's not really a universal solution for this from a product perspective.

You have other options to do this:

Build a new empty dataset as a standard. Instead of merging, APPEND data into this dataset, and map the data accordingly.
Set a model/script to check for duplicate fieldnames, then replace with string at the greatest length

IlkaIllers1

Yes, it exists under field mapping, but it needs to be done manually. If you have hundreds of fields and do several different merges, this is time consuming. I would like to remove non-essential fields with one click.

Same goes for data management - very time consuming. I would need to copy the original data (SOP not to delete fields from the original data), then manually delete all the layers in every single feature class. Sometimes I just want a layer that merges all the input layer and all the info I need can be found under the Merge_SRC field - I just need to know which feature class it came from.

From my experience, merging a lot of feature classes with append is even more difficult, the tool doesn't run.

Maybe I should clarify that we are talking about merging ~30 feature classes each with ~30-50 fields.

The script would only work for text fields, if the data type is differently, this would be more difficult?

ChristopherCounsell

Open Merge tool
Add input data
Select all fields by clicking the top field, holding shift, then the bottom field
Hit the red X to remove all selected from the field map
- You could also retain fields and use the properties to adjust the output field type
- There is a 'reset' button in the top-right too
Tick the 'Add source information to project'

IlkaIllers1

Yes, I know you can remove the fields you don't want. But you have to manually sort out the Shape_Length, Shape_Area, OBJECTID and Shape fields for keeping. It is just a repetitive task that I would have liked to get rid of.

mykemiller06

I would like to bulk add fields. Same process as bulk removal noted by Christopher, but in reverse. Is that functionality built in?

ChristopherCounsell

@mykemiller06 you can do this in a variety of ways

Fields view in Pro

Add fields (multiple) tool in pro

Add fields during merge or other geoprocessing tools options that give you a field map

mykemiller06

Right, but you have to check each field one at a time.

I want to highlight multiple fields and select them (tick the checkbox) to add to the field map list rather than checking every single field I want to add one by one.

I know you can highlight multiple fields after they are selected (checkbox ticked) at once and apply an Action when importing/merging, but there does not appear to be a method of "ticking the checkbox" for multiple highlighted fields in the Field Map.

ChristopherCounsell

@mykemiller06 given it's the exact opposite of this idea, you might be better served asking a new question (tool, workflow, desired outcome) and if the response is unsatisfactory, post a separate idea for this functionality.

I haven't had the issue you are encountering with merge so can't comment. Doing the above will open to more members of the community engaging on your query.