Preserving a GlobalID while moving data between Feature Classes

[alex_friant]

The spirit of this post is to gather other people's solutions, discuss ways of improving the suggested solution, and track future ArcGIS capabilities as they evolve for this problem.

This all started because we had existing data in a GDB that contained a Relationship Class (RC) between a Feature Class  (FC) and a Table (TBL) using the FC's GlobalID field. We wanted to move that data into a new GDB that had a new schema (changes in domains, fields, etc.).

The problem is that when you Append the old data into the new schema, the GlobalID in the new schema is different than the records from the old data. This breaks the relationship between the FC and the related TBL.

UPDATE 2:

I've got some bad news and I've got some good news. And some ugly details. But I also have steps/options.

The BAD news

The "New Solution" from my first update no longer works in Pro 3.x, as so kindly brought to our attention by @MichaelMannion a few days ago in this reply. Thanks Mike!

Basically, ArcGIS Pro no longer allows you to change the datatype for step #8 in the solution below. This breaks the solution I originally provided.

The GOOD news

ArcGIS Pro 3.x now respects the "Preserve GlobalID" environment option in the Append tool, but with very specific parameters. It works if the target database is either Enterprise Geodatabases and Mobile Geodatabases, but not File Geodatabases.

The UGLY details

In order for the Append tool to actually preserve the GlobalID from your source to your target, you still need to check the "Preserve GlobalID" environment option in the Append tool. 

You can only get it to work with Enterprise Geodatabases (EGDB) and Mobile Geodatabases (MGDB) as your targets, as no longer works with File Geodatabases (FGDB). Here's why.

The target GlobalID field must have and index that is configured as "unique". You can't make a GlobalID field that has a unique index in a FGDB.

Unique and ascending indexes are not supported for shapefiles or file geodatabases. These parameters are ignored when the tool is executed on a shapefile or file geodatabase data.

A 'unique' index can only happen if:

• the table/feature class is in an EGDB /MGDB
• you add the GlobalID to the table/feature class using ArcGIS Pro (3.x+)
  • alternatively, copy/pasting the empty target table/feature class into an EGDB or MGDB will automatically change the index to 'unique' for GlobalID in the paste target

When you follow those two parameters, the index that gets automatically created for the GlobalID field that is has the unique setting.

If your target field is a GUID datatype, it must also have a "unique index". You must use add the index in ArcGIS Pro, on an EGDB table/feature class, and make sure the "unique" checkbox is selected before you run the Add Attribute Index tool.

The STEPS for Enterprise Geodatabases/Mobile Geodatabases

Below I only refer to EGDB, but if you're using a MGDB the same steps apply.

1. Make sure your target table/feature class is in an EGDB
   1. If you have an empty table/feature class sitting in a FGDB but the indexes are not "unique", just copy them over to the Enterprise Geodatabase using ArcGIS Pro (3.x+)
2. Make sure your target field in the table/feature class has a "unique index"
   1. GlobalID
      1. If you already have a GlobalID field, but the index is not unique you need to replace your table/fc with a new one
         1. Export it to a local FGDB, but make sure to remove GlobalID following these steps.
         2. Delete the original target from the EGDB
         3. Replace the original target with a copy of your table/fc that has no GlobalID
         4. In ArcGIS Pro (3.x+) add GlobalID's to the target table/fc (right-click>Manage>check the box for  "Global IDs")
         5. Confirm that the index for the GlobalID field is in fact "unique"
   2. GUID
      1. If you already have a GUID, but the index is not unique, you need to delete the index
      2. Then you need to use the Add Attribute Index tool to create an index for the GUID, and make sure you check the box to have it be "unique"
      3. Confirm that the index for the GUID field is in fact "unique"
3. Now your EGDB target table/fc fields are ready!
4. Open the Append tool
   1. Under Environment, check the "Preserve GlobalID" box
   2. Under Parameters, append as usual

Once your data is in the updated schema, you can copy/paste it over to a File Geodatabase if you need to (for a deliverable or something).

The STEPS for File Geodatabases

All hope is not lost for users who must be restricted to File Geodatabase only. (But really, there is no reason to be afraid of the Mobile Geodatabase as a in between step, it works well!).

Thanks to user @DirtDogRoj in his excellently documented reply you can follow clearly outlined steps to achieve the same result. I think you could put it into Model Builder even as all the steps very systematic. You might like this method better just because! 😉

Pay no attention 2:

UPDATE 1:

I have a much, MUCH, better solution that I discovered and it doesn't appear to be documented anywhere. All the text below that has strikethrough you shouldn't pay attention to, as it was the old, convoluted solution.

New Solution:

1. Have your old data ready to load (your "loading" gdb - this is your source data)
2. Have your empty geodatabase with the new schema. (your "new" gdb. It's okay that it has the "GlobalID" field, there is a new way to populate it with the GlobalID values from your loading geodatabase in ArcGIS Pro.)
3. Open ArcGIS Pro
4. Make sure your two geodatabases are added to the project (the "loading" gdb and the "new" gdb)
5. Run the Append Tool
6. Use your loading gdb feature class as the source, and your new gdb feature class as the target
7. For the tool's Environment settings, make sure the "Preserve Global ID field" checkbox is checked
8. In the Append tool's Field Mapping section, under the "Properties" tab, change the target feature class Data Type for the GlobalID field from "GlobalID" ---to----> "GUID". (I know, this seems strange. The data type is set in the target feature class schema as "GlobalID", but if you don't change this to GUID, then the GlobalID values from your source feature class will not migrate over to your new FC. If you change this setting to "GUID", they will magically migrate over to your new FC! I don't believe this is documented. If someone sees this documented anywhere, please share in the comments!)
9. Repeat steps 4-8 for your related table.

Pay no attention 1:

After doing some searching, I discovered that there is a way to do this using and Enterprise Geodatabase (EGDB) and ArcGIS Pro.

I post my basic workflow here on how to preserve the FC's GlobalID values so that when you migrate the data over to the new schema, the GlobalID values stay the same in the new FC. This preserves the relationship between the FC and the TBL in the new schema.

Assumptions:

• You have a FC you need to migrate to a new schema and would like to preserve the GlobalIDs.
• You have access to an EGDB and ArcGIS Pro

Manual Steps:

1. Copy the source FC (Feature Class) to an EGDB (Enterprise Geodatabase)
   1. Rename each class to have "_OLD" appended to them
      • Note: You might need to deal with differences in domains at this point if they have the same name but have different contents
      • You don't want your final domains to be appended with "_1". If so, then after you copy them over, you will need to turn off the domains where they are used, and delete them in the EGDB
      • This doesn't affect final product because this is only happening in the source data
2. Prepare your new schema of your target FC to have a "GlobalID" that can be preserved:
   1. Take a copy of the FC empty schema
      1. Use X-Ray in ArcCatalog to remove the GlobalID field in the FC
      2. Create a new GDB using this new FC design
   2. Create a new "GlobalID" field manually (don't use the GP Toolbox) in the FC
      1. Use "GUID" datatype
      2. Do this in the FCL for the Global ID's that need to be preserved - this is normally only necessary on the FC where there is a oneway relationship
   3. Copy the FC to the EGDB
   4. In the newly copied FC in the EGDB…
      1. Create a new index for this new "GlobalID" field
      2. Make sure that it has "Unique" box checked, and "Ascending" too
3. Append records from the OLD FC to the NEW FC using ArcGIS Pro's Append tool
   1. Add the FC's to an ArcGIS Pro project
   2. Run the Append tool
      1. Make sure to have the "Preserve GlobalID" box checked under Environments
      2. For "Schema Type", use the "Use the Field Map to reconcile schema differences" option

At this point, you can now copy/paste that new FC back to your location of choice, and rebuild the RC so that it connects up with the new TBL. It turns out that the GUID's used in the related table to relate back to the FC are naturally preserved by using the Append tool in ArcCatalog, so performing the workflow above on the related TBL is unnecessary. Even though the TBL's GlobalID (not GUID) values change when moving the data, that's doesn't matter to us because they aren't used to create the relationship.

We don't do this often so we aren't going to take efforts to automate it, but I assume that might be possible.

[DirtDogRoj]

In my mind, it is the object relationship that is important, not the Global IDs themselves.  Here is how I preserve the relationship, not the IDs.  This could be a relationship between assets and inspections, sign poles and signs, sewer lines and manholes, or anything like that, via Global IDs.

Important Note: GlobalIDs may be important to you and your organization for historical reasons.  For example, you may have a third-party system which uses GlobalIDs in those databases and/or preserved documents (such as PDFs, paper files, spreadsheets, and emails) that reference them.  The process which follows will break those relationships, yet preserve current object relationships in ArcGIS.

In the steps that follow, a one-to-many or one-to-one relationship is assumed, and only between two tables.  With a little imagination, you can migrate many-to-many relationships and/or relationships involving more than two tables.  But that is beyond the scope of this post.

There is a Master class (M) and a detail class (D).  As a programmer, I like to use letters as variables: you insert the actual names of your data in place of M and D.  M is usually a feature class and D is usually a table.

1. Assume M has a GLOBALID column (of type GlobalID) and that D also has a FOREIGN key column (of type Text, length 38 or type GUID) which defines the relationship between M and D (i.e. D.FOREIGN contains values from M.GLOBALID)
2. Clear any selections in M and D
3. Add a new Long Integer field to M called TEMPID
4. Calculate M.TEMPID to be M.OBJECTID
5. Add a new Long Integer field to D called OTHERID
6. Join D to M using D.FOREIGN and M.GLOBALID (i.e. D's fields are on the left and M's fields are on the right)
7. Calculate D.OTHERID to be M.TEMPID
8. Remove the join
9. Use the Feature Class to Feature Class and/or the Table to Table tools to migrate M and D to your target workspace.  In doing so, M.GLOBALID and D.GLOBALID values will get overwritten, and that is okay.  They will remain GlobalID-type fields.  Make sure that D.FOREIGN stays Text 38 or GUID type.
10. M and D are now in their target workspace.  For the remainder of these steps, M and D refer to their new locations, and not their previous ones
11. Join D to M using D.OTHERID and M.TEMPID (i.e. again, D's fields are on the left and M's fields are on the right)
12. Calculate D.FOREIGN to be M.GLOBALID.  This will overwrite previous values
13. Remove the join
14. Delete the fields M.TEMPID and D.OTHERID
15. Define a new relationship between M and D using M.GLOBALID and D.FOREIGN

You are done.  M is related to D through GlobalID-type fields, though the old GlobalID values are gone.  ArcGIS will honor this and still produce new values for these fields.

Give me a kudos if this works for you!  Happy GIS'ing!

--Update: Modified some of the steps from experience: #1, #9

[alex_friant]

Your method would certainly work, and love your well formatted/visual documentation style!

Personally, I dislike adding/joining/calculating/dropping fields when ever possible. In both approaches, it is necessary to load the data into the new target schema, right? So if you're going to append that data, why not just flip the target datatype to GUID during the append step and be done with it instead of performing 15 additional steps after appending the data?

I guess I'm just lazy 😜

[DuncanKinnear]

Alex,

Thanks so much for this - solved my problem! I was trying to append records while preserving GlobalIDs and getting the 003340 error which told me to 'Add Attribute Index' - which didn't make any difference. When I saw your post I switched the data type to GUID, and it appended no problem at all. Bizarre this is not more obvious/documented!

Cheers

Duncan

[MichaelMannion]

• QUESTION: Has anyone successfully used the updated New Solution workflow from the original post with ArcGIS Pro 3.0?

Using Pro 3.0.2, I am unable to perform Step 8 because the Append > Field Map > Properties values are read-only (i.e. grayed out). This is true for all columns in the Output Fields list, not just GlobalID.

Based on the dates of other posts on this thread that report success, I decided to try with Pro 2.9 and it worked. Using the same machine, I uninstalled 3.0.2 > installed 2.9, so this should be the only difference between the failed and successful attempts.

After having read this thread a few times, I suspect that the New Solution workflow was incidental to the ArcGIS Pro 2.x behavior, and not an intentional feature. That's not to diminish the value of the solution - it's a great find by @alex_friant, and will solve my problem in the short term; thank you for taking the time to research and post this. Rather, my customer will need a long-term solution that we can use for future data model updates > data migrations. At some point, it will be impractical to keep an old ArcGIS Pro 2.9 instance around and we'll need a v3.0+ solution.

I'm going to try using Python with Pro 3.0 to set the target GlobalID column type to GUID in the field map. Perhaps the fact that the field map properties are read-only is just a UI issue, and not baked into the underlying app/API/ArcObjects. I'll post the result here, good or bad.

P.S. I did try just appending with Pro 3.0.2 by enabling the Preserve Global IDs tool environment setting and doing nothing else, but it failed with an... 000597? 000595?... something like that. I didn't capture a screenshot at the time. I was hoping that Preserve Global IDs just worked, without having to use the GUID trick, but it did not. At least not yet; I haven't tested all of the index permutations listed in the other posts, but I will if I don't find another solution.

[JosephHilliard94]

Hello!

Does the new mobile geodatabase solution for ArcGIS Pro 3.0 work for appending into Feature Layers? The previous solution (very helpful may I add!) for 2.9 worked for Feature Layers but can't get the new solution to work, any ideas?

[alex_friant]

Yes, mobile geodatabases are just as functional in that regard as other geodatabases. I can append to an MGDB and from an MGDB into a Hosted Feature Layer (Portal/AGOL). 

What's the specific workflow you are having problems with? And I apologize ahead of time if I misunderstood what you said 🙂

[JJ_Reyes]

@alex_friant - is the workflow you are describing with append only for when moving an entire feature class to a new database? I'm asking because I have an existing database (which has FC and related tables in a GlobalID based relationship) and when I append new data to an existing parent table, I have not had any problem with the GlobalID changing.

Additionally, my database is hosted in AGOL and I have successfully exported it as a FGDB, added new feature classes and republished it to the same layer without breaking anything regarding the GlobalIDs and their related data.

In this case it would seem to me the issue is only when trying to append features to a new FC. 

[JJ_Reyes]

@alex_friant - ok I'm answering my own question here after reading the first line of the documentation on Preserve Global IDs. Sorry for the email flood!

Tools that honor the Preserve Global IDs environment will reuse the Global ID values of the existing input dataset when appending rows to a target dataset.

Ie, if you are trying to move your data and keep the GlobalID the same. Not related to when you are adding completely new data to an existing dataset. That's not going to change GlobalIDs already in the dataset.

[tiztrain]

I copied my data into a mobile geodatabase and when I try to append I get the following error message:

000597: Failed to insert input record with FeatureID <value>.—ArcGIS Pro | Documentation

Anybody come across this a resolved it?

[alex_friant]

The four suggestions they offer will usually solve the problem. In my experience this issue is usually caused by data being longer in the input than the output field can accept. Rarely, it will be a geometry issue, so repair geometry fixes it.