topic Re: Preserving a GlobalID while moving data between Feature Classes in Data Management Questions

Preserving a GlobalID while moving data between Feature Classes

alex_friant — Fri, 27 Jan 2023 18:39:21 GMT

The spirit of this post is to gather other people's solutions, discuss ways of improving the suggested solution, and track future ArcGIS capabilities as they evolve for this problem.

This all started because we had existing data in a GDB that contained a Relationship Class (RC) between a Feature Class (FC) and a Table (TBL) using the FC's GlobalID field. We wanted to move that data into a new GDB that had a new schema (changes in domains, fields, etc.).

The problem is that when you Append the old data into the new schema, the GlobalID in the new schema is different than the records from the old data. This breaks the relationship between the FC and the related TBL.

UPDATE 2:

I've got some bad news and I've got some good news. And some ugly details. But I also have steps/options.

The BAD news

The "New Solution" from my first update no longer works in Pro 3.x, as so kindly brought to our attention by @MichaelMannion a few days ago in this reply. Thanks Mike!

Basically, ArcGIS Pro no longer allows you to change the datatype for step #8 in the solution below. This breaks the solution I originally provided.

The GOOD news

ArcGIS Pro 3.x now respects the "Preserve GlobalID" environment option in the Append tool, but with very specific parameters. It works if the target database is either Enterprise Geodatabases and Mobile Geodatabases, but not File Geodatabases.

The UGLY details

In order for the Append tool to actually preserve the GlobalID from your source to your target, you still need to check the "Preserve GlobalID" environment option in the Append tool.

You can only get it to work with Enterprise Geodatabases (EGDB) and Mobile Geodatabases (MGDB) as your targets, as no longer works with File Geodatabases (FGDB). Here's why.

The target GlobalID field must have and index that is configured as "unique". You can't make a GlobalID field that has a unique index in a FGDB.

Unique and ascending indexes are not supported for shapefiles or file geodatabases. These parameters are ignored when the tool is executed on a shapefile or file geodatabase data.

A 'unique' index can only happen if:

the table/feature class is in an EGDB /MGDB
you add the GlobalID to the table/feature class using ArcGIS Pro (3.x+)
- alternatively, copy/pasting the empty target table/feature class into an EGDB or MGDB will automatically change the index to 'unique' for GlobalID in the paste target

When you follow those two parameters, the index that gets automatically created for the GlobalID field that is has the unique setting.

If your target field is a GUID datatype, it must also have a "unique index". You must use add the index in ArcGIS Pro, on an EGDB table/feature class, and make sure the "unique" checkbox is selected before you run the Add Attribute Index tool.

The STEPS for Enterprise Geodatabases/Mobile Geodatabases

Below I only refer to EGDB, but if you're using a MGDB the same steps apply.

Make sure your target table/feature class is in an EGDB
1. If you have an empty table/feature class sitting in a FGDB but the indexes are not "unique", just copy them over to the Enterprise Geodatabase using ArcGIS Pro (3.x+)
Make sure your target field in the table/feature class has a "unique index"
1. GlobalID
  1. If you already have a GlobalID field, but the index is not unique you need to replace your table/fc with a new one
    1. Export it to a local FGDB, but make sure to remove GlobalID following these steps.
    2. Delete the original target from the EGDB
    3. Replace the original target with a copy of your table/fc that has no GlobalID
    4. In ArcGIS Pro (3.x+) add GlobalID's to the target table/fc (right-click>Manage>check the box for "Global IDs")
    5. Confirm that the index for the GlobalID field is in fact "unique"
2. GUID
  1. If you already have a GUID, but the index is not unique, you need to delete the index
  2. Then you need to use the Add Attribute Index tool to create an index for the GUID, and make sure you check the box to have it be "unique"
  3. Confirm that the index for the GUID field is in fact "unique"
Now your EGDB target table/fc fields are ready!
Open the Append tool
1. Under Environment, check the "Preserve GlobalID" box
2. Under Parameters, append as usual

Once your data is in the updated schema, you can copy/paste it over to a File Geodatabase if you need to (for a deliverable or something).

The STEPS for File Geodatabases

All hope is not lost for users who must be restricted to File Geodatabase only. (But really, there is no reason to be afraid of the Mobile Geodatabase as a in between step, it works well!).

Thanks to user @DirtDogRoj in his excellently documented reply you can follow clearly outlined steps to achieve the same result. I think you could put it into Model Builder even as all the steps very systematic. You might like this method better just because! 😉

Pay no attention 2:

~~UPDATE 1:~~

I have a much, MUCH, better solution that I discovered and it doesn't appear to be documented anywhere. All the text below that has strikethrough you shouldn't pay attention to, as it was the old, convoluted solution.

~~New Solution:~~

~~Have your old data ready to load (your "loading" gdb - this is your source data)~~
Have your empty geodatabase with the new schema. (your "new" gdb. It's okay that it has the "GlobalID" field, there is a new way to populate it with the GlobalID values from your loading geodatabase in ArcGIS Pro.)
~~Open ArcGIS Pro~~
~~Make sure your two geodatabases are added to the project (the "loading" gdb and the "new" gdb)~~
~~Run the Append Tool~~
~~Use your loading gdb feature class as the source, and your new gdb feature class as the target~~
~~For the tool's Environment settings, make sure the "Preserve Global ID field" checkbox is checked~~
In the Append tool's Field Mapping section, under the "Properties" tab, change the target feature class Data Type for the GlobalID field from "GlobalID" ---to----> "GUID". (I know, this seems strange. The data type is set in the target feature class schema as "GlobalID", but if you don't change this to GUID, then the GlobalID values from your source feature class will not migrate over to your new FC. If you change this setting to "GUID", they will magically migrate over to your new FC! I don't believe this is documented. If someone sees this documented anywhere, please share in the comments!)
~~Repeat steps 4-8 for your related table.~~

~~Pay no attention 1:~~

~~After doing some searching, I discovered that there is a way to do this using and Enterprise Geodatabase (EGDB) and ArcGIS Pro.~~

I post my basic workflow here on how to preserve the FC's GlobalID values so that when you migrate the data over to the new schema, the GlobalID values stay the same in the new FC. This preserves the relationship between the FC and the TBL in the new schema.

Assumptions:

~~You have a FC you need to migrate to a new schema and would like to preserve the GlobalIDs.~~
You have access to an EGDB and ArcGIS Pro

Manual Steps:

~~Copy the source FC (Feature Class) to an EGDB (Enterprise Geodatabase)~~
1. ~~Rename each class to have "_OLD" appended to them~~
  - ~~Note: You might need to deal with differences in domains at this point if they have the same name but have different contents~~
  - ~~You don't want your final domains to be appended with "_1". If so, then after you copy them over, you will need to turn off the domains where they are used, and delete them in the EGDB~~
  - ~~This doesn't affect final product because this is only happening in the source data~~
~~Prepare your new schema of your target FC to have a "GlobalID" that can be preserved:~~
1. ~~Take a copy of the FC empty schema~~
  1. ~~Use X-Ray in ArcCatalog to remove the GlobalID field in the FC~~
  2. ~~Create a new GDB using this new FC design~~
2. ~~Create a new "GlobalID" field manually (don't use the GP Toolbox) in the FC~~
  1. ~~Use "GUID" datatype~~
  2. ~~Do this in the FCL for the Global ID's that need to be preserved - this is normally only necessary on the FC where there is a oneway relationship~~
3. ~~Copy the FC to the EGDB~~
4. ~~In the newly copied FC in the EGDB…~~
  1. ~~Create a new index for this new "GlobalID" field~~
  2. ~~Make sure that it has "Unique" box checked, and "Ascending" too~~
~~Append records from the OLD FC to the NEW FC using ArcGIS Pro's Append tool~~
1. ~~Add the FC's to an ArcGIS Pro project~~
2. ~~Run the Append tool~~
  1. ~~Make sure to have the "Preserve GlobalID" box checked under Environments~~
  2. ~~For "Schema Type", use the "Use the Field Map to reconcile schema differences" option~~

At this point, you can now copy/paste that new FC back to your location of choice, and rebuild the RC so that it connects up with the new TBL. It turns out that the GUID's used in the related table to relate back to the FC are naturally preserved by using the Append tool in ArcCatalog, so performing the workflow above on the related TBL is unnecessary. Even though the TBL's GlobalID (not GUID) values change when moving the data, that's doesn't matter to us because they aren't used to create the relationship.

~~We don't do this often so we aren't going to take efforts to automate it, but I assume that might be possible.~~

Re: Preserving a GlobalID while moving data between Feature Classes

MitchHolley1 — Fri, 29 Sep 2017 17:21:51 GMT

Another idea is memorialize the GlobalIDs in the Feature Class (create new field called [GBL_GUID], GUID type --> Field Calculate the [GBL_GUID] field to the values in GlobalID). Now, if you export any data from that Feature Class in whatever database format.. the relationship will be maintained through the new [GBL_GUID] field.

Re: Preserving a GlobalID while moving data between Feature Classes

alex_friant — Tue, 05 Dec 2017 00:57:56 GMT

Great answer, thanks! Your solution is better because once the change is made, then I never have to deal with the issue again after changing schemas. Thanks again.

Re: Preserving a GlobalID while moving data between Feature Classes

alex_friant — Tue, 05 Dec 2017 17:03:19 GMT

After further testing, Mitch's solution won't work for us. The GUID field created doesn't autopopulate with a GUID in either ArcMap nor ArcGIS Online once deployed. Unless there is another trick to it, I'll have to stick to my original solution.

Re: Preserving a GlobalID while moving data between Feature Classes

ThomasColson — Tue, 05 Dec 2017 17:13:41 GMT

Will since you have an EGDB.....this is another one of those "Tech support will disown you" hacks #if they catch you doing it but...

In SQL Management Studio, change the object_flags to 4 (from 256) and sde_type to 14 (from 12) for the GlobalID column in SDE_column_registry for both feature classes. Then do you move. Remember to change them back!

Re: Preserving a GlobalID while moving data between Feature Classes

MitchHolley1 — Tue, 05 Dec 2017 18:08:36 GMT

Have your DBA to a SQL update to the rows every x amount of time.

Re: Preserving a GlobalID while moving data between Feature Classes

alex_friant — Tue, 05 Dec 2017 18:48:20 GMT

The data is to be hosted in AGOL eventually, so hacks won't work.

Re: Preserving a GlobalID while moving data between Feature Classes

alex_friant — Fri, 05 Jan 2018 18:50:53 GMT

Thomas, check out the update in my original post. I found a very simple, supported, non-hack way to do this using ArcGIS Pro.

Re: Preserving a GlobalID while moving data between Feature Classes

ThomasColson — Fri, 05 Jan 2018 19:07:38 GMT

very handy! I wonder if this can be mixed with some other python tools for archiving and downloading between AGOL and on-premise databases to achieve a true "two-way" sync without having to rely on flaky portal or pay for a data interop license.

Re: Preserving a GlobalID while moving data between Feature Classes

alex_friant — Fri, 05 Jan 2018 21:14:59 GMT

I don't see why not. The Append Tool should be scriptable, but I haven't had time to look into that yet. It's a good idea!

Re: Preserving a GlobalID while moving data between Feature Classes

PaulDavidson1 — Fri, 11 May 2018 18:06:38 GMT

This comment is somewhat off topic but related.

I have found the same situation when creating a Relationship class across a FeatureClass and say, an InspectionTable. I could not point the relationship as GLOBALID to GLOBALID.

Collector, Web Apps, just wouldn't allow for new records.

Now, we are using a BOTH direction in the relationship class so perhaps a one way would get rid of this.

The solution was to do what you did and create a GUID.

When you think this over, it actually makes sense.

I believe the following is at true statement:

All GLOBALIDs are GUIDs but not all GUIDs are GLOBALIDs

Hence, GlobalIDs are like an ObjectID and basically untouchable by the end user. The system will create & treate them as a Primary Key. At least for me, that concept helps explain this behavior.

Now not to hijack your thread, I'm going to post this question as a new thread if my Google search comes up empty handed... I have found that when I publish a layer with GLOBALIDs to ArcGIS Online (& I assume Portal but haven't verified that yet) the GLOBALIDs in ArcMap (and Pro but they were created in ArcMap) show up as all UpperCase. In AGOL, they are all lower case. It had been driving me nuts that I could not join on the GlobalIDs....

Researching that question brought me here as my top hit.

And the fact that you found is not documented is very strange. This is some fundamental stuff.

Re: Preserving a GlobalID while moving data between Feature Classes

ThomasColson — Fri, 11 May 2018 18:14:04 GMT

related to your case question: https://community.esri.com/message/769955-guid-switch-to-lower-case

Re: Preserving a GlobalID while moving data between Feature Classes

alex_friant — Fri, 11 May 2018 18:20:24 GMT

While this is a slightly different situation, it's similar in nature. Your feature class GLOBALID field needs to join to a GUID field in the standalone table. The standalone table should have it's own GLOBALID field, but that's just like the OBJECTID field: a unique id for each record. That's the way AGOL converts relationships between FC's and tables that join using OBJECTID, it adds a GUID and GLOBALID to the standalone table, a GLOBALID to the FC, and changes the relationship class to join this way FC.GLOBALID <----> TABLE.GUID

So it makes sense that it won't let you use GLOBALID on both sides of the join because the GLOBALID values need to be unique inside a Geodatabase across all items.

Re: Preserving a GlobalID while moving data between Feature Classes

MichaelWainright1 — Mon, 18 Feb 2019 20:42:09 GMT

Thank you! This is exactly what I needed. We were trying to Append from fgdb to SDE and getting a 999999 error when Preserve GlobalID was selected. Now that I think about it, makes sense that the target must be of data type GUID, though, preserving the GlobalID should automatically set the target to GUID data type, or at least throw a more useful error.

Re: Preserving a GlobalID while moving data between Feature Classes

FredSpataro — Mon, 11 Mar 2019 22:59:58 GMT

Alex,

Thanks for posting this!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

I was getting the same 9999999 Michael was seeing and couldn't figure out why.

Re: Preserving a GlobalID while moving data between Feature Classes

Anonymous User — Wed, 17 Apr 2019 13:43:05 GMT

Alex Friant I have attempted your steps to Preserve the GlobalID field when achieving data. In my case, the GlobalID were not preserved even with the box checked.

I do run into the same issue of the original problem as the append tool does not work unless the GlobalID field type is changed to GUID. However, in my case the GlobalID field is still not being preserved.

Have you ran into this issue as well?

UPDATED:

My solution turned out I needed to add an index to my archived feature class (Target Dataset) as the GlobalID field and set as unique. The append will run without making adjustments to the field map.

Re: Preserving a GlobalID while moving data between Feature Classes

alex_friant — Wed, 17 Apr 2019 16:58:36 GMT

Interesting. The "Add Global IDs" tool adds the index automatically, so perhaps it was added manually or the index was removed by accident? Either way, glad you got it figured out!

Re: Preserving a GlobalID while moving data between Feature Classes

Anonymous User — Wed, 17 Apr 2019 17:11:16 GMT

The Add Global IDs is something we have to do when creating the new feature class on the server. However, I don't believe it sets it to "unique" under the index properties so I have to go back in and add a new index pointing to the GlobalID and set it as "unique".

.....seems like unnecessary steps that should be already set in place.

Re: Preserving a GlobalID while moving data between Feature Classes

alex_friant — Wed, 17 Apr 2019 18:46:28 GMT

My GlobalID index is marked as "No" for Unique, but it still worked. Does that same FC has an ObjectID field with an index set to Unique = 'Yes'? Mine does. I downloaded the FC from AGOL at one point, so perhaps that's

Re: Preserving a GlobalID while moving data between Feature Classes

Anonymous User — Wed, 17 Apr 2019 19:06:26 GMT

My ObjectID field are showing unique as well. My target dataset was created as a blank FC on the sde. My input dataset was from AGOL. I did eventually download it from AGOL with the same results of Append failing. When I look at the index for the AGOL dataset the Global is set to "Yes" for unique. When I look at my target the GlobalID is set to "No" and I had to manually create the new index and set unique to "Yes". I'm assuming the append must be looking at this index for the GlobalID field and was failing because the settings between the two indexes did not match up between the two classes.