Distributed Collaboration Sync Error

KevinPiraino2

I am looking for some guidance or confirmation on an error I have been encountering when attempting to sync a workspace that participates in a distributed collaboration between our Orgs. Portal and AGOL sites.

Basically, we are attempting to share Portal feature services (where the data is hosted in our MSSQL Server) to AGOL as hosted feature services; the AGOL hosted feature service would be a "copy" of the Portal feature service.

Some additional background:

The feature service / data we are attempting to share is originates from a Parcel Fabric enabled database.
Our MSSQL server has two instance, a production (editing) and publication instance. Every night our production instance is replicated/copied and overwrites our publication instance.
The feature services I am trying to share to the collaboration are built using data from the publication instance.

When we try to sync the workspace immediately after the feature service is published and shared (same day), the sync completes successfully. When we try to sync the workspace the next day, the sync fails. I am wondering if the nightly replicate / copy routine, where the production instance of the database is replicated / copied and overwrites the publication instance of the database, is breaking/blowing away the "replica" that is created by the collaboration/sync process. Thus when the workspace sync is executed the same day the feature service is published/added to the workspace, the workspace has no issue syncing the content, but when the sync is executed the following day (or any day after) the sync fails due to no reference of the "replica" in the database.

The specific error I am receiving from the Portal Admin logs is: "Failed to obtain sync/replica info for Feature Service item 'ITEMID', URL 'PORTALURL/arcgis/rest/services/FeatureServiceName/FeatureServer' while collecting item list for collaboration workspace id 'WORKSPACEID'. Item exists but will not be synchronized."

Is there a way to confirm that a replica has been created at the database level (in SQL Server)? Is there a way to confirm my theory about the replica being broken/blown away by our nightly SQL replicate/copy routine?

Infrastructure Configuration:

Using ArcGIS Enterprise 11.3
Enterprise Geodatabase version 11.3.0.52636
SQL Server version 2022

Collaboration & Workspace Configuration:

Distributed Collaboration with AGOL site as HOST and Portal site as GUEST.
Workspace set as SEND ONLY (from Portal to AGOL).
Workspace set with "Feature layers and views in my portal are sent as 'COPIES'".
1. "If unable to share as copies share as references" is UNCHECKED.

Portal Feature Service Configuration:

Geodatabase workspace set to "Branch Version".
Version Management UNCHECKED.
Editing "Enabled" (Add, Delete, Update).
Enabled Sync CHECKED.
Sync - Version Creation --> NONE.

TimoT

Hi @KevinPiraino2,

It’s been a while since I've worked with branch-versioned data, especially in a distributed collaboration context, but I can offer a few thoughts.

I believe your Version Creation setting should be set to “Create a version for each downloaded map,” even though your workspace is set to Send Only. I believe this setting is required when working with versioned data in distributed collaborations, but I’m not 100% certain. It would be great if someone else could chime in and confirm or provide additional insight. Although your scenario doesn’t exactly match the one in the document below, this document provides some useful guidance on preparing versioned data for distributed collaborations:

Prepare feature layer data for use in a distributed collaboration—ArcGIS Pro | Documentation

If you're trying to confirm your theory about the replica being broken by the nightly replicate/copy routine, here are a few things you could try or gather further information on:

With your initial successful syncs, are they only for newly published feature services that haven’t been shared to the collaboration workspace?
What happens if you sync the feature layer twice on the same day, before the nightly replicate/copy process runs? If your theory is correct, the sync should work both times since the replica would still be in place. Ensure there is a change in the feature layer before doing the re-sync test.
What error messages appear in the sync status collaboration logs? You can find these at /portal/sharing/rest endpoint > Home > Portals > Self > Collaborations > [COLLABORATION-ID] > Workspaces > [WORKSPACE-ID] > Sync Status
Cross checking your sync status logs, are your initial successful syncs real-time or scheduled syncs?

Regards,

Timo

KevinPiraino2

Timo,

A couple answers to your questions and comments:

I believe your Version Creation setting should be set to “Create a version for each downloaded map.”
- I have tested this and I do not believe the configuration is necessary even for a branch versioned database. I was able to successfully publish, share to the collaboration (with AGOL creating the hosted "copy" layer), make an edit in the Portal layer, and sync the workspace successfully.

With your initial successful syncs, are they only for newly published feature services that haven’t been shared to the collaboration workspace?
1. Yes, I did not test this workflow using another already published feature service. I did not want to use the production Parcel Fabric feature service to test this against and we do not have any other feature services in our publication side to test against.
What happens if you sync the feature layer twice on the same day, before the nightly replicate/copy process runs? If your theory is correct, the sync should work both times since the replica would still be in place. Ensure there is a change in the feature layer before doing the re-sync test.
1. If I sync the feature service twice on the same day, it works fine and is able to successfully sync. Even when performing an edit on the Portal side, the edit does come through on the AGOL side (as long as its within the same day).
What error messages appear in the sync status collaboration logs? You can find these at /portal/sharing/rest endpoint > Home > Portals > Self > Collaborations > [COLLABORATION-ID] > Workspaces > [WORKSPACE-ID] > Sync Status
1. The message from the "Sync Status" page:
```
Unable to sync item '46bb45cb5a0f4086b40bf2e0e4de77e9'. Error retrieving Sync info or replicas for peer 'neJvtQ4PXvnQ86MJ'.
```
Cross checking your sync status logs, are your initial successful syncs real-time or scheduled syncs?
1. The interesting thing about the real-time vs. scheduled sync, is that even on an initial sync, the workspace tries to perform both the real-time and scheduled sync. I'm not sure if that's intended or if I just set up the workspace incorrectly, but on the initial sync it will attempt both and one will inevitably fail (because its trying to create two AGOL layers with the same ID). After the initial sync though, its fine and the scheduled sync that is performed on the same day succeeds, but any subsequent syncs on following days fails.

TimoT

Unfortunately, not too much more I can add. Looking back at your original post, your theory certainly sounds plausible at this stage.

I hope someone else can chime in and provide you with further insight or some sort of SQL query to confirm the replica state at geodatabase level.

KevinPiraino2

Timo,

The only other thing I will add regarding this is that we do not currently have "replica tracking" enabled on the feature datasets/feature classes we are trying to share into the workspace. While I'm not entirely sure that this is the reason for the issue, it is another configuration option I need to look into. Upon initial testing, it didn't appear that not having "replica tracking" enabled mattered, but reading into the ESRI documentation a bit further makes it sound like it may be necessary.

I also found a different ESRI Community post discussing what happens behind the scenes when "replica tracking" is enabled.

Replica Tracking GP tool: https://pro.arcgis.com/en/pro-app/3.4/tool-reference/data-management/enable-replica-tracking.htm

Replica Tracking behind the scenes info: https://community.esri.com/t5/arcgis-pro-questions/enable-replica-tracking/td-p/1382804