Agol -> Portal distributed syncs break after an Agol -> Agol partner collaboration edit

ReedHunter · ‎10-29-2021

After sharing hosted copied featureclass data from 1. (guest) portal <= (distributed collaboration) => to 2.(host) Agol site <= (Partnered Collaboration) => and then to 3. (partner) Agol site, syncs back from 2 to 1 fail as soon as a partner at 3 makes an edit to the hosted featureclass.

If nobody has thoughts on what might resolve this, I'll file a bug with Esri tech support within the week.

Steps:

Create a distributed collaboration between your Portal server and Agol. The workspace should support "send and receive" for its data.
Create a partnered collaboration between your Agol site and another organization's Agol site. Make certain that the group members may update items.
Create a hosted featureclass in your portal's distributed collaboration workspace, sharing as a copy and not just as a reference. In our situation we created the featureclass logged in with admin privileges, and owner of the featureclass was the same as the owner/creator of the distributed collaboration as well as the partnered collaboration.
Confirm that the featureclass you added in portal is showing up in your Agol site's distributed collaboration workspace.
Make edits within both the Agol and portal sites, and then initiate a sync from the portal side's workspace. Confirm that edits from both sides were synced.
Now share the same hosted featureclass from your Agol site to the "group" for your partnered collaboration. Make sure that the sharing of the featureclass is set to "owner and shared with groups".
Make another edit to the featureclass within your portal site, and confirm from a member of the partnered organization that they can see the edit(s) you just made. In our case we made an insert, and update, and a delete to first a point feature class, and the 2nd time around to a polygon featureclass.
Have a member of the partner organization make some edit to the hosted featureclass from within their Agol site. We did this using the built in web mapping application's editing tools.

Expected Result:

The result I expected was for the edits from the partner organization to show up a few minutes later in my portal copy of the feature class.

Result:

The edits from the partner Agol organization were visible when viewing the featureclass within ArcGIS Online, but no edits sync back to the portal copy of the data.
Further attempts to make edits to the Portal copy of the data are successful, but do not sync up to Agol.
Further attempts to edit the data in Agol are successful, but do not sync down to Portal. In other words the break is at the distributed collaboration sync relationship.
This break is at the featureclass level, not at the entire collaboration. I know this because I've run this test twice with the same collaborations, and the 2nd hosted featureclass synced beautifully across the distributed collaboration, showing up across the partnered collaboration after the 1st hosted featureclass was failing to sync.
When accessing the sync logs at https://webadaptorhost.domain.com/webadaptorname/portaladmin/logs/queryfilter/, the following error is reported:

{

"type": "WARNING",

"message": " Cannot create replica for item '7d67e6c4c7054b4eb46027eebb9c80e5' for peer portal 'IYrj3otxNjPsrTRD' as maximum number ('6') of replicas have been created and not yet synchronized. To share the data for this feature layer, please un-share and re-share the item with the group.",

"time": 1629487329136,

"source": "Sharing",

"machine": "<For security I removed our host server name>",

"user": "",

"code": 219999,

"elapsed": "",

"process": "5856",

"thread": "1",

"methodName": "",

"requestID": ""

}

]

}

When I follow the following instructions on geonet to pull up more info on the portal logs while in debug mode, portal doesn't detect any errors:

Login to portal's admin site to be ready to switch portal's logging from Warning to Debug mode at https://webadaptorhost.domain.com/webadaptorname/portaladmin/ -> logs -> settings -> edit
Connect another web page to https://webadaptorhost.domain.com/webadaptorname/sharing/rest. Click on Home > Portals > Self > Scroll to the bottom on the page > Collaboration. Click on the relevant collaboration id > Collaboration Workspaces > Click on the relevant collaboration workspace id > Sync Status
Have a 3rd web page connect to your Portal site's collaboration workspace site to be ready to manually sync the workspace.
Switch your portal to Debug mode, then manually trigger the workspace sync, monitor the Sync Status for a few minutes, and then switch your portal back to warning mode when you're done.

Following those instructions at 11:20 on Oct 29, I saw these results.:

Status IdSync Execution TypeStatusStartedEndedModifiedStatus Summary

94fab80dc82640a8b4e68c8ad2ef0926scheduledcompletedOct 29, 2021, 11:22:33 AMOct 29, 2021, 11:24:20 AMOct 29, 2021, 11:24:20 AMCompleted sync successfully.

436748d0ad914011aa90853070ccc6c5scheduledcompletedOct 29, 2021, 11:20:16 AMOct 29, 2021, 11:22:07 AMOct 29, 2021, 11:22:07 AMCompleted sync successfully.

14b2f3915626424a97e35e97a04f10b1scheduledcompletedOct 29, 2021, 10:20:16 AMOct 29, 2021, 10:22:13 AMOct 29, 2021, 10:22:13 AMCompleted sync successfully.

7316d07dd5ed439d8abd3b33f865ed70scheduledcompletedOct 29, 2021, 9:20:16 AMOct 29, 2021, 9:22:06 AMOct 29, 2021, 9:22:06 AMCompleted sync successfully.

e263112e51e54f5c82de1d950473c300scheduledcompletedOct 29, 2021, 8:20:16 AMOct 29, 2021, 8:22:09 AMOct 29, 2021, 8:22:09 AMCompleted sync successfully.

f75ac0ef3f984e688b547f3d38453376scheduledcompletedOct 29, 2021, 7:20:16 AMOct 29, 2021, 7:22:09 AMOct 29, 2021, 7:22:09 AMCompleted sync successfully.

d80f5c057a6c4f11968d60a2a725a2a3scheduledcompletedOct 29, 2021, 6:20:16 AMOct 29, 2021, 6:22:15 AMOct 29, 2021, 6:22:15 AMCompleted sync successfully.

9c83a8453b6e434cbd90af47a2fe925dscheduledcompletedOct 29, 2021, 5:20:16 AMOct 29, 2021, 5:22:06 AMOct 29, 2021, 5:22:06 AMCompleted sync successfully.

c677b80873e04551a7bcc62326ee8b7fscheduledcompletedOct 29, 2021, 4:20:16 AMOct 29, 2021, 4:22:14 AMOct 29, 2021, 4:22:14 AMCompleted sync successfully.

28909ae4418a43fcabc3d7b3bcecc2f6scheduledcompletedOct 29, 2021, 3:20:16 AMOct 29, 2021, 3:22:07 AMOct 29, 2021, 3:22:07 AMCompleted sync successfully.

ce068189cf3a41138fd373e776555af4scheduledin_progressSep 11, 2021, 9:20:16 AMN/ASep 11, 2021, 9:20:16 AMCompleted initial validation. Syncing items to participants.

94077618cf654853801bfc966beefa98scheduledin_progressAug 30, 2021, 5:20:16 AMN/AAug 30, 2021, 5:20:16 AMCompleted initial validation. Syncing items to participants.

8cb980156ab749c4ba5fdf4b844bbcc9 realtime completed Jul 20, 2021, 4:31:39 PM Jul 20, 2021, 4:32:05 PM Jul 20, 2021, 4:32:05 PM Completed sync successfully.

The messages say that the syncs are successful, but the edits from the partnered Agol organization on our Agol site are still missing from the copy on Portal.

PierreloupDucroix · ‎04-04-2022

Hi,

Did you find a solution to your problem ?

We have the same error and log message (max number of replica), and apparently, the replica created by the collaboration for one item is not removed.

We can see them in rest/services/folder/service_name/FeatureServer/replicas

Even after unsharing/resharing the item, after 6 synchronizations, the collab fails to sync again.

Regards

ReedHunter · ‎04-14-2022

I did get a resolution on this a few weeks ago by visiting the product islands at the Esri Dev Summit in Palm Springs. In the end it turned out to be nothing more than a browser cache problem. I used a different computer than I had the first few times, so of course the problem did not reproduce when I sat with the technician. Back when I first wrote this blog I did try several forced reloads of the web app, but I don't recall deleting all temporary files.

The Esri technician informed me that when you set up a partnered collaboration, what's different from a distributed collaboration is that there is only one data store in the cloud. Distributed collaborations have multiple data stores, so the data has to be replicated between them. Partnered collaborations are about setting up user accounts to access just one data store, so as soon as the other organization's editor makes the edits, they should be visible to your own map viewer because you're both referencing the same feature class from the same database.

I hope this helps.

SBerg_VHB · ‎08-23-2023

Hello. I am wondering if anyone has been able to determine the general causes for the collaboration sync error "maximum number ('6') of replicas have been created and not yet synchronized."

What would cause this state?

Assuming a service is in a collaboration group, I would expect...1 (?) replica constantly synced and no more (?).

So this indicates a problem with permissions on the existing replicas or permissions deleting then when they are unused...or an unknown bug...or reflects that the sync failed due to other issues (ie. data > 1GB?)...just spectulating but welcome all ideas!

Thanks,

Sam

StefanUseldinger · ‎12-12-2023

We face the same issue in the context of a Portal collaboration.