Portal Collaborations Failing

[DeanHowell1]

We have a large number of collaborations coming from a 10.7.1 ArcGIS Enterprise portal to a 10.8.1 ArcGIS Enterprise Portal and for some reason all of our collaborations are failing to sync.

WARNING	A scheduled job synchronizing items of collaboration 'Hosted_to_Collaboration_Test_Dean', '7282d8213eda42318907a737de9a7fef' and collaboration workspace '988e475bec1d4d44ba7b9612be6cf52f' with participant '89b42d53-8794-413f-8c65-dbb607513ec3' has failed.	2021-08-23T09:17:15,558	Sharing	10.10.57.70	admin	219999	305428
WARNING	A scheduled job synchronizing items of collaboration 'Hosted_to_Collaboration_Test_Dean', '7282d8213eda42318907a737de9a7fef' and collaboration workspace '988e475bec1d4d44ba7b9612be6cf52f' with participant '89b42d53-8794-413f-8c65-dbb607513ec3' has failed.	2021-08-23T09:17:15,558	Sharing	10.10.57.70	admin	219999	305428
WARNING	The import of replication package on portal with id 'df397694-47c7-43de-b0dc-73cafff46a48' failed.

There is also a regular entry in the logs

WARNING

Validation failed for hosting server 'https://portal/server'.

Any suggestions where to begin troubleshooting?

Regards

Dean

[HenryLindemann]

Hi Dean,
You can navigate to the collaboration workspace in the rest .
https://server.co.za/portal/sharing/rest/portals/0123456789ABCDEF/collaborations/04ddb40f20734a878cc..."COLLABORATION ID"/workspaces/070ff68f5e334b838c5a082f8c525f"Workspace ID"
At this location you will find sync status there is detailed logs as to which item failed, look for the itemID.

Typical failures are schema changes and large datasets 10 000 records + for schema changes you will have drop the item in the destination.
Regards
Henry

[DeanHowell1]

Thanks @HenryLindemann it is happening with all the collaborations but the logs have certainly given me additional insights into the issue.

[HenryLindemann]

HI Dean
Are you using Windows Authentication Perhaps ? What is the error in the logs ?

[DeanHowell1]

The logs are letting me know there seems to be an issue importing the replication package so could be a permission issue somewhere in the system. 

[AdamBakiera]

When you say "could be a permission issue somewhere in the system." What exactly do you mean? We are experiencing a similar error and it's very bizarre because our Collaboration has been syncing without issues for well over a year. About a week ago, we started seeing errors and we're not sure why. 

[DeanHowell1]

Hello @AdamBakiera what we found is that there was a space issue on one of the drives. Somewhere there is a setting that sets a threshold and for us it was 10GB and one of our AWS FSx drives had got below this threshold which triggered something that stop the collaborations continuing to fill the drive. Once I sorted the space issue the collaborations kept working. I hope that helps.

[DavidColey]

Hi @AdamBakiera  - we did too.  Last week, on Sept 10,  our collaborations-by-reference from our 10.9 portal to our host agol also began failing 8 of them failed to sync for 6 days.  The replication crpk files were failing to import on the agol side and would time out after 2 to 3 hours.

This morning, on 09/16 they all suddenly succeeded.  I did nothing, they just started working

[AdamBakiera]

Ha! That's literally exactly what happened here. I was preparing screenshots for a tech support call and this morning I saw the sync went through. Very strange...

[DavidColey]

So clearly something on the esri side . . . I know they are prepping for the Sept agol update, maybe that had / has something to do with it . . .