Distributed Collaboration - how to troubleshoot "Exporting item failed" errors

[tmichael_wpjwa]

I have a distributed collaboration set up between ArcGIS Enterprise and ArcGIS Online.  The collaboration is currently synchronizing 16 feature layers as copies.

I am seeing frequent failures with the collaboration syncing one or more layers.  The Collaboration Sync Status area of /portal/sharing/rest currently looks like this:

Digging in to one of the "completed_with_failures" records, I see messages like this:

[{  "taskStatusId": "9f50f1eba2404a04a61df6f351163539",  "results": {    "succeededCount": 15,    "failedCount": 1,    "notProcessedCount": 0  },  "failed": [{    "itemId": "823cac5574374d7d840e77eb67580ba8",    "itemTitle": "Base_Parcel_Collaboration",    "message": "Failed to export item '823cac5574374d7d840e77eb67580ba8' into a replication package : 'Exporting item failed'"  }],  "notProcessed": [],  "created": 1709654574646 }]

Looking at the "completed_with_failures" errors, it appears that it is not always the same layer that causes the issue.  In many cases it's a parcel layer that isn't changing anyway.

I increased the log levels for both Portal and Server to DEBUG and found entries similar to this in the portal logs:

Failed to synchronize replica. Failed to synchronize a replica on hosted feature layer 'https://gis.<redacted>.com/server/rest/services/Distributed_Collaboration/Base_Parcel_Collaboration/FeatureServer'. {"resultUrl":"","replicaName":"","submissionTime":1709653058000,"lastUpdatedTime":1709653067000,"status":"Failed","error":{"code":408,"description":"SyncTools service error.(null)"}}. Failed to synchronize a replica on hosted feature layer 'https://gis.<redacted>.com/server/rest/services/Distributed_Collaboration/Base_Parcel_Collaboration/FeatureServer'. {"resultUrl":"","replicaName":"","submissionTime":1709653058000,"lastUpdatedTime":1709653067000,"status":"Failed","error":{"code":408,"description":"SyncTools service error.(null)"}}

So it looks like the sync service is having an issue with...something(?), but I'm not sure where else to look.  Hoping someone here has an idea on where to look / what to try to track down these intermittent errors.

Additional details:

ArcGIS Enterprise 11.1.0, fully patched

Geodatabase is running on SQL Server 2019 in AWS RDS

Collaboration is syncing 16 feature services as copies.  Each service contains a single layer from the geodatabase.

Sync issue is intermittent between layers, but the most frequent errors appear to be with the parcel layers.

Thanks!

[ahargreaves_FW]

Go check your portal logs. Perhaps you are exceeding the 1GB limit on collaborations that the doc fails to tells you about?

[tmichael_wpjwa]

I'm familiar with those errors, and have encountered them before when first learning about collaborations, but I'm not getting them in this case.  Appreciate the suggestion though!

[tmichael_wpjwa]

Since the original message I have also changed the following settings to see if any changed the behavior:

- In ArcGIS Server Manager, adjusted the shared instanced settings > number of shared instances per machine from 4 to 8.

- Increased the Maximum Instances of the SyncTools service to 6.

- Changed the services that are included in the error messages most often from shared to dedicated instances.

None of these changes seem to have made a difference.

[DamonNelton]

Any updates on this topic - We are having similar issues. When we first set everything up it worked but then the Syncing stopped working. We never changed anything so unclear why?. Our errors are similar to these. Thanks Damon

[tmichael_wpjwa]

No updates.  I ended up building my own "monitoring" tools in an ArcGIS Notebook that sends me the collaboration status (error messages) once a day, and the same layers consistently fail.  As I mentioned above the layers that fail most frequently are also layers that don't change very often, so I haven't been too concerned about it.  Also, I'm currently blocked from fully leveraging a distributed collaboration for my purposes by the 1GB feature layer size limit, so I haven't really felt a need to pursue these errors much further if this isn't the solution I'll be using.

[AmyRoust]

There is a bug in Enterprise 11.1 for multi-machine deployments that is fixed in 11.3:

113-IssuesAddressedList_06112024.pdf (esri.com)

In addition to edit sync failures, I was getting a lot of those generic "failed to export item" errors while still on Enterprise 11.1. Upgrading to 11.3 resolved my issue.

One other thing to check: make sure that you aren't hiding any fields in your attribute table. In particular, the sync needs all of the editor tracking fields as well as global ID. I got that generic error many times when I had hidden those fields from the attribute table before publishing to Portal.

Good luck!

[tmichael_wpjwa]

Thanks for sending, I hadn't seen that.  That doesn't apply to me specifically because I'm running a single-machine deployment, but it gives me hope at least that the issue will resolve itself after we upgrade to 11.3.  I'm looking to perform that upgrade this fall.

[AmyRoust]

Here's another issue that I ran into today. I have a user named gis_viewer in our SQL database that I was using to connect to the enterprise database with my data. I can publish services to Portal with that database connection, but it's missing a permission that will prevent the data from copying via the collaboration. I don't know what the permission is, but I think it requires more than SELECT. Maybe someone from Esri like @ReeseFacendini can provide the link to that resource?