Portal Collaboration Failed (Failed to export item 'xxxx' into a replication package)

[Alexwang]

Hello, we set up collaborations between our portals (10.5.1). Although some times i have to manually run the sync to get the contents synced, most times we are pretty happy with it. In the past couple of days, one of our users tried to share an map image layer (map service as the source) and it didn't get through. When I tried manually running the sync, unsharing/resharing it like what i used to do, it still didn't go through. I noticed whenever I ran the sync, the following warning message appeared in the portal log. 

Failed to export item '57116782cc064eca9078cc87c02b9a55' into a replication package

Based on my understanding, data for only hosted feature layers gets replicated during collaboration while for map services, just, just a reference is shared. But why it's trying to replicate the data for this case? Since I couldn't make it work, our user tried republishing the service and re-shared it and it was able to get through. But republishing the service is not a solution for us since it will change the portal item IDs and mess up any existing map/apps using the item. 

Can anyone please help why the sync didn't work? Any workarounds or solutions?

[AlexandrosTamilos]

Hello,

Did you figure this out? I'm having the same issue. I've noticed that it happens when I add hosted features with multiple layers and tables.

Thanks!

[KingstonArcGIS_Online]

Having the same issue with only 2 features in my UAT env. One un-versioned (works) and one versioned (which has this error)

[Natalie_Runyan]

no solution here, yet, but I am working on this with tech support.

Same error, in a collaboration between ArcGIS Online (AGOL) and our Enterprise Portal.  Ours started failing without warning on 6/10/2022, after months of syncing fine.  The package fails on all services not just one.  (The only thing that changed around the same time that syncs started failing was generating a portal token - using the Portal REST directory, https://<portal>/<webadaptor>/sharing/rest/GenerateToken - for a specific user, *using our own portal as the referrer*.)

If we delete the destination content (the *copy* created on the host from the portal), the first sync works, but any subsequent ones fail.  Also, it seems to only push a copy of the map services, not the feature services.  

Configuration

Fails on push of portal services-to-AGOL

Task Name:	Export replication package
Task Status:	failed

1. We have server federated with enterprise portal at 10.9.1 Windows
2. Source data in SQL geodatabase registered with the GIS Server
3. services with Global IDs, and archiving enabled (no versioning), and (both with or without replica tracking enabled)
4. feature services published with Pro 2.9, with sync enabled
5. Collaboration set to send and receive, but workspace set to copy only (not reference, and *unchecked create reference if unable to copy*)
6. all items shared with "public" and the collaboration groups

Fails on push of hosted service in AGOL-to-Portal

Export and download peer replication package
Task Status: failed

1. created a hosted service for testing
2. initial copy is made to the portal, but subsequent updates fail

What we Tried So Far

1. Deleted the entire collaboration and built a fresh one, using a different geodatabase source; ; sync still fails at the Export Replica step, both ways. -- this seems to point to an issue with the portals instead of the data or the collaboration per se.
2. Full ReIndex of Portal; still fails; sync still fails at the Export Replica step, both ways.  Even though the index numbers matched, tried this just in case.
3. In each, Portal and AGOL:  Organization > Settings > Security > Trusted Servers > Added corresponding servers from both sides; sync still fails at the Export Replica step, both ways.
4. Confirmed could add the feature service to AGOL from portal as a "new item from URL"; works with no issues.  So, AGOL can reach the services in our portal directly.

TIP:  Esri recommended that anytime I force a sync, to do it from the REST API for the Portal (ex. https://<portal>/<portalwebadaptor>/sharing/rest/portals/<portalOrgID>/collaborations/<collaboration... "sync" and then "sync status"

[SrijanaTuladhar]

Hi Natalie_Runyan,

We are having the same issue. Were you able to resolve the issue in your end?  Thanks for your help. 

[HollyTorpey_LSA]

Any solutions? We're having a similar issue as well after upgrading to Enterprise 11.0. Our send-only workspaces still sync, but our send-and-receive workspace fails on the "Export and download peer replication package" task, citing an inability to unpack the replication package to the temporary location.

Interestingly, the last successful sync happened in the middle of our upgrade process, after upgrading Portal but before upgrading Server. For that reason, I believe the failure is caused by some change that occurred during our Server or DataStore upgrades. 

Here's the log text for the failed task:

Task Id: 8519dab454cb4548b424b5e67402f7cc">
Participant portal Id: jrlD8VS1h8uKjsE8
Task Name: Export and download peer replication package
Task Status: failed
Started: Jan 18, 2023, 2:12:20 PM
Ended: Jan 18, 2023, 2:12:24 PM
Modified: Jan 18, 2023, 2:12:24 PM
Status Summary: Exception: java.lang.RuntimeException: com.esri.gw.GWException: Could not unpack replication package to temporary location.

Updated to add: It turned out that the problem was insufficient space on the Enterprise installation drive following the upgrade. Portal has a temp directory there where it tries to temporarily save replication packages during the sync process. Examining the Portal debug logs was key to figuring this out ([your portal URL]/portaladmin/logs/queryFilter).

[Sarah_of_Lubbock]

I saw that you were having issues with you send/receive workspace, but not your send workspace - I was running into that as well.
I just turned the send/receive to just send - and it fixed everything!  At the moment, we don't need to receive any data, so this works for us.
Thanks!

[GrantSmith122]

Hello,

I have been having a similar issue to this, and wanted to share my experience. Overall I have found ESRI collaborations to be extremely flaky in their functionality and the size of attachments in layers can be very important.

We have an AGOL and an Enterprise 11.1 account. We currently have a project where we need to share data from AGOL to Enterprise for use in an offline Field Maps app (just view the data), and then need to share collected feature layer data back to AGOL. In essence we have 8 hosted feature layers going from AGOL to Enterprise, and 2 hosted feature layers going from Enterprise to AGOL. All have attachments.

When we initially set up a collaboration, we had one workspace set up with Send and Receive Content enabled so we could pass data back and forth in one workspace via one Group on the AGOL and Enterprise side of things (nice and organized). The first sync of the workspace worked when the datasets were small and tidy, then subsequent syncs started failing with similar errors to what everyone is discussing in this thread. 

I went back and forth with ESRI support for a while, added some SSL certificates to Portal to make sure it trusted AGOL, burnt the whole collab down and restarted, etc. etc. Same results of intermittently working, or failing altogether. We were also seeing replication package items in our Portal that were left over/not deleted after the failed sync, which were 1.7 GB in size. That would indicate a problem since replication packages can't be larger than 1 GB, but we had no idea why the package was that large.

I then dove into the hosted feature layers themselves. In the item overview page, they were all listed as a few MB in size (2.895 MB, 1.758 MB, 985 KB, etc). Way way way under the 1.7 GB size of the replication packages. When I looked at the data table and viewed the size of all the attachments, I found in many cases that the first few records in a table had attachments that equalled or exceeded the listed size value listed in the item overview page. One hosted feature layer was listed as 2.895 MB, but after adding up all the attachment sizes, the layer was actually more like 187 MB. Whoopsie, that would've been nice to know.

I then split the collaboration workspace into three new workspaces, one to send data to AGOL, and two to send data to Enterprise from AGOL. Syncs now work successfully.

I wouldn't consider this a solution, more like a workaround. The 1 GB replication package limit seems arbitrary and ridiculous, especially in this day and age of complex datasets with lots of attributes, attachments, and constant collection/editing workflows. If people have successful syncs of a workspace, then all the sudden see it failing, take a quick look at the layers in question and make sure it isn't size-related if you have a lot of attachments. 

Collaborations themselves should be such useful tools, but they seem nefarious to me sometimes. Arbitrary size limits, not very helpful logs, and inconsistent performance. Maybe they'll be updated in future AGOL / Enterprise releases.