Workforce Replica Sync issue

[JakeMatthys]

We use the ESRI disaster recovery tools to export our enterprise site from our primary machine to our backup machine.  Last week we needed to switch to the backup machine and after doing so we found that users were able to reload their Workforce project (which successfully creates a replica) but they were unable to sync.

The ArcGIS Server error log is below.

Before we open an ESRI helpdesk ticket, I just wanted to confirm if we should expect the Workforce mobile app should be able to sync when the project is exported from server A and imported to server B?

Thanks,

 - Jake

SEVERE

Dec 1, 2022, 6:56:08 PM

Error executing tool. Sync Feature Service Replica Job ID: jafc5f825644b4f1c948ead2fc6c2c703 : ErrorMsg@SyncGPService:{"code":500,"description":"Failed to synchronize."} Failed to execute (Sync Feature Service Replica).

System/SyncTools.GPServer

SEVERE

Dec 1, 2022, 6:56:07 PM

An error occurred.

System/SyncTools.GPServer

SEVERE

Dec 1, 2022, 6:56:07 PM

Failed to synchronize.

System/SyncTools.GPServer

SEVERE

Dec 1, 2022, 6:56:07 PM

Error: Failed to import data changes to replica.

System/SyncTools.GPServer

SEVERE

Dec 1, 2022, 6:56:07 PM

Failed to import data changes to replica. Failed to import data changes to replica.

System/SyncTools.GPServer

SEVERE

Dec 1, 2022, 6:56:07 PM

Initializing replica message handler with delta file: C:\Users\svc_agsprod\AppData\Local\Temp\tmp-ags-directories9706684956999726321689407314548600\arcgissystem\arcgisuploads\services\Hosted\workforce_8212341464d54f7e9911cff278bbbd1b.FeatureServer\i5204a867-43bd-4063-8011-80368ca09aec\du_cu4sdk7yiatozx2zkcv4p5jpfe.geodatabase failed.

System/SyncTools.GPServer

SEVERE

Dec 1, 2022, 6:56:07 PM

Failed to open delta database: C:\Users\svc_agsprod\AppData\Local\Temp\tmp-ags-directories9706684956999726321689407314548600\arcgissystem\arcgisuploads\services\Hosted\workforce_8212341464d54f7e9911cff278bbbd1b.FeatureServer\i5204a867-43bd-4063-8011-80368ca09aec\du_cu4sdk7yiatozx2zkcv4p5jpfe.geodatabase. Additional Error:

[cadams]

I am experiencing the same issue but in my case I have a highly available ArcGIS Enterprise deployment in Azure which means we cannot move the ArcGIS Server directories to a local machine. However, as a test, I created a share with a much shorter path lenth on one of the ArcGIS Server VMs, moved the arcgissystem folder to this share and syncing worked fine.

We are using Azure files which gives this path structure, currently always 260 characters in length for each workforce project:

\\xxxxxxxxxxxxxxxxxx.file.core.windows.net\esri-prod\HA\arcgisserver\directories\arcgissystem\arcgisuploads\services\Hosted\workforce_51b0b99b94e44a0dbb6dd3bde08ec93f.FeatureServer\i9ec6a959-fc4d-4fd6-89f6-4fc58c3a4800\du_7msoerznist6hekxqhjq6vvw6a.geodatabase

@MattMoore when you say 'Windows and ESRI will both barf when directory paths get close to 260 characters', do you know if there is a hard limit? This is important for us as we could reduce the file path by several characters if we rename the base URL and name of the share. Otherwise we will have to look at something like DFS namespaces

[MattMoore]

Assuming your GIS servers are joined to your domain, a simple DNS alias (ex: \\gisfiles.domainname) could shorten your path to "\\xxxxxxxxxxxxxxxxxx.file.core.windows.net". You shouldn't have to rename the base URL to your Azure file storage.  Hopefully you have lower test environments you can test against as you will have to update the ArcGIS Server configuration directories to hit the DNS alias. 

[ahagopian_coj]

Hello!  We are having a similar issues but our errors don't say anything about the delta database.  How did you get the full path name?  I have looked at ours in the replica database and it is 168 characters long but am wondering if I don't have the true length of the pathname.  Thanks!

[JakeMatthys]

We retrieved the error from our ArcGIS Server log when users were attempting to sync (and or create replicas, I can't remember which).  The error on the original post is from our ArcGIS Server.

[MattMoore]

yes turn on verbose or debug logging on your federated ArcGIS Server machine and you should be able to see what the issue is when initiating a sync.

[JakeMatthys]

Thanks @MattMoore.  The issue was just as you described.  We are unsure how our Enterprise instance was configured with the directory that it was using because our configuration files that we use to build the site specify a different (shorter) path.  Thanks for pointing us in the right direction, this resolved an issue that has plagued us for a few weeks. 

[MattMoore]

Good deal, happy to help!