Workforce Replica Sync issue

[JakeMatthys]

We use the ESRI disaster recovery tools to export our enterprise site from our primary machine to our backup machine.  Last week we needed to switch to the backup machine and after doing so we found that users were able to reload their Workforce project (which successfully creates a replica) but they were unable to sync.

The ArcGIS Server error log is below.

Before we open an ESRI helpdesk ticket, I just wanted to confirm if we should expect the Workforce mobile app should be able to sync when the project is exported from server A and imported to server B?

Thanks,

 - Jake

SEVERE

Dec 1, 2022, 6:56:08 PM

Error executing tool. Sync Feature Service Replica Job ID: jafc5f825644b4f1c948ead2fc6c2c703 : ErrorMsg@SyncGPService:{"code":500,"description":"Failed to synchronize."} Failed to execute (Sync Feature Service Replica).

System/SyncTools.GPServer

SEVERE

Dec 1, 2022, 6:56:07 PM

An error occurred.

System/SyncTools.GPServer

SEVERE

Dec 1, 2022, 6:56:07 PM

Failed to synchronize.

System/SyncTools.GPServer

SEVERE

Dec 1, 2022, 6:56:07 PM

Error: Failed to import data changes to replica.

System/SyncTools.GPServer

SEVERE

Dec 1, 2022, 6:56:07 PM

Failed to import data changes to replica. Failed to import data changes to replica.

System/SyncTools.GPServer

SEVERE

Dec 1, 2022, 6:56:07 PM

Initializing replica message handler with delta file: C:\Users\svc_agsprod\AppData\Local\Temp\tmp-ags-directories9706684956999726321689407314548600\arcgissystem\arcgisuploads\services\Hosted\workforce_8212341464d54f7e9911cff278bbbd1b.FeatureServer\i5204a867-43bd-4063-8011-80368ca09aec\du_cu4sdk7yiatozx2zkcv4p5jpfe.geodatabase failed.

System/SyncTools.GPServer

SEVERE

Dec 1, 2022, 6:56:07 PM

Failed to open delta database: C:\Users\svc_agsprod\AppData\Local\Temp\tmp-ags-directories9706684956999726321689407314548600\arcgissystem\arcgisuploads\services\Hosted\workforce_8212341464d54f7e9911cff278bbbd1b.FeatureServer\i5204a867-43bd-4063-8011-80368ca09aec\du_cu4sdk7yiatozx2zkcv4p5jpfe.geodatabase. Additional Error:

[MattMoore]

We are having the same issue. Were you able to ever find a fix for this?

[JakeMatthys]

We are still having this issue.  We have opened an ESRI helpdesk ticket, but have not made any progress.  Will post updates back to this thread.

[MattMoore]

Thanks for replying. We are actively engaging the Workforce product development team. Ill post back here if we get resolution to this problem.

[JakeMatthys]

Please let them know that other users are having this issue and please post back to this thread if you get any resolution.  We will do the same if we make progress with this issue.  Thanks again @MattMoore 

[MattMoore · Accepted solution]

Jake, was able to figure out the issue in our environment. Someone set the federated ArcGIS Server's instance configuration directories to a DFS share that had a long root path. Example: \\long.local.domain.name\dfsabc\PTI\things\servername\arcgisserver.  Found some errors in the verbose AGS logs that stated Failed to open delta database: \\long.local.domain.name\dfsabc\PTI\things\servername\arcgisserver\directories\arcgissystem\arcgisuploads\services\Hosted\workforce_dc0b60ddbb49451dba857d5e10df9659.FeatureServer\if8f03b3f-7185-4f4c-93a4-2001748e7d09\du_zgogxolq7mrst52w7ecjvsjw2d.geodatabase. Additional Error: 

Windows and ESRI will both barf when directory paths get close to 260 characters. In this environment this was the issue. Once the ArcGIS Server directories were moved to the local disk of the server everything in Workforce started syncing properly. 

Not sure if this will be your issue but it was certainly the culprit in our environment. 

[MattMoore]

This looks to be your issue as well. The path in your log to the sql lite geodatabase ESRI uses for the replica sync is 273 characters. Shorten that path and you should be good to go.

[JakeMatthys]

Thanks @MattMoore, We will look into this to see if it is also our issue.

 - Jake

[MattMoore]

Should be able to use ArcGIS Server manager to change all the configuration and input directory locations. It will move all the content for you. I'd recommend "C:\arcgisserver" since ESRI decided that adding 190+ characters to the path where it uploads the replica GDBs was a good idea. 

192 characters

directories\arcgissystem\arcgisuploads\services\Hosted\workforce_dc0b60ddbb49451dba857d5e10df9659.FeatureServer\if8f03b3f-7185-4f4c-93a4-2001748e7d09\du_zgogxolq7mrst52w7ecjvsjw2d.geodatabase

Should be able to sync all existing replicas created in workforce afterwards.

[JakeMatthys]

Thanks @MattMoore ... we are going to test this solution today...