topic Re: Workforce Replica Sync issue in ArcGIS Workforce Questions

Workforce Replica Sync issue

JakeMatthys — Mon, 05 Dec 2022 21:47:27 GMT

We use the ESRI disaster recovery tools to export our enterprise site from our primary machine to our backup machine. Last week we needed to switch to the backup machine and after doing so we found that users were able to reload their Workforce project (which successfully creates a replica) but they were unable to sync.

The ArcGIS Server error log is below.

Before we open an ESRI helpdesk ticket, I just wanted to confirm if we should expect the Workforce mobile app should be able to sync when the project is exported from server A and imported to server B?

Thanks,

- Jake

SEVERE

Dec 1, 2022, 6:56:08 PM

Error executing tool. Sync Feature Service Replica Job ID: jafc5f825644b4f1c948ead2fc6c2c703 : ErrorMsg@SyncGPService:{"code":500,"description":"Failed to synchronize."} Failed to execute (Sync Feature Service Replica).

System/SyncTools.GPServer

SEVERE

Dec 1, 2022, 6:56:07 PM

An error occurred.

System/SyncTools.GPServer

SEVERE

Dec 1, 2022, 6:56:07 PM

Failed to synchronize.

System/SyncTools.GPServer

SEVERE

Dec 1, 2022, 6:56:07 PM

Error: Failed to import data changes to replica.

System/SyncTools.GPServer

SEVERE

Dec 1, 2022, 6:56:07 PM

Failed to import data changes to replica. Failed to import data changes to replica.

System/SyncTools.GPServer

SEVERE

Dec 1, 2022, 6:56:07 PM

Initializing replica message handler with delta file: C:\Users\svc_agsprod\AppData\Local\Temp\tmp-ags-directories9706684956999726321689407314548600\arcgissystem\arcgisuploads\services\Hosted\workforce_8212341464d54f7e9911cff278bbbd1b.FeatureServer\i5204a867-43bd-4063-8011-80368ca09aec\du_cu4sdk7yiatozx2zkcv4p5jpfe.geodatabase failed.

System/SyncTools.GPServer

SEVERE

Dec 1, 2022, 6:56:07 PM

Failed to open delta database: C:\Users\svc_agsprod\AppData\Local\Temp\tmp-ags-directories9706684956999726321689407314548600\arcgissystem\arcgisuploads\services\Hosted\workforce_8212341464d54f7e9911cff278bbbd1b.FeatureServer\i5204a867-43bd-4063-8011-80368ca09aec\du_cu4sdk7yiatozx2zkcv4p5jpfe.geodatabase. Additional Error:

Re: Workforce Replica Sync issue

MattMoore — Mon, 30 Jan 2023 16:22:24 GMT

We are having the same issue. Were you able to ever find a fix for this?

Re: Workforce Replica Sync issue

JakeMatthys — Mon, 30 Jan 2023 17:16:19 GMT

We are still having this issue. We have opened an ESRI helpdesk ticket, but have not made any progress. Will post updates back to this thread.

Re: Workforce Replica Sync issue

MattMoore — Mon, 30 Jan 2023 17:21:40 GMT

Thanks for replying. We are actively engaging the Workforce product development team. Ill post back here if we get resolution to this problem.

Re: Workforce Replica Sync issue

JakeMatthys — Mon, 30 Jan 2023 23:53:35 GMT

Please let them know that other users are having this issue and please post back to this thread if you get any resolution. We will do the same if we make progress with this issue. Thanks again @MattMoore

Re: Workforce Replica Sync issue

MattMoore — Tue, 31 Jan 2023 02:23:51 GMT

Jake, was able to figure out the issue in our environment. Someone set the federated ArcGIS Server's instance configuration directories to a DFS share that had a long root path. Example: \\long.local.domain.name\dfsabc\PTI\things\servername\arcgisserver. Found some errors in the verbose AGS logs that stated Failed to open delta database: \\long.local.domain.name\dfsabc\PTI\things\servername\arcgisserver\directories\arcgissystem\arcgisuploads\services\Hosted\workforce_dc0b60ddbb49451dba857d5e10df9659.FeatureServer\if8f03b3f-7185-4f4c-93a4-2001748e7d09\du_zgogxolq7mrst52w7ecjvsjw2d.geodatabase. Additional Error:

Windows and ESRI will both barf when directory paths get close to 260 characters. In this environment this was the issue. Once the ArcGIS Server directories were moved to the local disk of the server everything in Workforce started syncing properly.

Not sure if this will be your issue but it was certainly the culprit in our environment.

Re: Workforce Replica Sync issue

MattMoore — Tue, 31 Jan 2023 12:29:21 GMT

This looks to be your issue as well. The path in your log to the sql lite geodatabase ESRI uses for the replica sync is 273 characters. Shorten that path and you should be good to go.

Re: Workforce Replica Sync issue

JakeMatthys — Tue, 31 Jan 2023 13:35:20 GMT

Thanks @MattMoore, We will look into this to see if it is also our issue.

- Jake

Re: Workforce Replica Sync issue

MattMoore — Tue, 31 Jan 2023 14:35:14 GMT

Should be able to use ArcGIS Server manager to change all the configuration and input directory locations. It will move all the content for you. I'd recommend "C:\arcgisserver" since ESRI decided that adding 190+ characters to the path where it uploads the replica GDBs was a good idea.

192 characters

directories\arcgissystem\arcgisuploads\services\Hosted\workforce_dc0b60ddbb49451dba857d5e10df9659.FeatureServer\if8f03b3f-7185-4f4c-93a4-2001748e7d09\du_zgogxolq7mrst52w7ecjvsjw2d.geodatabase

Should be able to sync all existing replicas created in workforce afterwards.

Re: Workforce Replica Sync issue

JakeMatthys — Tue, 31 Jan 2023 15:53:45 GMT

Thanks @MattMoore ... we are going to test this solution today...

Re: Workforce Replica Sync issue

JakeMatthys — Wed, 01 Feb 2023 15:41:10 GMT

Thanks @MattMoore. The issue was just as you described. We are unsure how our Enterprise instance was configured with the directory that it was using because our configuration files that we use to build the site specify a different (shorter) path. Thanks for pointing us in the right direction, this resolved an issue that has plagued us for a few weeks.

Re: Workforce Replica Sync issue

MattMoore — Wed, 01 Feb 2023 16:46:07 GMT

Good deal, happy to help!

Re: Workforce Replica Sync issue

cadams — Thu, 22 Jun 2023 08:58:58 GMT

I am experiencing the same issue but in my case I have a highly available ArcGIS Enterprise deployment in Azure which means we cannot move the ArcGIS Server directories to a local machine. However, as a test, I created a share with a much shorter path lenth on one of the ArcGIS Server VMs, moved the arcgissystem folder to this share and syncing worked fine.

We are using Azure files which gives this path structure, currently always 260 characters in length for each workforce project:

\\xxxxxxxxxxxxxxxxxx.file.core.windows.net\esri-prod\HA\arcgisserver\directories\arcgissystem\arcgisuploads\services\Hosted\workforce_51b0b99b94e44a0dbb6dd3bde08ec93f.FeatureServer\i9ec6a959-fc4d-4fd6-89f6-4fc58c3a4800\du_7msoerznist6hekxqhjq6vvw6a.geodatabase

@MattMoore when you say 'Windows and ESRI will both barf when directory paths get close to 260 characters', do you know if there is a hard limit? This is important for us as we could reduce the file path by several characters if we rename the base URL and name of the share. Otherwise we will have to look at something like DFS namespaces

Re: Workforce Replica Sync issue

MattMoore — Thu, 22 Jun 2023 12:44:19 GMT

Assuming your GIS servers are joined to your domain, a simple DNS alias (ex: \\gisfiles.domainname) could shorten your path to "\\xxxxxxxxxxxxxxxxxx.file.core.windows.net". You shouldn't have to rename the base URL to your Azure file storage. Hopefully you have lower test environments you can test against as you will have to update the ArcGIS Server configuration directories to hit the DNS alias.

Re: Workforce Replica Sync issue

ahagopian_coj — Wed, 02 Aug 2023 15:49:56 GMT

Hello! We are having a similar issues but our errors don't say anything about the delta database. How did you get the full path name? I have looked at ours in the replica database and it is 168 characters long but am wondering if I don't have the true length of the pathname. Thanks!

Re: Workforce Replica Sync issue

JakeMatthys — Wed, 02 Aug 2023 17:25:34 GMT

We retrieved the error from our ArcGIS Server log when users were attempting to sync (and or create replicas, I can't remember which). The error on the original post is from our ArcGIS Server.

Re: Workforce Replica Sync issue

MattMoore — Thu, 10 Aug 2023 12:09:40 GMT

yes turn on verbose or debug logging on your federated ArcGIS Server machine and you should be able to see what the issue is when initiating a sync.