Download Sync not working in AGS 10.5.1

[JoeHershman]

Is anyone running a 10.5.1 ArcGIS Server and able to sync both directions?  We have two different systems both 10.5.1 and neither are successfully syncing data.  Data seems to upload from client without issue but the download is failing.  This also causes a JobStatus.Failed to be returned.  Because the job returns a failure, even though the edits did get sent correctly the replica does not think they were and so the outstanding edits just continues to increase and be resent.

I did have bidirectional sync working successfully prior to our upgrade to 10.5.1 so I have seen it work.

This is the message set returned from every replica we try to Sync.  This happens with 10 replicas on my side and even more on our other system.  I have looked through the AGS logs and there is not much there, other than it failed

2017-08-31 08:48:40,547 DEBUG - Job started.
2017-08-31 08:48:40,547 DEBUG - Creating delta database to path: 
2017-08-31 08:48:40,547 DEBUG - No delta database to upload.
2017-08-31 08:48:40,547 DEBUG - Creating server job.
2017-08-31 08:48:40,547 DEBUG - Sending request start sync geodatabase on the server.
2017-08-31 08:48:40,547 DEBUG - Received response for start sync geodatabase on the server.
2017-08-31 08:48:40,547 DEBUG - Getting server job status.
2017-08-31 08:48:40,547 DEBUG - Sending request get server sync job status.
2017-08-31 08:48:40,547 DEBUG - Received response for get server sync job status.
2017-08-31 08:48:40,547 DEBUG - Delaying job status for 0 seconds.
2017-08-31 08:48:40,547 DEBUG - Getting server job status.
2017-08-31 08:48:40,547 DEBUG - Sending request get server sync job status.
2017-08-31 08:48:40,547 DEBUG - Received response for get server sync job status.
2017-08-31 08:48:40,547 DEBUG - Delaying job status for 2 seconds.
2017-08-31 08:48:40,547 DEBUG - Getting server job status.
2017-08-31 08:48:40,547 DEBUG - Sending request get server sync job status.
2017-08-31 08:48:40,547 DEBUG - Received response for get server sync job status.
2017-08-31 08:48:40,547 DEBUG - Delaying job status for 0 seconds.
2017-08-31 08:48:40,547 DEBUG - Getting server job status.
2017-08-31 08:48:40,547 DEBUG - Sending request get server sync job status.
2017-08-31 08:48:40,547 DEBUG - Received response for get server sync job status.
2017-08-31 08:48:40,547 DEBUG - Delaying job status for 4 seconds.
2017-08-31 08:48:40,547 DEBUG - Getting server job status.
2017-08-31 08:48:40,547 DEBUG - Sending request get server sync job status.
2017-08-31 08:48:40,547 DEBUG - Received response for get server sync job status.
2017-08-31 08:48:40,547 DEBUG - Delaying job status for 4 seconds.
2017-08-31 08:48:40,547 DEBUG - Getting server job status.
2017-08-31 08:48:40,547 DEBUG - Sending request get server sync job status.
2017-08-31 08:48:40,547 DEBUG - Received response for get server sync job status.
2017-08-31 08:48:40,547 DEBUG - Delaying job status for 4 seconds.
2017-08-31 08:48:40,547 DEBUG - Getting server job status.
2017-08-31 08:48:40,548 DEBUG - Sending request get server sync job status.
2017-08-31 08:48:40,548 DEBUG - Received response for get server sync job status.
2017-08-31 08:48:40,548 DEBUG - Delaying job status for 4 seconds.
2017-08-31 08:48:40,548 DEBUG - Getting server job status.
2017-08-31 08:48:40,548 DEBUG - Sending request get server sync job status.
2017-08-31 08:48:40,548 DEBUG - Received response for get server sync job status.
2017-08-31 08:48:40,548 DEBUG - Error while handling get server sync job status. Job error 500 Failed to synchronize.
2017-08-31 08:48:40,548 DEBUG - Job failed. Job error 22 User defined failure. Error while handling get server sync job status. Job error 500 Failed to synchronize.‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

[JoeHershman]

I went back and ran a test on one of the 10.3.1 servers I still have access to.  On this server the data synchronized both directions successfully and success message was returned.  In 10.5.1 they have added new parameters to the Rest API.  I should note one difference the 10.3.1 is unsecure, it is not federated

New in 10.5.1

• The createReplica operation supports a targetType parameter which includes a targetType named server. Replicas with a targetType of server can support syncing between 2 servers.
• If the targetType is server, the createReplica operation also allows you to specify a syncDirection.
• The createReplica response now includes the replicaID and targetType in all cases when async=true.

Just guessing it seems that the rest API defaults these new values incorrectly and so the replicas from 10.5.1 do not have the correct settings.  

Does Quartz 100.1 offline support 10.5.1?  The documentation gives minimum versions but I see nothing about a maximum version.

akajanus-esristaff‌, WCrick-esristaff‌

[HamishMills2]

Hi Josh,

I think this is now a known bug with 10.5.1, we found the same issue (https://community.esri.com/thread/200887-sync-fails-at-1051-with-500-error ).

Cheers

[JasonWarzinik]

Testing a 10.6 setup and seeing the same exact issue.  Collector Sync fails but I do see the features created in the geodb, anyone figure out a workaround or fix?

[HamishMills2]

Hey Jason, as far as we have seen the only work around is to switch to a versioning workflow. However if you implement versioning you also then need to manage the offline version that is created per user or map. 

With this bug we have generally seen that although collector syncs fail consistently, any data going from mobile device back to main database works. The problem seems to be that any changes going back to your device from the database do not complete. i.e. the posting of changes back to base works but the reconcile back down does not. So as with some of our clients they have chosen to simply tell users to ignore the error for now and if they need changes from the default database, delete the map and re-download it.

Hope this is of some help to you. 

Cheers

Hamish

[JoeHershman]

Not meant to sound critical to you.  Changing your entire business process falls well outside what I would ever describe as a work-around.  In my opinion, a versioning workflow is impossible to manage in a large deployment.  Not to mention esri's recommends that the non-versioned archive enabled workflow.  To me the only workaround is not upgrading beyond 10.5.

Esri needs to get it together.  This is a known issue, it has been submitted and it is documented by numerous people and yet it persists.  Downloading new data is fine in a small deployment with small datasets but it is not in a large deployment (you cannot ask a few hundred people to just delete and download fresh data).  Plus if versioned that means you now need to manage the versions that were just removed.