Experiencing Timeouts when loading Assignments

[JakeMatthys]

We have a process that loads assignments from our enterprise work management solution into our Workforce project.  This process has been consistently running with out issues for the past month, until today.  We have been receiving the following timeout error when the process attempts inserts and updates to the Assignments service.  

I would estimate that 30% of our transactions are successful and the remaining 60% fail with the following error:  {"updateResults":[{"objectId":4813,"globalId":"d11b36a8-0b2b-452c-80b9-f5ee3c4e80fc","success":false,"error":{"code":1000,"description":"Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding."}}]}.

Can anyone in the community offer any methods to help troubleshoot this issue?  Or is it possible that the an ESRI server is experiencing issues today?  We have even tested by manually adding an assignment (to eliminate our load process as a cause) from the REST endpoint and had a 50% success rate and received the same error on the failures.

Thanks,

  -Jake

[CraigGillgrass]

Hi Jake,

Can I ask that you contact Tech Support on this issue?  There were updates to ArcGIS Online last night; I don't know if the issue you are seeing is due to those updates; but Tech Support is your best resource to try and track this down.

Craig

[JakeMatthys]

Hi Craig,

I wanted to follow up with you on this issue.  We have initiated a case (# 01865488) with ESRI support.  I have done some follow up tests and wanted to share my findings with you.  I have attempted to Add/Update our WF Assignments hosted web service using three methods (C# web api calls, rest endpoint user interface, and the WF Dispatch web page) and all attempts have failed due to a timeout.  I have also attempted to add WF Users to the Dispatcher and Worker web services using the WF project configuration web page and was also unsuccessful because of a timeout issue.  I have repeated these tests on 3 different WF projects, same timeout error each time.  We currently do not have any issues submitting Survey123 forms, so it seems that the timeouts are only occurring to our hosted web services that are associated with a WF project.  Our log file shows that the timeout errors first occurred Dec 14 19:15PST.  At this point both our Survey123 and WF hosted services were down until Dec 14 20:55PST (which aligns with ArcGIS Online Health Dashboard ), at which time our process began communication with our Survey123 hosted web services, but the WF hosted web services continued to time out from that point on.  Have you had any reports of this issue from other WF users or does this seem to be isolated to our organization's AGOL portal?

Thanks,

 - Jake

[CraigGillgrass]

Jake,

While we have gotten a report on an issue related to Workforce that coincided with the Online update; it did not deal with timeouts.  We haven't gotten any other reports of timeout issues after the 4.4 update.

I appreciate the extra information - please make sure this makes its way to Tech Support so they can follow-up - they are your best resource on this.

Craig

[JakeMatthys · Accepted solution]

Hi Craig, 

Just to follow up, the performance of our feature services related to WF projects returned to normal this week.  Not really sure of a cause.  We worked with support, but it was difficult to perform detective work after the fact.

Just wanted to keep you in the loop, no need for a followup.

 - Jake