ArcGIS Server Manager ("Expired" vs Timeout)

[RyanSutcliffe]

I am having some trouble understanding how ArcGIS Server Manager defines "Timeouts", specifically in the stats view vs "Expired" services in the Logs.

My confusion: All our services use the default timeout value specified in the pooling tab of the Service: 60 seconds. I've experienced requests timeout in our apps and in our ArcGIS Server Error Logging I can see corresponding logs for these. The Error says:

"Error performing query operation. Wait time of the request to the service [service name] has expired"

However if I look at the Stats view, ArcGIS Server tells me that:

There are zero incidents of service timeouts:

And more confusingly, if I go to  look at the "Maximum Response Time (Seconds)", I see that ArcGIS Server reports that it is spending 3-4 minutes services requests: 

Maximum Response Time well above 60 seconds

What is going on here? What is the distinction ArcGIS Server is making here? I would expect to:

- Never see a maximum response time above 60 seconds

- See timeout statistics corresponding to instances of "expired" services in Logs. Isn't an "expired" service, a result of a service request taking over 60 seconds (essentially a timeout?)

I've tried looking for guidance, clarification in ArcGIS Server documentation (ex here) but have not found anything. 

Additional Notes:

- We are not (to my knowledge) using geoprocessing services, or exporting data via ArcGIS Server. There may be instances of occasional accessing of services via ArcGIS Desktop, but they are primarily accessed via ArcGIS JavaScript API Web Applications.

- We are using ArcGIS Server 10.71

- We use an NGinx proxy server which also has configuration for timeouts but I believe they are overridden to be above 60 seconds.

- We are seeing these timeouts in the app both in draw operations of layers within the service and in query operations to the service from ArcGIS JavaScript API. Interestingly, the query calls that timeout don't seem to appear in the logs of ArcGIS Server. (Still investigating). We even override the requestOptions to ensure that clientside the request will wait beyond 60 seconds: 

let queryTask = new QueryTask({
          url: baseUrl,
          requestOptions: {timeout: 180000}, //allow app to wait for 180 seconds before giving up
        });

[RyanSutcliffe]

A colleague just pointed out the importance of the distinction between usage time and wait time as discussed in the Tuning and Configuring Services link also referenced above. The idea being that difference between logs and stats above relate to this. I will do some testing later when I am in office to try to understand and welcome links that might help clarify this.

[RyanSutcliffe · Accepted solution]

Mulling it over a bit it makes sense. It seems:

The distinction between the error messages and the stats posted above are indeed related to usage time and wait time. As stated in link: Wait time is the time  before ArcGIS Server is ready to handle a request; Usage time is how long ArcGIS Server takes to process the request.

 

The Timout stats relates to usage time which in our case is left at default 600 seconds (10min). Since none of our services ever need that much time to process, it correctly reports nothing.

The MaxResponseTime also relates to usage time which we see has gone over 60 seconds in some instances (but not beyond 10mins).

The log messages of course relate to "Wait Time". These are failures of ArcGIS Server to get to starting our job before its own limit of 60 seconds. (It sure would be nice for ArcGIS Server to have a way to view this in the stats view.)

After some testing I determined that setting a `timeout` length in a query call in the JavaScript API can reduce the length that the request will wait but it can not extend the request beyond what is specified by the server. E.g: Server timeout is 60 seconds, JavaScript timeout is 90 seconds, the request will be cancelled by server after 60 seconds. If the JavaScript timeout was 30 seconds you'll see the request canceled in the browser. No error or log will show in ArcGIS Server for the later case.

The last item I haven't been yet able to test is browser default timeouts. All browsers have timeout limits so I presume ArcGIS Server when it is running a long processing job, it has a mechanism to keep the connection open (sending some sort of intermediate response info or whatnot) that will prevent the browser from timing out.

I suspect that the browser would override a really long wait time by ArcGIS Server though. For example FireFox seems to default to 90 seconds timeout. If you set ArcGIS Server to allow 180 seconds wait time I think that the request would still timeout clientside. And ArcGIS Server logs would record nothing. But this is unconfirmed.

Finally for clarity: a log related to a Usage timeout says:

---

Processing request took longer than the usage timeout for service [service name]'. Server request timed out. Check that the usage timeout is appropriately configured for such requests.

[RyanSutcliffe]

ESRI Canada reminded me that for geoprocessing jobs, the way you usually deal with long running requests is to make them async so that they are a series of network requests and avoid the browser timeout limits.