Help Understanding System Log Parser Output

[ZachBodenner]

Hello,

I’m hoping for some help demystifying ArcGIS System Log Parser. For starters, I’m hoping for a little more explanation about the difference between the Elapsed Time (AVG) in the Statistics by Resource tab, Wait Time (AVG) in the Wait Time (Queue Time) tab, and Instance Creation Time (AVG) in the Instance Creation Time tab. I think I might not understand server processes enough to know the difference between some of these statistics because they seem like they measure the same thing. What’s the difference between wait time and elapsed time? I’m not sure.

 Additionally, what factors impact the time it takes for a server to create an instance? For example, I’m looking at a server resource that I have configured to have 2 minimum services running, yet the instance creation time is listed as 55 seconds on average. If I have 2 ArcSOCs dedicated to that service, why would it take so long to create?

Finally (though I have a lot more questions I’ll end here to avoid too long a post), in the Summary tab, no matter how far back I set my query in the Parser tool, the Data TimeSpan field seems to max out at a certain distance into the past. Is there an action that resets the .log file so that I can only see back so far? Restarting the ArcGIS Server instance or rebooting the server, for example?

Thanks!

Edit: PS I have read through JBoyle's help blogs (https://community.esri.com/t5/implementing-arcgis-blog/arcgis-server-tuning-and-optimization-with-sy...   and   https://community.esri.com/t5/implementing-arcgis-blog/system-log-parser-statistics-and-service/ba-p...) and I'm still a little confused.

[AaronLopez]

Hi @ZachBodenner,
Maybe I can help answer some of your questions...

"What is the difference between the Elapsed Time (AVG) in the Statistics by Resource tab, Wait Time (AVG) in the Wait Time (Queue Time) tab, and Instance Creation Time (AVG) in the Instance Creation Time tab?"
I'll start with elapsed times. These entries in the ArcGIS Server log (code 100004) are not response times per se, but an elapsed time. This values marks the duration the ArcSOC.exe process spent working on the request. This time also includes data access time but does not include wait time or creation time (or transport time).

Wait time or queue time is the duration of the request spent waiting to get to an available ArcSOC.exe process just to start working. Typically, this time is low (subsecond). When it is higher than a second, that is important information to note and usually suggests that the service does not have enough maximum number of instances available. Since there is not enough during that moment, requests start queueing...this waiting can impact the user experience.

Instance creation time (or start time) is the duration of the overall request time spent "waiting" (a different wait than the other previous one) for an available ArcSOC.exe (service instance) to start up. Sometimes there are enough instances ready to handle incoming requests, sometimes not.
Some deployments have configurations (for dedicated services) that use different values for the minimum and maximum number of instances...the default is a min of 1 and a max of 2. In the case where a lot of requests for a service come in all at once, a demand is (first) put on the number of instances that need to be running to meet this need. If not enough are running, more are started until the maximum is reached.
All service start up times are different but these log values helps identify the cost. Sometimes it is expensive and the start up times are very long. If you feel the times are too long and the user experience is impacted, increase the service instance minimum.

Another strategy when seeing long waiting times and/or instance creation times for a service is to change its type from dedicated to shared. This is great to do for seldom used services and more optimal than setting the instance to a min of 0 and max of 1.
While switching to a shared service type has several benefits, the service capability has to be supported (as a shared service) and the service needs to be published through ArcGIS Pro.
  
Note, there is no entry in the ArcGIS Server logs that show transport time. In other words, the time it takes to transfer the response from ArcGIS Server to the Web Adaptor and then to the client.
However, using System Log Parser to analyze the Web Adaptor access logs, show request times in the report that are closer to actual response times.   

"What factors impact the time it takes for a server to create an instance? For example, I’m looking at a server resource that I have configured to have 2 minimum services running, yet the instance creation time is listed as 55 seconds on average. If I have 2 ArcSOCs dedicated to that service, why would it take so long to create?"
The number of instances available to the service do not directly impact the creation time.
But the composition and complexity of the MXD or ArcGIS Pro project are two factors that can impact the creation time.
As for places to start troubleshooting the 55 seconds, try opening up the MXD or ArcGIS Pro project and examining it. Perhaps one layer, at the default extent, is showing too much data and detail. You might improve this by zooming in closer, turning off the layer, or optimizing the layer (e.g. missing index, simply the geometry).

"Finally (though I have a lot more questions I’ll end here to avoid too long a post), in the Summary tab, no matter how far back I set my query in the Parser tool, the Data TimeSpan field seems to max out at a certain distance into the past. Is there an action that resets the .log file so that I can only see back so far? Restarting the ArcGIS Server instance or rebooting the server, for example?"

Running System Log Parser from the command line can yield more options to run the query for a longer duration of time. It can go really far back...though by default, ArcGIS Server only keeps logs for 90 days. If the Data TimeSpan field appears to max out it might be due to configuration of ArcGIS Server (this setting can be adjusted from Manager).
When creating a report against a long duration of time, it is highly recommended to select the Optimized Analysis Type and if available, use the ArcGIS Server Log Query (File System) for faster IO log reading.

Hope that helps.