System Log Parser - Statistics and Service Optimization

[JacobBoyle412]

• What is System Log Parser?
• Introduction to Statistics Used In System Log Parser
• Fields of the Statistics Collected
• Identifying Opportunities to Tune Service Performance
• Best Practices for Services

What is System Log Parser?

System Log Parser is an ArcGIS for Server (10.1+) log query and analyzer tool to help you quickly quantify the "GIS" in your deployment. When run, it connects to an ArcGIS for Server instance on port 6080/6443/443 as a publisher (or an administrator), retrieves the logs from a time duration (specified as an input), analyzes the information then produces a spreadsheet version of the data that summarizes the service statistics. The command-line version of System Log Parser (slp.exe) is used by ArcGIS Monitor for data capture.

Note:   This post is a second in a series on System Log Parser, please see ArcGIS Server Tuning and Optimization with System Log Parser to learn how to setup your server for System Log Parser and an overview of the report.

 

Introduction to Statistics Used In System Log Parser

There are several statistical categories you should be familiar with when using System Log Parser. (definitions from Wikipedia)

Percentile (P) - a measure used in statistics indicating the value below which a given percentage of observations in a group of observations falls. For example, the 20th percentile is the value (or score) below which 20% of the observations may be found. 

Average (avg) -   is a single number taken as representative of a list of numbers. Different concepts of average are used in different contexts. Often "average" refers to the arithmetic mean, the sum of the numbers divided by how many numbers are being averaged. In statistics, mean, median, and mode are all known as measures of central tendency, and in colloquial usage, any of these might be called an average value. 

Maximum (Max) - the largest value of the function within a given range.

Minimum (Min) - the smallest value of the function within a given range.

Standard Deviation (Stdev) -    measure that is used to quantify the amount of variation or dispersion of a set of data values. A l...

Fields of the Statistics Collected

Field
Definition

Resource	Requested resource or service (Service REST endpoint)
Capability	The ArcGIS capability of the resource
Method	The function performed by the resource (What was accessed)
Count	The number of requests for this resource
Count Pct	Count percentage based on total service requests
Avg	The average time (in seconds) spent processing request
Min	The time (in seconds) of the shortest request
P5, P25, P50, P75	The percentile grouping of the time (in seconds)
P95	95% of all responses occur between 0 seconds and the value displayed in this column per service
P99	99% of all responses occur between 0 seconds and the value displayed in this column per service
Max	The time (in seconds) of the longest request
Stdev	The standard deviation of time (in seconds)
Sum	The total time (in seconds) spent processing requests per resource
Sum Pct	The total time (in seconds) spent processing requests

We're going to focus on 2 key statistics, P95 and Max.  As we learned above, P95 signifies the response time for the fastest 95% of all requests and Max signifies the maximum draw time per request per service and method.

Identifying Opportunities to Tune Service Performance

In the example below, I've sorted P95 and Max values over 1/2 second.  User experience drops the longer your draw-time takes. 

I've highlighted any Max draw time over 1/2 second in red and any P95 draw time over 1/2 second in yellow.  These are the services and layers I'd focus on cleaning up, focusing first on getting the P95 value below 1/2 second first. 

In the next section you'll find starting points to tune and optimize your services.

Another column worth reviewing is the Sum Pct.  this column factors in the number of requests for each service and the respective average time, then weights that in against all the other services.

For example:   

1. One service may have thousands of more requests than all others but it has fast times (Sum Pct should be low)
2. Another service may have just a small handful of requests but very slow times (Sum Pct should be high). In this case, this service would be a good candidate to for tuning.

Best Practices for Services

Below are some links to get you started on service tuning and SOC management.

• Tune and Configure Services
• Tune Services Using Best Practices
• Shared and Dedicated Instances
• Server Software Performance (GIS Wiki)
• ArcGIS Enterprise: Performance and Scalability Best Practices

In addition to the above, data source performance should be looked at if adjustments to the service do not help enough. You can look at:

• Geodatabase Design Steps 
• Increasing the Frequency of Updating Statistics on Tables
• Rebuilding Index Tables
• Proper Parent-Child Version Design
• Database Compression
• Monitor the Database

I hope you find this helpful, do not hesitate to post your questions here: https://community.esri.com/thread/231451-arcgis-architecture-series-tools-of-an-architect

Note: The contents presented above are recommendations that will typically improve performance for many scenarios. However, in some cases, these recommendations may not produce better performance results, in which case, additional performance testing and system configuration modifications may be needed.

[MehdiPira1]

Very useful! Thanks @JacobBoyle412 

[Anonymous User]

Exactly what we needed!!! Thank you @JacobBoyle412 !

[NickMiller2]

Does anyone have a sample bat file that schedules the running of System Log Parser that they can share?

[ZachBodenner]

What does it mean when a Username in a system log parser output is given as "Anonymous User?" At first I though it was just an indicator that a publicly available service was accessed by...a member of the public. But in the report I ran recently I have Anonymous User listed as an accessor for internal services.

[AaronLopez]

> Does anyone have a sample bat file that schedules the running of System Log Parser that they can share?
Hello @NickMiller2, 

What is the log source you are wanting to parse with SLP? ...I'll list examples for using several of the common ones. Select one of the slp.exe commands below and arguments and put them into a bat file.

ArcGIS Server Log Query (File System)
slp.exe -f AGSFS -i \\myserver.domain.com\c$\arcgis\arcgisserver\logs\MYSERVER.DOMAIN.COM -eh now -sh 30day -a optimized -validate true

ArcGIS Server Log Query (Web)
slp.exe -f AGS -s https://myserver.domain.com/server -u gisadmin -p Myp@ssword -eh now -sh 7day -a optimized -validate true

Internet Information Services Log Query
slp.exe -f IIS -i \\myserver.domain.com\c$\inetpub\logs\LogFiles\W3SVC1 -eh now -sh 30day -a optimized -validate true

All of the examples commands above use the Optimized report which is the recommended Analysis Type. The Optimized report has tremendous memory savings over the other analysis types, especially if you are reading 30days worth of logs. If you are wanting to query ArcGIS Server logs but can only use the web, reading 30day might be difficult (due to the nature of the REST Admin API that is used behind-the-scenes). 

To assist in troubleshooting, the "-validate true" is passed into the slp.exe command in print any potential errors to the console that might be encountered. Otherwise, slp.exe runs "silently".

For in-depth troubleshooting, you can optionally pass in "-apploglevel DEBUG". This will create a unique run log of the slp.exe execution...with alot of detail. The log file is typically found in:
C:\Users\gisadmin\Documents\System Log Parser\Logs\Application_agsfsx2_20220702T220628_k6a7xrhk.log

Hope this helps.

[AaronLopez]

> What does it mean when a Username in a system log parser output is given as "Anonymous User?"
> At first I though it was just an indicator that a publicly available service was accessed by...a member
> of the public. But in the report I ran recently I have Anonymous User listed as an accessor for
> internal services.
Hi ZachBodenner,

If an ArcGIS Enterprise internal service is shared to Everyone (public) and is accessed without the member actually logging in, the user entry in the log is typically just anonymous. Other log entries (which SLP might use based on the analysis being performed, like wait time, instance creation time, arrival time, etc...) may alternatively list, Anonymous user. But for case of the "Statistics By User" worksheet, I usually just see anonymous listed for non-authenticated entries. 

There also some one-off user entries that I cannot directly explain...other than SLP is just reporting on what is in the logs. For example, some user entries are listed as: 0123456789abcdef::[username] or null. The alpha-numeric listing in front of the user account is not seen often and I believe is just observed with a few services. The 'null' entry is another version of anonymous...but isolated to just one or two particular services. 

Hope that helps.

[AaronLopez]

The following is an additional resource which may help provide information on the "anonymous" entries observed in the ArcGIS Enterprise (e.g., ArcGIS Server) logs:

ArcGIS Enterprise Analysis with System Log Parser: Understanding Anonymous Entries for the User Name...