ArcGIS Enterprise Analysis with System Log Parser's Optimized Analysis Type (Beginner)

[AaronLopez]

System Log Parser's Optimized Analysis Type

Although you are probably familiar with using System Log Parser (SLP) to read logs and help you quantify your ArcGIS Enterprise usage, there is a relatively new feature in this popular, free utility that can make the effort easier and the analysis more powerful.  

This feature is the "Optimized" report option which is a new parameter in the GUI listed under Analysis Type. For log queries such as ArcGIS Server (Web), ArcGIS Server (File System), Internet Information Services (IIS) and Elastic Load Balancer (AWS), it is now the default. This option is not yet available for Cloud Front and Azure log queries. For backward compatibility, Simple, WithOverviewCharts and Complete types are still an option, but for the best analysis experience, they are not recommended.

• The Optimized Analysis Type option shown in the GUI:

A System Log Parser spreadsheet report generated using the Optimized option:

Also Available From the Command-Line

The GUI (SystemLogsGUI.exe) is convenient for creating reports but the command line version (slp.exe) is great for automating System Log Parser analysis. The Optimized report option is available from both!

Optimized Report Origins

Over the years, one of the most requested enhancements for SLP has been to improve the speed and memory usage (of the machine running it) when parsing logs for very busy Sites or large time spans (e.g. several weeks or more). The Optimized report directly addresses these two items. Performance is measurably faster and memory orders of magnitude lower. With the ability to search greater spans of time with a quicker executions, more powerful statistics can be performed as the report can analyze more requests.

For the best results, use the Optimized report with ArcGIS Server (File System), Internet Information Services (IIS) or Elastic Load Balancer (AWS) log queries. Memory usage savings can still be obtained with ArcGIS Server (Web) log queries, but the act of retrieving large amounts of logs through the REST Admin API is still the performance bottleneck.

Optimized Report Contents

System Log Parser reports focus on grouping the values from the logs into key time-based categories. While the Optimized report for the different log sources (ArcGIS Server, IIS, or ELB) has some commonality, there can be additional statistics depending on which was utilized. For example, if the Optimized report was used for querying ArcGIS Server (Web) or ArcGIS Server (File System) logs, wait time, instance creation time, and arrival time will each be statistically broken down in additional to elapsed time. 

System Log Parser Support

For support contact SystemTestTool@esri.com

Note: System Log Parser is developed from Esri's Professional Services but is not a product of Esri.  

Latest Version

Bug fixes and new features are always being added to SLP.

The latest version can be found here: System Log Parser (0.12.17.0) 

[notebooker]

Hi Aaron,

is it possible to let the application (GUI version) persist Log Location and Report Folder? Every time I start the app, it has default values.

Thank you. 

[AaronLopez]

Hello @notebooker,
In the System Log Parser GUI it is not possible to persist those values.

However, everything you can do in the GUI you can also accomplish in the command-line version, slp.exe.
With PowerShell (or bat) scripts, you could persist the necessary inputs and make the execution easier.

Hope that helps.

Aaron

[WenfengLi]

I generated reports successfully. However most tables are empty (such as users, waiting time, statistics.....). I have the ArcGIS log level set to INFO. Is it the problem? Should set to to higher level of logging? 

[ghaskett_aaic]

Hello.

Do we know if there are any compatibility issues with this tool and the ArcGIS Enterprise 11.2 software?

I recently set this up and ran it on one of our servers, however it doesn't look like it is working properly.

It completed and shows a start time and end time for the query and timing for the Start/End Time for Data (Coordinated Universal Time), but it doesn't show a Start/End Time for Data (Central Standard Time), plus the Data TimeSpan value is 0 along with a lot of the other values from Unique Resources Requested down.

Am I not filling something out right or is there an issue with 11.2?

Thanks,

[AaronLopez]

Hi @ghaskett_aaic,
My guess is that your 11.2 Site and System Log Parser are fine.
There are two conditions that exist that can lead to the situation of the generated report listing 0 hits.

• The LogLevel of ArcGIS Server is set to a value of either Warning, Error or Info
  • The statistics in the System Log Parser report are based on elapsed time request entries, but this requires the LogLevel to be set to Fine
    
  • Verbose and Debug work too but they not recommended for Production sites due to how much detail is recorded
    
• There were no requests to the Site during the specified log query duration (e.g.,  2024-03-01T10:21:57 through 2024-03-05T10:21:57)
  
  • For a quick check, authenticate to the Site from the REST service endpoint, select a service, pan or zoom around a few times, then generated the report with an end time that includes those requests (e.g., end time = now)
  • This report should include your requests that were just made to the Site
    

Hope that helps
Aaron

[AnjulPandey]

@AaronLopezI would appreciate it if you could suggest the following -

• Is there a way to obtain usage statistics of vector tile service published on ArcGIS Server?
• Has anyone created a script or notebook to collect a list of all the vector tile packages with their storage size and usage statistics?
• Logs for map and vector tiles generated by publishing services on ArcGIS Server?
• Logs for how many features were edited by the user?
• Logs of how many vector or raster tiles were downloaded by the user.

[AaronLopez]

Hi @AnjulPandey,

• Is there a way to obtain usage statistics of vector tile service?
  There might be some usage statistics through the ArcGIS Web Adaptor access logs (IIS log), if it is available.
  The information here would be counts (how many times was a vector tile service requested) and performance (how long were users waiting to down load the tiles). System Log Parser should be able to address this the the Internet Information Services Log Query capability.
• Has anyone created a script or notebook to collect a list of all the vector tile packages with their storage size?
  Not that I am aware of but I have not specifically looked for tools that address that functionality.
  
• Logs for map and vector tiles generated by publishing services?
  Is this similar to the first item? Are you after usage statistics or publishing statistics.
  If so, System Log Parser would summarize requests for such data (usage and publishing) from the ArcGIS Web Adaptor access log. 
  But it is worth pointing out that for publishing statistics, only the job's GUID would be listed in the report.
• Logs for how many features were edited by the user?
  This is interesting. Would this be hosted feature services, traditional or both?
  For traditional feature services, this might involve queries to the enterprise geodata to understand what, if anything has changed since the check.
  Although I am not sure if any observability or reporting tools do this, I have not looked.
  Assuming the edited data was through the applyEdits function of a feature service, System Log Parser would list if that method was called. Of course, this does not mean the data was changed...just that the applyEdits function was called.
• Logs of how many vector or raster tiles were downloaded by the user?
  Map, Vector and Raster tile requests through a web adaptor are reported like any other request. System Log Parser would be able to report on usage statistics (e.g., counts and performance) from the ArcGIS Web Adaptor access logs. But, these log sources do not typically record the ArcGIS User that made the request. So the "who" would not be available.
  

Hope that helps
Aaron