Latest Contributions by JacobBoyle412

Amazon and Esri recently published a whitepaper outlining the steps needed to setup and configure Amazon AppStream 2.0 and ArcGIS Pro.  Through testing, Esri and AWS outline the various classes of AppStream hosts: ArcGIS 2D Workloads – stream.compute.large, stream.memory.large. Compute and Memory optimized instances are perfectly suited for ArcGIS Pro workloads that does not require a GPU. ArcGIS 3D Workloads (Normal) – stream.graphics-design.xlarge. Graphics Design instances are ideal for delivering applications such as ArcGIS Pro that rely on hardware acceleration of DirectX, OpenGL, or OpenCL. Powered by AMD FirePro S7150x2 Server GPUs and equipped with AMD Multi user GPU technology, instances start from 2 vCPU, 7.5 GiB system memory, and 1 GiB graphics memory, to 16 vCPUs, 61 GiB system memory, and 8 GiB graphics memory. ArcGIS 3D Workloads (High res) – stream.graphics-design.2xlarge or stream.graphics-pro.4xlarge. The Graphics Pro instance family offers three different instance types to support the most demanding graphics applications. Powered by NVIDIA Tesla M60 GPUs with 2048 parallel 4 processing cores, there are three Graphics Pro instances types starting from 16 vCPUs, 122 GiB system memory, and 8 GiB graphics memory, to 64 vCPUs, 488 GiB system memory, and 32 GiB graphics memory. These instance types are ideal for graphic workloads that need a massive amount of parallel processing power for 3D rendering, visualization, and video encoding, including applications such as ArcGIS Pro. Please find the full whitepaper here: https://d1.awsstatic.com/product-marketing/AppStream2.0/Amazon%20AppStream%202.0%20ESRI%20ArcGIS%20Pro%20Deployment%20Gu…    ... View more

What is System Log Parser? System Log Parser is an ArcGIS for Server (10.1+) log query and analyzer tool to help you quickly quantify the "GIS" in your deployment. When run, it connects to an ArcGIS for Server instance on port 6080/6443/443 as a publisher (or an administrator), retrieves the logs from a time duration (specified as an input), analyzes the information then produces a spreadsheet version of the data that summarizes the service statistics. ArcGIS Monitor uses the command-line version of System Log Parser (slp.exe) for data capture. System Log Parser supports the following service types: Feature Services Geoprocessing Services Network Analyst Services Geocode Services KML Services Stream Services GeoData Services Map Services Workflow Manager Services Geometry Services Image Services   Globe Services Mobile Services   System Log Parser (github.com), a free-standing application or Add-on for ArcGIS Monitor, is an effective tool for diagnosing and reviewing infrastructure functionality. Getting Started This section will configure ArcGIS Server to collect logs at the level needed for the tool and set up System Log Parser to generate a report (MS Excel). 1.   Ensure the following conditions are met on the machine you’ll be running System Log Parser from: 64-bit Operating System: Windows 7 (64 bit), Windows 8.x, Windows 10 Windows Server 2008 64 bit, Windows Server 2012, Windows Server 2016 RAM: 4 GB Microsoft .NET Framework 4.5 or 4.6 Microsoft Excel 2010 or newer (or appropriate .xlsx viewer). 2.   Set your ArcGIS Server logs to Fine on EACH server you’d like to get metrics on. Complete instructions on how to change ArcGIS Server log levels can be found here:  Specify Server Log Settings Note:    I recommend running the logging at FINE for AT LEAST one week prior to running System Log              Parser. This should give you a fairly clear picture of a typical weeks load. 3.   Download and extract System Log Parser here: github.com 4.   Extract the .zip file. Note:    This is BOTH the user interface and the Add-on for ArcGIS Monitor.  We will be focused on the user interface version for this exercise. 5.   Launch System Log Parser 6.   Browse to the location you extracted System Log Parser 7.   In the System Log Parser for ArcGIS folder, locate and launch SystemLogsGUI.exe Note:    You may be prompted that Windows has protected your PC.  If you do get this prompt, please click              More info and then click Run Anyway. Configuring System Log Parser The following outlines the configuration required to set up System Log Parser to analyze a week's worth of logs. Note:    The System Log Parser will automatically access logging for all clusters that are part of an ArcGIS              Server Site. If you have multiple ArcGIS Server Sites configured Click the ArcGIS Server (Web) button to display the following: Fill out the above form as indicated below: 1.   Enter the Server URL. The typical syntax with ArcGIS Server 10.2 or higher is: https://<host_name>:<port_number>/arcgis   The typical syntax with ArcGIS Server 10.1 is: https://<host_name>:<port_number>/ArcGIS Note:    If your URL structure is different, enter it. 2.   Enter the ArcGIS Server Manager user name with a publisher or better permissions.  3.   Enter the user's password 4.   Check this box if you are accessing a Site federated to Portal for ArcGIS Note:   Consider using a web adapter address for the Server URL:  https://<webadaptor_name>/server Note:   If accessing over the internet, this assumes that the web adapter was registered with administrative access to ArcGIS Server 5.   Check this box if you use IWA(Integrated Windows Authentication) 6.   If needed, specify a token(advanced option) 7.   Select an End Time (Now) 8.   Select Start Time (1 week) 9.   Select Analysis Type (Complete) Simple: Provides only the Service Summary page data.  Note: This mode will also generate a list of the underlying data source by service and layer in the service.  WithOverviewCharts: Provides the Service Summary page plus charts of Request Count, Average Request Response Time, Instance Creation Time, Wait Time (Queue Time), and Max Request Response Time. Complete: Provides a Service Summary page plus all data and charts in separate tabs for all services. ErrorsOnly: Provides a report of just the errors. VerboseMode: Provides full verbose log analysis (Limited to 12 hours). 10.   Select Report Type (Spreadsheet) 11.   Specify where to output the report (Default is your My Documents location) Click Analyze Logs.  This process can take a few minutes or longer; this all depends on the number of transactions logged. Review the System Log Parser report   When System Log Parser finishes running, it will open the report in Excel if present.  If you run this from a machine without Microsoft Excel, move it to a computer with Excel and open it. You will note a summary tab and several tabs listed across the bottom of the spreadsheet.  We'll cover each in further detail below, by tab. Summary When the Excel report opens, you will see the Summary tab. The below screen grab shows what server this was run against and some summary statistics. Statistics On the bottom of the Excel report, select the Statistics tab to view a table of all services by layer and service types. This is where we'll spend most of our time.  Please read the rest of this post, then click here. Resources On the bottom of the Excel report, select the Resources tab to view several charts: Top 20 Resources by Count Top 20 Resources by Average Response Time Top 20 Resources by Maximum Response Time Methods On the bottom of the Excel report, select the Methods tab to view several charts: Top 20 Methods by Count Top 20 Methods by Average Response Time Top 20 Methods by Maximum Response Time Queue Time At the bottom of the Excel report, select the Queue Time tab to view any services that had to wait for an ArcSOC to return a result. In an ideal setting the below is the desired value: Users At the bottom of the Excel report, select the Users tab to view a chart of the top 20 users by request count. Time At the bottom of the Excel report, selects the Time tab to view a chart of requests by day. Throughput per Minute At the bottom of the Excel report, select the Throughput per Minute tab to few a minute-by-minute breakdown of requests. Below is a sample of what information can be found on the tab: Elapsed Time of All Resources At the bottom of the Excel report, select the Elapsed Time of All Resources tab to view a chronological listing of all requests from the time period the System Log Parser report was generated. I'd also like to thank Aaron Lopez‌ for his help and continued development of this invaluable tool.  Note: The contents presented above are recommendations that will typically improve performance for many scenarios. However, in some cases, these recommendations may not produce better performance results, in which case, additional performance testing and system configuration modifications may be needed. I hope you find this helpful, do not hesitate to post your questions here: ArcGIS Architecture Series: Tools of an Architect. ... View more

What is System Log Parser? System Log Parser is an ArcGIS for Server (10.1+) log query and analyzer tool to help you quickly quantify the "GIS" in your deployment. When run, it connects to an ArcGIS for Server instance on port 6080/6443/443 as a publisher (or an administrator), retrieves the logs from a time duration (specified as an input), analyzes the information then produces a spreadsheet version of the data that summarizes the service statistics. The command-line version of System Log Parser (slp.exe) is used by ArcGIS Monitor for data capture.   Note:   This post is a second in a series on System Log Parser, please see ArcGIS Server Tuning and Optimization with System Log Parser to learn how to setup your server for System Log Parser and an overview of the report.   Introduction to Statistics Used In System Log Parser   There are several statistical categories you should be familiar with when using System Log Parser. (definitions from Wikipedia)   Percentile (P) - a measure used in statistics indicating the value below which a given percentage of observations in a group of observations falls. For example, the 20th percentile is the value (or score) below which 20% of the observations may be found.    Average (avg) -   is a single number taken as representative of a list of numbers. Different concepts of average are used in different contexts. Often "average" refers to the arithmetic mean, the sum of the numbers divided by how many numbers are being averaged. In statistics, mean, median, and mode are all known as measures of central tendency, and in colloquial usage, any of these might be called an average value.    Maximum (Max) - the largest value of the function within a given range.   Minimum (Min) - the smallest value of the function within a given range.   Standard Deviation (Stdev) -    measure that is used to quantify the amount of variation or dispersion of a set of data values. A low standard deviation indicates that the data points tend to be close to the mean (also called the expected value) of the set, while a high standard deviation indicates that the data points are spread out over a wider range of values.   Fields of the Statistics Collected   Field Definition Resource  Requested resource or service (Service REST endpoint) Capability  The ArcGIS capability of the resource Method  The function performed by the resource (What was accessed) Count The number of requests for this resource Count Pct  Count percentage based on total service requests Avg  The average time (in seconds) spent processing request Min The time (in seconds) of the shortest request P5, P25, P50, P75 The percentile grouping of the time (in seconds) P95 95% of all responses occur between 0 seconds and the value displayed in this column per service P99 99% of all responses occur between 0 seconds and the value displayed in this column per service Max The time (in seconds) of the longest request Stdev The standard deviation of time (in seconds) Sum The total time (in seconds) spent processing requests per resource Sum Pct The total time (in seconds) spent processing requests   We're going to focus on 2 key statistics, P95 and Max.  As we learned above, P95 signifies the response time for the fastest 95% of all requests and Max signifies the maximum draw time per request per service and method.   Identifying Opportunities to Tune Service Performance   In the example below, I've sorted P95 and Max values over 1/2 second.  User experience drops the longer your draw-time takes.    I've highlighted any Max draw time over 1/2 second in red and any P95 draw time over 1/2 second in yellow.  These are the services and layers I'd focus on cleaning up, focusing first on getting the P95 value below 1/2 second first.  In the next section you'll find starting points to tune and optimize your services.   Another column worth reviewing is the Sum Pct.  this column factors in the number of requests for each service and the respective average time, then weights that in against all the other services.     For example:    One service may have thousands of more requests than all others but it has fast times (Sum Pct should be low) Another service may have just a small handful of requests but very slow times (Sum Pct should be high). In this case, this service would be a good candidate to for tuning.   Best Practices for Services   Below are some links to get you started on service tuning and SOC management. Tune and Configure Services Tune Services Using Best Practices Shared and Dedicated Instances Server Software Performance (GIS Wiki) ArcGIS Enterprise: Performance and Scalability Best Practices           In addition to the above, data source performance should be looked at if adjustments to the service do not help enough. You can look at: Geodatabase Design Steps  Increasing the Frequency of Updating Statistics on Tables Rebuilding Index Tables Proper Parent-Child Version Design Database Compression Monitor the Database   I hope you find this helpful, do not hesitate to post your questions here: https://community.esri.com/thread/231451-arcgis-architecture-series-tools-of-an-architect   Note: The contents presented above are recommendations that will typically improve performance for many scenarios. However, in some cases, these recommendations may not produce better performance results, in which case, additional performance testing and system configuration modifications may be needed. ... View more