BLOG
|
Hi @ChiefKeefSosa300 (tq), These are really thoughtful questions, thank for asking them! The statements about changing the output format to BMP or utilizing another the spatial reference are mentioned with respect to the that specific test. But they need more context... I favor PNGs over BMPs as the former (Portable Network Graphics) is generally much faster to generate than the latter (Bitmap Image). For example, an export map request that creates a PNG is about 87KB. Using a BMP for the same area, the resulting image is around 15MB. The drastic difference in size would impact scalability (as more network bandwidth is required per request). That said, if your application or workflow requires BMP, then that is the way to go and your benchmark should reflect that. There is nothing wrong with using BMP as an image format for a vector data benchmark, but you should get more throughput by using PNG. As for the spatial reference, this test used bounding boxes in a geographic coordinate system of "WGS 1984" (WKID 4326) because that is what the data (and published service) was originally in. Of course, there are many different spatial references. The goal when possible, is to try to avoid projecting-on-the-fly. In other words, if the data (and service) are in "WGS 1984" and you ask for "NAD27" (using the appropriate NAD27 bounding box), ArcGIS Server should appropriately fulfil the request. However, while very convenient, this transformation between coordinate systems comes at a performance cost. If you use this test and data as a benchmark, "WGS 1984" (WKID 4326) are an optimal choice. But, if you are wanting to build a benchmark test against your data that is in another coordinate system, I would use bounding boxes in that coordinate system to ensure you can get the best performance. So to conclude, does using BMP or another coordinate system fail the test, not at all. Performance would be impacted, but you still have a good "measuring stick". As long as the test parameters are known and kept constant between runs over time, you'll have a reliable benchmark regardless of the options used. Hope this helps. Aaron
... View more
10-04-2023
05:45 PM
|
1
|
0
|
700
|
BLOG
|
Hi @Jen_Zumbado-Hannibal, Both tools are similar but also different. The 10.8.1 python script listed on the arcgis.com page, focuses on retrieving usage statistics (total number of requests, maximum and average response time, and total timed-out requests). The soccer captures some of that info, but the primary purpose of it is to help the GIS analyst or administrator understand the optimal instance configuration of a service. In other words, the max instance of a dedicated service was set to 12...but was the system able to reach this maximum for the duration of interest? If the captured busy instances data constantly matches or is drastically less than the maximum you can then choose a more optimal configuration to improve scalability or save memory. Hope that helps. Aaron
... View more
09-26-2023
07:15 PM
|
1
|
0
|
2176
|
BLOG
|
Hi @ChiefKeefSosa300 (tq), Would you be able to delete one of the "image/png Validation" elements and try playing the test again? These validations items are looking to see if the content returned from ArcGIS Server looks like a png image (this is the default image format this test requests). If it does not match a png signature, it fails the request. Failing a request whose response is not expected is a common strategy when you want to know if your test encounters an error. Aaron
... View more
09-26-2023
06:56 PM
|
0
|
0
|
736
|
BLOG
|
Hi @ChiefKeefSosa300, I was able to test the portal_administration1 JMeter project against a 11.0 deployment and it worked. Can you confirm that the value for the "PortalServerName" only contains the hostname of the machine running Portal for ArcGIS (e.g., portalserver.domain.com)? It should not contain any protocol or spaces after (e.g., http://portalserver.domain.com ). Also, can you verify that you can connect to the portal through a web browser on the same machine running the test? This can be through 7443 or 443. Don't forget to use the web adaptor's portal instance name if connecting to 443 (e.g., portal). Thanks. Aaron
... View more
09-08-2023
11:30 AM
|
1
|
0
|
500
|
BLOG
|
Hi @ChiefKeefSosa300, I have tried this test to create members with ArcGIS Enterprise 10.9, 10.9.1 and 11.1. I do not recall the results with 11.0 specifically. Did you encounter an issue? Thanks. Aaron
... View more
08-23-2023
10:17 PM
|
0
|
0
|
547
|
BLOG
|
Hi @ErickTGG, Any observability tool could be used to capture operation data like CPU and memory usage from a performance/load test. And, if these reports and filters already exist and can be consumed that is definitely a good strategy. One thing to keep in mind is that by default, many monitoring tools capture metrics for long term analysis. A common polling rate is every 1 minute or 5 minutes since they are expecting to store a lot of data (and this frequency can help with the manageability and scalability of that system). As such, that "lower" resolution is generally not ideal for examining the impact of a load test. A typical load test runs for 1-- 2 hours. For such a short duration, higher capture intervals like 5, 10, or 20 seconds, are usually favored so a more detailed impact on the deployment's resources can be seen. This might be adjustable with most monitoring tools...not sure. In cases where this higher resolution is not an option or maybe there is no monitoring tool even available, having the testing framework capture this type of data can be extremely valuable for the analysis. Hope that helps. Aaron
... View more
06-21-2023
04:50 PM
|
0
|
0
|
1463
|
BLOG
|
ArcSOC Availability and Utilization Optimizing the ArcSOC instance availability and utilization for your service is a good strategy for helping users obtain fast response times and lower wait times from their dynamic requests to your Site. It can also benefit server resource utilization like memory as the service is not running a lot of instances that it will never use. But optimizing the minimum and maximum number of instances for your dedicated services is not a one-time job. Usage patterns of your services can change over time so the task of collecting this information is something that you will want and to revisit periodically as a GIS administrator. Before diving into how to observe ArcSOC instance activity statistics, let’s review some of the key details of the two ArcSOC-based service types in ArcGIS Server and how they play into this discussion: Dedicated Shared Note: Wait time is the duration the request spends in a “queue” on the server until an ArcSOC instance is available to start working on it. Dedicated Instance Pool Services Dedicated services (e.g., non-hosted and non-shared services) are a mainstay ArcGIS resource in deployments as many applications depend on the capabilities such as geoprocessing, Branch Version editing, and Utility Network workflows (all which require dedicated services). While such services are very versatile and are a major pillar in ArcGIS because of the functionality they provide, as a GIS administrator you need to periodically examine, adjust, and configure the number of ArcSOC instances (minimum and maximum) to take full advantage of your available resources as your Site and users grow. To learn more about ArcSOC instances see: Understand service instances Limitations of Shared Instance Pool Services Shared service instances are great! They are truly a game changer for helping admins manage the demand for many services with finite resources. However, their restrictions and requirements limit which service capabilities can be used with them. Geoprocessing, Branch Version editing, and Utility Network, for example, are currently not supported through shared services (or through hosted Services). This leaves dedicated services as the only choice for such functionality. Configured ArcSOC Instance Availability vs Instance Demand For services that are very popular, critical and/or have the requirement of running under the dedicated service type, understanding the optimal instance setting is important for several reasons. If the maximum number of active instances is too high, memory is wasted (as well as cost). The over allocation of ArcSOC instances is an important but unique case as its impact would not show up in the analysis of just response times. Alternatively, having the maximum too low can impact performance (in the form of higher response times and longer wait times) as users might be frequently waiting for an already busy ArcSOC instance to become free. Of course, setting the instance minimum and maximum to different values has a trade-off too. For critical services where performance is paramount, having the user’s request wait while an instance needs to be started can be time consuming and will affect performance. So, for predictable performance on essential services, it is recommended to set the minimum and maximum number of instances to the same value. As listed on Introduction to service instances: Accordingly, it's important for ArcGIS Server administrators to monitor the number of instances their site is running, and to limit running instances when performance is inhibited by memory usage. Configuring the service’s availability (through its instance minimums and maximums) and the impact of those settings from the user’s demand on the service is key to an optimally running Site. There is a mutual relationship between configuring the service’s availability (through the instance minimums and maximums) and the direct affect that has to requests coming in to a dedicated service. While finding the optimal setting is an ongoing task, there are some tools and resources to help administrators tackle the challenge. ArcGIS Server Service Report The ArcGIS Server Service Report (introduced in 10.1) is one of those lesser-known REST Admin API gems. This resource can help monitor a Site by providing a configurable summary of all the services in a folder. It is Generally a fast-performing request (depending on the number of services in the folder being requested). The instance service statistics section of the returned response is extremely valuable as it lists details on the ArcSOC instances (min, max, busy) across the whole deployment (e.g., the ArcGIS Server Site). By periodically polling this endpoint, one can get up-to-the-second insight of the service instance configuration vs the demand…as it happens. With such information, better decisions can be made about optimizing machine and service resources. In turn, this can help improving response times and lower wait times. Note: In ArcGIS Server, instance service statistics information and the Statistics page in Manager are actually different resources though they are providing similar views of the same data. The service statistics provide raw access to the instance values (Site-wide and per machine) as well as more detail. The Statistics page is an interface for creating reports from some of that information. Automating the Service Report Collection with Soccer Any script, program or tool that regularly observes the Service Report endpoint of the folder of interest would suffice. However, if you are looking for a free, existing tool then Soccer is recommended. (Arc)SOC ScannER or Soccer is a utility for scanning and reading the services' statistics on a specific ArcGIS Server folder. It parses the collected data and writes it out to a CSV file for additional post-capture analysis (e.g., creating charts in a spreadsheet to visual the usage). It takes advantage of the REST Admin's Service Report resource in ArcGIS Server to gather this info. The original goal of soccer was to capture the report endpoint output of a specific folder in ArcGIS Server and save the ArcSOC instance statistics (e.g., Running, Busy, Maximum, etc...) for each service. Currently, soccer is a command-line only utility. It is made available in the .NET 6.0 portable runtime for Windows (win-x64), Linux (linux-x64) and macOS (osx-x64). For simplicity, running soccer only requires 3 inputs (other parameters can be passed in to extend functionality): soccer.exe -s "[https://ArcGISServer/ServerWebAdaptor]" -f [FolderToScan] -t "[PreGeneratedArcGISToken]" For example: soccer.exe -s "https://gisserver.domain.com/server" -f "Gas" -t "APLeyWOcKZp9stZ_C01DQ.." Note: A pre-generated ArcGIS Server token can be obtained from Portal. Typically, the generateToken URL is: https://gisserver.domain.com/portal/sharing/rest/generateToken and the Webapp URL would then be: https://gisserver.domain.com/server/admin. Set the Expiration value to something appropriate for your expected monitoring duration. Standard Out while running from a command-window: Connected to: "https://gisserver.domain.com/server " (Gas) Press Ctrl-C twice to stop... Sleeping 5 seconds... When running, soccer connects to the Service Report endpoint of the ArcGIS Server folder specified, collects the data, writes it to a local CSV file, then sleeps. After the sleep duration has elapsed, it repeats the process. Soccer will keep collecting until the process has been manually stopped (Ctrl-C). Note: To collect on the root ArcGIS Server folder, use either: -f “/” or -f “” Analyzing the CSV File Sample contents as seen from a simple text viewer: DateTime,Epoch,IntervalSeconds,Host,Folder,ServiceName,Type,Provider,Running,Busy,Maximum,Free,NotCreated,Initializing,Transactions,TotalBusyTime,ServicesCollected,ResponseTimeMilliseconds,ContentLength,ConfiguredState,RealTimeState,Message
5/2/2023 1:12:45 AM,1682989965310,5,gisserver.domain.com,Gas,Gas_Utility_Network,MapServer,ArcObjects11,32,0,32,32,0,0,0,0,2,121.6224,6065,STARTED,STARTED,success
5/2/2023 1:12:45 AM,1682989965310,5,gisserver.domain.com,Gas,Landbase_PostgreSQL,MapServer,ArcObjects11,32,0,32,32,0,0,0,0,2,121.6224,6065,STARTED,STARTED,success
5/2/2023 1:12:50 AM,1682989970469,10,gisserver.domain.com,Gas,Gas_Utility_Network,MapServer,ArcObjects11,32,0,32,32,0,0,0,0,2,117.901,6065,STARTED,STARTED,success
5/2/2023 1:12:50 AM,1682989970469,10,gisserver.domain.com,Gas,Landbase_PostgreSQL,MapServer,ArcObjects11,32,0,32,32,0,0,0,0,2,117.901,6065,STARTED,STARTED,success
5/2/2023 1:12:55 AM,1682989975606,15,gisserver.domain.com,Gas,Gas_Utility_Network,MapServer,ArcObjects11,32,0,32,32,0,0,0,0,2,109.6187,6065,STARTED,STARTED,success
5/2/2023 1:12:55 AM,1682989975606,15,gisserver.domain.com,Gas,Landbase_PostgreSQL,MapServer,ArcObjects11,32,0,32,32,0,0,0,0,2,109.6187,6065,STARTED,STARTED,success
5/2/2023 1:13:00 AM,1682989980733,20,gisserver.domain.com,Gas,Gas_Utility_Network,MapServer,ArcObjects11,32,0,32,32,0,0,0,0,2,125.1686,6065,STARTED,STARTED,success
5/2/2023 1:13:00 AM,1682989980733,20,gisserver.domain.com,Gas,Landbase_PostgreSQL,MapServer,ArcObjects11,32,0,32,32,0,0,0,0,2,125.1686,6065,STARTED,STARTED,success
5/2/2023 1:13:05 AM,1682989985874,25,gisserver.domain.com,Gas,Gas_Utility_Network,MapServer,ArcObjects11,32,0,32,32,0,0,0,0,2,116.3949,6065,STARTED,STARTED,success
5/2/2023 1:13:05 AM,1682989985874,25,gisserver.domain.com,Gas,Landbase_PostgreSQL,MapServer,ArcObjects11,32,0,32,32,0,0,0,0,2,116.3949,6065,STARTED,STARTED,success Note: By design, columns such as IntervalSeconds and ResponseTimeMilliseconds will show duplicate values if more than one service exists in the folder being observed. The collected data is a typical CSV, but there are some important fields which will be helpful for quickly analyzing the ArcSOC activity from out service(s) of interest: IntervalSeconds ServiceName Running Busy Maximum By opening the CSV file in a spreadsheet, other services residing in the same folder can be easily filtered out through the ServiceName column. The IntervalSeconds, Running, Busy, Maximum values can then be plotted to visualize (e.g., through the Scatter with Smooth Line chart ) and show the instance configuration versus incoming demand. In this case, the Gas_Utility_Network service was set to min/max instance configuration of 32/32. The "polished" chart below: Note: The Maximum and Running represent the ArcSOC service configuration instance maximum and minimum, respectively. In this case, Running has the same values and is plotted “behind” Maximum. Busy represents the number of instances that were active (due to requests from users) across all the machines in the Site. The “goal” of tuning and optimization in this case, would be to avoid the Busy values from constantly reaching the Maximum. If this were to happen, it means there was not enough ArcSOCs available for the service to meet the user demand as the instances were always busy. User requests would then most likely be encountering increased response times and wait times in the process. If the requests wait too long in the system, they timeout (typically after 60 seconds). Having a lot of service request timeouts would negatively impact the user experience. Based on the observations from this monitored duration, the rise and fall of the Busy column values did not come anywhere close to the maximum number of instances that were available…which from one perspective was good. However, this also indicated that there was a measurable number of instances running and taking up memory but not being used…which was not ideal. To optimize system resources going forward, the instance configuration of this service could have been set to a lower value that is closer to the (expected) peak usage (e.g., somewhere between 18 – 24). Use 18 to conserve memory with the potential of encountering several instances where user wait longer for their requests to be fulfilled Use 24 to favor performance over memory usage Final Thoughts Having an “optimized” service setting for the minimum and maximum number of instances does not guarantee that users will never encounter slow performance. User needs and habits change over time, so monitoring this information is something that will need to be done periodically. In the real-world, there are conditions where service wait times can still be encountered in a deployment…even with ample instances available and plenty of system resources. It is not realistic (or possible) to eliminate service wait time completely but is instead more practical to try to reduce it where possible. Optimizing the service instances is something admins can directly control that has an impact on wait times. Understanding and periodically evaluating the ArcSOC instance configuration (for dedicated services) and its relationship to the user demand is key to helping GIS administrators better plan and manager their Site by optimizing performance and efficiently utilizing resources.
... View more
05-17-2023
10:29 AM
|
11
|
4
|
4164
|
BLOG
|
Hello @notebooker, In the System Log Parser GUI it is not possible to persist those values. However, everything you can do in the GUI you can also accomplish in the command-line version, slp.exe. With PowerShell (or bat) scripts, you could persist the necessary inputs and make the execution easier. Hope that helps. Aaron
... View more
05-16-2023
04:36 PM
|
0
|
0
|
3843
|
BLOG
|
Hello @venugopalinfotech, > Does LR application supports GIS applications to do performance testing. Thanks to ArcGIS Server (and ArcGIS Enterprise), the majority of GIS functions take place via HTTPS. This is were most modern testing frameworks (e.g., JMeter, Load Runner) can be used to "playback" web traffic to exercise your workflow of interest. > If LR supports GIS applications, what are the steps that needs to be taken into consideration while recording the scripts. Keep in mind that testing an application involves a different set of strategies and techniques that testing services. Recording application workflow traffic from a web browser and converting that traffic into test framework logic varies from tool to tool. But it is still very doable with most modern frameworks. Test services (e.g., map, feature, geoprocessing, etc...) is more atomic as you are just focusing on one service at a time. To get the most value out of this effort typically requires a wider range of test input data (e.g. AOI extents) that exercise the service to a greater degree. There are several strategies for doing this with JMeter here: Performance Engineering: Load Testing ArcGIS Enterprise > I have found that just for ‘Login and Logout’, the application has generated around 5000 lines of script code. Yes, but I would guess that the high majority of these lines might be to support requests for static items (e.g., css, png, js files). While overall these static items are what the user's browser is issuing, as the tester, you can probably reduce this to a handful of dynamic calls that are responsible for the login and logout logic. > Is there any performance tool that supports GIS (WebAppBuilder, ESRI -ArcGIS) applications apart from LR. JMeter does this (tests HTTPS) pretty well and its free! I recommend looking at some of our Community Articles on testing ArcGIS with JMeter: Performance Engineering: Load Testing ArcGIS Enterprise From what I understand, Load Runner is a good tool too for testing HTTPS. Hope this helps. Aaron
... View more
01-25-2023
02:33 PM
|
2
|
0
|
4070
|
BLOG
|
Thank you to all the MVP's for your help in the Esri Community. You rock!
... View more
01-25-2023
02:04 PM
|
7
|
0
|
4846
|
BLOG
|
Hello Bernie, I just tried downloading the zip and was able to open it with the expected contents. Can you try download it from another computer and/or another web browser? @NoahMayer ...a good point about the "Save link as" option. I was not aware. @berniejconnors, If you click the SystemLogParser.zip link (from the latest page), and select Download, this should give you then full zip file.
... View more
10-27-2022
11:45 AM
|
0
|
0
|
1997
|
BLOG
|
Capturing Hardware Utilization During an Apache JMeter Load Test Measured transaction throughput and response times are critical data of any ArcGIS Enterprise load test, but captured hardware utilization of the deployment machines provide vital information as well. Together, these test artifacts allow for the proper analysis of the capabilities and efficiencies of the Site. Determining that an application or feature service has achieved a particular level of throughput is good, but confirming the scalability characteristics while also examining the captured processor utilization of the tested workload is even better. This Article will discuss several ways to capture the machine hardware utilization. This resource usage is a great compliment to the results of an Apache JMeter load test of ArcGIS Enterprise and can help further the analysis. The Article will focus on the most common scenarios using free tools and utilities for Windows and Linux. Capturing Strategies Unfortunately, there is no one size fits all for capturing hardware utilization in a load test. Given the potential for a multitude of different environments (Windows, Linux, Cloud, Kubernetes, Docker, etc...) or lack of required access and permissions, there may be cases where you will need to alter your strategy and methodology in order to capture and monitor the usage information from your hardware while the Apache JMeter tests run. For example, perfmon (e.g. Performance Monitor) is free and included with every instance of Microsoft Windows and can work great when all the machines in the environment are on Windows. It is the frontend for several Windows technologies (e.g. WMI, DCOM) and all of them together offer a versatile framework for monitoring hardware utilization in "real-time" with graphs or scripting a resource capture to a data file. However, perfmon is not available for Linux so another tool (e.g. dstat) would be needed to capture the equivalent information. Of course, even with an all Windows setup, challenges can still arise with the perfmon approach, such as with restrictive cloud environments. But, are there any other ways to record this valuable machine data? We'll explore the answer to that question later in this Article. What Information Should the Load Test Capture? When GIS work such as generating a map image or calculating a Geoprocessing service request is sent to an ArcGIS Enterprise deployment, the server machines that work on the response will utilize various hardware resources to complete the job. Ideally, most of the work happens in the central processor unit (CPU) but this is not always the case. Regardless of the technology used to monitor and capturing usage for your load test, there are four primary hardware counter categories that all server machines have: Processor Memory Network Disk Each category has a variety of metrics that can be used for the showing utilization of that resource. Its not necessary (or recommended) to capture every available processor metric during a test...just the vital ones. In other words, the ones needed to answer typical load test questions or the particular story you are after. Typically, some important metrics include the following: Processor Utilization percentage Memory Available (MB or bytes) Used (MB or bytes) Used percentage Network Send (MB or bytes) Received (MB or bytes) Disk Idle percentage Queue length Read (MB or bytes) Write (MB or bytes) Note: If your servers have multiple disks, it is highly recommend to collect on all of them but separately. In other words, treat the C drive, D drive and E drive as different resources when capturing using. It technically possible to have multiple active Network devices but it is not very common. The Performance Monitor interface collecting from various hardware metrics that have been added manually added (to perfmon): Note: Some methodologies for capturing the utilization will save the values in bytes or while other use megabytes. Either is fine, but it is important to understand which one is being captured for analysis and report presentation of the data. Note: Together with the transaction throughput (e.g. transactions/sec) and transaction response times (e.g. seconds), the metrics listed above make up the bulk of a load test's Key Performance Indicators (KPIs). KPIs help evaluate and benchmark the success of a particular test or effort against defined test goals (e.g. the Test Plan). Sample Interval Typically, a load test (with a primary focus on scalability) does not run more than an hour or two. Since that is considered a relatively short duration, it makes sense to capture the hardware utilization with a fast interval (e.g. short frequency) of 5 to 10 seconds. This helps ensure that enough data is obtained for performing a thorough analysis. If the capture interval is 1 minute or longer, some significant impacts to the hardware resources (from the test) might be missed. Note: Monitoring software may already be running on your network that is periodically collecting hardware utilization from your ArcGIS Enterprise servers. If any generated reports are available, they can be used for load test analysis as long as the capture interval is on a short frequency (e.g. 5 to 10 seconds). For load test analysis, you are interested in good metric detail for a few hours as opposed to broad metric coverage of a full day. The Difference Between Capturing and Monitoring The terms "capturing" and "monitoring" hardware utilization are often used interchangeably. However, for Apache JMeter testing purposes, this Article is focused on capturing the usage over monitoring it. The key difference here is the capturing means we are recording or saving the information for easier post-test analysis. Test Clients (Load Generator Machines) When capturing hardware utilization from a load test against ArcGIS Enterprise, it is a best testing practice to also measure usage from the Test Client machine(s). Understandably, most of the focus is put into observing the server resources, but the hardware from the test clients can be a bottleneck too. If these machines do not have enough capacity (e.g. processor, memory, network or disk) the ability to generate load against your deployment will be negatively impacted! Understand Your Server Baselines If you are not very familiar with the hardware of your ArcGIS Enterprise deployment, it is recommended to watch the resource utilization while the system is idle. This "usage" will help paint a mental picture of the best case scenario. In other words, what does the machine look like when the software and services are running but no one is using it. In such a scenario, resource activity from all four hardware categories should be minimal. A load tests does not need to record this information of the "idle state" but it is good to be aware of what it looks like. Note: Using tools like Task Manager (for Windows) or top (for Linux) are great tools for manually getting a quick view of system resource utilization. However, these are not ideal saving the usage from all the categories to a file. Task Manager's view of system resource utilization (Windows): top's view of system resource utilization (Linux): Common Utilization Capturing Examples Using Perfmon (To Capture Utilization Outside of Load Test) Assuming the required permissions are available for your user account in Windows, perfmon is one of the easiest ways to record hardware utilization for load testing analysis. Performance Monitor (and the supporting technologies) can capture the usage from the local machine or multiple remote servers, simultaneously. Note: As a reminder, the metrics available to perfmon within each hardware group are plentiful. There is also a large amount of software metrics that can be captured. But in this case, less is more. Only a small handful of counters are needed for most situations. As mentioned earlier, where you can run into considerations: The ArcGIS Enterprise deployment is running on Linux perfmon is not available for Linux The easiest workaround is to use another methodology to collect the information (discussed later) The deployment is in the cloud Running erfmon in the cloud to collect from other deployment machines (in the cloud) is technically possible but behind-the-scenes it uses networking protocols that are typically blocked in the environment The easiest workaround here is to run perfmon manually on each cloud machine and collect the generated outputs when the test has completed Launching perfmon in Windows is as simple as executing: Run --> perfmon From there you can add each machine as well as each respective hardware metric and counter. While this is very powerful, it can be time consuming and potentially prone to typos! Instead, the recommended strategy for this methodology is to use a PowerShell script where all of these items are predefined. The following will gather key perfmon metrics (that are already defined) from the four primary hardware categories and will do this for four different machines (that you can specify): # ==============================================================================
# Updated: 2022-08-02
# Filename: ps_get_all_continuous.ps1
# Version: 0.1.0
# Desc: Continuously collect various metrics and save (binary) values to a file
# Notes: It is recommended to start the metric collection 20 seconds before test starts
# as counter initialization can take some time
# ==============================================================================
Write-Output -InputObject 'Collecting all metrics (continuous)...'
# Variables
# Windows Perfmon Counter(s)
$counters = "Processor(_Total)\% Processor Time","Memory\Available MBytes","Memory\Committed Bytes","Memory\% Committed Bytes In Use","Network Interface(*)\Bytes Received/sec","Network Interface(*)\Bytes Sent/sec","LogicalDisk(_Total)\% Idle Time","LogicalDisk(_Total)\Disk Read Bytes/sec","LogicalDisk(_Total)\Disk Write Bytes/sec"
# Machines to collect from
$computers = @('server1.domain.com','server2.domain.com','server3.domain.com','server4.domain.com')
# Time in seconds between each sample collection
$sampleinterval = 10
$date = (Get-Date)
$dyndate = '{0:yyyyMMddTHHmmss}' -f $date # 20220303T140044
# File name
$filename = "perfmon_all_continuous"
# Output file type (choices are blg, csv, or tsv)
$outputfiletype = "blg"
# Output file
$outputfile = '{0}_{1}.{2}' -f $filename, $dyndate, $outputfiletype
# Get-counter
Write-Output -InputObject ('Start DateTime: {0}' -f $date)
Get-counter -Computername $computers -Counter $counters -Continuous -Sampleinterval $sampleinterval | Export-counter -Force -FileFormat $outputfiletype -Path $outputfile The metrics are collected every 10 seconds until you hit Ctrl-C in the PowerShell window. The information is saved in a binary formatted file that you can open in Windows (for further analysis or transformation): Note: It is recommended to start this Powershell script about 20 or 30 seconds before your start the load test as counter initialization is not immediate and takes a moment or two. You can open the *.blg file at any time during the capture, but the latest information is not flushed and saved to the file until you hit Ctrl-C from the PowerShell window where the script is running. Double-clicking and opening the *.blg file should look similar to the following: The end result is an output that looks very similar to what we captured. However, in this case, the process has been automated (which is a good thing). From this interface in Performance Monitor, you can select/deselect metrics for specific isolation, export the graph as an image, save the raw data to another format like CSV, or conduct usage analysis from all of the collected machines. Pretty powerful! Note: Performance Monitor scripts do not directly integrate into JMeter. Note: For cloud deployments, you may have to run the perfmon script on each respective server manually. Using dstat (To Capture Utilization Outside of Load Test) dstat is a free command line tool (written by dag@wieers.com) available for (or with) most Linux distributions that captures (and monitors) hardware utilization. Although the utility has not been updated for several years, it is fairly robust and easy to script. dstat terminal output with running it with no options: The following will run on your Linux server and capture hardware usage to a csv file every 5 second until you manually stop it with Ctrl-C: [gisadmin@pesrv04 ~]$ dstat -tTcmn --disk-util -d --nocolor --noheaders --output dstat_20220823.csv 5 Note: With dstat, the captured processor usage is "idle time percentage", not usage percentage. To get the usage percentage, subtract 100 from the value . However, the disk utilization is in a usage percentage. Additionally, the output above includes a presentation format so the raw values (not shown) for memory, network and disk are saved as bytes. Note: dstat scripts would need to be run manually and separately on every Linux machine in the deployment. They do not directly integrate into JMeter. Once a load test is done and the scripts stopped, the CSV files can be collected and analyzed (e.g. plotted or charted in a spreadsheet program). Using Apache JMeter (To Capture Utilization Directly From Load Test) Although it has a few more steps (as well as its own hurdles) than perfmon and dstat scripts, using Apache JMeter to capture hardware utilization has one main advantage: Consistency. The same JMeter Test Plan can be used to capture the same hardware metrics from Windows and Linux (and Mac) This is accomplished through several free pieces of software: The jp@gc - PerfMon Metrics Collector extension that is added to JMeter This component is used to define what metrics to collect and from which machines The PerfMon Server Agent tool that runs in a Java Runtime Environment (JRE) on each server in the deployment This component does the actual collection on each server and sends the data back to the JMeter The jp@gc - PerfMon Metrics Collector component can be easily added to the Test Plan through the JMeter Plugins Manager (found under "PerfMon (Servers Performance Monitoring): The PerfMon Server Agent component is relatively easy to add but a few extra steps to get it to work right. Note: Due to a bug in the sigar-amd64-winnt library, the ServerAgent crashes if you use the out-of-the-box library with the JDK or JRE greater than version 8. The workaround is to patch the library or use an up-to-date version 8 JRE. OpenLogic provides free, periodically updated v8 OpenJDK (and JRE) releases to download for Windows, Linux and Mac. Although the JDK would work, the JRE is recommended in this case as it is a smaller package to download but still provides the environment needed for the agent to run. Note: Either the 32bit or 64bit JRE will work for the ServerAgent. Note: The OpenLogic JDK or JRE can also be used to run Apache JMeter itself. The ServerAgent and OpenLogic JRE unzipped in the same folder: The OpenLogic JRE moved into the ServerAgent folder: Open the startAgent.bat file in an editor and update it with the name of the JRE directory that was copied into the ServerAgent folder: off
set JAVAPATH=.\openlogic-openjdk-jre-8u342-b07-windows-64\bin
%JAVAPATH%\java -jar %0\..\CMDRunner.jar --tool PerfMonAgent %* Save the changes The C:\JMeterAgent\ServerAgent-2.2.3 folder now has a self-contained ServerAgent that can be copied over to the appropriate machines for collecting their hardware utilization To start the agent in Windows, double-click on the startAgent.bat The agent will launch into a command window: The agent will not start "collecting" metrics until directed by the Apache JMeter Test Plan By default, the agent and JMeter will communicate with each other on TCP and UDP port 4444 Note: The jp@gc - PerfMon Metrics Collector uses the name "perfmon" but does not actually utilize Windows Performance Monitor technology. The SampleWorldCities Test Plan With Metric Collection Support To download the Apache JMeter Test Plan used in this Article see: sampleworldcities5.zip Opening the Test Plan in Apache JMeter should look similar to the following: Adjust the User Defined Variables to fit your environment This test defines four different machines to collect hardware utilization from: A web adaptor The requests are also sent to this endpoint An ArcGIS Server A Portal for ArcGIS The Test Client (the machine running JMeter) The hostname of this machine is auto-detected by JMeter when the test runs Components of the Test Plan The jp@gc - PerfMon Metrics Collector extension is the only element of interest here as this test is practically identical to previous Article Test Plans for SampleWorldCities. jp@gc - PerfMon Metrics Collector Extension As mentioned earlier, the extension is what tells each running agent what it want them to collect. The Test Plan available in this Article has already added the counters for each of the four primary hardware categories. Note: The metrics parameters will work for Linux and Mac. However, the Disks IO metrics are in the extension are configured to collect on the C drive (e.g. C\:\) which does not exist in Linux. This would need to be changed to /. Validate the JMeter-to-ServerAgent Connectivity Start the test from the GUI by click the play button (green triangle) If all the ServerAgents are running and reachable on all the intended machines in the ArcGIS Enterprise deployment, they should be able to collect utilization and send the information back to the Test Plan As JMeter receives this information it will display the information in the Chart tab Let this collect for a few second to validate connectivity then stop the test Note: When the test is run from the command line, the Filename containing the data of the collected metrics will be based on variables in the bat script that define the "results_" file. Once the test has completed, you can use this "jp@gc - PerfMon Metrics Collector" element to load the raw values and either isolate particular counters or generate chart images. This file will start with the string "perfmon_". The results and perfmon files both end with a *.jtl extension but are just CSV files. Test Execution The load test should be run in the same manner as a typical JMeter Test Plan. See the runMe_withP.bat script included with the sampleworldcities5.zip project for an example on how to run a test as recommended by the Apache JMeter team. The runMe_withP.bat script contains a jmeterbin variable that will need to be set to the appropriate value for your environment This script contains a little more logic than the runMe.bat found in other Test Plan Articles This extra logic was added so the results file and the file containing the collected hardware end with the same runname Contents of the runMe_withP.bat script: echo off
rem Scripted JMeter Test Plan execution
rem Utilizing ApacheJMeter.jar for invocation
rem
rem 2022/08/25.1
rem
rem With environment variable to JMeter support (for PerfMon extension)
rem ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
rem *** Variables ***
rem Set JMeter memory to min/max of 4GB (adjust based on your test client resources)
set heap=-Xms4g -Xmx4g -XX:MaxMetaspaceSize=256m
rem Location of %JAVA_HOME%\bin
set javadir=%JAVA_HOME%\bin
rem Location of Apache JMeter bin
set jmeterbin=C:\apache-jmeter-5.5\bin
rem Location of JMeter Test Plan root folder (e.g. the folder where the Test Plan resides)
set projectdir=%~dp0
rem Name of the JMeter Test Plan (without the JMX file extension)
set testname=sampleworldcities5
rem String appended to results file of each test run
set runname=testrun1
rem Proxy settings
rem set proxyhost=http://localhost
rem set proxyport=8888
rem ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
rem *** Start your engines ***
color 20
rem Set environment variable to reference in test (e.g. ${__P(env.run)})
set jvm_args="-Denv.runname=%testname%_%runname%"
echo on
rem *** Test started ***
%javadir%\java.exe --illegal-access=warn %jvm_args% %heap% -jar %jmeterbin%\ApacheJMeter.jar -Jjmeter.save.saveservice.autoflush=false -n -f -t "%projectdir%\%testname%.jmx" ^
-l "%projectdir%\results\results_%testname%_%runname%.jtl" ^
-j "%projectdir%\logs\debug_%testname%_%runname%.log" ^
-e -o "%projectdir%\reports\%testname%_%runname%" ^
rem uncomment as needed
rem -H %proxyhost% -P %proxyport% ^
rem *** Test completed ***
echo off
color 40
ping localhost -n 5
color 07 Note: It is always recommended to coordinate the load test start time and duration with the appropriate personnel of your organization. This ensures minimal impact to users and other colleagues that may also need to use your on-premise ArcGIS Enterprise Site. Additionally, this helps prevent system noise from other activity and use which may "pollute" the test results. Note: For several reasons, it is strongly advised to never load test services provided by ArcGIS Online. General Methodology Guide This guide is not intended to be comprehensive covering every hardware utilization capture scenario. Scenario #1 Servers Windows Environment On-premise Collecting Machine (Test Client) Windows Description Capturing Windows usage from Windows Difficulty Easy/Moderate Recommended Methodology Windows Perfmon (via PowerShell script) Scenario #2 Servers Windows/Linux Environment On-premise Collecting Machine (Test Client) Windows Description Capturing Windows/Linux usage from Windows Difficulty Moderate Recommended Methodology JMeter Perfmon Extension and ServerAgent Scenario #3 Servers Linux Environment On-premise Collecting Machine (Test Client) Windows/Linux Description Capturing Linux usage from Windows/Linux Difficulty Easy/Moderate Recommended Methodology dstat command or script Scenario #4 Servers Windows/Linux Environment Cloud Collecting Machine (Test Client) Windows Description Capturing Windows/Linux usage from Windows Difficulty Moderate Recommended Methodology Cloud Watch/Azure Monitor (not covered in this Article) Scenario #5 Servers Windows Environment Cloud Collecting Machine (Test Client) Windows Description Capturing Windows usage from Windows Difficulty Moderate Recommended Methodology Run Windows Perfmon (via PowerShell script) on each machine in the deployment Scenario #6 Servers Linux Environment Kubernetes Collecting Machine (Test Client) Windows Description Capturing Linux contain/pod usage from Windows Difficulty Moderate/Hard Recommended Methodology Environment dashboard or custom resource endpoint (not covered in this Article) Common Utilization Collection Challenges Capturing hardware utilization of deployment machines for a load test is a best testing practice but it is not always possible. The most typical challenges are: Permissions The most common Not having been granted access to or the ability to capture usage from remote servers is the most common challenge Environment/Location Collecting utilization from machines on the local network is one thing, collecting usage from machines in the cloud is another thing Operation System Sometimes the OS presents its own *hurdles* for capturing the utilization Technical Certain environments (e.g. Kubernetes/Docker) may not have the same APIs for collecting the usage as a traditional (physical or virtual) machine Final Thoughts Simply put, there are many ways to capture hardware utilization information from the machines in an ArcGIS Enterprise deployment to analyze with your load test results. Which is the best way to capture it...any way that works best were you can quickly record the info and use it for effectively analysis. Although no one Article can cover every situation, environment and scenario, this one lists several methodologies for capturing this data for common ones.
... View more
08-25-2022
11:06 PM
|
0
|
2
|
3325
|
BLOG
|
System Log Parser's ServiceDetails Analysis Type Looking for a quick way to summarize the details of all your ArcGIS Enterprise services? If so, then System Log Parser's ServiceDetails Analysis Type is highly recommended. This feature performs a scan of your Site through the REST Admin API and presents the gathered information into a spreadsheet report for easy digestion. The report is a great resource to find details on your service provider statistics, service configuration info, or data source analysis. Selecting ServiceDetails from the GUI The ServiceDetails Analysis Type can be selected from the type right of the GUI: Note: The ServiceDetails Analysis Type is only available with an ArcGIS Server (Web) log query since it must make use of the REST Admin API Selecting ServiceDetails from the Command Line The ServiceDetails Analysis Type is also available from the command-line. For example: slp.exe -s https://yourwebadaptor.domain.com/server -u gisadmin -p Myp@ssword -pfl true -validate true -a servicedetails The ServiceDetails Report The upper section in the Summary worksheet of the ServiceDetails report: The lower section detailing Service Provider Statistics: The Site Details worksheet provides a listing of the hardware (CPU and Memory) from the machines running ArcGIS Server There is also table containing a verbose breakdown on the configured parameters of each service Additionally, there are two Datasource worksheets (not shown) providing information and analysis on where the services are looking for their respective data Latest Version Bug fixes and new features are always being added to SLP. The latest version can be found here: System Log Parser (0.12.17.0)
... View more
08-01-2022
12:36 PM
|
2
|
0
|
2006
|
BLOG
|
Hi MarcovanderHeide, Thank you for the kind feedback! I have not yet looked into SAML authentication with ArcGIS Enterprise but I do have a simple Integrated Windows Authentication/Single Sign On test that might help. Please take a look at the sampleworldcities6.zip Test Plan. A few items worth mentioning: The NTLM authorization is handled by the HTTP Authorization Manager element. This is set to read in the credentials defined in the User Defined Variables section, but this could also be CSV Data Set Config file driven. For me, there were a few ways to tell that I was logging in as the intended domain user The services REST endpoint would display the "Logged in user" up at the top of the page The HTML response body is shown below: Another verification was the last request in the test, an export map call to the SampleWorldCities that is only accessible to members of organization (e.g. the ArcGIS Enterprise Site): Note: Even with IWA/SSO, a token is still generated and used to access the service.
... View more
07-27-2022
04:16 PM
|
0
|
0
|
2100
|
BLOG
|
System Log Parser's GUI and Command Line The System Log Parser (SLP) utility is great for being able to analyze many different "ArcGIS Enterprise" log sources to help quantify the usage of your Site. It is free and includes an easy-to-use graphical user interface (GUI) called SystemLogsGUI.exe. But, did you know it can also be run from the command line via the slp.exe that is included with the download? Command line capable execution makes a good fit for automation through PowerShell or good old fashioned bat scripts. The slp.exe executable includes all of the same capabilities. However, you can override many of the limits imposed by the GUI such as: the end time and start time. You can even access features that are not available from the GUI like: searching with specific UTC dates, customizing the name of the report, and enabling the application debug log (for troubleshooting). The Most Popular Log Sources The exact command and options will vary slightly depending and the log source you are wanting to use for the analysis. Let's start with the most popular log sources using some recommended options: ArcGIS Server Log Query (File System) C:\SystemLogParser> slp.exe -f AGSFS -i "\\myserver.domain.com\c$\arcgis\arcgisserver\logs\MYSERVER.DOMAIN.COM" -eh now -sh 30day -a optimized -validate true or C:\SystemLogParser> slp.exe -f AGSFS -i "\\myserver.domain.com\c$\arcgis\arcgisserver\logs\MYSERVER.DOMAIN.COM" -eh now -sh 30day -a optimized -validate true The -f AGSFS parameter specifies the system log parser read function...ArcGIS Server File System. The -i "[location of logs]" parameter specifies the file system or network location of the folder containing the logs. The -eh now and -sh 30day parameters specify the end hour and start hour. In this case, the end time (most recent time) will stop "now" (defined as the date time of whenever the command is executed) and the start time (furthest back in time) will begin 30 days ago. Note: There is no authentication with this method, but access to the logs is granted via the network share or local file system permission. Only read permission is required. Note: The file system is the fastest and most scalable way to read the ArcGIS Server logs. Note: For ArcGIS Server Log Query (File System), it is possible to read logs from multiple servers into one report. This can also be accomplished with the "-i" option by separating each path with a comma. For example: C:\SystemLogParser> slp.exe -f AGSFS -i "\\myserver1.domain.com\c$\arcgis\arcgisserver\logs\MYSERVER1.DOMAIN.COM,\\myserver2.domain.com\c$\arcgis\arcgisserver\logs\MYSERVER2.DOMAIN.COM,\\myserver3.domain.com\c$\arcgis\arcgisserver\logs\MYSERVER3.DOMAIN.COM" -eh now -sh 30day -a optimized -validate true ArcGIS Server Log Query (Web) C:\SystemLogParser> slp.exe -f AGS -s https://myserver.domain.com/server -u siteadmin -p AdminP@ssword -eh now -sh 7day -a optimized -validate true or C:\SystemLogParser> slp.exe -f AGS -s https://myserver.domain.com/server -u gisadmin -p Myp@ssword -pfl true -eh now -sh 7day -a optimized -validate true The -f AGS parameter specifies the system log parser read function...ArcGIS Server Web. The -s [ArcGIS Server instance URL], -u [administrator user], and -p [administraor password] parameters specify the options needed to authenticate to your ArcGIS Server. Use the -pfl true parameter if user performing the query is using a Portal Federated Login. The -eh now and -sh 30day parameters specify the end hour and start hour. The end time will stop "now" (defined as the date time of whenever the command is executed) and the start time will begin 7 days ago. Note: While technically possible to read 30 days worth of logs through the web, it is not recommended, even when using the Optimized report. The reason is because for busy Sites, the overall process can be very resource intensive on the server machines with tokens or individual log requests timing out. Note: Unlike reading the logs from the file system, you do not need to specify each individual server in the Site as the ArcGIS Server's REST Admin API does this automatically. Internet Information Services Log Query C:\SystemLogParser> slp.exe -f IIS -i \\myserver.domain.com\c$\inetpub\logs\LogFiles\W3SVC1 -eh now -sh 30day -a optimized -validate true The -f IIS parameter specifies the system log parser read function...Internet Information Services. The -i "[location of logs]" parameter specifies the file system or network location of the folder containing the logs. The -eh now and -sh 30day parameters specify the end hour and start hour. In this case, the end time (most recent time) will stop "now" (defined as the date time of whenever the command is executed) and the start time (furthest back in time) will begin 30 days ago. Note: There is no authentication with this method, but access to the logs is granted via the network share or local file system permission. Only read permission is required. Note: The values included in them are a nice representation of "response times" which can help describe the experience the users encountered with respect to performance. AWS ELB (Amazon Web Services, Elastic Load Balancer) C:\SystemLogParser> slp.exe -f ELB -s mybucket1 -reg USWest2 -u ABC123DEF456 -p n4b5n74v75 -eh now -sh 7day -a optimized -validate true The -f ELB parameter specifies the system log parser read function...Elastic Load Balancer. The -s [bucket name], -u [access key], -p [secret key], and -reg [AWS region] parameters specify the options needed to authenticate to your account. The -eh now and -sh 7day parameters specify the end hour and start hour. In this case, the end time (most recent time) will stop "now" (defined as the date time of whenever the command is executed) and the start time (furthest back in time) will begin 7 days ago. Note: System Log Parser can query large time spans (e.g. greater than 7 days) of ELB logs, but to significantly improve the download concurrency and reading performance for such large queries (e.g. 30days worth of logs), it is recommended to run slp.exe through the Get-SLPReport.ps1 PowerShell script provided with the System Log Parser zip. This script utilizes multiple PowerShell threads to quickly download and unzip the logs. It then uses the ELBFS log source function (-f ELBFS) to read them from the local disk. Note: The Get-SLPReport.ps1 script requires at least PowerShell version 7 to run. Useful Universal Options The -a optimized parameter All of the examples commands above use the Optimized report which is the recommended Analysis Type. Note: The Optimized report has tremendous memory savings over the other analysis types, especially if you are reading 30days worth of logs. The -validate true parameter To assist in troubleshooting, the -validate true is passed into the slp.exe command to print any warnings or errors to the console that occur. Without it, the default mode of slp.exe is to run silently. The -o false parameter When running log query through a scheduled script (e.g. Task Manager), it is usually desirable to suppress the automatic opening of the generated report. Setting -o to false will create the report but not open it when slp.exe completes. The -apploglevel DEBUG parameter For in-depth troubleshooting, you can optionally pass in -apploglevel DEBUG. This will create a unique run log of the slp.exe execution...with a large amount of detail. The log file (for an ArcGIS Server file system log query) would typically be found in a location similar to the following: C:\Users\gisadmin\Documents\System Log Parser\Logs\Application_agsfsx2_20220702T220628_k6a7xrhb.log The -d "C:\Users\[username]\Desktop\my slp reports" parameter slp.exe allows the location of report to be set at generation time with the "-d" switch. If the "my slp reports" directory does not exist, it will be created. The user running slp.exe will need write permission to "C:\Users\gisadmin\Desktop", otherwise an error will be thrown. Note: In the case above, using -d "C:\Users\[username]\Desktop\my slp reports" will create the folder, if it does not exist. However, if it does not exist and a slash is appended (-d "C:\Users\[username]\Desktop\my slp reports\"), slp.exe will throw an error. The -n "myreport123.xlsx" parameter slp.exe allows the name of the report to be set at generation time with the "-n" switch. Note: The file extension (e.g. ".xlsx") is not added automatically so be sure to include it if using the -n option. The -endstring "2022/05/21 12:00:00 PM" and -startstring "2022/05/21 9:30:00 AM" parameters slp.exe allows for surgical time ranges. You can pass exact end and start times (in UTC) which can be helpful for troubleshooting or when looking for specific date in the past. For examples, a particular spike in response times that occurred 10 days ago but only lasted for a few hours. In this case, you most likely don't need everything from 10 days ago until right now, but instead want to focus just on the event. Note: Both the -endstring and -startstring should be passed strings that define time in the Coordinated Universal Time or UTC timezone. Other Log Sources slp.exe supports several other log sources such as Azure (-f azure), CloudFront (-f cloudfront) and Apache Tomcat (-f tomcat). However, these sources do not yet support the Optimized Analysis Type and querying large durations of times against busy Sites can result in some time consuming and resource intensive (on the machine running slp.exe) log parsing. Apache Tomcat A typical tomcat installation does not usually have the access logs enabled by default. However, they can be activated with a relatively simple change to the server.xml file. Open server.xml in your favorite text editor Ensure the the following is added and/or not commented out: <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs" fileDateFormat="yyyy-MM-dd.HH" pattern="%h %l %u %t "%r" %s %b %v %p %D" prefix="localhost_access_log." resolveHosts="false" suffix=".txt"/> Save edits Restart Tomcat Note: In this example, the suffix is set to ".txt" but using ".log" will also work with slp.exe Latest Version Bug fixes and new features are always being added to SLP. The latest version can be found here: System Log Parser
... View more
07-24-2022
10:22 PM
|
7
|
3
|
3449
|
Title | Kudos | Posted |
---|---|---|
3 | 2 weeks ago | |
1 | 07-01-2022 03:26 PM | |
1 | 3 weeks ago | |
1 | 06-14-2024 11:00 PM | |
5 | 05-01-2024 11:17 AM |
Online Status |
Offline
|
Date Last Visited |
3m ago
|