Latest Contributions by AaronLopez

System Log Parser's GUI and Command Line The System Log Parser (SLP) utility is great for being able to analyze many different "ArcGIS Enterprise" log sources to help quantify the usage of your Site. It is free and includes an easy-to-use graphical user interface (GUI) called SystemLogsGUI.exe. But, did you know it can also be run from the command line via the slp.exe that is included with the download? Command line capable execution makes a good fit for automation through PowerShell or good old fashioned bat scripts. The slp.exe executable includes all of the same capabilities. However, you can override many of the limits imposed by the GUI such as: the end time and start time. You can even access features that are not available from the GUI like: searching with specific UTC dates, customizing the name of the report, and enabling the application debug log (for troubleshooting). The Most Popular Log Sources The exact command and options will vary slightly depending and the log source you are wanting to use for the analysis. Let's start with the most popular log sources using some recommended options: ArcGIS Server Log Query (File System)   C:\SystemLogParser> slp.exe -f AGSFS -i "\\myserver.domain.com\c$\arcgis\arcgisserver\logs\MYSERVER.DOMAIN.COM" -eh now -sh 30day -a optimized -validate true   or   C:\SystemLogParser> slp.exe -f AGSFS -i "\\myserver.domain.com\c$\arcgis\arcgisserver\logs\MYSERVER.DOMAIN.COM" -eh now -sh 30day -a optimized -validate true   The -f AGSFS parameter specifies the system log parser read function...ArcGIS Server File System. The -i "[location of logs]" parameter specifies the file system or network location of the folder containing the logs.  The -eh now and -sh 30day parameters specify the end hour and start hour. In this case, the end time (most recent time) will stop "now" (defined as the date time of whenever the command is executed) and the start time (furthest back in time) will begin 30 days ago. Note: There is no authentication with this method, but access to the logs is granted via the network share or local file system permission. Only read permission is required. Note: The file system is the fastest and most scalable way to read the ArcGIS Server logs. Note: For ArcGIS Server Log Query (File System), it is possible to read logs from multiple servers into one report. This can also be accomplished with the "-i" option by separating each path with a comma. For example:   C:\SystemLogParser> slp.exe -f AGSFS -i "\\myserver1.domain.com\c$\arcgis\arcgisserver\logs\MYSERVER1.DOMAIN.COM,\\myserver2.domain.com\c$\arcgis\arcgisserver\logs\MYSERVER2.DOMAIN.COM,\\myserver3.domain.com\c$\arcgis\arcgisserver\logs\MYSERVER3.DOMAIN.COM" -eh now -sh 30day -a optimized -validate true   ArcGIS Server Log Query (Web)   C:\SystemLogParser> slp.exe -f AGS -s https://myserver.domain.com/server -u siteadmin -p AdminP@ssword -eh now -sh 7day -a optimized -validate true   or   C:\SystemLogParser> slp.exe -f AGS -s https://myserver.domain.com/server -u gisadmin -p Myp@ssword -pfl true -eh now -sh 7day -a optimized -validate true   The -f AGS parameter specifies the system log parser read function...ArcGIS Server Web. The -s [ArcGIS Server instance URL], -u [administrator user], and -p [administraor password] parameters specify the options needed to authenticate to your ArcGIS Server. Use the -pfl true parameter if user performing the query is using a Portal Federated Login. The -eh now and -sh 30day parameters specify the end hour and start hour. The end time will stop "now" (defined as the date time of whenever the command is executed) and the start time will begin 7 days ago. Note: While technically possible to read 30 days worth of logs through the web, it is not recommended, even when using the Optimized report. The reason is because for busy Sites, the overall process can be very resource intensive on the server machines with tokens or individual log requests timing out. Note: Unlike reading the logs from the file system, you do not need to specify each individual server in the Site as the ArcGIS Server's REST Admin API does this automatically. Internet Information Services Log Query   C:\SystemLogParser> slp.exe -f IIS -i \\myserver.domain.com\c$\inetpub\logs\LogFiles\W3SVC1 -eh now -sh 30day -a optimized -validate true   The -f IIS parameter specifies the system log parser read function...Internet Information Services. The -i "[location of logs]" parameter specifies the file system or network location of the folder containing the logs.  The -eh now and -sh 30day parameters specify the end hour and start hour. In this case, the end time (most recent time) will stop "now" (defined as the date time of whenever the command is executed) and the start time (furthest back in time) will begin 30 days ago. Note: There is no authentication with this method, but access to the logs is granted via the network share or local file system permission. Only read permission is required. Note: The values included in them are a nice representation of "response times" which can help describe the experience the users encountered with respect to performance. AWS ELB (Amazon Web Services, Elastic Load Balancer)   C:\SystemLogParser> slp.exe -f ELB -s mybucket1 -reg USWest2 -u ABC123DEF456 -p n4b5n74v75 -eh now -sh 7day -a optimized -validate true   The -f ELB parameter specifies the system log parser read function...Elastic Load Balancer. The -s [bucket name], -u [access key], -p [secret key], and -reg [AWS region] parameters specify the options needed to authenticate to your account. The -eh now and -sh 7day parameters specify the end hour and start hour. In this case, the end time (most recent time) will stop "now" (defined as the date time of whenever the command is executed) and the start time (furthest back in time) will begin 7 days ago. Note: System Log Parser can query large time spans (e.g. greater than 7 days) of ELB logs, but to significantly improve the download concurrency and reading performance for such large queries (e.g. 30days worth of logs), it is recommended to run slp.exe through the Get-SLPReport.ps1 PowerShell script provided with the System Log Parser zip. This script utilizes multiple PowerShell threads to quickly download and unzip the logs. It then uses the ELBFS log source function (-f ELBFS) to read them from the local disk. Note: The Get-SLPReport.ps1 script requires at least PowerShell version 7 to run. Useful Universal Options The -a optimized parameter All of the examples commands above use the Optimized report which is the recommended Analysis Type. Note: The Optimized report has tremendous memory savings over the other analysis types, especially if you are reading 30days worth of logs. The -validate true parameter To assist in troubleshooting, the -validate true is passed into the slp.exe command to print any warnings or errors to the console that occur. Without it, the default mode of slp.exe is to run silently. The -o false parameter When running log query through a scheduled script (e.g. Task Manager), it is usually desirable to suppress the automatic opening of the generated report. Setting -o to false will create the report but not open it when slp.exe completes. The -apploglevel DEBUG parameter For in-depth troubleshooting, you can optionally pass in -apploglevel DEBUG. This will create a unique run log of the slp.exe execution...with a large amount of detail. The log file (for an ArcGIS Server file system log query) would typically be found in a location similar to the following: C:\Users\gisadmin\Documents\System Log Parser\Logs\Application_agsfsx2_20220702T220628_k6a7xrhb.log The -d "C:\Users\[username]\Desktop\my slp reports" parameter slp.exe allows the location of report to be set at generation time with the "-d" switch. If the "my slp reports" directory does not exist, it will be created. The user running slp.exe will need write permission to "C:\Users\gisadmin\Desktop", otherwise an error will be thrown. Note: In the case above, using -d "C:\Users\[username]\Desktop\my slp reports" will create the folder, if it does not exist. However, if it does not exist and a slash is appended (-d "C:\Users\[username]\Desktop\my slp reports\"), slp.exe will throw an error.  The -n "myreport123.xlsx" parameter slp.exe allows the name of the report to be set at generation time with the "-n" switch. Note: The file extension (e.g. ".xlsx") is not added automatically so be sure to include it if using the -n option. The -endstring "2022/05/21 12:00:00 PM" and -startstring "2022/05/21 9:30:00 AM" parameters slp.exe allows for surgical time ranges. You can pass exact end and start times (in UTC) which can be helpful for troubleshooting or when looking for specific date in the past. For examples, a particular spike in response times that occurred 10 days ago but only lasted for a few hours. In this case, you most likely don't need everything from 10 days ago until right now, but instead want to focus just on the event. Note: Both the -endstring and -startstring should be passed strings that define time in the Coordinated Universal Time or UTC timezone. Other Log Sources slp.exe supports several other log sources such as Azure (-f azure), CloudFront (-f cloudfront) and Apache Tomcat (-f tomcat). However, these sources do not yet support the Optimized Analysis Type and querying large durations of times against busy Sites can result in some time consuming and resource intensive (on the machine running slp.exe) log parsing.  Apache Tomcat A typical tomcat installation does not usually have the access logs enabled by default. However, they can be activated with a relatively simple change to the server.xml file. Open server.xml in your favorite text editor Ensure the the following is added and/or not commented out: <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs" fileDateFormat="yyyy-MM-dd.HH" pattern="%h %l %u %t &quot;%r&quot; %s %b %v %p %D" prefix="localhost_access_log." resolveHosts="false" suffix=".txt"/> Save edits Restart Tomcat  Note: In this example, the suffix is set to ".txt" but using ".log" will also work with slp.exe Latest Version Bug fixes and new features are always being added to SLP. The latest version can be found here: System Log Parser      ... View more

System Log Parser's Optimized Analysis Type Although you are probably familiar with using System Log Parser (SLP) to read logs and help you quantify your ArcGIS Enterprise usage, there is a relatively new feature in this popular, free utility that can make the effort easier and the analysis more powerful.   This feature is the "Optimized" report option which is a new parameter in the GUI listed under Analysis Type. For log queries such as ArcGIS Server (Web), ArcGIS Server (File System), Internet Information Services (IIS) and Elastic Load Balancer (AWS), it is now the default. This option is not yet available for Cloud Front and Azure log queries. For backward compatibility, Simple, WithOverviewCharts and Complete types are still an option, but for the best analysis experience, they are not recommended. The Optimized Analysis Type option shown in the GUI: A System Log Parser spreadsheet report generated using the Optimized option: Also Available From the Command-Line The GUI (SystemLogsGUI.exe) is convenient for creating reports but the command line version (slp.exe) is great for automating System Log Parser analysis. The Optimized report option is available from both! Optimized Report Origins Over the years, one of the most requested enhancements for SLP has been to improve the speed and memory usage (of the machine running it) when parsing logs for very busy Sites or large time spans (e.g. several weeks or more). The Optimized report directly addresses these two items. Performance is measurably faster and memory orders of magnitude lower. With the ability to search greater spans of time with a quicker executions, more powerful statistics can be performed as the report can analyze more requests. For the best results, use the Optimized report with ArcGIS Server (File System), Internet Information Services (IIS) or Elastic Load Balancer (AWS) log queries. Memory usage savings can still be obtained with ArcGIS Server (Web) log queries, but the act of retrieving large amounts of logs through the REST Admin API is still the performance bottleneck. Optimized Report Contents System Log Parser reports focus on grouping the values from the logs into key time-based categories. While the Optimized report for the different log sources (ArcGIS Server, IIS, or ELB) has some commonality, there can be additional statistics depending on which was utilized. For example, if the Optimized report was used for querying ArcGIS Server (Web) or ArcGIS Server (File System) logs, wait time, instance creation time, and arrival time will each be statistically broken down in additional to elapsed time.  Report System Log Parser Bugs Contact [email protected] Note: The System Log Parser tool is developed at Esri but is not a supported product.   Latest Version Bug fixes and new features are always being added to SLP. The latest version can be found here: System Log Parser    ... View more

Administration Automation with Apache JMeter Apache JMeter is a great load testing tool, but it's a fantastic automation framework too! There are many ArcGIS Enterprise administrative workflows and automation solutions for your portal. This Article focuses on using JMeter to call the ArcGIS REST API in order to carry out user management tasks that would be tedious for large numbers of members. Thankfully, the JMeter's GUI makes the test setup and REST request building easy. The User Administration Test Plans To download the Apache JMeter Test Plan used in this Article see: portal_administration1.zip  This project includes 6 Test Plans for ArcGIS Enterprise 10.9/10.9.1: Add a new user A simple, basic test that just adds new members Add a new user with a few options A test that adds new members but allows the Start page and a Portal Group to be specified Add a new user with more options A test that adds new members (Start page and Group) but can set Add-on licenses Set the security question/answer for new users Set the security question and answer for newly added members that have not logged in Disable a user A test that disables users Enable a user A test that enables users The CSV Data Set Config of Users For convenience, all the Test Plans in the project work off the same list of users from the same file. In the Test Plans, this is referenced by the CSV Data Set Config element named "Users File". The Text File List of Users  The included text file contains user information for working with 10 different members. However, it can be adjusted and/or expanded to suite the needs of your organization. There are many different options for the role and userLicenseTypeId fields These choices can also impact the Add-on licenses as some automatically include specific entitlements Note: It is recommended to first run the test plans with a small list of users in order to see if everything is configured correctly for your Site. Administrator Login With the exception of one, all the included Test Plans have logic to log in as a built-in Portal for ArcGIS administrator at the beginning of the test. For efficiency this action is only executed once (at the start) per each test thread. Note: When connecting to the Portal for ArcGIS component of ArcGIS Enterprise, the Test Plans will be sending requests directly to the "arcgis" instance on port 7443. Add a New User (portal_users_add1) The portal_users_add1 Test Plan is simple way to add new members to Portal for ArcGIS.  The administrator credentials are specified from the User Defined Variables section of the Test Plan Except where noted, this step is performed at the beginning of all the included tests Once the test authenticates as the administrator, it calls the createUser function and repeats it for each line in file containing the list of users Since this test uses a single HTTP Request to create the user it is the fastest and most scalable way to add new members This test only adds users, it does not perform any other duties such as joining a member to a group, setting the Start page, or selecting add-on licenses For convenience, the username is appended to all user-based transactions and requests This assists troubleshooting if a particular iteration of the test could not add a specific user All of the included tests follow this design pattern This test is similar to the process used on the Example: Add members to the portal resource, command line utility and Add members from a file feature built into Portal for ArcGIS. Add a New User (portal_users_add2) The portal_users_add2 Test Plan is an easy way to add new members to Portal for ArcGIS but includes a few options. In addition to creating the user, this test allows the administrator to set additional properties like the Start page (also known as the landing page) and a Portal Group. This test expands the user creation process by 3 additional requests per user If creating thousands of users, you may notice this test takes longer to complete than portal_users_add1 This is due to the fact that more work is taking place Note: A new member can actually be added to more than one Portal Group on creation. However, for simplicity, portal_users_add2 only adds the user to one group and the same group is used for all members. The group used is defined from the PortalGroupId User Defined Variable. This GUID Id needs to be manually looked up from your Portal for ArcGIS Site. If you do not wish to add the user to a Group, simply disable the setProperties request in the test. Add a New User (portal_users_add3) The portal_users_add3 Test Plan is an automated way to add new members to Portal for ArcGIS with the most options for an administrator. This test allows you to set the Start page and Portal Group but adds the ability to specify Add-on licenses like ArcGIS Pro and Extensions and certain User type extensions. Immediately after the administrator authentication, the test makes a call to retrieve GUIDs for the ArcGIS Pro and User type extensions These GUIDs will be used later when assigning the licenses to the users The add-on licenses add several more requests to the user creation process Although powerful, these additional requests can add time to the overall task as they are performed for each member that is created The test is configured to assign the user: ArcGIS Pro Advanced and all available Extensions (as of 10.9/10.9.1) All User type extensions Note: There are other Add-on licenses such as Applications and ArcGIS Runtime extensions that were not included in the portal_users_add3 Test Plan. Many of these other licenses would require there own specific HTTP request. Again, while this can be convenient and powerful, it can add time to process of adding each user. There are also some licenses like App bundles which were not included in the test as they are automatically included with user license type (e.g. Creator). Set the Security Question/Answer for New Users (portal_users_update_profile1) The portal_users_update_profile1 Test Plan is a little unique. It is the only test in the project which does not log in as a Portal for ArcGIS administrator. Instead, it logs in as each user and assumes it is performing the initial login for each member as it will set their security question and answer. Presetting the security question and answer is completely optional Your organization may instead prefer to have each user set  these values when they first log in Disable a User (portal_users_disable1) The portal_users_disable1 Test Plan is an automated way for taking a list of users and disabling their membership to the portal. Once the account is disabled the user cannot log in. This is a less destructive function than delete.  Disabling users is fairly straight-forward and performed with one REST call to disableUsers Note: For simplicity, the disableUsers request in the portal_users_disable1 Test Plan is only disabling one member at a time. However, for each call to the disableUsers function, the request will accept groups of users for improved efficiency. As of 10.9/10.9.1, disableUsers accepts up to 25 users at a time. Note: The portal_users_disable1 test can be executed over the same users successfully. From the point of view of ArcGIS Enterprise, it is just disabling the member(s) again. Enable a User (portal_users_enable1) The portal_users_enable1 Test Plan is an automated way for taking a list of users and enabling their membership to the portal. Once the account is enabled the user cannot log in. Enabling users is fairly straight-forward and performed with one REST call to enableUsers Note: For simplicity, the enableUsers request in the portal_users_enable1 Test Plan is only enabling one member at a time. However, for each call to the enableUsers function, the request will accept groups of users for improved efficiency. As of 10.9/10.9.1, enableUsers accepts up to 25 users at a time.  Note: The portal_users_enable1 test can be executed over the same users successfully. From the point of view of ArcGIS Enterprise, it is just enabling the member(s) again. The Thread Group Configuration Unlike the previous Apache JMeter Article tests that are time-dominant, the Test Plans in this project are iteration based. In other words, when creating or disabling a specific users, we only need to work on the users of interest from the list once. The Thread Group "step load" configuration that is included by default with every Apache JMeter installation includes a very convenient Loop Count setting to specify exactly how many iteration the Test Plan should execute The Loop Count setting should match the number of lines in the "Users File" that contain members to be added/disabled/enabled Note: All Test Plans in the project are configured with the same Thread Group setting. Additionally, all of the included tests are executed with one concurrent test thread. Test Execution Also, unlike the previous Apache JMeter Article tests that are executed from the command-line, you can probably get away with running this administrative automation Test Plans right from the GUI. Of course, this depends on how many users you are planning to create, disable, or enable. If you are working with a few hundred, then the GUI would be fine. However, if you plan to create thousands or tens of thousands of users (or more), you will want to run the Test Plans from the command-line for the best usage efficiency of the test workstation resources. See the runMe.bat script included with the portal_administration1.zip project for an example on how to run a test as recommended by the Apache JMeter team. This script is configured to run portal_users_add3, but can easily be adjusted to running any of the tests. The runMe.bat script contains a jmeterbin variable that will need to be set to the appropriate value for your environment Note: It is always recommended to coordinate the start time with the appropriate personnel of your organization. This ensures minimal impact to users and other colleagues that may also need to use your on-premise ArcGIS Enterprise Site.  Validating the Test Plans If the test is being run from the GUI, there are several listeners that have been added to all of the included Test Plans that offer immediate feedback on the status. The View Results Tree element offers a convenient way to quickly examine the status of each transaction (e.g. "Create User Account -- portalpublisher2") and its respective requests (e.g. "/arcgis/portaladmin/security/users/createUser--portalpublisher2") Thanks to the Response Assertion rule elements added to each request, the green check mark is trusted indicator of a successful transaction or request The View Results in Table element offers a handy way to see status of each transaction and its response time all from one table Troubleshooting a Command-line Test Execution As mentioned earlier, when working with large amounts of users, the recommended approach is to run the Test Plans from the command-line. However, administrators will be very interested to understand which users, if any, encountered errors through the automation. It is here that the JMeter Test Report can offer great insight. From the Request Summary pie chart on the initial page of the Test Report, you can quickly see if any errors were encountered  If errors were encountered from the test run, the Statistics table (bottom of the first report page) can make the failed user requests easily to find when sorting by the FAIL column Final Thoughts There are many frameworks, tools and utilities out there to perform administrative task automation for ArcGIS Enterprise. Most likely, they all have their own strengths. Apache JMeter is handy as it provide a graphical interface for building and adjusting the REST requests need to perform the functions. The HTML/JavaScript reports which can be automatically created at the end of a test report are a nice bonus for understanding if whole job was successful or which particular parts failed. To download the Apache JMeter Test Plan used in this Article see: portal_administration1.zip  A Quick Word on Using Multiple Threads All of the included tests could be configured to use multiple, concurrent threads for faster execution. This is fine from a technical point of view, but all of these test perform write operations to the internal database for the Portal for ArcGIS component. As with any database, such operations can be resource intensive and can only go so fast. Using too many concurrent threads may actually slow down the performance of these tests. A Quick Word on Deleting Users The tests included with the project did not include a delete user operation. Deleting a member from the portal is permanent (without backups being available) and such tools that automate this action should use caution. Additionally, some users may have uploaded a plethora of content to the portal. This content would need to be delete or transferred to another user before removing that member.       Apache JMeter released under the Apache License 2.0. Apache, Apache JMeter, JMeter, the Apache feather, and the Apache JMeter logo are trademarks of the Apache Software Foundation.   ... View more

Why Test a Asynchronous Geoprocessing Service? The most popular reason is probably, "because you have been asked to test it". As a tester, GIS administrators may look to you to show how a geoprocessing (GP) model behaves as a service under load. Many GP models are built to perform long-running, critical and complex tasks. Since they are a resource often found on ArcGIS Enterprise deployments (as a service), this makes the understanding of their performance and scalability profiles key.  Asynchronous Geoprocessing Service Testing Challenges The hardest part of the load testing an asynchronous geoprocessing service is the loop logic and being able to handle the different states of the job. The loop should not be too aggressive and you need to exit the loop under the right conditions. Appropriately marking a test iteration as failed based on the job status or passed based on the output results from the task, is also critical. How to Test an Asynchronous Geoprocessing Service? The basic steps for load testing an asynchronous GP service are: Provide inputs for geoprocessing task Submit job   Capture unique job ID Perform an initial job status check Loop on the job status check If job succeeded Exit loop Sleep for a short duration If job succeeded Examine results If available, download output/data At a minimum, the Apache Test Plan should handle this logic. However, there are some enhancements that can be added to this process, like:  Maximum number of job status checks in the loop Maximum number of test iterations Marking the job as failed if a non-successful status is returned The Test Plan included in the Article provides these additional features. The "Summarize Invasive Species" Geoprocessing Model and Dataset The understanding of the process in this Article is most effective if the steps can be reproduced. For that, we can turn to a modern ArcGIS Pro GP model and dataset that is free and publicly available. The Share a web tool -- Summarize Invasive Species package is available from arcgis.com. View of the New Zealand data from ArcGIS Pro (with Topographic Basemap): This Article will not cover the details of configuring or publishing this model as a service in ArcGIS Enterprise. For information on such actions, see: Share a web tool   Test Data Generation Although the JMeter Test Plan will utilize some data to make the inputs to the model/service dynamic and more realistic, this will be pre-generated and automatically included with the test. The reason is to focus on the test logic (and not the data generation). The Asynchronous Geoprocessing Service Test Plan  To download the Apache JMeter Test Plan used in this Article see: async_gp1.zip  Opening the Test Plan in Apache JMeter should look similar to the following: Adjust the User Defined Variables to fit your environment Note: The test has different variables for the name of the (GP) service and (GP) tool. When published, they often use the same name (e.g. "SummarizeInvasiveSpecies") but do not have to. Components of the Test Plan SubmitJob  It all start with "submit the job". This is probably the easiest part of the test. Once this request has been sent, the service returns a job id and the server can begin to work on the task. This id is captured from the Regular Expression Extractor element. Note: The job id is unique to each job (and test thread). It is used in every subsequent request in the test. CSV Data Set Config We briefly skip to the bottom to mention the CSV Data Set Config element. The inputs are important and required to submit a job, but the generation of its data but is not a heavy area of focus of this testing Article. Contents example of the inputs.csv file: Note: The full input.csv file is included with the async_gp1 Test Plan. InitialJobStatus The initial job status transaction has one HTTP request element inside that is used to find and populate a variable (jobStatus) with the current state of the submitted task. This value will be used to enter the upcoming while loop. As of ArcGIS Enterprise 10.9, there are just a few different values that the status of a job can have; these states reflect the job's life cycle: esriJobSubmitted esriJobWaiting esriJobExecuting esriJobSucceeded esriJobFailed esriJobTimedOut esriJobCancelling esriJobCancelled Ideally, we are expecting that from the status perspective, the job's states will be: esriJobSubmitted --> (esriJobWaiting -->) esriJobExecuting --> esriJobSucceeded Accounting for the other values is what makes the testing of the service both fun and tricky. LoopJobStatus  The job status loop (also known as the while loop) has several parts to it. They all play an important role for periodically examining the status of the (test thread's) unique job id and then properly handling the state when it changes (e.g. succeeds or fails or just takes too long). While loop logic Job status check Short sleep timer There are also some nice-to-have extras (mentioned earlier as enhancements): Response assertion check  Maximum loop check WhileLoop As long as the returned job status is either "esriJobSubmitted", "esriJobWaiting", or "esriJobExecuting", the loop will continue running. Just checking against these three states helps keep the loop logic simple. Note: A job status of "esriJobSucceeded" will exit the loop. This is a good thing and what we want the test logic to encounter. JobStatusCheck The job status check is an HTTP request asking for the current value of the task. It does the same thing as the request inside the InitialJobStatus, but in a loop. Note: Since this element is inside a loop and there will most likely be multiple occurrences, it is important not to give the HTTP Request the same name as the Transaction. This helps avoid confusion in the analysis and reporting. ResponseAssertion  As mentioned, this logic is a nice-to-have. The value of the job status from the LoopJobStatus request is immediately checked. If it is "esriJobSubmitted", "esriJobWaiting", "esriJobExecuting" or "esriJobSucceeded", the request will be marked as successful. If any other job status states appear (e.g. "esriJobFailed", "esriJobTimedOut"), the request (and the loop transaction) will be marked as failed, which is a good thing. This design favors a simple approach to the testing logic. Note: The "esriJobSucceeded" is not a condition in the while loop but it is a value we look for with the ResponseAssertion rule to determine a successful request. SleepWhileLoop  The sleep timer is critical. Without it, too many status check requests are sent to the service which causes unnecessary load. Since the job status request is fast and light-weight but the overall task is long-running, it makes sense to delay each state check by a second or two (the test sleep variable is set to 2000ms). This is exactly what this timer does. Note: The Test Plan sleep variable, WhileLoopSleep is set to 2000 (ms), which is 2 seconds. IfWhileLoopMax The while loop iteration checking logic is also a nice-to-have. It has several parts it and the logic is carried out independently for each job. IfWhileLoopMax -- This element verifies whether the job status check loop has executed more than the allowed maximum amount of iterations (WhileLoopMax variable...default is 300); if the limit has been reached it carries out the following test elements: WhileLoopMaxReached -- An HTTP request identical to LoopJobStatus JSR223Listener -- This logic purposely fails the WhileLoopMaxReached request that was just sent FlowControlAction -- This logic ends the job status check loop by immediately stopping the current, individual test thread CounterWhileLoop -- There is a counter that is incremented for every iteration of the job status check  Note: The purpose of the IfWhileLoopMax logic is to stop the job, fail the loop operation and make it easy for the tester to see that a job is taking "too long" to execute the task. The Test Plan uses 2000 (ms) and 300 for the WhileLoopSleep and WhileLoopMax variables, respectively. This would allow for about a 10 minute job execution time. If your jobs run times are longer, please adjust these, as needed. IfJobSucceeded  Now out of the while loop (finally!), the test logic checks the last known status of the job. If it succeeded, it carries out some additional logic. GetParamUrl -- this HTTP request is identical to the LoopjobStatus (and the InitialJobStatus) elements The server response for this request is examined differently as it is parsed for the value of the paramUrl string DownloadOutput The value of the paramUrl variable is added to the end of the unique job request and the contents are downloaded. DownloadOutput -- this HTTP Request downloads the content whose name was based on the value of the paramUrl variable populated from the previous request ResponseAssertion -- a response assertion is added looking for the key word "rings" to validate that the contents actually contain a geometry Note: This step is optional, but it represents the full delivery of task...the data specific to the submitted job. Verifying the job's output contained geometry data helps the test show that the service was working as expected. Other jobs may produce an entirely different output, adjust the ResposneAssertion logic as needed. IfTestIterationMax The test iteration check is also a nice-to-have feature for load testing an asynchronous geoprocessing service. Its logic is very similar to the IfWhileLoopMax check. However, it keeps track of the total number of jobs (successful or failed) across all test threads. IfTestIterationMax -- This element verifies whether the amount of executed tests (e.g. jobs submitted) is more than the allowed maximum amount of iterations (TestIterationMax variable...default is 2500); if the limit has been reached it carries out the following test elements: TestIterationMaxReached -- An HTTP request identical to LoopJobStatus JSR223Listener -- This logic purposely fails the TestIterationMaxReached request that was just sent FlowControlAction -- This logic ends the load test by immediately stopping the all test threads CounterTestIteration -- There is a counter that is incremented for every test iteration (in other words, one job submitted equals one test iteration) Note: The purpose of the IfTestIterationMax logic is to stop the after a specific number job have been sent from the load test. Not all tests need to utilize this feature or hit this maximum. However, if you are experimenting with the test logic, it is a good strategy to set the maximum to a low value until you have verified that things are behaving as expected. Otherwise, your test might send many long-running jobs to the service at once, which in turn, could take a while to complete. This feature helps avoid that scenario. The Thread Group Configuration The JMeter Test Plan is currently configured for a 30 minute load test with each step lasting a little under 2 minutes.  Different environments and data may require an alternative setting to achieve the desired test results, adjust as needed The average "Summarize Invasive Species" job in this example takes between 8 -18 seconds If your service is significantly longer, you should adjust the Thread Group Configuration to produce a step duration longer than 2 minutes in order to obtain a decent sampling (per each step) Validating the Test Plan As a best testing practice, it is always a good idea to validate the results coming back from the server before applying the actual load. Use the View Results Tree listener to assist with the validation The Test Plan includes a View Results Tree Listener but it is disabled by default Enable it to view the results From the GUI, Start the test Transactions  Select and expand one of the "LoopJobStatus" Transactions The results should resemble the following: In this example, the LoopJobStatus transaction above contained 7 status check requests that all completed successfully and because of this, the job also succeeded The response time of the loop (e.g. the job) was just over 12 seconds (12066 ms) As more pressure is applied (e.g. via the load test), each job will require more status checks which will in turn, take longer to complete By design, this will show up as longer response times for the LoopJobStatus operation The response time of the LoopJobStatus transaction is a great measuring stick for judging the overall performance and scalability of an asynchronous geoprocessing service Note: Generally speaking, the "job status loop" component of an asynchronous geoprocessing service test will represent the bulk of time for every test iteration. All the other operations (SubmitJob, InitialJobStatus, DownloadOutput, etc...) typically happen very quickly. Requests Expand one of the "DownloadOutput" Transactions Select one of the https requests The results should resemble the following: The contents from the DownloadOutput request is helpful for validating that the job was able to produce an expected output In this case, it is a geometry formatted in JSON Based on the GP model used as the service, this geometry summarizes the range of invasive grass species near locations where people may come into contact with the grasses and facilitate their spread Note: Other geoprocessing services may produce a different type of output than the JSON shown in the example above. Test Execution The load test should be run in the same manner as a typical JMeter Test Plan. See the runMe.bat script included with the async_gp1.zip  project for an example on how to run a test as recommended by the Apache JMeter team.  The runMe.bat script contains a jmeterbin variable that will need to be set to the appropriate value for your environment Note: It is always recommended to coordinate the load test start time and duration with the appropriate personnel of your organization. This ensures minimal impact to users and other colleagues that may also need to use your on-premise ArcGIS Enterprise Site. Additionally, this helps prevent system noise from other activity and use which may "pollute" the test results. Note: For several reasons, it is strongly advised to never load test services provided by ArcGIS Online. JMeter Report Throughput Curves As expected for a long running job, the throughput of the LoopJobStatus operation is low The peak throughput appeared to occur at the 11:48 mark At this time, the service produced about 0.4 transactions/second This equated to around 1,440 jobs through the system per hour Note: The JobStatusCheck request is selected to disable its rendering in the chart. Since this was a fast request, it showed higher throughput than the LoopJobStatus operation, but that is not what we are interested in for understanding the scalability of the service. Performance Curves The performance of the job at the beginning of the test was about 14 seconds At the point where the throughput maxed out (the 11:48 mark), the performance had increased to over 21 seconds Despite an increased load that produced longer and longer response times, the service and system continue to complete the jobs successfully Final Thoughts While there are many geoprocessing models out there that perform a variety of different tasks, this Article can be used as a guide on how to load test an asynchronous GP service. As with many things related to testing, geoprocessing services are easy to apply load against. However, since each job has a life cycle that needs to be tracked, the test logic has to account for this changing job status. It is this characteristic of the service that introduces some complexity to the Test Plan. That said, Apache JMeter is a feature-rich testing framework that helps testers meet this challenge. To download the Apache JMeter Test Plan used in this Article see: async_gp1.zip  To download the geoprocessing package and data used in this Article see: Share a web tool  A Quick Word on Sizing The focus of previous testing articles has typically not been to offer strategies and techniques on capacity planning. However, generally speaking, long-running jobs like ones from an asynchronous geoprocessing service are relatively easy to size. For example, if the average job duration for a service is something long like 30 seconds and your ArcGIS Server machine has 8 CPU cores, you would be able to support 8 concurrent users (before queueing begins to occur). In other words, for the situation just mentioned, the rough sizing would be: 1 job  == 1 core == 1 user (where the response times are > 1 second) Note: This assumes the minimum number of instances for the GP service would be set to the number of available CPU cores (e.g. 8 on an 8 core machine). This setting would be done for predictable performance and maximum throughput. Would things fall over with more than 8 concurrent jobs are going? Most likely not, this is just when queuing starts to occur. Whenever queuing start to happens, the response times of the job completions will become a little longer (e.g. slower) for all the running jobs. Knowing this, do you still have to load test the GP? Most likely yes. All geoprocessing models, data and inputs are different, behave different and can use the available hardware in various ways. Your test will show this impact under load, which will be very important to understand. A Quick Word on Instances While it is possible to have the dedicated GP service instances set to use all the CPU cores on ArcGIS Server, a GIS Administrator may intentionally chose to not go with that configuration. Since the jobs of the GP service could be very long running and resource intensive, an alternate deployment strategy might be to purposely set the maximum number of instances for the GP service to a lower value so other users can use different services without waiting (due to resource constraints). A Quick Word on the Location of the arcgisjobs Folder While not a focus of this Article, the location of the arcgisjobs folder can have an impact on performance and/or scalability of the geoprocessing service. This location is where the service will temporarily read and write data as the job is being processed. The final output of each job is also stored in this location. For extremely busy Sites where thousands of jobs are concurrently being carried out from multiple ArcGIS Servers, consider the storage capabilities (e.g. I/O speed, reliability) of this location.       Apache JMeter released under the Apache License 2.0. Apache, Apache JMeter, JMeter, the Apache feather, and the Apache JMeter logo are trademarks of the Apache Software Foundation.   ... View more