The most popular reason is probably, "because you have been asked to test it". As a tester, GIS administrators may look to you to show how a geoprocessing (GP) model behaves as a service under load. Many GP models are built to perform long-running, critical and complex tasks. Since they are a resource often found on ArcGIS Enterprise deployments (as a service), this makes the understanding of their performance and scalability profiles key.
The hardest part of the load testing an asynchronous geoprocessing service is the loop logic and being able to handle the different states of the job. The loop should not be too aggressive and you need to exit the loop under the right conditions. Appropriately marking a test iteration as failed based on the job status or passed based on the output results from the task, is also critical.
The basic steps for load testing an asynchronous GP service are:
At a minimum, the Apache Test Plan should handle this logic. However, there are some enhancements that can be added to this process, like:
The Test Plan included in the Article provides these additional features.
The understanding of the process in this Article is most effective if the steps can be reproduced. For that, we can turn to a modern ArcGIS Pro GP model and dataset that is free and publicly available.
The Share a web tool -- Summarize Invasive Species package is available from arcgis.com.
Although the JMeter Test Plan will utilize some data to make the inputs to the model/service dynamic and more realistic, this will be pre-generated and automatically included with the test. The reason is to focus on the test logic (and not the data generation).
Note: The test has different variables for the name of the (GP) service and (GP) tool. When published, they often use the same name (e.g. "SummarizeInvasiveSpecies") but do not have to.
It all start with "submit the job". This is probably the easiest part of the test. Once this request has been sent, the service returns a job id and the server can begin to work on the task. This id is captured from the Regular Expression Extractor element.
Note: The job id is unique to each job (and test thread). It is used in every subsequent request in the test.
We briefly skip to the bottom to mention the CSV Data Set Config element. The inputs are important and required to submit a job, but the generation of its data but is not a heavy area of focus of this testing Article.
Note: The full input.csv file is included with the async_gp1 Test Plan.
The initial job status transaction has one HTTP request element inside that is used to find and populate a variable (jobStatus) with the current state of the submitted task. This value will be used to enter the upcoming while loop.
The job status loop (also known as the while loop) has several parts to it. They all play an important role for periodically examining the status of the (test thread's) unique job id and then properly handling the state when it changes (e.g. succeeds or fails or just takes too long).
There are also some nice-to-have extras (mentioned earlier as enhancements):
As long as the returned job status is either "esriJobSubmitted", "esriJobWaiting", or "esriJobExecuting", the loop will continue running. Just checking against these three states helps keep the loop logic simple.
Note: A job status of "esriJobSucceeded" will exit the loop. This is a good thing and what we want the test logic to encounter.
The job status check is an HTTP request asking for the current value of the task. It does the same thing as the request inside the InitialJobStatus, but in a loop.
Note: Since this element is inside a loop and there will most likely be multiple occurrences, it is important not to give the HTTP Request the same name as the Transaction. This helps avoid confusion in the analysis and reporting.
As mentioned, this logic is a nice-to-have. The value of the job status from the LoopJobStatus request is immediately checked. If it is "esriJobSubmitted", "esriJobWaiting", "esriJobExecuting" or "esriJobSucceeded", the request will be marked as successful. If any other job status states appear (e.g. "esriJobFailed", "esriJobTimedOut"), the request (and the loop transaction) will be marked as failed, which is a good thing. This design favors a simple approach to the testing logic.
Note: The "esriJobSucceeded" is not a condition in the while loop but it is a value we look for with the ResponseAssertion rule to determine a successful request.
The sleep timer is critical. Without it, too many status check requests are sent to the service which causes unnecessary load. Since the job status request is fast and light-weight but the overall task is long-running, it makes sense to delay each state check by a second or two (the test sleep variable is set to 2000ms). This is exactly what this timer does.
Note: The Test Plan sleep variable, WhileLoopSleep is set to 2000 (ms), which is 2 seconds.
The while loop iteration checking logic is also a nice-to-have. It has several parts it and the logic is carried out independently for each job.
Note: The purpose of the IfWhileLoopMax logic is to stop the job, fail the loop operation and make it easy for the tester to see that a job is taking "too long" to execute the task. The Test Plan uses 2000 (ms) and 300 for the WhileLoopSleep and WhileLoopMax variables, respectively. This would allow for about a 10 minute job execution time. If your jobs run times are longer, please adjust these, as needed.
Now out of the while loop (finally!), the test logic checks the last known status of the job. If it succeeded, it carries out some additional logic.
The value of the paramUrl variable is added to the end of the unique job request and the contents are downloaded.
Note: This step is optional, but it represents the full delivery of task...the data specific to the submitted job. Verifying the job's output contained geometry data helps the test show that the service was working as expected. Other jobs may produce an entirely different output, adjust the ResposneAssertion logic as needed.
The test iteration check is also a nice-to-have feature for load testing an asynchronous geoprocessing service. Its logic is very similar to the IfWhileLoopMax check. However, it keeps track of the total number of jobs (successful or failed) across all test threads.
Note: The purpose of the IfTestIterationMax logic is to stop the after a specific number job have been sent from the load test. Not all tests need to utilize this feature or hit this maximum. However, if you are experimenting with the test logic, it is a good strategy to set the maximum to a low value until you have verified that things are behaving as expected. Otherwise, your test might send many long-running jobs to the service at once, which in turn, could take a while to complete. This feature helps avoid that scenario.
The JMeter Test Plan is currently configured for a 30 minute load test with each step lasting a little under 2 minutes.
As a best testing practice, it is always a good idea to validate the results coming back from the server before applying the actual load.
Note: Generally speaking, the "job status loop" component of an asynchronous geoprocessing service test will represent the bulk of time for every test iteration. All the other operations (SubmitJob, InitialJobStatus, DownloadOutput, etc...) typically happen very quickly.
Note: Other geoprocessing services may produce a different type of output than the JSON shown in the example above.
The load test should be run in the same manner as a typical JMeter Test Plan.
See the runMe.bat script included with the async_gp1.zip project for an example on how to run a test as recommended by the Apache JMeter team.
Note: It is always recommended to coordinate the load test start time and duration with the appropriate personnel of your organization. This ensures minimal impact to users and other colleagues that may also need to use your on-premise ArcGIS Enterprise Site. Additionally, this helps prevent system noise from other activity and use which may "pollute" the test results.
Note: For several reasons, it is strongly advised to never load test services provided by ArcGIS Online.
Note: The JobStatusCheck request is selected to disable its rendering in the chart. Since this was a fast request, it showed higher throughput than the LoopJobStatus operation, but that is not what we are interested in for understanding the scalability of the service.
While there are many geoprocessing models out there that perform a variety of different tasks, this Article can be used as a guide on how to load test an asynchronous GP service. As with many things related to testing, geoprocessing services are easy to apply load against. However, since each job has a life cycle that needs to be tracked, the test logic has to account for this changing job status. It is this characteristic of the service that introduces some complexity to the Test Plan. That said, Apache JMeter is a feature-rich testing framework that helps testers meet this challenge.
The focus of previous testing articles has typically not been to offer strategies and techniques on capacity planning. However, generally speaking, long-running jobs like ones from an asynchronous geoprocessing service are relatively easy to size. For example, if the average job duration for a service is something long like 30 seconds and your ArcGIS Server machine has 8 CPU cores, you would be able to support 8 concurrent users (before queueing begins to occur).
In other words, for the situation just mentioned, the rough sizing would be:
Note: This assumes the minimum number of instances for the GP service would be set to the number of available CPU cores (e.g. 8 on an 8 core machine). This setting would be done for predictable performance and maximum throughput.
Would things fall over with more than 8 concurrent jobs are going? Most likely not, this is just when queuing starts to occur. Whenever queuing start to happens, the response times of the job completions will become a little longer (e.g. slower) for all the running jobs.
Knowing this, do you still have to load test the GP? Most likely yes. All geoprocessing models, data and inputs are different, behave different and can use the available hardware in various ways. Your test will show this impact under load, which will be very important to understand.
While it is possible to have the dedicated GP service instances set to use all the CPU cores on ArcGIS Server, a GIS Administrator may intentionally chose to not go with that configuration. Since the jobs of the GP service could be very long running and resource intensive, an alternate deployment strategy might be to purposely set the maximum number of instances for the GP service to a lower value so other users can use different services without waiting (due to resource constraints).
While not a focus of this Article, the location of the arcgisjobs folder can have an impact on performance and/or scalability of the geoprocessing service. This location is where the service will temporarily read and write data as the job is being processed. The final output of each job is also stored in this location. For extremely busy Sites where thousands of jobs are concurrently being carried out from multiple ArcGIS Servers, consider the storage capabilities (e.g. I/O speed, reliability) of this location.
Apache JMeter released under the Apache License 2.0. Apache, Apache JMeter, JMeter, the Apache feather, and the Apache JMeter logo are trademarks of the Apache Software Foundation.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.