Load Test an Asynchronous Geoprocessing Service Using Apache JMeter (Advanced)

AaronLopez · ‎02-18-2022

Why Test a Asynchronous Geoprocessing Service?

The most popular reason is probably, "because you have been asked to test it". As a tester, GIS administrators may look to you to show how a geoprocessing (GP) model behaves as a service under load. Many GP models are built to perform long-running, critical and complex tasks. Since they are a resource often found on ArcGIS Enterprise deployments (as a service), this makes the understanding of their performance and scalability profiles key.

Asynchronous Geoprocessing Service Testing Challenges

The hardest part of the load testing an asynchronous geoprocessing service is the loop logic and being able to handle the different states of the job. The loop should not be too aggressive and you need to exit the loop under the right conditions. Appropriately marking a test iteration as failed based on the job status or passed based on the output results from the task, is also critical.

How to Test an Asynchronous Geoprocessing Service?

The basic steps for load testing an asynchronous GP service are:

Provide inputs for geoprocessing task
- Submit job
  - Capture unique job ID
Perform an initial job status check
Loop on the job status check
- If job succeeded
  - Exit loop
- Sleep for a short duration
If job succeeded
- Examine results
  - If available, download output/data

At a minimum, the Apache Test Plan should handle this logic. However, there are some enhancements that can be added to this process, like:

Maximum number of job status checks in the loop
Maximum number of test iterations
Marking the job as failed if a non-successful status is returned

The Test Plan included in the Article provides these additional features.

The "Summarize Invasive Species" Geoprocessing Model and Dataset

The understanding of the process in this Article is most effective if the steps can be reproduced. For that, we can turn to a modern ArcGIS Pro GP model and dataset that is free and publicly available.

The Share a web tool -- Summarize Invasive Species package is available from arcgis.com.

View of the New Zealand data from ArcGIS Pro (with Topographic Basemap):

This Article will not cover the details of configuring or publishing this model as a service in ArcGIS Enterprise. For information on such actions, see:
- Share a web tool

Test Data Generation

Although the JMeter Test Plan will utilize some data to make the inputs to the model/service dynamic and more realistic, this will be pre-generated and automatically included with the test. The reason is to focus on the test logic (and not the data generation).

The Asynchronous Geoprocessing Service Test Plan

To download the Apache JMeter Test Plan used in this Article see: async_gp1.zip
Opening the Test Plan in Apache JMeter should look similar to the following:
- Adjust the User Defined Variables to fit your environment

Note: The test has different variables for the name of the (GP) service and (GP) tool. When published, they often use the same name (e.g. "SummarizeInvasiveSpecies") but do not have to.

Components of the Test Plan

SubmitJob

It all start with "submit the job". This is probably the easiest part of the test. Once this request has been sent, the service returns a job id and the server can begin to work on the task. This id is captured from the Regular Expression Extractor element.

Note: The job id is unique to each job (and test thread). It is used in every subsequent request in the test.

CSV Data Set Config

We briefly skip to the bottom to mention the CSV Data Set Config element. The inputs are important and required to submit a job, but the generation of its data but is not a heavy area of focus of this testing Article.

Contents example of the inputs.csv file:

Note: The full input.csv file is included with the async_gp1 Test Plan.

InitialJobStatus

The initial job status transaction has one HTTP request element inside that is used to find and populate a variable (jobStatus) with the current state of the submitted task. This value will be used to enter the upcoming while loop.

As of ArcGIS Enterprise 10.9, there are just a few different values that the status of a job can have; these states reflect the job's life cycle:
- esriJobSubmitted
- esriJobWaiting
- esriJobExecuting
- esriJobSucceeded
- esriJobFailed
- esriJobTimedOut
- esriJobCancelling
- esriJobCancelled
Ideally, we are expecting that from the status perspective, the job's states will be:
- esriJobSubmitted --> (esriJobWaiting -->) esriJobExecuting --> esriJobSucceeded
Accounting for the other values is what makes the testing of the service both fun and tricky.

LoopJobStatus

The job status loop (also known as the while loop) has several parts to it. They all play an important role for periodically examining the status of the (test thread's) unique job id and then properly handling the state when it changes (e.g. succeeds or fails or just takes too long).

While loop logic
Job status check
Short sleep timer

There are also some nice-to-have extras (mentioned earlier as enhancements):

Response assertion check
Maximum loop check

WhileLoop

As long as the returned job status is either "esriJobSubmitted", "esriJobWaiting", or "esriJobExecuting", the loop will continue running. Just checking against these three states helps keep the loop logic simple.

Note: A job status of "esriJobSucceeded" will exit the loop. This is a good thing and what we want the test logic to encounter.

JobStatusCheck

The job status check is an HTTP request asking for the current value of the task. It does the same thing as the request inside the InitialJobStatus, but in a loop.

Note: Since this element is inside a loop and there will most likely be multiple occurrences, it is important not to give the HTTP Request the same name as the Transaction. This helps avoid confusion in the analysis and reporting.

ResponseAssertion

As mentioned, this logic is a nice-to-have. The value of the job status from the LoopJobStatus request is immediately checked. If it is "esriJobSubmitted", "esriJobWaiting", "esriJobExecuting" or "esriJobSucceeded", the request will be marked as successful. If any other job status states appear (e.g. "esriJobFailed", "esriJobTimedOut"), the request (and the loop transaction) will be marked as failed, which is a good thing. This design favors a simple approach to the testing logic.

Note: The "esriJobSucceeded" is not a condition in the while loop but it is a value we look for with the ResponseAssertion rule to determine a successful request.

SleepWhileLoop

The sleep timer is critical. Without it, too many status check requests are sent to the service which causes unnecessary load. Since the job status request is fast and light-weight but the overall task is long-running, it makes sense to delay each state check by a second or two (the test sleep variable is set to 2000ms). This is exactly what this timer does.

Note: The Test Plan sleep variable, WhileLoopSleep is set to 2000 (ms), which is 2 seconds.

IfWhileLoopMax

The while loop iteration checking logic is also a nice-to-have. It has several parts it and the logic is carried out independently for each job.

IfWhileLoopMax -- This element verifies whether the job status check loop has executed more than the allowed maximum amount of iterations (WhileLoopMax variable...default is 300); if the limit has been reached it carries out the following test elements:
- WhileLoopMaxReached -- An HTTP request identical to LoopJobStatus
- JSR223Listener -- This logic purposely fails the WhileLoopMaxReached request that was just sent
- FlowControlAction -- This logic ends the job status check loop by immediately stopping the current, individual test thread
CounterWhileLoop -- There is a counter that is incremented for every iteration of the job status check

Note: The purpose of the IfWhileLoopMax logic is to stop the job, fail the loop operation and make it easy for the tester to see that a job is taking "too long" to execute the task. The Test Plan uses 2000 (ms) and 300 for the WhileLoopSleep and WhileLoopMax variables, respectively. This would allow for about a 10 minute job execution time. If your jobs run times are longer, please adjust these, as needed.

IfJobSucceeded

Now out of the while loop (finally!), the test logic checks the last known status of the job. If it succeeded, it carries out some additional logic.

GetParamUrl -- this HTTP request is identical to the LoopjobStatus (and the InitialJobStatus) elements
- The server response for this request is examined differently as it is parsed for the value of the paramUrl string

DownloadOutput

The value of the paramUrl variable is added to the end of the unique job request and the contents are downloaded.

DownloadOutput -- this HTTP Request downloads the content whose name was based on the value of the paramUrl variable populated from the previous request
ResponseAssertion -- a response assertion is added looking for the key word "rings" to validate that the contents actually contain a geometry

Note: This step is optional, but it represents the full delivery of task...the data specific to the submitted job. Verifying the job's output contained geometry data helps the test show that the service was working as expected. Other jobs may produce an entirely different output, adjust the ResposneAssertion logic as needed.

IfTestIterationMax

The test iteration check is also a nice-to-have feature for load testing an asynchronous geoprocessing service. Its logic is very similar to the IfWhileLoopMax check. However, it keeps track of the total number of jobs (successful or failed) across all test threads.

IfTestIterationMax -- This element verifies whether the amount of executed tests (e.g. jobs submitted) is more than the allowed maximum amount of iterations (TestIterationMax variable...default is 2500); if the limit has been reached it carries out the following test elements:
- TestIterationMaxReached -- An HTTP request identical to LoopJobStatus
- JSR223Listener -- This logic purposely fails the TestIterationMaxReached request that was just sent
- FlowControlAction -- This logic ends the load test by immediately stopping the all test threads
CounterTestIteration -- There is a counter that is incremented for every test iteration (in other words, one job submitted equals one test iteration)

Note: The purpose of the IfTestIterationMax logic is to stop the after a specific number job have been sent from the load test. Not all tests need to utilize this feature or hit this maximum. However, if you are experimenting with the test logic, it is a good strategy to set the maximum to a low value until you have verified that things are behaving as expected. Otherwise, your test might send many long-running jobs to the service at once, which in turn, could take a while to complete. This feature helps avoid that scenario.

The Thread Group Configuration

The JMeter Test Plan is currently configured for a 30 minute load test with each step lasting a little under 2 minutes.

Different environments and data may require an alternative setting to achieve the desired test results, adjust as needed
- The average "Summarize Invasive Species" job in this example takes between 8 -18 seconds
  - If your service is significantly longer, you should adjust the Thread Group Configuration to produce a step duration longer than 2 minutes in order to obtain a decent sampling (per each step)

Validating the Test Plan

As a best testing practice, it is always a good idea to validate the results coming back from the server before applying the actual load.

Use the View Results Tree listener to assist with the validation
- The Test Plan includes a View Results Tree Listener but it is disabled by default
  - Enable it to view the results
From the GUI, Start the test

Transactions

Select and expand one of the "LoopJobStatus" Transactions
- The results should resemble the following:

In this example, the LoopJobStatus transaction above contained 7 status check requests that all completed successfully and because of this, the job also succeeded
The response time of the loop (e.g. the job) was just over 12 seconds (12066 ms)
As more pressure is applied (e.g. via the load test), each job will require more status checks which will in turn, take longer to complete
- By design, this will show up as longer response times for the LoopJobStatus operation
The response time of the LoopJobStatus transaction is a great measuring stick for judging the overall performance and scalability of an asynchronous geoprocessing service

Note: Generally speaking, the "job status loop" component of an asynchronous geoprocessing service test will represent the bulk of time for every test iteration. All the other operations (SubmitJob, InitialJobStatus, DownloadOutput, etc...) typically happen very quickly.

Requests

Expand one of the "DownloadOutput" Transactions
Select one of the https requests
- The results should resemble the following:

The contents from the DownloadOutput request is helpful for validating that the job was able to produce an expected output
- In this case, it is a geometry formatted in JSON
  - Based on the GP model used as the service, this geometry summarizes the range of invasive grass species near locations where people may come into contact with the grasses and facilitate their spread

Note: Other geoprocessing services may produce a different type of output than the JSON shown in the example above.

Test Execution

The load test should be run in the same manner as a typical JMeter Test Plan.

See the runMe.bat script included with the async_gp1.zip project for an example on how to run a test as recommended by the Apache JMeter team.

The runMe.bat script contains a jmeterbin variable that will need to be set to the appropriate value for your environment

Note: It is always recommended to coordinate the load test start time and duration with the appropriate personnel of your organization. This ensures minimal impact to users and other colleagues that may also need to use your on-premise ArcGIS Enterprise Site. Additionally, this helps prevent system noise from other activity and use which may "pollute" the test results.

Note: For several reasons, it is strongly advised to never load test services provided by ArcGIS Online.

JMeter Report

Throughput Curves

As expected for a long running job, the throughput of the LoopJobStatus operation is low
- The peak throughput appeared to occur at the 11:48 mark
  - At this time, the service produced about 0.4 transactions/second
    - This equated to around 1,440 jobs through the system per hour

Note: The JobStatusCheck request is selected to disable its rendering in the chart. Since this was a fast request, it showed higher throughput than the LoopJobStatus operation, but that is not what we are interested in for understanding the scalability of the service.

Performance Curves

The performance of the job at the beginning of the test was about 14 seconds
- At the point where the throughput maxed out (the 11:48 mark), the performance had increased to over 21 seconds
- Despite an increased load that produced longer and longer response times, the service and system continue to complete the jobs successfully

Final Thoughts

While there are many geoprocessing models out there that perform a variety of different tasks, this Article can be used as a guide on how to load test an asynchronous GP service. As with many things related to testing, geoprocessing services are easy to apply load against. However, since each job has a life cycle that needs to be tracked, the test logic has to account for this changing job status. It is this characteristic of the service that introduces some complexity to the Test Plan. That said, Apache JMeter is a feature-rich testing framework that helps testers meet this challenge.

To download the Apache JMeter Test Plan used in this Article see: async_gp1.zip
To download the geoprocessing package and data used in this Article see: Share a web tool

A Quick Word on Sizing

The focus of previous testing articles has typically not been to offer strategies and techniques on capacity planning. However, generally speaking, long-running jobs like ones from an asynchronous geoprocessing service are relatively easy to size. For example, if the average job duration for a service is something long like 30 seconds and your ArcGIS Server machine has 8 CPU cores, you would be able to support 8 concurrent users (before queueing begins to occur).

In other words, for the situation just mentioned, the rough sizing would be:

1 job == 1 core == 1 user (where the response times are > 1 second)

Note: This assumes the minimum number of instances for the GP service would be set to the number of available CPU cores (e.g. 8 on an 8 core machine). This setting would be done for predictable performance and maximum throughput.

Would things fall over with more than 8 concurrent jobs are going? Most likely not, this is just when queuing starts to occur. Whenever queuing start to happens, the response times of the job completions will become a little longer (e.g. slower) for all the running jobs.

Knowing this, do you still have to load test the GP? Most likely yes. All geoprocessing models, data and inputs are different, behave different and can use the available hardware in various ways. Your test will show this impact under load, which will be very important to understand.

A Quick Word on Instances

While it is possible to have the dedicated GP service instances set to use all the CPU cores on ArcGIS Server, a GIS Administrator may intentionally chose to not go with that configuration. Since the jobs of the GP service could be very long running and resource intensive, an alternate deployment strategy might be to purposely set the maximum number of instances for the GP service to a lower value so other users can use different services without waiting (due to resource constraints).

A Quick Word on the Location of the arcgisjobs Folder

While not a focus of this Article, the location of the arcgisjobs folder can have an impact on performance and/or scalability of the geoprocessing service. This location is where the service will temporarily read and write data as the job is being processed. The final output of each job is also stored in this location. For extremely busy Sites where thousands of jobs are concurrently being carried out from multiple ArcGIS Servers, consider the storage capabilities (e.g. I/O speed, reliability) of this location.

Apache JMeter released under the Apache License 2.0. Apache, Apache JMeter, JMeter, the Apache feather, and the Apache JMeter logo are trademarks of the Apache Software Foundation.