This tutorial contains strategies and procedures for creating and running a dynamic load test against an ArcGIS Server service and assumes you are familiar with some of the basics of Apache JMeter. If you are new to Apache JMeter, please see our Performance Testing with Apache JMeter (An Introduction).
Before diving into the details on constructing and validating a data driven load test, it’s important to first discuss a few testing strategies specific to JMeter. These tips are different that items listed in our Recommended Strategies for Load Testing an ArcGIS Server Deployment Article which is intended for use with any testing tool (but also applicable to JMeter).
One tactic that can help with managing multiple tests and multiple reports in JMeter is to implement a simple but consistent folder structure for storing the various elements of each test. The primary driver for a JMeter test is the JMX file…this is where all the testing logic resides. But as you continue to add features to your test and run it multiple times, you will expand beyond the management of just the single JMX file.
A little organization goes a long way…manually creating a few key directories can help manage the various JMeter files that will populate the hard drive over time.
Note: This approach is done from the Operating System’s file system and not directly from Apache JMeter.
A recommended folder structure:
Within the "project" folder, 5 empty directories would be created: datasets, logs, reports, results and uploads.
Like the folder structure mentioned above, utilizing User Defined Variables is a test construction strategy that favors portability, reusability and Test Plan sharing. Some common variables used with an ArcGIS service test are:
In the JMeter Test Plan, these variables can be easily referenced with the notation like: ${ServiceName} which would automatically be replaced with the name of service at execution time. Using variables is less error prone than using hard-coded values if you ever plan to use the test in another environment. When the test is then run against another deployment, the tester only needs to adjust the variable definition. For Test Plans with many defined HTTP Request samples, this can be a real time saver.
There are several choices when it comes to defining your step load logic. They are all similar and offer the same basic functionality but do have some subtle differences. Thread Group is a good, safe choice that offers maximum test portability as it is included with the core JMeter product.
That said, using the JMeter Plugins Manager (installed as part of the Performance Testing with Apache JMeter (An Introduction)) to add the “Custom Thread Groups” can provide the test environment with some other useful options in this arena.
From the Plugins Manager within JMeter (Options-->Plugins Manager):
Once the “Custom Thread Groups” plugin is installed, a recommended alternative over Thread Group is the bzm – Concurrency Thread Group as it allows for a straight-forward step configuration that visually renders the defined pressure which is a great validation feature from a testing point of view.
Adding Transactions to your test can be a helpful way of grouping one or more similar requests that belong to the same “operation”. For example, the loading of an application, a navigational map zoom or form search. Using a transaction like this can greatly aid analysis as it isolates the operation (e.g. form search) into its own logical “container” with only the requests responsible for its function. Whereas, throwing many requests, for many different functions or services into one large group can make the post-test study very difficult. With transactions, analysis can be conducted to understand the performance of each operation, respectively.
Note: Map applications and workflows vary; some operations will have transactions composed of many requests, others just one request.
Note: The performance of different operations in a test does not always scale in the same way; some may do better than others; being able to identify poor performing operations (by a unique transaction name) is good analysis.
Ensuring the expected response is coming back from the remote server is just as import as sending the request. For various reason, when testing an ArcGIS resource like a service, relying on just the HTTP status code (e.g. HTTP 200) is insufficient.
One way to validate the response is to look for key words in the Header or Body. If a PNG image is requested but a textual error message is returned instead, a Response Assertion rule looking for “image/png” in the Header will mark the attempt as failed. If it is present in the Header, it is considered a passed request.
Note: For every HTTP Request that needs to be examined in the JMeter Test Plan, a Response Assertion rule must be added.
Using the strategies above, a flexible and versatile JMeter Test Plan can be created that calls for a different area of interest from an ArcGIS map service with every request. A “dynamic test” is ideal as it can make the server resources work harder than if the same data is requested every time. Such a test is more representative of real-world conditions.
As a performance tester, we want to take something like the export map request signature below and turn it into a dynamic JMeter load test:
https://yourwebadaptor.domain.com/server/rest/services/SampleWorldCities/MapServer/export?dpi=96&transparent=true&format=png32&layers=show%3A0%2C1%2C2&bbox=-108.76228873935%2C31.0409016308382%2C-88.8526618315487%2C46.9686031570791&bboxSR=4326&imageSR=4326&size=1477%2C827&f=image
JMeter offers several methodologies for constructing HTTP payloads in a Test Plan such as:
This Article will focus on the first method of showing the manually practice for creating a JMeter test, how to assemble the components of the request and how to make it's behavior “dynamic”. This data driven test will call the SampleWorldCities map service (running on your local deployment) through the export function.
Note: By many measures, SampleWorldCities is considered a small and very light-weight map service, but given that it is ubiquitous with ArcGIS Server deployments, it is ideal for using it to walk through the creation of a load test.
With the bzm – Concurrency Thread Group plugin, the configured step load logic can be easily seen and visually confirmed in the chart graphic above. This test will be set to run for 20 minutes, will increase pressure by using 10 different steps (adding 1 concurrent test thread at a time every 2 minutes), hit a maximum of 10 concurrent test threads then will stop.
Now we want to create a data driven HTTP request that will ask for a different spatial extent with each request. This is done with a regular JMeter HTTP Request.
A request signature for export map such as:
https://yourwebadaptor.domain.com/server/rest/services/SampleWorldCities/MapServer/export?dpi=96&transparent=true&format=png32&layers=show%3A0%2C1%2C2&bbox=-108.76228873935%2C31.0409016308382%2C-88.8526618315487%2C46.9686031570791&bboxSR=4326&imageSR=4326&size=1477%2C827&f=image
contains several URL components that need to go to different places on the HTTP Request page (to maximize flexibility and maintenance). Let’s start with the key/value pairs of the request by separating the URL parameters.
Below the parameters section has been modified to contain several JMeter variables instead of the original values. It is okay that some parts will reference JMeter variable names that have not yet created in the test.
dpi=96&transparent=true&format=png32&layers=show%3A0%2C1%2C2&bbox=${bbox_9244649}&bboxSR=${sr_9244649}&imageSR=${sr_9244649}&size=${width_9244649}%2C${height_9244649}&f=image
This will help put in place several pieces necessary to make certain parts of the HTTP Request dynamic.
bbox,width,height,mapUnits,sr,scale
"-108.76228873935,31.0409016308382,-88.8526618315487,46.9686031570791",1280,1024,esriDecimalDegrees,4326,9244649
"23.1282001284589,34.337748761293,43.03782703626,50.2654502875339",1280,1024,esriDecimalDegrees,4326,9244649
"-117.052786808947,35.3545948648786,-97.1431599011459,51.2822963911194",1280,1024,esriDecimalDegrees,4326,9244649
"-100.83549484975,36.0637839078502,-80.9258679419492,51.991485434091",1280,1024,esriDecimalDegrees,4326,9244649
"17.7030587822399,34.3302705716379,37.6126856900409,50.2579720978788",1280,1024,esriDecimalDegrees,4326,9244649
"32.2827904735659,37.2792197897781,52.192417381367,53.2069213160189",1280,1024,esriDecimalDegrees,4326,9244649
"0.222469446585188,35.0562794764081,20.1320963543862,50.9839810026489",1280,1024,esriDecimalDegrees,4326,9244649
"100.262489726575,20.8660021309943,120.172116634376,36.7937036572352",1280,1024,esriDecimalDegrees,4326,9244649
"72.4186084471192,22.1748804724666,92.3282353549203,38.1025819987075",1280,1024,esriDecimalDegrees,4326,9244649
"-61.2094273153097,-26.7402777397378,-41.2998004075086,-10.812576213497",1280,1024,esriDecimalDegrees,4326,9244649
"-72.5908131920005,-26.6270515481087,-52.6811862841995,-10.6993500218679",1280,1024,esriDecimalDegrees,4326,9244649
"99.7474167083529,20.7088254145549,119.657043616154,36.6365269407958",1280,1024,esriDecimalDegrees,4326,9244649
"102.59837658124,20.384726879328,122.508003489041,36.3124284055689",1280,1024,esriDecimalDegrees,4326,9244649
"80.8239612278688,23.4521454643548,100.73358813567,39.3798469905956",1280,1024,esriDecimalDegrees,4326,9244649
"-105.812716128284,31.0536390717042,-85.9030892204829,46.9813405979451",1280,1024,esriDecimalDegrees,4326,9244649
Note: This info is great for validating the test playback but also useful for performance troubleshooting
Many changes were made since the last Save.
To download the Apache JMeter Test Plan used in this Article see: sampleworldcities1.zip
To download a version of this Apache JMeter Test Plan with more detail covering additional map scales see: sampleworldcities2.zip
The Apache JMeter team recommends to not run load test from the GUI but to instead invoke the test from the command-line for optimal performance and resource utilization. Since this process can involve several switches, parameter adjustments and checks that are worth discussing in detail, it is a procedure that deserves its own Article called: Running an Apache JMeter Load Test from Command-line mode (Beginner/Intermediate).
Note: Please coordinate with your GIS team if your Apache JMeter test will be sending requests to a server that might impact other users. A load test should be scheduled to run during non-peak business hours.
For a set of additional testing tutorials (of increasing complexity), see: Performance Engineering: Load Testing ArcGIS Enterprise
Apache JMeter released under the Apache License 2.0. Apache, Apache JMeter, JMeter, the Apache feather, and the Apache JMeter logo are trademarks of the Apache Software Foundation.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.