Creating a Load Test in Apache JMeter Against a Network Analyst Route Service (Intermediate/Advanced)
Network Analyst Route
Simply put, the Network Analyst route solver is used for finding the quickest way to get from one place to another. This traveled path might just involve a start and end location but could optionally stop at several locations while also asking the solver to generate turn-by-turn directions for each route in the solution.
Note: Route functionality is available with a Network Analyst license.
Load Testing a Network Analyst Route Service
Network Analyst is packed with many capabilities and features for route solving. Such solutions can be executed through ArcGIS Pro but many times it is consumed through an ArcGIS service. Since it provides industry-leading technology for route solutions, it is logical to want to load test your locally running (route) solver service to see its scalability potential.
There are several types of analysis provided by the Network Analyst extension, this Article uses routes as they are very easy to work with...the only required inputs are at least two valid stops points. This characteristic makes it a good choice for demonstrating how to generate data and use it in a load test against a route service.
Note: The walkthrough in this Article used ArcGIS Pro 2.9 with Network Analyst services that ran in an ArcGIS Enterprise 10.9 deployment.
How to Test a Network Analyst Route Service?
Network Analyst ArcGIS Pro Tutorial Data
The understanding of the processes in this Article are most effective if the steps can be followed using the same data. For such a task, the Network Analyst team has made a great set of data available.
There is a tutorial found on arcgis.com called Network Analyst ArcGIS Pro Tutorial Data. Zipped, it is about 132MB and consists of Network Analyst data for several different cities: San Diego, and Paris, and San Francisco. The Geographic Coordinate System is: WGS 1984 (WKID: 4326). The data is publicly accessible.
Note: The examples in this Article will focus on the San Diego dataset.
- View of the San Diego Streets data from ArcGIS Pro (with Topographic Basemap):
- The Streets, Walking_Pathways or Network Dataset (NewSanDiego_ND) layers do not need to enabled to utilize the Network Analyst capabilities
- In the example above, they are enabled to act as a point of reference of the San Diego streets
- This Article will not cover the details of creating, configuring or publishing a network dataset in ArcGIS Enterprise. For information on such tasks, see:
- Create a network dataset tutorial that specifically uses this San Diego geodatabase
- Publish routing services
Note: The route solver examples in this Article use a map service (with the network analysis capability) as opposed to a geoprocessing service. The map service uses synchronous execution.
Test Data Generation
This testing effort will require valid stop points to use within the JMeter test.
As with other JMeter Articles on Community, we need good test data to get the most value from the results. And like before, the Load Testing Tools make short work of this job. There is even a specific tool for creating route data.
Version 1.3.0 adds some nice enhancements to the "Generate Data (Solve Route)" tool.
Making the Tools Available from ArcGIS Pro
Once the load-testing-tools project has been downloaded to your machine, place the unzipped folder in a directory that is accessible or made accessible by ArcGIS Pro. If you have a previous version of the Load Testing Tools already installed, this updated version can be placed along side it (although with a different folder name) or completely replace the previous version.
For example:
- Place the load-testing-tools folder in C:\Users\[username]\Documents\ArcGIS
- Use the Add Folder Connection from Catalog in ArcGIS Pro to list the contents of this directory:
The "Generate Data (Solve Route)" tool can create test data from the (map) service, a local copy of the data or the data within an enterprise geodatabase. For this example, any data in WGS 1984 (WKID: 4326) with an area of interest focusing around San Diego could be used.
Launch the Generate Data (Solve Route) Tool
- Launching the Generate Data (Solve Route) tool should present an interface similar to the following:
- In its simplest form, only the path of csv file, which will contain the stop points, needs to be specified
- However, while we want to generate random points to use as the stops, we would like to avoid creating them in the bays, lakes or ocean
- This is where the optional Constraining Polygon parameter comes in
- This input field can be used to reference a data layer to spatially limit where the points are generated
- This is where the optional Constraining Polygon parameter comes in
- In actuality, we will adjust all of the default values
- However, while we want to generate random points to use as the stops, we would like to avoid creating them in the bays, lakes or ocean
- View of the polygon (in pink) outlining the area of interest of the San Diego streets data in ArcGIS Pro:
Note: This polygon was created manually and is not included with the San Diego dataset
- To download the SanDiegoPolygon shapefile used in this Article see: SanDiegoPolygon.zip
Note: From a testing point of view, the polygon does not need to include every segment of the streets layer
The Generate Data (Solve Route) Tool Inputs
- Adjust the Number of Tests to:
- 1000
- Adjust the Stops Per Test to:
- 2
- Point the Constraining Polygon to:
- SanDiegoPolygon
- Set the Output to a file path location where the results will get written:
- C:\Users\[username]\Documents\ArcGIS\Projects\NetworkAnalystMap1\sandiegostops1.csv
- Click Run to execute the tool
- Examining the CSV file will reveal the generated stop data
- This data will be used directly in the Apache JMeter test as input
- Viewing the file in a text editor should show something similar to the following:
- The features of the route solver are amazingly vast and could accept other spatial data, for example:
- Barriers, Polyline Barriers, and Polygon Barriers are other inputs that could be passed into a request parameter
- The generation of these other inputs for route solver requests will not be covered in this Article
- Barriers, Polyline Barriers, and Polygon Barriers are other inputs that could be passed into a request parameter
Spatially Visualize the Generated Points
The generated points that are used for the stops in the requests can be added to the ArcGIS Pro project to spatially view their location.
- From ArcGIS Pro, use Catalog to locate and open the file geodatabase inside the project
- Locate the random_pts feature class
- Add the feature class to the Current Map:
The Route Solver Test Plan
- To download the Apache JMeter Test Plan used in this Article see: route_solver1.zip
- Opening the Test Plan in Apache JMeter should look similar to the following:
- Adjust the User Defined Variables to fit your environment
Note: The Apache JMeter release used for this Article was 5.4.3 (this version provides critical security updates for Apache Log4j2). It is strongly recommended that all Apache JMeter deployments run on the latest release.
HTTP Request
The route solve test is simple and fairly straight-forward. All of the test logic can be found within one JMeter HTTP Request object. Following the testing style used in previous Articles, this request item is placed inside a Transaction Controller.
The key/value pairs for the request in this JMeter test are based on two factors:
- The functionality available in the published Network Analyst service (and underlying data)
- The values in this test were taken directly from the default ones used from the REST endpoint of the published San Diego service, for example:
- The version of ArcGIS Enterprise (ArcGIS Server)
- Some versions add new capabilities
- This test is based on the published service from the San Diego dataset and ArcGIS Enterprise 10.9
- Some versions add new capabilities
Different network datasets may have different request parameter options available or populated, by default. Some parameters if enabled (like returnDirections), will tell the solver to return more information. This in turn asks the service to do more work which will increase the response time of the request.
Note: The view of the HTTP Request from the Table of Content (left side of Test Plan) will appear as a mix of JMeter variables and strings. This is by design. This values will become populated on playback (in the View Results Tree object and raw results file).
The Thread Group Configuration
The JMeter Test Plan is configured for a load test of 20 minutes. With this test example using two stops for each route request , the solver should perform well and return a good handful of samples (e.g. responses from the server) for each step.
- Different environments and data may require an alternative setting to achieve the desired test results, adjust the test thread settings as needed
Validating the Test Plan
As a best practice, it is always a good idea to validate the results coming back within the JMeter GUI before executing the actual load test from the command-line.
- Use the View Results Tree listener to assist with the validation
- The Test Plan for this Article includes a View Results Tree Listener but it is disabled
- Enable it to view the results when the test is played from the GUI
- The Test Plan for this Article includes a View Results Tree Listener but it is disabled
- From the GUI, Start the test
- Let the test run for 20 seconds or so
- Click Stop
- Let the test run for 20 seconds or so
Transactions
- Select one of the "Route" Transactions
- The View Results Tree section should resemble the following:
- In this example, all transactions completed successfully
- Sometimes when stopping the playback, the last Transactions in the View Results Tree may fail as it was stopped "mid-request"; this is safe to ignore
Requests
- Expand one of the "Route" Transactions
- Select the HTTPS request within it
- The results should resemble the following:
- In this example, the selected request completed successfully (as indicated by the green check mark)
- The success of the parent Transaction already indicated this status
- From the Sampler result tab, take a quick glance at the Size in bytes field
- In this example, the Request Size was about 15KB which usually means good geometry data was returned; in other words, the responses were not "empty" and is more proof that it was successful
- Examine the URL of the request
- As mentioned earlier, this value of the request URL becomes populated at runtime
- Click on Response data tab and Response Body sub-tab
- This shows a textual view of the data returned from the request:
Note: The route geometries returned are commonly rendered in web browser based JavaScript applications. Although Apache JMeter is a (test) client, it does not spatially render these geometry responses from the server in that way.
Test Execution
The load test should be run in the same manner as a typical JMeter Test Plan.
See the runMe.bat script included with the route_solver1.zip project for an example on how to run a test as recommended by the Apache JMeter team.
- The runMe.bat script contains a jmeterbin variable that will need to be set to the appropriate value for your environment
- If Network Analyst route service was published as dedicated, adjust the minimum and maximum instances accordingly prior to running the load test
- For more information see: Configure service instance settings
- The published route service used in this Article was dedicated with the maximum instances set to 4
- The ArcGIS Server component was running on a system with 4 CPU cores
Note: It is always recommended to coordinate the load test start time and duration with the appropriate personnel of your organization. This ensures minimal impact to users and other colleagues that may also need to use your on-premise ArcGIS Enterprise Site. Additionally, this helps prevent system noise from other activity and use which may "pollute" the test results.
Note: For several reasons, it is strongly advised to never load test ArcGIS Online.
JMeter Report
Throughput Curve
- The auto-generated JMeter Report can provide insight into the throughput of the route service under load
- Since each Route Transaction contained one request, both metrics (request and transaction) virtually showed the same value; this is expected given the design of the test
- In this case, the peak throughput for the two stop route solves was about 15 transactions/second
- Given the environment tested, this equates to around 54,000 route solves/hour
Performance Curves
- The auto-generated JMeter Report can also provide insight into the performance of the route service under load
- Since each Route Transaction contained one request, both metrics (request and transaction) virtually showed the same value; this is expected given the design of the test
- The performance of the route requests was good and under 1 second throughout the load test
- Where the throughput first peaked at 15 transactions/second is where the response time was measured
- At this point in the test, the average response time was about 333 ms or 0.33 seconds
- Where the throughput first peaked at 15 transactions/second is where the response time was measured
- It may also be helpful to see the plotted response times with respect to the step load (configured threads)
- Previous charts showed values with respect to time
Final Thoughts
The Apache JMeter Test Plan in this Article represents a programmatic approach for applying load to a Network Analyst route service. One of the strengths of this test is that it easy to configure and maintain.
The auto-generated JMeter report provides charts and summaries that can be used to quickly analyze the performance and scalability of the route service.
- To download the Apache JMeter Test Plan used in this Article see: route_solver1.zip
- To download the San Diego dataset used in this Article see: Network Analyst ArcGIS Pro Tutorial Data
Apache JMeter released under the Apache License 2.0. Apache, Apache JMeter, JMeter, the Apache feather, and the Apache JMeter logo are trademarks of the Apache Software Foundation.