Cached map services are a popular and recommended way to provide a well performing presentation of static data. The cache service type is a proven technology, but there may still be requirements to test it under load to observe its scalability first hand on a specific deployment architecture. While cached map services perform well, serving up thousands of simultaneous tile requests can be resource intensive on the server hardware.
Note: Due to the fast rate of delivery and consumption of the resource, load testing cached map services can also be intensive on the hardware utilization of the test client workstation.
Compared to the load testing of the export map function, proper testing of a cached map service introduces several challenges as the request composition with each map screen changes. Since the underlying cache scheme is using a grid design, the map extents of some pans or zooms may pull down more or less tile images than others. Accounting for this real-world behavior of the cache service makes the test logic more complex than if it were exercising the export map function.
The test logic should also be dynamic and cover a decent area of interest. Converting a HAR file of captured cache tile requests into a test might be quick and easy to do but does not show a realistic scalability of the service. This is due to the small sample of tile requests being used over and over again.
Generally speaking, requests for individual cache tiles are fast...very fast. Due to this behavior, the test logic also needs to perform well, scale with the service and have minimal overhead on the test client
The steps in this Article should work with any existing cached map service on your local ArcGIS Enterprise deployment. However, if one if not available, it is recommended to give the Natural Earth dataset a look for the task.
Although the steps should work with any data, the walkthrough of the process in this Article might be more effective if they can be directly followed. In such cases, it is great turning to the Natural Earth datasets which provides some decent map detail (at smaller scales) covering the whole world.
Note: It is recommended to become familiar with some of metadata details of the cached map service as the load testing effort will require knowledge of some of that information (e.g. xorigin, yorigin, tileCols, tileRows, and spatial reference as well as the scales that contain tiles).
With a cached map service available, the next step would be to generate test data over an area of interest.
As with other JMeter Articles on Community, we need good test data to get the most value from the results. And like before, the Load Testing Tools package (for ArcGIS Pro) makes short work of this job. There is even a specific tool for creating bounding box data to use with a cached map services.
Note: Version 1.3.0 of Load Testing Tools added the "Generate Bounding Boxes (Precision)" tool.
Download and unzip the package then make that folder available to your ArcGIS Pro project.
The CSV Data Set Config elements in JMeter are used to reference the newly generated test data from the file system. The current version of the Test Plan is built to utilize 3 different CSV files (one for each map scale data file).
Note: Other that the User Defined Variables and the setting of the Filename in the CSV Data Set Config elements, there should not be anything else that requires editing or changing in the Test Plan. The test logic is listed below just to explain how the values in the HTTP Request become populated.
To avoid more complex JMeter test logic, 24 fixed map cache levels of detail are placed inside a class in a JSR223 Sampler test element. That "complex alternative" would be to connect the endpoint of the service at the start of the test and pull down the cache tile metadata. Putting HTTP logic into JSR223 Samplers is technically doable, but not the route I chose.
Levels Of Detail -- JSR223 Sampler (Full Logic):
// FileServer class
import org.apache.jmeter.services.FileServer
public class Lod{
int level
double resolution
double scale
double tolerance
}
public class MyLodList1{
public List<Lod> LodList = new ArrayList()
MyLodList1(){
// Based on ArcGIS Online Map Scales
// https://services.arcgisonline.com/arcgis/rest/services/World_Street_Map/MapServer
//
// Spatial Reference: 102100 (3857)
Lod lod = new Lod()
lod = new Lod()
lod.level = 0
lod.resolution = 156543.03392800014 //11
lod.scale = 591657527.591555
lod.tolerance = 0.5
this.LodList.add(lod)
lod = new Lod()
lod.level = 1
lod.resolution = 78271.51696399994 //11
lod.scale = 295828763.795777
lod.tolerance = 0.5
this.LodList.add(lod)
lod = new Lod()
lod.level = 2
lod.resolution = 39135.75848200009 //11
lod.scale = 147914381.897889
lod.tolerance = 0.25
this.LodList.add(lod)
lod = new Lod()
lod.level = 3
lod.resolution = 19567.87924099992 //11
lod.scale = 73957190.948944
lod.tolerance = 0.5
this.LodList.add(lod)
lod = new Lod()
lod.level = 4
lod.resolution = 9783.93962049996 //11
lod.scale = 36978595.474472
lod.tolerance = 0.5
this.LodList.add(lod)
lod = new Lod()
lod.level = 5
lod.resolution = 4891.96981024998 //11
lod.scale = 18489297.737236
lod.tolerance = 0.5
this.LodList.add(lod)
lod = new Lod()
lod.level = 6
lod.resolution = 2445.98490512499 //11
lod.scale = 9244648.868618
lod.tolerance = 0.5
this.LodList.add(lod)
lod = new Lod()
lod.level = 7
lod.resolution = 1222.9924525624949 //13
lod.scale = 4622324.434309
lod.tolerance = 0.5
this.LodList.add(lod)
lod = new Lod()
lod.level = 8
lod.resolution = 611.49622628137968 //14
lod.scale = 2311162.217155
lod.tolerance = 0.5
this.LodList.add(lod)
lod = new Lod()
lod.level = 9
lod.resolution = 305.74811314055756 //14
lod.scale = 1155581.108577
lod.tolerance = 0.5
this.LodList.add(lod)
lod = new Lod()
lod.level = 10
lod.resolution = 152.87405657041106 //14
lod.scale = 577790.554289
lod.tolerance = 0.5
this.LodList.add(lod)
lod = new Lod()
lod.level = 11
lod.resolution = 76.437028285073239 //15
lod.scale = 288895.277144
lod.tolerance = 0.5
this.LodList.add(lod)
lod = new Lod()
lod.level = 12
lod.resolution = 38.21851414253662 //14
lod.scale = 144447.638572
lod.tolerance = 0.5
this.LodList.add(lod)
lod = new Lod()
lod.level = 13
lod.resolution = 19.10925707126831 //15
lod.scale = 72223.819286
lod.tolerance = 0.5
this.LodList.add(lod)
lod = new Lod()
lod.level = 14
lod.resolution = 9.5546285356341549 //16
lod.scale = 36111.909643
lod.tolerance = 0.5
this.LodList.add(lod)
lod = new Lod()
lod.level = 15
lod.resolution = 4.77731426794937 //14
lod.scale = 18055.954822
lod.tolerance = 0.05
this.LodList.add(lod)
lod = new Lod()
lod.level = 16
lod.resolution = 2.388657133974685 //15
lod.scale = 9027.977411
lod.tolerance = 0.025
this.LodList.add(lod)
lod = new Lod()
lod.level = 17
lod.resolution = 1.1943285668550503 //16
lod.scale = 4513.988705
lod.tolerance = 0.025
this.LodList.add(lod)
lod = new Lod()
lod.level = 18
lod.resolution = 0.5971642835598172 //16
lod.scale = 2256.994353
lod.tolerance = 0.005
this.LodList.add(lod)
lod = new Lod()
lod.level = 19
lod.resolution = 0.29858214164761665 //17
lod.scale = 1128.497176
lod.tolerance = 0.005
this.LodList.add(lod)
lod = new Lod()
lod.level = 20
lod.resolution = 0.14929107082380833 //17
lod.scale = 564.248588
lod.tolerance = 0.0025
this.LodList.add(lod)
lod = new Lod()
lod.level = 21
lod.resolution = 0.07464553541190416 //17
lod.scale = 282.124294
lod.tolerance = 0.0005
this.LodList.add(lod)
lod = new Lod()
lod.level = 22
lod.resolution = 0.03732276770595208 //17
lod.scale = 141.062147
lod.tolerance = 0.0005
this.LodList.add(lod)
lod = new Lod()
lod.level = 23
lod.resolution = 0.01866138385297604 //17
lod.scale = 70.5310735
lod.tolerance = 0.0005
this.LodList.add(lod)
}
}
MyLodList1 mylods = new MyLodList1()
List<Lod> LodList = mylods.LodList
vars.putObject("LodList",LodList)
The JSR223 Samplers inside the GetMapTile Transaction is the logic responsible for taking a bounding box and transforming it into the corresponding cache tiles.
Note: JSR223 Samplers using Groovy are generally executed quickly and add very little overhead to the test
GetMapTile -- JSR223 Sampler A1 (Full Logic):
// Script to process a CSV file (from Load Testing Tools) with lines in the following format:
// bbox,width,height,mapUnits,sr,scale
// FileServer class
import org.apache.jmeter.services.FileServer
import org.apache.commons.math3.util.Precision
//import java.math.BigDecimal
// GetMapTile
bbox_var = vars.get("bbox_A")
String[] bboxParts = bbox_var.split(',')
double xmin = Double.parseDouble(bboxParts[0])
double ymin = Double.parseDouble(bboxParts[1])
double xmax = Double.parseDouble(bboxParts[2])
double ymax = Double.parseDouble(bboxParts[3])
width_var = vars.get("width_A")
height_var = vars.get("height_A")
// Use map scale resolution (map units per pixel) to determine tile level
double mapresolution = 0
int resolutionprecision = 10
mapresolution = Precision.round((Math.abs(xmax - xmin) / Double.parseDouble(width_var)), resolutionprecision)
scale_var = vars.get("scale_A")
double bbox_scale_double = Double.parseDouble(scale_var)
// Map units per pixel
double tileresolution = 0
double lod_resolution = 0
double scale = 0
int tilelevel = 0
LodList = vars.getObject("LodList") // Assuming cached map service has a Spatial Reference of 102100 (3857)
boolean firstIteration = true;
for(int i = 0; i < LodList.size; i++)
{
lod_resolution = Precision.round(LodList[i].resolution, resolutionprecision)
tileresolution = lod_resolution
tilelevel = LodList[i].level
scale = LodList[i].scale
if (mapresolution >= lod_resolution)
{
break
}
}
tileCols_var = vars.get("TileCols")
cols = Double.parseDouble(tileCols_var)
tileRows_var = vars.get("TileRows")
rows = Double.parseDouble(tileRows_var)
// Origin of the cache (upper left corner)
xorigin_var = vars.get("Xorigin")
xorigin = Double.parseDouble(xorigin_var)
yorigin_var = vars.get("Yorigin")
yorigin = Double.parseDouble(yorigin_var)
// Get minimum tile column
double minxtile = (xmin - xorigin) / (cols * tileresolution)
// Get minimum tile row
// From the origin, maxy is minimum y
double minytile = (yorigin - ymax) / (rows * tileresolution)
// Get maximum tile column
double maxxtile = (xmax - xorigin) / (cols * tileresolution)
// Get maximum tile row
// From the origin, miny is maximum y
double maxytile = (yorigin - ymin) / (rows * tileresolution)
// Return integer value for min and max, row and column
int mintilecolumn = (int)Math.floor(minxtile)
int mintilerow = (int)Math.floor(minytile)
int maxtilecolumn = (int)Math.floor(maxxtile)
int maxtilerow = (int)Math.floor(maxytile)
Scheme_var = vars.get("Scheme")
WebServerName_var = vars.get("WebServerName")
ServerInstanceName_var = vars.get("ServerInstanceName")
ServiceName_var = vars.get("ServiceName")
ServiceType_var = vars.get("ServiceType")
def cacheRequest
def tilePaths = []
int count = 0
for (int row = mintilerow; row <= maxtilerow; row++)
{
// for each column in the row, in the map extent
for (int col = mintilecolumn; col <= maxtilecolumn; col++)
{
cacheRequest = ("/").concat(ServerInstanceName_var).concat("/rest/services/").concat(ServiceName_var).concat("/").concat(ServiceType_var)
cacheRequest = cacheRequest.concat("/tile").concat("/").concat(tilelevel.toString()).concat("/").concat(row.toString()).concat("/").concat(col.toString())
count++
tilePaths.add(cacheRequest)
}
}
def requestCount = count.toString()
vars.putObject("RequestCount_A",requestCount)
vars.putObject("TilePaths_A",tilePaths)
There are several components needs for this part of the Test Plan. With the bounding box translated into the corresponding cache tiles and assembled into a list of URLs, a third JSR223 is needed to place each URL into a variable inside a loop. The loop logic takes place inside the Cache Tiles transaction.
All of the test logic above exists just for this component of the test. For each map scale, there is only one HTTP Request! This simple design favors readability and maintainability.
Note: The HTTP Requests contains a Response Assertion element to validate the items returned from the server. If the content type of the response is image/jpeg or image/png, then the request will pass. However, some VectorTileServer caches may return a Protocolbuffer Binary Format (*.pbf) file. In these cases, the Patterns to Test would need to be manually expanded to the following: image/jpeg || image/png || application/octet-stream || application/x-protobuf
The JMeter Test Plan is currently configured for a relatively short test of 20 minutes. Cached map services perform well, so a lot of throughput will be taking place within each step (2 minutes per step) and from the test overall.
As a best practice, it is always a good idea to validate the results coming back before executing the actual load test.
Note: Once visual validation and debugging is complete, it is recommended to disable the View Results Tree element before executing the load test
The load test should be run in the same manner as a typical JMeter Test Plan.
See the runMe.bat script included with the cache_tiles1.zip project for an example on how to run a test as recommended by Apache JMeter.
Note: It is always recommended to coordinate the load test start time and duration with the appropriate personnel of your organization. This ensures minimal impact to users and other colleagues that may also need to use your on-premise ArcGIS Enterprise Site. Additionally, this helps prevent system noise from other activity and use which may "pollute" your test results.
Note: For several reasons, it is strongly advised to never load test ArcGIS Online.
Note: "Peak throughput" is a point in a test where no higher throughput can be achieved. This does not mean that is the maximum amount of pressure the service will support without "falling over". Generally speaking, if additional users ask for cache tiles after the system has reached peak throughput (e.g. you run the step load configuration higher), the service will still fulfill their requests but they will just wait longer for the responses to return (due to queueing).
The Apache JMeter Test Plan in this Article represents a programmatic approach for applying load to an ArcGIS cached map service. One of the strengths of this test is that it easy to build, configure and maintain.
The auto-generated JMeter report provides charts and summaries that can be used to analyze the performance and scalability of the cached map service.
Every cached service is different. But generally speaking, the performance and scalability of a cached service can be affected by a variety of factors:
Apache JMeter released under the Apache License 2.0. Apache, Apache JMeter, JMeter, the Apache feather, and the Apache JMeter logo are trademarks of the Apache Software Foundation.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.