Community

Creating a Load Test in Apache JMeter Against a Cached Map Service (Advanced)

2052
4
01-18-2022 12:03 PM
AaronLopez
by Esri Contributor
Esri Contributor
2 4 2,052
‎01-18-2022 12:03 PM

Why Test a Cached Map Service?

Cached map services are a popular and recommended way to provide a well performing presentation of static data. The cache service type is a proven technology, but there may still be requirements to test it under load to observe its scalability first hand on a specific deployment architecture. While cached map services perform well, serving up thousands of simultaneous tile requests can be resource intensive on the server hardware.

Note: Due to the fast rate of delivery and consumption of the resource, load testing cached map services can also be intensive on the hardware utilization of the test client workstation.

Cached Map Service Testing Challenges

Compared to the load testing of the export map function, proper testing of a cached map service introduces several challenges as the request composition with each map screen changes. Since the underlying cache scheme is using a grid design, the map extents of some pans or zooms may pull down more or less tile images than others. Accounting for this real-world behavior of the cache service makes the test logic more complex than if it were exercising the export map function.

The test logic should also be dynamic and cover a decent area of interest. Converting a HAR file of captured cache tile requests into a test might be quick and easy to do but does not show a realistic scalability of the service. This is due to the small sample of tile requests being used over and over again.

Generally speaking, requests for individual cache tiles are fast...very fast. Due to this behavior, the test logic also needs to perform well, scale with the service and have minimal overhead on the test client

How to Test a Cached Map Service?

The steps in this Article should work with any existing cached map service on your local ArcGIS Enterprise deployment. However, if one if not available, it is recommended to give the Natural Earth dataset a look for the task.

The Natural Earth Dataset

Although the steps should work with any data, the walkthrough of the process in this Article might be more effective if they can be directly followed. In such cases, it is great turning to the Natural Earth datasets which provides some decent map detail (at smaller scales) covering the whole world.

  • Download the Natural Each dataset here 
    • The download above is a subset of the larger Natural_Earth_quick_start.zip and includes a modified MXD for ArcMap 10.8.1 and ArcGIS Pro 2.8 project
      • Either can be used to publish and create a cached map service to ArcGIS Enterprise
  • The Natural Earth subset of data should look similar to the following when opened in ArcGIS Pro (or ArcMap)

arcgispro_naturalearth_dataset.png

  • This Article will not cover the details of creating, configuring or publishing a cached map service in ArcGIS Enterprise. For information on such actions, see:

Note: It is recommended to become familiar with some of metadata details of the cached map service as the load testing effort will require knowledge of some of that information (e.g. xorigin, yorigin, tileCols, tileRows, and spatial reference as well as the scales that contain tiles).

Test Data Generation

With a cached map service available, the next step would be to generate test data over an area of interest.

As with other JMeter Articles on Community, we need good test data to get the most value from the results. And like before, the Load Testing Tools package (for ArcGIS Pro) makes short work of this job. There is even a specific tool for creating bounding box data to use with a cached map services.

Note: Version 1.3.0 of Load Testing Tools added the "Generate Bounding Boxes (Precision)" tool.

Download and unzip the package then make that folder available to your ArcGIS Pro project.

The Generate Bounding Boxes (Precision) Tool

  • Launching the Generate Bounding Boxes (Precision) tool should present an interface similar to the following:

loadtestingtools_generateboundboxesprecision1.png

  • Before running the tool, let's adjust the input to target the data generation process to:
    • Specific map scales (in this case three different scales)
      • Scales 4622324.434309 and 1155581.108577 were kept
      • Scale 2311162.217155 was added
      • The number of records to be generated was adjusted the reflect larger map scales
        • As the scale number goes down, we want to tool to generate more boxes 
    • A specific area of interest (optional)
      • A polygon of the United States was added to a new map
      • This feature was set as the Constraining Polygon

cachemap_uspolygon_arcgispro.png

  • Click Run
    • Tool execution may take a few moments

Visualizing the Generated Data in ArcGIS Pro

  • The Contents screen will populate by adding a new feature classes that is visually representing the generated data
    • Not all the generated map scales will be immediately seen

cachemap_generatedbboxes_arcgispro.png

Visualizing the Generated Data in a Text Editor

  • Using the file system explorer, navigate to the ArcGIS Pro project used for generating the data and open one of the csv files using your favorite text editor
    • The file contents should look similar to the following:

cachemap_generatedbboxes_notepad.png

  • The Apache JMeter test will be configured to convert each of these bound boxes into the corresponding cache map tiles

The Cached Map Service Test Plan 

  • To download the Apache JMeter Test Plan used in this Article see: cache_tiles1.zip 
  • Opening the Test Plan in Apache JMeter should look similar to the following:
    • Adjust the User Defined Variables to fit your environment
      • Xorigin, Yorigin, TileCols, TileRows are properties of the created map cache that can be found on the REST endpoint page of the service
        • TileCols and TileRows are typically found under Tile Info Height and Width

cachemap_jmeter_testplan.png

Components of the Test Plan

CSV Data Set Config

The CSV Data Set Config elements in JMeter are used to reference the newly generated test data from the file system. The current version of the Test Plan is built to utilize 3 different CSV files (one for each map scale data file).

cachemap_jmeter_csvdatasetconfig.png

Note: Other that the User Defined Variables and the setting of the Filename in the CSV Data Set Config elements, there should not be anything else that requires editing or changing in the Test Plan. The test logic is listed below just to explain how the values in the HTTP Request become populated.

Levels Of Detail List Logic

To avoid more complex JMeter test logic, 24 fixed map cache levels of detail are placed inside a class in a JSR223 Sampler test element. That "complex alternative" would be to connect the endpoint of the service at the start of the test and pull down the cache tile metadata. Putting HTTP logic into JSR223 Samplers is technically doable, but not the route I chose.

  • There is only one JSR223 Sampler inside the Levels Of Detail Transaction
  • This item is executed only once, at the start of each test thread
    • The element contains 24 fixed cache levels of detail, with level 0 starting at scale 591657527.591555 
      • If your cache scheme starts at a different scale for 0, then the JSR223 Sampler will need to be manually adjusted
  • This JSR223 Sampler does not need to be edited to run the test
    • This assumes cached map service has a Spatial Reference of 102100 (3857)

cachemap_jmeter_levelofdetail_jsr223.png

Levels Of Detail -- JSR223 Sampler (Full Logic):

 

 

 

 

 

// FileServer class
import org.apache.jmeter.services.FileServer

public class Lod{
   int level
   double resolution
   double scale
   double tolerance 
}

public class MyLodList1{
	public List<Lod> LodList = new ArrayList()	
	
	MyLodList1(){
		// Based on ArcGIS Online Map Scales
		// https://services.arcgisonline.com/arcgis/rest/services/World_Street_Map/MapServer
		//
		// Spatial Reference: 102100 (3857)
		Lod lod = new Lod()

		lod = new Lod()
		lod.level = 0
		lod.resolution = 156543.03392800014 //11
		lod.scale = 591657527.591555
		lod.tolerance = 0.5
		this.LodList.add(lod)
		
		lod = new Lod()
		lod.level = 1
		lod.resolution = 78271.51696399994 //11
		lod.scale = 295828763.795777
		lod.tolerance = 0.5
		this.LodList.add(lod)
		
		lod = new Lod()
		lod.level = 2
		lod.resolution = 39135.75848200009 //11
		lod.scale = 147914381.897889
		lod.tolerance = 0.25
		this.LodList.add(lod)
		
		lod = new Lod()
		lod.level = 3
		lod.resolution = 19567.87924099992 //11
		lod.scale = 73957190.948944
		lod.tolerance = 0.5
		this.LodList.add(lod)
		
		lod = new Lod()
		lod.level = 4
		lod.resolution = 9783.93962049996 //11
		lod.scale = 36978595.474472
		lod.tolerance = 0.5
		this.LodList.add(lod)
		
		lod = new Lod()
		lod.level = 5
		lod.resolution = 4891.96981024998 //11
		lod.scale = 18489297.737236
		lod.tolerance = 0.5
		this.LodList.add(lod)
		
		lod = new Lod()
		lod.level = 6
		lod.resolution = 2445.98490512499 //11
		lod.scale = 9244648.868618
		lod.tolerance = 0.5
		this.LodList.add(lod)
		
		lod = new Lod()
		lod.level = 7
		lod.resolution = 1222.9924525624949 //13
		lod.scale = 4622324.434309
		lod.tolerance = 0.5
		this.LodList.add(lod)
		
		lod = new Lod()
		lod.level = 8
		lod.resolution = 611.49622628137968 //14
		lod.scale = 2311162.217155
		lod.tolerance = 0.5
		this.LodList.add(lod)

		lod = new Lod()
		lod.level = 9
		lod.resolution = 305.74811314055756 //14
		lod.scale = 1155581.108577
		lod.tolerance = 0.5
		this.LodList.add(lod)

		lod = new Lod()
		lod.level = 10
		lod.resolution = 152.87405657041106 //14
		lod.scale = 577790.554289
		lod.tolerance = 0.5
		this.LodList.add(lod)

		lod = new Lod()
		lod.level = 11
		lod.resolution = 76.437028285073239 //15
		lod.scale = 288895.277144
		lod.tolerance = 0.5
		this.LodList.add(lod)

		lod = new Lod()
		lod.level = 12
		lod.resolution = 38.21851414253662 //14
		lod.scale = 144447.638572
		lod.tolerance = 0.5
		this.LodList.add(lod)

		lod = new Lod()
		lod.level = 13
		lod.resolution = 19.10925707126831 //15
		lod.scale = 72223.819286
		lod.tolerance = 0.5
		this.LodList.add(lod)

		lod = new Lod()
		lod.level = 14
		lod.resolution = 9.5546285356341549 //16
		lod.scale = 36111.909643
		lod.tolerance = 0.5
		this.LodList.add(lod)

		lod = new Lod()
		lod.level = 15
		lod.resolution = 4.77731426794937 //14
		lod.scale = 18055.954822
		lod.tolerance = 0.05
		this.LodList.add(lod)

		lod = new Lod()
		lod.level = 16
		lod.resolution = 2.388657133974685 //15
		lod.scale = 9027.977411
		lod.tolerance = 0.025
		this.LodList.add(lod)

		lod = new Lod()
		lod.level = 17
		lod.resolution = 1.1943285668550503 //16
		lod.scale = 4513.988705
		lod.tolerance = 0.025
		this.LodList.add(lod)
		
		lod = new Lod()
		lod.level = 18
		lod.resolution = 0.5971642835598172 //16
		lod.scale = 2256.994353
		lod.tolerance = 0.005
		this.LodList.add(lod)

		lod = new Lod()
		lod.level = 19
		lod.resolution = 0.29858214164761665 //17
		lod.scale = 1128.497176
		lod.tolerance = 0.005
		this.LodList.add(lod)

		lod = new Lod()
		lod.level = 20
		lod.resolution = 0.14929107082380833 //17
		lod.scale = 564.248588
		lod.tolerance = 0.0025
		this.LodList.add(lod)

		lod = new Lod()
		lod.level = 21
		lod.resolution = 0.07464553541190416 //17
		lod.scale = 282.124294
		lod.tolerance = 0.0005
		this.LodList.add(lod)

		lod = new Lod()
		lod.level = 22
		lod.resolution = 0.03732276770595208 //17
		lod.scale = 141.062147
		lod.tolerance = 0.0005
		this.LodList.add(lod)

		lod = new Lod()
		lod.level = 23
		lod.resolution = 0.01866138385297604 //17
		lod.scale = 70.5310735
		lod.tolerance = 0.0005
		this.LodList.add(lod)
		
	}
}



MyLodList1 mylods = new MyLodList1()
List<Lod> LodList = mylods.LodList
vars.putObject("LodList",LodList)

 

 

 

 

 

GetMapTile Logic

The JSR223 Samplers inside the GetMapTile Transaction is the logic responsible for taking a bounding box and transforming it into the corresponding cache tiles. 

  • There is one JSR223 Sampler for each map scale (e.g. one for each corresponding CSV Data Set Config)
    • CSV Data Set Config A --> JSR223 Sampler A1
  • This is executed with every test thread iteration
    • This is executed frequently...every time a new bounding box is read in
  • These JSR223 Samplers do not need to be edited to run the test

Note: JSR223 Samplers using Groovy are generally executed quickly and add very little overhead to the test

cachemap_jmeter_getmaptile_jsr223.png

 GetMapTile -- JSR223 Sampler A1 (Full Logic):

 

 

 

 

 

// Script to process a CSV file (from Load Testing Tools) with lines in the following format:
// bbox,width,height,mapUnits,sr,scale

// FileServer class
import org.apache.jmeter.services.FileServer
import org.apache.commons.math3.util.Precision
//import java.math.BigDecimal

// GetMapTile
     
bbox_var = vars.get("bbox_A")
String[] bboxParts = bbox_var.split(',')
double xmin = Double.parseDouble(bboxParts[0])
double ymin = Double.parseDouble(bboxParts[1])
double xmax = Double.parseDouble(bboxParts[2])
double ymax = Double.parseDouble(bboxParts[3])

width_var = vars.get("width_A")
height_var = vars.get("height_A")
// Use map scale resolution (map units per pixel) to determine tile level
double mapresolution = 0
int resolutionprecision = 10
mapresolution = Precision.round((Math.abs(xmax - xmin) / Double.parseDouble(width_var)), resolutionprecision)

scale_var = vars.get("scale_A")
double bbox_scale_double = Double.parseDouble(scale_var)

// Map units per pixel
double tileresolution = 0
double lod_resolution = 0
double scale = 0
int tilelevel = 0
LodList = vars.getObject("LodList") // Assuming cached map service has a Spatial Reference of 102100 (3857)
boolean firstIteration = true;
for(int i = 0; i < LodList.size; i++)
{
     lod_resolution = Precision.round(LodList[i].resolution, resolutionprecision)
	tileresolution = lod_resolution
	tilelevel = LodList[i].level
	scale = LodList[i].scale		
	
     if (mapresolution >= lod_resolution)
     {
     	break
	}
}

tileCols_var = vars.get("TileCols")
cols = Double.parseDouble(tileCols_var)
tileRows_var = vars.get("TileRows")
rows = Double.parseDouble(tileRows_var)
// Origin of the cache (upper left corner)
xorigin_var = vars.get("Xorigin")
xorigin = Double.parseDouble(xorigin_var)
yorigin_var = vars.get("Yorigin")
yorigin = Double.parseDouble(yorigin_var)

// Get minimum tile column
double minxtile = (xmin - xorigin) / (cols * tileresolution)
// Get minimum tile row
// From the origin, maxy is minimum y
double minytile = (yorigin - ymax) / (rows * tileresolution)
// Get maximum tile column
double maxxtile = (xmax - xorigin) / (cols * tileresolution)
// Get maximum tile row
// From the origin, miny is maximum y
double maxytile = (yorigin - ymin) / (rows * tileresolution)

// Return integer value for min and max, row and column
int mintilecolumn = (int)Math.floor(minxtile)
int mintilerow = (int)Math.floor(minytile)
int maxtilecolumn = (int)Math.floor(maxxtile)
int maxtilerow = (int)Math.floor(maxytile)

Scheme_var = vars.get("Scheme")
WebServerName_var = vars.get("WebServerName")
ServerInstanceName_var = vars.get("ServerInstanceName")
ServiceName_var = vars.get("ServiceName")
ServiceType_var = vars.get("ServiceType")
def cacheRequest
def tilePaths = []
int count = 0
for (int row = mintilerow; row <= maxtilerow; row++)
{
	// for each column in the row, in the map extent
     for (int col = mintilecolumn; col <= maxtilecolumn; col++)
     {
          cacheRequest = ("/").concat(ServerInstanceName_var).concat("/rest/services/").concat(ServiceName_var).concat("/").concat(ServiceType_var)
          cacheRequest = cacheRequest.concat("/tile").concat("/").concat(tilelevel.toString()).concat("/").concat(row.toString()).concat("/").concat(col.toString())
          count++
          tilePaths.add(cacheRequest)
     }
}


def requestCount = count.toString()
vars.putObject("RequestCount_A",requestCount) 
vars.putObject("TilePaths_A",tilePaths)

 

 

 

 

 

Cache Tile Loop and Path Population

There are several components needs for this part of the Test Plan. With the bounding box translated into the corresponding cache tiles and assembled into a list of URLs, a third JSR223 is needed to place each URL into a variable inside a loop. The loop logic takes place inside the Cache Tiles transaction. 

  • There is one JSR223 Sampler for each map scale 
    • CSV Data Set Config A --> JSR223 Sampler A2
  • These JSR223 Samplers do not need to be edited to run the test
  • There is a Loop Controller added to only ask for the actual number of tiles per bounding box since this amount can change extent to extent
    • The number of tiles that correspond to each bound box vary by extent but also but the map resolution (1920x1080)
      • Higher screen resolutions require more tiles
    • The Loop Controller contains the following elements:
      • Counter
      • JSR223 Sampler
      • HTTP Request

Loop Controller

cachemap_jmeter_cachetiles_loopcontroller.png

Counter

cachemap_jmeter_cachetiles_counter.png

JSR223 Sampler

cachemap_jmeter_cachetiles_jsr223sampler.png

HTTP Request

All of the test logic above exists just for this component of the test. For each map scale, there is only one HTTP Request! This simple design favors readability and maintainability.

cachemap_jmeter_cachetiles_httprequest.png

Note: The HTTP Requests contains a Response Assertion element to validate the items returned from the server. If the content type of the response is image/jpeg or image/png, then the request will pass. However, some VectorTileServer caches may return a Protocolbuffer Binary Format (*.pbf) file. In these cases, the Patterns to Test would need to be manually expanded to the following: image/jpeg || image/png || application/octet-stream || application/x-protobuf

The Thread Group Configuration

The JMeter Test Plan is currently configured for a relatively short test of 20 minutes. Cached map services perform well, so a lot of throughput will be taking place within each step (2 minutes per step) and from the test overall.

  • Different environments may require an alternative pressure configuration to achieve the desired test results, adjust as needed

cachemap_jmeter_threadgroup.png

Validating the Test Plan

As a best practice, it is always a good idea to validate the results coming back before executing the actual load test.

  • Use the View Results Tree listener to assist with the validation
    • The Test Plan includes a View Results Tree Listener but it is disabled by default
      • Enable it to view the results
  • From the GUI, Start the test

Transactions

  • Select one of the "Cache Tiles" Transactions
    • The results should resemble the following:

cachemap_jmeter_validation_transaction.png

  • In this example, all the transactions completed successfully (e.g. the green checkmark)
    • Cache Tiles (map scale: 4622324.434309)
    • Cache Tiles (map scale: 2311162.217155)
    • Cache Tiles (map scale: 1155581.108577)
  • Selecting one of the transactions and the Sampler result element lists some key information
    • Take a quick glance at the Size in bytes
      • In the example above, the Transaction size was over 50KB which suggests decent tile data (for this dataset) was being returned and the responses were not all "blank" images
    • The Number of samples in the transaction was 80
      • Since there is a JSR223 Sampler with every tile request, this actually resulted in 40 tiles being downloaded
    • The Load time shows 62 (ms), meaning it only took 0.062 seconds to pull down 40 tile images

Requests

  • Expand the selected Transaction
    • In this example, Cache Tiles (map scale: 1155581.108577)
  • Select one of the HTTPS requests
    • The results should resemble the following:

cachemap_jmeter_validation_request1.png

  • In this example, the select request completed successfully (e.g. the green checkmark)
  • Take a quick glance at Load time
    • In this example, the individual tile request only took 2 ms (0.002 seconds) to download
  • Clicking on the Response data tab allows you to preview the requested tile:

cachemap_jmeter_validation_request2.png

Note: Once visual validation and debugging is complete, it is recommended to disable the View Results Tree element before executing the load test

Test Execution

The load test should be run in the same manner as a typical JMeter Test Plan.

See the runMe.bat script included with the cache_tiles1.zip  project for an example on how to run a test as recommended by Apache JMeter. 

  • The runMe.bat script contains a jmeterbin variable that will need to be set to the appropriate value for your environment

Note: It is always recommended to coordinate the load test start time and duration with the appropriate personnel of your organization. This ensures minimal impact to users and other colleagues that may also need to use your on-premise ArcGIS Enterprise Site. Additionally, this helps prevent system noise from other activity and use which may "pollute" your test results.

Note: For several reasons, it is strongly advised to never load test ArcGIS Online.

JMeter Report

  • The auto-generated JMeter Report can provide insight into the throughput of the cached map service under load
    • This report is auto-generated from the command-line options passed in from the runMe.bat script

Throughput Curve

  • The JMeter Report for a cached map service load test may appear sluggish and slow when viewed in a web browser
    • This is due to the default nature of its composition, which attempts to render every unique request in some of the charts
      • In a test such as this, there will be many
      • From the chart legend, select all JSR223 Sampler items to disable their rendering (as they may skew the scale)
  • In this case, the peak throughput for any one of the given map scale transactions of cached tiles was about 15 transactions/second
    • Since 3 map scales were tested, the total transactions per second achieved was 45 transactions/second
      • This equated to around 162,000 cache transactions/hour 
    • The peak throughput appear to occur at the 10:34 mark

cachemap_jmeter_report_transactionspersecond.png

Performance Curve

  • The performance of the cache throughput was good at roughly 120 ms or 0.12 seconds
    • This was observation was taken where the peak transactions/sec occurred at the 10:34 mark

cachemap_jmeter_report_responsetimesovertime.png

Note: "Peak throughput" is a point in a test where no higher throughput can be achieved. This does not mean that is the maximum amount of pressure the service will support without "falling over". Generally speaking, if additional users ask for cache tiles after the system has reached peak throughput (e.g. you run the step load configuration higher), the service will still fulfill their requests but they will just wait longer for the responses to return (due to queueing).

Final Thoughts

The Apache JMeter Test Plan in this Article represents a programmatic approach for applying load to an ArcGIS cached map service. One of the strengths of this test is that it easy to build, configure and maintain.

The auto-generated JMeter report provides charts and summaries that can be used to analyze the performance and scalability of the cached map service.

  • To download the Apache JMeter Test Plan used in this Article see: cache_tiles1.zip

Additional Items Worth Mentioning

Every cached service is different. But generally speaking, the performance and scalability of a cached service can be affected by a variety of factors:

  • Deployment architecture
    • The location of the cache data with respect to the ArcGIS tile handler(s)
    • Cache data storage disk technology and speed
  • Network bandwidth
    • Between the cache data storage and ArcGIS tile handler(s)
    • Between the ArcGIS tile handler(s) and ArcGIS Web Adaptor(s)
    • Between the ArcGIS Web Adaptor(s) and Test Client
  • The processor speed and number of processing cores
    • The delivery of cache tiles is quick but under heavy load the overall process utilizes CPU resources from the ArcGIS tile handler and ArcGIS Web Adaptor (if it exists in the deployment) hosting technology (e.g. Microsoft's Internet Information Services service)
  • Different data can perform differently
    • The average tile size (e.g. size on disk)
      • Smaller tile sizes that contain less data might perform differently that larger more detailed tiles
    • Tested map scales
      • Even for the same dataset, map scale 36111.909643 may have "heavier" cache tiles than map scale 1155581.108577

Assumptions and Constraints

  • JDK 17 or greater will not work with this (JMeter 5.4.x) Test Plan
    • Running on these JDK releases will throw the following error: org.codehaus.groovy.GroovyBugError: BUG! exception in phase 'semantic analysis' in source unit 'Script161.groovy' Unsupported class file major version 61
    • Using JDK 16 or earlier avoids this error
      • The reason is because JMeter 5.4.x only supports JDK 16 (or earlier)
    • If JDK 17 or greater is required for your environment, you must use JMeter 5.5 (which supports JDK 17)
  • On-Demand Cache is not enabled
    • Might work but has not been tested
  • Single Fused Map Cache is TRUE
  • The cache Storage Format is COMPACT
  • Image format of the tiles are in JPG or PNG
    • Due to the Response Assertion rule to validate the return from the server
  • The included Test Plan should work with a cached service for
    • Map
    • Image
      • The ServiceType variable (under User Defined Variables) would need to be changed
      • Not heavily tested
    • Vector
      • The ServiceType variable (under User Defined Variables) would need to be changed
      • VectorTile service tile images can be in Protocolbuffer Binary Format (*.pbf)
        • The Response Assertion rule would need to expand to include application/octet-stream or application/x-protobuf
      • The JSR223 Samplers within the GetMapTile transaction would need to be adjusted to add ".pbf" to the end of the cacheRequest variable
      • Not heavily tested

 

 

 

Apache JMeter released under the Apache License 2.0. Apache, Apache JMeter, JMeter, the Apache feather, and the Apache JMeter logo are trademarks of the Apache Software Foundation.

cache_tiles1.zip
cache_tiles2.zip
4 Comments