Running an Apache JMeter Load Test from Command-line mode (Beginner/Intermediate)
Why Run Test Plans from Command-line mode?
The primary reason for running a load test from Command-line (CLI) mode and not through the GUI mode is that the later can decrease JMeter's capabilities. Using the GUI to run the test can consume additional CPU and memory that can negatively impact the test results. The CLI mode environment is the optimal choice for the test execution and is the recommended methodology by the Apache JMeter team.
Running a test through a command windows might be slightly old school, but it is still very effective.
Verifying the Pre-Test Checklist
It is a good testing practice to utilize a pre-test checklist to help get the most effective use of your test time.
Recommended checklist items include:
- Coordinating the load test start time and duration with administrative staff
- Ensures minimal impact of the test to users and other colleagues for the ArcGIS Enterprise Site
- Helps prevent system noise from other activity and use which may "pollute" the test results
- When testing a traditional (e.g. dedicated) ArcGIS Service, ensure the minimum and maximum instances are set appropriately
- This is necessary if your test goal is trying to understand the maximum achievable throughput from the service
- For max throughput, a general rule of thumb is to set the maximum instances to the number of CPU cores
- For predictable performance, set the minimum instances to the number of maximum instances
- Adjusting instances will cause a service to restart (plan accordingly)
- Increasing minimum instances of a service will require additional physical memory from the ArcGIS Server machine
- This is necessary if your test goal is trying to understand the maximum achievable throughput from the service
- If the Test Plan contains a Listener like the View Results Tree, ensure to disable it from the GUI
- Running Listeners can increase the consumption of test client resources and potentially impact the test
- Validate the Thread Group for step load logic and test duration
- If the test is scheduled for a specific window, ensure the test will be set to run the expected length of time
- Ensuring the ArcGIS Server Log Level is not set to DEBUG or VERBOSE.
- While these log levels can be helpful for troubleshooting issues, they can impact performance and scalability of the service(s) being tested
Batch Script Walkthrough
Since the command-line is utilized for running the Apache JMeter load test, it is advantageous to assemble the actions and environment prep into a batch script (e.g. *.bat file in Windows). This favors repeatability and maintenance.
Variables
Using variables are handy ways to improve the maintenance of a script. While hard-coding values is technically fine, it can hinder readability if paths for certain items are very long. The following sections are the primary components our batch will utilize variables.
Memory
By default, Apache JMeter (version 5.4.1) runs with a minimum and maximum of 1GB of memory. This is adequate for simple tests running on older machine hardware, but some tests require more complex logic or multiple data files and more memory might be needed to ensure the tests results are not impacted
Note: Many load tests focus on the resource utilization of the server hardware. However, the test client capabilities can also be a bottleneck that limits the ability of accomplish the test goals.
Increasing the default memory can be easily with the following
- set heap=-Xms4g -Xmx4g -XX:MaxMetaspaceSize=256m
JMeter and Project Paths
We want to tell the batch script where to find Apache JMeter and our Test Plan.
Setting the path to the JMeter bin folder is straightforward
- set jmeterbin=C:\apache-jmeter-5.4.1\bin
As it the (manually created) project folder where the jmx file (e.g. the Test Plan) will reside
- set projectdir=C:\JMeter Tests\sampleworldcities3
For some extendibility, a variable is created for the name of the jmx file separate from the project folder (without the jmx extension and even if they utilize the same name)
- set testname=sampleworldcities3
For more convenience, a variable is created that will be appended to each run. For multiple reasons, a typical load test can be run several times, with each having a slightly different option enabled within the Test Plan. This can help keep track of different runs of the same test.
- set runname=run1
Test Execution and Switches
Thanks to the utilization of the variables mentioned above, the execution of JMeter with a passed in Test Plan as a command-line switch can easily be reviewed in just a few lines:
%jmeterbin%\jmeter -n -f -t "%projectdir%\%testname%.jmx" ^
-l "%projectdir%\results\results_%testname%_%runname%.jtl" ^
-j "%projectdir%\logs\debug_%testname%_%runname%.log" ^
-e -o "%projectdir%\reports\%testname%_%runname%" ^
- The -n switch runs JMeter in commad-line mode
- The -t "path_to_jmx" switch tells JMeter where on the file system to find the jmx file.
- The -l "path_to_jtl" switch tells JMeter where to log samples, this is the results file and the most important artifact of the test run.
- The -j "path_to_log" switch says where to store the JMeter run log (e.g. environments information). The -e switch tells JMeter to generate a test report (dashboard) once the test has completed.
- The -o "path_to_report_folder" switch tells JMeter where to generated the report (dashboard).
- The -f switch tells JMeter to force delete existing results files and report folder if they are present before starting the test. Altering the "runname" variable in the script is the easiest way to ensure a new results file and report are created with each test run.
- The caret character or ^ is an "escape" character and is added to the Windows batch script so the next character (newline) is interpreted as an ordinary character. This helps with readability.
Putting It All Together
The complete batch script is as follows:
echo off
rem Scripted JMeter Test Plan execution
rem Utilizing jmeter.bat for invocation
rem
rem 2021/06/07.1
rem
rem ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
rem *** Variables ***
rem Set JMeter memory to min/max of 4GB (adjust based on your test client resources)
set heap=-Xms4g -Xmx4g -XX:MaxMetaspaceSize=256m
rem Location of %JAVA_HOME%\bin (currently not used)
rem set javadir=C:\jdk-16.0.1\bin
rem Location of Apache JMeter bin
set jmeterbin=C:\apache-jmeter-5.4.1\bin
rem Location of JMeter Test Plan root folder (e.g. the folder where the Test Plan resides)
set projectdir=C:\JMeter Tests\sampleworldcities3
rem Name of the JMeter Test Plan (without the JMX file extension)
set testname=sampleworldcities3
rem String appended to results file of each test run
set runname=run1
rem ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
echo on
rem *** Test started ***
%jmeterbin%\jmeter -n -f -t "%projectdir%\%testname%.jmx" ^
-l "%projectdir%\results\results_%testname%_%runname%.jtl" ^
-j "%projectdir%\logs\debug_%testname%_%runname%.log" ^
-e -o "%projectdir%\reports\%testname%_%runname%" ^
rem *** Test completed ***
echo off
Running the Load Test
For the first first runs of your test, it is recommend to execute it from command window that is already open. This way, if there are immediate issues invoking any the commands in the batch script, they will persist on the screen so they can be corrected more easily (e.g. a typo).
- From Windows, you can invoke a Command Prompt window by clicking Start, then typing cmd.
- Once this window appears, enter in the path to your batch script:
- "C:\JMeter Tests\sampleworldcities3\runMe.bat", hit Enter
- The script will run for the configured duration then return to the command prompt
- "C:\JMeter Tests\sampleworldcities3\runMe.bat", hit Enter
- Once this window appears, enter in the path to your batch script:
Running from command-line mode does not provide the console with real-time statistics of the test execution. Such functionality can be obtained but is not covered in this Article.
Test Artifacts
Test artifacts are items created from the load test that can be used for analysis. Items like report and the JMeter Text Logs (JTL) files which contain the raw test results are typically the most important to retain. The run log can also be useful as its listing can be used to confirm the configured memory of the environment for the test run as well as updates to which step was being executed and when. After multiple runs of the same test, where a different runname in the script was used, it is easy to wind up with many jtl files and report folders. This is where following the testing strategy of utilizing a project folder for management is key.
Note: The report generated by JMeter is HTML and JavaScript based. If some components of the report are blank or do not appear to render, try adjusting the security settings of the browser or open in another browser, if available.
- The initial page and Response Times Over Time section of a JMeter generated report
To download the Apache JMeter Test Plan used in this Article see: sampleworldcities3.zip
This test requires the Custom Thread Groups plugin to be installed in JMeter.
Preparing for Analysis
The generated report is one of the best places to start for conducting analysis in order to determine if the test goals were achieved. Since this analysis of a load test can be an involved process that is worth discussing in detail, it is a procedure that deserves its own Article.
For a set of additional testing tutorials (of increasing complexity), see: Performance Engineering: Load Testing ArcGIS Enterprise
Apache JMeter released under the Apache License 2.0. Apache, Apache JMeter, JMeter, the Apache feather, and the Apache JMeter logo are trademarks of the Apache Software Foundation.