Latest Contributions by RandallWilliams

I’ve been doing some load testing against ArcGIS Server recently, and it came to mind that while we often tell users to use load testing tools prior to going public with a GIS application, we don’t go into too much detail into how to go about setting up those tests. In this blog, I’ll describe how to configure and run tests using an open source Java tool called JMeter to assist with load testing your GIS Server installation and application.First, what is JMeter? From the JMeter website: “The Apache JMeter™ desktop application is open source software, a 100% pure Java application designed to load test functional behavior and measure performance. It was originally designed for testing Web Applications but has since expanded to other test functions. “What can I do with it? “Apache JMeter may be used to test performance both on static and dynamic resources (files, Servlets, Perl scripts, Java Objects, Data Bases and Queries, FTP Servers and more). It can be used to simulate a heavy load on a server, network or object to test its strength or to analyze overall performance under different load types. You can use it to make a graphical analysis of performance or to test your server/script/object behavior under heavy concurrent load." That’s all great, but here’s what I do with it: I use JMeter to simulate the load of a group of users concurrently working with ArcGIS Server. These simulated users are SUPERHUMAN, as my JMeter tests fire off requests at the rate of hundreds or thousands per minute. I recently tested memory utilization of 10.1 after sending literally 100 million + query feature requests to ArcGIS Server on an AMI over the course of a week or so. (You’ll be happy to know that I found that memory was well managed during this test). Remember that JMeter is not a browser. While the server is going to complete whatever operation you submit to it, JMeter will not render the response graphically. You get results in the form of a PASS/FAIL notification. Nice. Let’s put that together and run a test. Allow me to walk you through it.What do I need? Great question. You’ll need: Your ArcGIS Server machine. You can test Java or .Net, on any supported OS. Somewhere to install JMeter. This can be the same machine as your ArcGIS Server, but I find that you’ll get more accurate results if JMeter is installed somewhere else. That’s because as it runs, the JMeter application’s Java container will chew up a good bit of RAM, which may skew your results. JMeter is a pure Java tool, so it can run on Windows and Linux. For my testing, I’m running JMeter on Win08. Fiddler and Firebug. I find that the Fiddler tool is helpful for troubleshooting failed requests and the like. It’s also good for helping to build requests to be used in your test plan. Fiddler and Firebug (add-on for the Firefox browser) are both excellent lightweight tools for testing and debugging. The Java Runtime Environment (JRE) installed and an environment variable pointing to JAVA_HOME. Where do I start? There’s not really an installer for JMeter, everything you need is inside the application folder. The first thing to do is open the JMeterbin folder, and have a look at the JMeter.bat file.  On line 71, find the line that reads as follows:set HEAP=-Xms512m –Xmx512m This variable defines the amount of memory dedicated to the Java Heap. I’d increase this to 1024, devoting a full gigabyte of memory to the JVM, like so:set HEAP=-Xms1024m -Xmx1024m Otherwise, particularly on larger tests, you run the risk of going over the Java Heap threshold, which can cause the JVM dedicated to JMeter to fail. After you’ve downloaded JMeter, installed the JRE, and set the JAVA_HOME environment variable to point to your Java install location, it’s time to fire up JMeter. To do this, open a command prompt and navigate down into the JMeter BIN directory (the same place where JMeter.bat lives). Call JMeter.bat from a command line, and the JMeter application will open. With JMeter open, it’s time to first define some defaults. To do that, right click on the Test Plan icon (It looks like a beaker) and add an HTTP Request Defaults Element.  (Add>Config Element>HTTP Request Defaults) Now we will populate some of these parameters on the HTTP Request Defaults page. Server name or IP: that’s self-explanatory. Insert the IP address or hostname of the machine that runs ArcGIS Server (or the proxy that points to server).  I don’t usually define any connect or response timeouts here in JMeter. The path will be the path to your SOAP or REST endpoint – to the operation you want to perform (query, export map, etc.). The parameters that are sent with the request are similar to the parameters that are defined if you’re querying through the REST Services Directory. I find it helpful to use the Fiddler tool to capture the URLs of queries sent either through the service’s REST page or your web application to assist with populating these parameters. For instance, you may issue a query based on a set of X/Y coordinates with the intention of retrieving some attributes for a given feature. In that case, you may capture a URL similar to the following in Fiddler: http://randall:6080/arcgis/rest/services/usa/MapServer/0/query?where= &text=&objectIds=&time=&geometry=-119.78861600000%2C35.30318900030 &geometryType=esriGeometryPoint&inSR=4269&spatialRel=esriSpatialRel Intersects&relationParam=&outFields=*&returnGeometry=true&maxAllowable Offset=&outSR=&returnIdsOnly=false&returnCountOnly=false&orderByFields =&groupByFieldsForStatistics=&outStatistics=&returnZ=false&returnM=false &f=pjson In this URL, I’m issuing a query to layer ‘0’ in a service called ‘usa’ serviced from port 6080 on machine called ‘randall’. I’m not using any real SQL operators (the where clause), nor am I looking for a specific text attribute (text). I don’t care about the object ID, or the time element. I do care about the geometry, that’s where my XY values are specified. In fact, the only elements I care about in this test is the spatial reference (4326), the spatial relationship (intersect), the outfields (*, since I want them all), and I’m submitting the geometry as a point. Finally, for this test, I would like for the server to respond to the requests I send in JSON format.  With that in mind, I can truncate the URL that I’ll be parsing a bit, to make setting up the JMeter test a bit more manageable: http://localhost:6080/ArcGIS/rest/services/usa/MapServer/0/query?&geometry =-149.54518800000%2C70.42196900010&geometryType=esriGeometryPoint&inSR=4269 &spatialRel=esriSpatialRelIntersects&returnCountOnly=false&returnIdsOnly= false&returnGeometry=true&outFields=*&f=json Once I’ve gathered these defaults, I putthese inputs into my JMeter project: Next, I’ll need to set up some simulated users. While we’ve got the request defaults down, we want to test with a good many values – in other words, we don’t want our application to just take the same request over and over, that’s too easy. It’s best if we can test with many different samples.  In JMeter, this is accomplished by creating ‘thread groups’. Right-click on the test plan and select ‘Add’, then ‘Threads (Users). In my case, we’re simulating a load of 16 users, so each thread group will contain 2 threads. I want to see how much load my application can take before it fails, so I’m setting this test to loop forever. Once the Thread Groups are set up, we’ll need to set up the actual requests, which contain the defaults we set up earlier, as well as the variables that we want to pass in our requests. To that end, right click on a Thread Group; choose ‘Add’, ‘Config Element’, and then ‘HTTP request defaults’. This screen should look familiar; it’s exactly like the defaults we originally created. In this case, the objective of the test is to see how much load the server can handle if many hundreds of thousands of query requests are sent to gather the attributes of a feature based on XY coordinates passed through the query. So, for each request sent by JMeter, the only query element that changes from request to request will be the input point geometry. To that end, I’ve compiled a .CSV file, and set up a CSV Data Set Configuration element in my test project (Add>Config element>CSV data set Config). Your data source doesn’t have to be CSV, you can use a database table if you prefer – use a JDBC connection for that. Per the figure above, my CSV file is called xycord.csv (I’ve got the CSV stored in the same folder as my test plan – if yours is stored somewhere else, set the filename to the absolute path where it’s located. Be sure to set the variable names properly – if there are spaces and commas in your CSV file; make sure that this element reflects those spaces. The comma delimitation here only separates variable names in the CSV – the actual delimiter used in the CSV file is described in the ‘Delimiter’ parameter. Since I’m allowing my test to run until I force it to stop, I’m recycling the requests and am not stopping when JMeter detects that the end of the file is reached. For my test, I’ve replicated each HTTP request parameter for each of the users that I’m simulating in the thread groups I’ve set up. Now it’s time to test. Save the test plan. In JMeter, go to ‘Run’, then ‘Start’. Click on one of the reports. With luck, your requests will run. If the error percentage you get is 100%, then you’re likely shooting bad requests. You can examine the requests you’re sending to the server by working with the Fiddler tool: If Fiddler is installed on the same machine as JMeter, then in the main HTTP request defaults page, you may force traffic to go through the Fiddler proxy so that requests and subsequent responses are captured. By default, Fiddler listens on port 8888, so you can set up JMeter as follows: When you start your JMeter run again, Fiddler will capture the traffic, which will allow you to see what JMeter is passing so that you may make adjustments according to your needs.A Few Tips Use as few listeners as possible. For smaller tests, the graphical counters are great. These counters are memory intensive, however, and I’ve crashed my JVM with larger tests when running with counters. I just use the Summary Report – it’s got the information I need. Consider using JMeter from a command line interface once your configuration is set up. The Command Line option is minimally resource intensive. That’s about it for this installment. I’ll follow up with additional testing workflows as I can.Randall W. - Server Support Analyst ... View more

Are you working to put together a migration plan for the transition to ArcGIS 10.1 for Server? Because there are significant changes in the software architecture at ArcGIS 10.1 for Server, a number of hints and tips have been put together to help you through the process. Good to Know At ArcGIS 10.1 for Server, service configuration files from previous versions will no longer work. (At previous releases, service configuration files would remain valid.) Version 10.1 uses a different mechanism for map services called the Service Definition (SD), which is not a text configuration file as it was in previous versions. Previously created map caches will remain valid. ArcGIS for Server 10.1 no longer uses SOM and SOC.  At 10.0, you had to manage accounts for each, but at 10.1, you’ll only have to manage one account. The concept of a distributed installation with multiple SOC machines attached to one or more SOM boxes is also going away. Each GIS server is its own installation. GIS server machines and services may be clustered together using tools available in ArcGIS Server Manager to offer better computing power, but each ArcGIS for Server machine is effectively autonomous. There is no longer a differentiation between the Java and .Net versions of ArcGIS for Server.  Now it’s just ArcGIS for Server. The product installs with its own application server; this server should be dedicated to ArcGIS for Server and nothing else. Esri has created applications, called “Web Adapters,” that link your current server of choice (IIS or a J2EE application server) to the GIS server instance. These applications are small broker components that basically forward requests from your web server to the internal server used with ArcGIS. When you create an administrative connection to ArcGIS for Server 10.1, you’ll have the option to package and copy data in the Service Definition and push the whole package up to a GIS server. This means that an SD is portable – a user in one location can create a service definition and ship it to the GIS server owner without having to worry about an account having access to data in the remote location. Ways to Prepare It’s important to maintain a parallel environment between your current implementation and an install of version 10.1. If you don’t have the budget to maintain multiple ArcGIS for Server machines, your organization should, at the minimum, stand up ArcGIS for Server 10.1 on a desktop class box. DCOM in ArcGIS for Server is going away, which means that local connections will no longer be allowed. If your current applications use local connections for operations like editing in the ADF, you’ll want to plan on replacing those local connections with feature services. There’s no longer an application builder in ArcGIS Server Manager as the ADF has been phased out. There are great options available in the Flex and Silverlight viewers. I especially like the Silverlight viewer. If you haven’t seen it, I recommend logging into the Beta Community and downloading it. In preparing for the differences in ArcGIS for Server 10.1, we’d like to encourage you to get involved in the Beta Community and learn the new workflows, features and functionality. Take notes of what does and doesn’t work in your organization, and if you find an issue that you believe is a bug, report it to us so that we can confirm and log it. If nothing else, go ahead and replicate your current services in 10.1 so as to minimize any downtime associated with the transition to version 10.1.Randall W. - Server Usage Support Analyst ... View more

Most of us here are relatively familiar with ArcGIS Server for .Net. However, in support our exposure to the Java product is relatively limited, simply because it seems that fewer customers have implemented ArcGIS Server for Java. In this post, I'll try to cover a topic that comes up every now and then in support, how to secure both OGC and Esri GIS Service endpoints provided by the same instance of ArcGIS Server. This concept is similar to but implemented differently than the ArcGIS Server for .Net help topic "Multiple ArcGIS Server Web Instances for Security". Before we begin, lets review a couple of best practices (things that are good to know) that we should understand prior to engaging in any configurations. First, have a quick look at the documentation regarding exporting the ArcGIS Server 10 for Java web service and REST handlers. The Tomcat web server that ships with ArcGIS Server for Java should be used for development and ArcGIS Server administration. It's not really meant for handling the load of production work. We recommend using ArcGIS Server manager to export your REST, web service, and token applications out to a standard .WAR file and then deploying those .WAR files to your favorite J2EE web server for production use. Second, regarding security and OGC services: It's important to recognize that there is no provision in the OGC spec for token based authentication with OGC services. This means that if you'd like to secure your WMS, WFS, or WCS Services, you'll need to use a more standard authentication mechanism other than the Esri provided token service, like HTTP BASIC or DIGEST authentication. Out of the box,  without deploying the GIS Server handlers to an external J2EE server, ArcGIS Server for Java will be limited to a single authentication mechanism. However, as we'll discuss below, we can work around this limitation by deploying multiple web services handlers - each with it's own authentication mechanism. To accommodate this configuration, we'll use Tomcat 7 (either 32- or 64-bit is fine, just be sure that if you've got the same bit level Java installed as whichever Tomcat you choose). While you're setting up Tomcat, since we'll be working with authentication, you'll also want to stand up HTTPS/SSL. Otherwise, your credentials and tokens will be passed in clear text over the wire. In my testing, I generally create a SSL repository and self signed certificate using the keytool command. SSL is required when working with the token service with ArcGIS Server for Java, and there is not a supported way to disable this requirement like there is in the .Net product. After Tomcat is installed and configured with SSL, set up your security store. For this example, we'll set ArcGIS Server to use ArcGIS Token Authentication and enable security for your services. Test to verify that security is working. Next, we'll export the SOAP (web services), REST, and token service handlers, and then deploy them to the external instance of Tomcat and test to verify that the services can be reached. When I export my web handlers, I usually stick with generic application names, like 'REST', 'Services', and 'Tokens'. When you're configuring the SOAP and REST handlers, make sure that you enable authentication using the ArcGIS token service, and make sure that the token service URL is pointing to the token service handler deployed on the "external" instance of Tomcat. After they've been successfully deployed to Tomcat, test to verify that you can reach your REST endpoints and service WSDLs. Keep in mind that unless a new context path is explicitly defined, these applications will deploy to the root of the web server. That means that your URLs to your service endpoints will look similar to these (using Tomcat's default HTTPS port): https://myserver:8443/services?wsdl https://myserver:8443/rest/services Once you've tested to verify that the service handlers are working as expected, including requiring authentication to the secured services, it's time to deploy another services handler. This one we'll use to access our secure OGC services. In this case, since we know that OGC services do not support token based authentication, we'll configure the services handler to use basic or digest authentication. We can use the same security store created earlier when we secured the services, but we can't use the token service. Export this application as a .WAR, and name it something like "OGC". Keep in mind that since we're now using an external instance of Tomcat, we'll also need to make a quick modification to the tomcat-users.xml file. Otherwise, Tomcat won't be able to recognize the users and roles you set with the internal Tomcat and service handlers that were created using the ArcGIS Server Manager GUI experience. By default, you'll find that file here: C:Program FilesApache Software FoundationTomcat 7.0conf Open Tomcat-Users.xml with a text editor and uncomment the users and roles section, then add lines to this file to match the users and you've created you created in the GIS Server security store, like this: Finally, give the Apache Tomcat service a restart so that Tomcat recognizes the new users and roles. You should then be able to pass credentials to reach your OCG services, like this: https://myserver:8443/OGC/myservice/mapserver/wmsserver?request=getcapabilities That's it! To summarize, because OGC services do not support token based authentication, we essentially exported two sets of service handlers: one for our REST services using the token service, and another for our OGC services using basic or digest authentication. Services delivered over SOAP may be authenticated using either endpoint, depending on the requirements of the client application.Randall W. - Server Support Analyst ... View more

Recently a user wanted to be able to standardize the ArcGIS Explorer for Desktop application for GIS clients in his organization. He found that once ArcGIS Explorer was configured locally on his machine, he was able to share the default .NMF map document file with users, but the .NMF only held the configuration information for layers in the table of contents - not for add-ins or geoprocessing tools. The Application Configuration Manager could be used to allow or prevent the usage of specific ArcGIS Explorer functionality like the analysis tools or to add new functionality, but couldn't be used to persist a URL to a specific geoprocessing service. The user thought it would be great if ArcGIS Explorer could be provided to users in such a way that the references to add-ins and geoprocessing tasks remained in place for users installing ArcGIS Exolorer for Desktop thus saving clients the time it may take to set up the environment as desired. There are two places that ArcGIS Explorer for Desktop keeps reference to data layers: The default .NMF document itself The add-ins folder After saving the default document in ArcGIS Explorer, use these simple steps to homogenize the Explorer experience for all employees in the department. The admin should: Copy the folder "C:Users<username>AppDataRoamingESRIarcgis explorerAddIns" to a place accessible to network users. Copy the file "C:Users<username>DocumentsArcGIS ExplorerMy Default Map.nmf" to a place accessible to network users. 2.  The users should: Install ArcGIS Explorer. Copy the ad-ins folder referenced above to the corresponding location on their local machine. Copy the 'My Default Map.NMF' to the corresponding location on their local machine. When ArcGIS Explorer for Desktop is started on the user's machine, the experience should be identical to that of the application set up by the administrator.Randal W. - ArcGIS for Server Support Analyst ... View more

ArcGIS for AutoCAD is a great tool for loading basemap layers  provided by ArcGIS Servers to AutoCAD projects. In fact, it has gotten so popular that users have requested documentation regarding ways to load the ArcGIS for AutoCAD interface into AutoCAD at application initialization.  To that end, the ArcGIS for AutoCAD team has provided the following workflow for automatically load this tool. Happy CADding! The ArcGIS for AutoCAD toolbar doesn’t automatically load when installed on a machine with AutoCAD. This can cause some confusion, or it can just cause a hassle when you need to reload AutoCAD every day or multiple times a day. It can be more useful to set up AutoCAD to automatically load this toolbar by editing the acad.lsp file. By default, this file is located in the directory: C:Program FilesAutodeskAutoCADsupport. Here are the steps to configure this file: Open Notepad as an administrator and navigate to C:Program FilesAutodeskAutoCADsupport. Open ‘acad.lsp’ (for AutoCAD 2011 this file is called ‘acad2011.lsp’). In this LSP file, add the following lines: ;;; Automatically loads the ArcGIS for AutoCAD toolbar. (command "_ribbon") (command "netload" "C:\Program Files\ArcGIS for AutoCAD 250\ArcGISForAutoCAD") Save the changes and close Notepad. Opening AutoCAD should now also automatically load the ArcGIS for AutoCAD toolbar. For steps to manually load the ArcGIS for AutoCAD toolbar, please refer to the web help topic Loading ArcGIS for AutoCAD.Randall W. and Scott. P - ArcGIS Server Support Analysts ... View more