Latest Contributions by AaronLopez

Branch Versioning Branch versioning is a type of versioning for an enterprise geodatabase that aligns with the ArcGIS Enterprise Web GIS model. It uses a service-oriented design to support multiuser editing workflows and long transaction scenarios through web feature layers.  When datasets are registered as branch versioned in the geodatabase, there is an option to enable the Version Management capability as the resource is being shared/published. This creates a version management endpoint that can perform the creation of new named versions as well as the administration of existing ones. Web users can query and edit data of their own named version through the web feature layer or reconcile and post the changes into the default parent version for other members to utilize. In short, branch versioning is built for the modern web and uses a simplified data model that is optimized for today's clients like ArcGIS Pro. For more information see:  Versioning types -- branch versioning Version management Version administrator   Why Test Branch Versioning? Since branch versioning is the data model for multi-user editing over the web with ArcGIS Enterprise and is a major components for frameworks such as utility network and parcel fabric, the understanding of its performance profile can greatly benefit GIS administrators. Branch Versioning Editing Testing Challenges While the exercising of branch versioning queries from JMeter is not too difficult, things become more complex when editing is involved: For example: When a new named version is created, it needs be referenced in the test by the associated GUID in some places and by the name in others Any time a "write" operation like takes place against the data like creating a new named version, applying edits, reconciling or posting, a new historicalMoment is generated Any web feature layer queries that take place after a write operation will want to use the updated historicalMoment to retrieve the most up-to-date information from the version management system Prior to adding new data, the test needs to ask the system to reserve objectIDs and use the returned numeric value as the starting point In addition to objectIDs, items being inserted or updated also have globalIDs to keep track of To properly perform write operations, an editing lock needs to be acquired that utilizes a unique sessionID With editing through a load test, it is easy to see the additional moving parts that the Test Plan needs to keep track of. Of course, when this happens in ArcGIS Pro, this complexity is handled automatically for the user. Note: The historic moment is one of the strengths of branch versioning as it allows a user a mechanism for viewing data for a particular date and time. Lock acquisition is another key feature of branch versioning which allows the system to have multiple readers for each named version or just one exclusive editor. How to Test Branch Versioning Editing? The branch versioning editing workflow was conducted in ArcGIS Pro 3.2 Final (and ArcGIS Enterprise 11.2). While the operations were executed in ArcGIS Pro, the HTTP traffic (between ArcGIS Pro and ArcGIS Enterprise) was captured and later converted from a HAR file to a JMX file (JMeter Test Plan). At a high level, the workflow operations in this Article contained the following steps which represent a user's (branch versioning) editing session in ArcGIS Pro: Portal Member Authentication Open the Project Create a new named version Switch to the named version Zoom in to an area of interest Insert a new point Update attributes of the new point Insert a new polygon Update attributes of the new point Save Edits Reconcile with Default Post the changes back to Default Zoom out (back to the initial map scale) The general steps in this Article should work with any service data that is registered as branch versioned. Though the test workflow makes specific use of a point layer and a polygon layer that existed in the utilized dataset. The dataset tested was the free and publicly available Natural Earth. View of the Natural Earth service from ArcGIS Pro: Southern California, USA The Natural Earth Dataset The Natural Earth datasets provides some decent map detail (at smaller scales) covering the whole world. Download the Natural Each dataset here  The download above is a subset of the larger Natural_Earth_quick_start.zip and includes a modified MXD for ArcMap 10.8.1 and ArcGIS Pro 2.8 project Either can be used to publish and create a cached map service to ArcGIS Enterprise The ArcGIS Pro Feature Cache ArcGIS Pro uses a feature cache to improve the display performance when navigating to areas of interest that have already been visited on the map. If any editing takes place this cache is refreshed. Feature caching is an all-around good performance strategy. However, when capturing the HTTP traffic for this workflow, the feature cache was turned off. The Editing Workflow via ArcGIS Pro This section lists the editing workflow steps from an ArcGIS Pro visualization point of view. Open the Project The resources of the project are based on the web feature layers from the published service Create a new, public named version and switch to it When manually conducting the editing workflow, the value of "edits_20231211_2" was used but within the JMeter test, this value will be dynamic generated and unique to better allow for more automated execution After opening the project, ArcGIS Pro will issue multiple queries to the various layers listed in the table. When a new version is created and ArcGIS Pro switches to it, similar looking layers queries will be sent again but these are actually for the new version. Note: Although the editing workflow starts with "Portal Member Authentication", the logic for this operation was manually adjusted in the Test Plan to remove some additional requests and reduce complexity Zoom in to an area of interest (where the inserts will take place) In this example, the area of interest is Southwest of Palm Springs, California, USA Insert a new point From ArcGIS Pro, select the Edit tab from the top menu; click on Create (under the Features section); then from the list of Create Features Templates select "ne_10m_populated_places" Then, click on an area in the map to insert the point Note: After clicking in the map, the point was actually inserted into the geodatabase. However, since the Save operation has not been executed, the "Branch Moment" for the named version has not yet been updated. Once save has taken place, the edits become "official". The next step involved changing attributes of the newly inserted point as it was an opportunity to later show a feature update within the JMeter Test Plan. Alternatively, you could have just used the Feature Template to pre-populate the attribute fields and then perform the insert. Update attributes of the new point With the newly added point still selected, click Attributes (under Selection), then populate several of the fields. Attributes to update SCALERANK, NATSCALE and LABELRANK rank was given arbitrary numeric values The NAME field was specifically set to "Otisburg" To commit the changes, click Apply Insert a new polygon From ArcGIS Pro, select the Edit tab from the top menu; click on Create (under the Features section); then from the list of Create Features Templates select "ne_10m_urban_areas" Then click on an area of the map and draw a polygon around the "Otisburg point" Update attributes of the new polygon With the newly added polygon still selected, click Attributes (under Selection), then populate several of the fields. scalerank was given an arbitrary numeric value The featurecla field was specifically set to "It's a little bitty place" Note: When adding the point and polygon, several feature queries were being issued as the cursor moved around the map. These queries are the result of the Snapping engine. With snapping enabled (the default option) the additional requests are expected behavior. The captured workflow used in this Article left this option enabled. The functionality can be turned off from the Snapping management if the additional requests are not desired in the captured traffic. Save Edits With the branch versioning edits complete, the changes were internally finalized by the system once Save was clicked Reconcile with Default Reconciling with default is the process of resolving conflicts that result from merging parent version data (e.g., from default) into the edit session  The captured workflow used the Reconcile defaults (as captured by ArcGIS Pro): Detect conflicts by attribute (instead of object) Abort if conflicts were detected was false Issue a Post (after the reconcile) automatically was false Run the job asynchronously was false Since this editing workflow is relatively simple, there should be no existing attributes that conflict the edits that were performed Post the changes back to Default Post is the process of merging the edit session data into the default version The inserts will now become part of default and any new named versions that are created after The captured workflow used the Post defaults (as captured by ArcGIS Pro): Run the job asynchronously was false Note: If the captured traffic is executed in the Test Plan multiple times (e.g., a load test), the "same" items will appear multiple times in default (as expected). From a visual perspective, it might be difficult to "see" these new objects spatially in the default version as they would all be using the same geometry. However, the labeling of the point and polygon layers might give hints that multiple occurrences exist for items using the same geometry. Note: A few words about an asynchronous reconcile or post; with ArcGIS Pro 3.2, the default execution from the ribbon menu is to utilize synchronous execution. If asynchronous for an operation is used instead, it follows a different HTTP request pattern (since it become a submitted job whose status must be periodically polled). Additionally, the action utilizes a separate ArcGIS Server service (e.g., the System GP services like: SyncTools, UtilityNetworkTools, or VersionManagementTools) to carry out the appropriate asynchronous task. Zoom out (back to the initial map scale) Finally, the session ends by navigating back to the original map scale of 1:1,000,000 to help provide a visual point of reference of the edits that were performed The Branch Versioning Editing JMeter Test Plan To download the Apache JMeter Test Plan used in this Article see: branch_versioning_editing1.zip  Opening the Test Plan in Apache JMeter should look similar to the following: The tests steps closely follow the workflow performed in ArcGIS Pro: Portal Member Authentication Open the Project Create a new named version Switch to the named version Zoom in to an area of interest Insert a new point Update attributes of the new point Insert a new polygon Update attributes of the new point Save Edits Reconcile with Default Post the changes back to Default Zoom out (back to the initial map scale) Components of the Test Plan In order to help keep this branching versioning editing Test Plan as simple as possible, only a few options exist under User Defined Variables that need to be configured. A single HTTP Header Manager element is utilized and just one Response Assertion exists to throw messages if the word "error" occurs in any response. CSV Data Set Config The test does make use of a CSV Data Set Config elements in JMeter to order to authenticated and utilize a different Portal member when executed under load. This item is location inside the Thread Group. Note: As administrators, testers and analysists, the JMeter Test Plan could be as complex as needed to fit your needs. It is not uncommon to having an editing test have the actual changes (e.g., geometries inserts or attributes updates) use a CSV file to make the effort more dynamic and realistic. However, one of the goals of this Article is help show how the editing can be done by keeping the complexity of the test down to a minimum. (User Test Thread) Initialization Transaction Once the user or more accurately, the test thread, starts but before authentication takes place, we want JMeter to generate values for a few items and assign them to variables: sessionID Used for acquiring locks and for write operations Unique for each user's ArcGIS Pro session gdbVersion geodatabase Version name identifier Used to help create the named version's "official name" Once created, ArcGIS Enterprise will also generate a GUID associated to the new named version A unique name for each named version A GUID for our point to be added Unique for every added point Unique for each users ArcGIS Pro session A GUID for our polygon to be added Unique for every added polygon  Unique for each users ArcGIS Pro session These are defined through a User Parameters element. Open ArcGIS Pro Project Transaction When expanded, the Open ArcGIS Pro Project transaction will list the requests issued when the aprx file was launched. There are typically many requests for such an operation as ArcGIS Pro gathers meta-data on the service and layers. In the middle of this operation are a few requests calling a special GUID resource. These are calls to the default version in the geodatabase. Typically, "BD3F4817-9A00-41AC-B0CC-58F78DBAE0A1" is the GUID associated with Default. There is also startReading request to obtain a read lock that utilizes the sessionID generated earlier in the test. Create Version and Switch Version Transaction The Create Version operation does a lot of important work in the Test Plan. It ensures that the edits will have a place to go (the named version) until they are intentionally merged with Default. Upon creating the new version, the versionGUID, versionName and branch versioning moment are returned which need to be captured and stored into additional test variables since they will all be used in subsequent requests. The versionName is the official vanity designation of the named version. While the test generated a value for "gdbVersion", the system adds some addition information to its final form. For example: gdbVersion value (generated by JMeter): version_O5AHEMD89CYTH7GW versionName value (returned by ArcGIS Enterprise): USER007.version_O5AHEMD89CYTH7GW Note: Create (version) is available through the service's VersionManagementServer capability. Remember, when creating the new named version, ArcGIS Pro was instructed to switch to it. Immediately following create, there were requests issued against the new version using its GUID and well as several queries. These queries had specific key/value parameters that depended on the new branch versioning moment and name of the new version that was returned (from the response of create). Add Point Spatially Transaction Like the create operation, Add Point Spatially contains several critical pieces. This particular action is the first to "start" the edit session for the long transaction by sending a startEditing request which acquires a write lock against the named version. Since an insert is taking place, an objectID for the point is also required. The reserveObjectIDs call asks the system to return an objectID starting point for that particular layer. The test requested 100 IDs and the system responded with a firstObjectID to start with (which is captured to a JMeter variable) followed by an another number. The second number is the continuous, reserved range of IDs (the firstObjectID is the first item in this range). Note: While the GlobalID GUIDs can be generated by the Test Plan, ObjectIDs cannot. ObjectIDs must be allocated by the web feature layer.   Note: A reserveObjectIDs request asking for 100 allocated ObjectIDs does not guarantee that 100 continuously IDs were reserved. The system will respond with the range available from that initial request. Additional reserveObjectIDs calls might be needed to obtain the necessary amount of IDs if the tested workflow is planning to perform multiple inserts that exceed the allocation. For such a situation, the test plan would need to handle the objectID logic itself. ArcGIS Pro does this automatically for the editing user. With the ObjectID for the point reserved and captured, the insert can now take place. Web feature layer editing (adds, deletes, and updates) happen through the applyEdits function. This request requires the named version name and sessionID as well as the edit payload. This payload lists of the edits to be performed specifying the layer ID, operation (e.g., adds), attributes and geometry. The pointGlobalID (GUID) generated at the start of the test is also used. The value for the edits key:     [{"id":1,"adds":[{"attributes":{"OBJECTID":${pointObjectID},"SCALERANK":null,"NATSCALE":null,"LABELRANK":null,"FEATURECLA":null,"NAME":null,"NAMEPAR":null,"NAMEALT":null,"DIFFASCII":null,"NAMEASCII":null,"ADM0CAP":null,"CAPIN":null,"WORLDCITY":null,"MEGACITY":null,"SOV0NAME":null,"SOV_A3":null,"ADM0NAME":null,"ADM0_A3":null,"ADM1NAME":null,"ISO_A2":null,"NOTE":null,"LATITUDE":null,"LONGITUDE":null,"CHANGED":null,"NAMEDIFF":null,"DIFFNOTE":null,"POP_MAX":null,"POP_MIN":null,"POP_OTHER":null,"RANK_MAX":null,"RANK_MIN":null,"GEONAMEID":null,"MEGANAME":null,"LS_NAME":null,"LS_MATCH":null,"CHECKME":null,"MAX_POP10":null,"MAX_POP20":null,"MAX_POP50":null,"MAX_POP300":null,"MAX_POP310":null,"MAX_NATSCA":null,"MIN_AREAKM":null,"MAX_AREAKM":null,"MIN_AREAMI":null,"MAX_AREAMI":null,"MIN_PERKM":null,"MAX_PERKM":null,"MIN_PERMI":null,"MAX_PERMI":null,"MIN_BBXMIN":null,"MAX_BBXMIN":null,"MIN_BBXMAX":null,"MAX_BBXMAX":null,"MIN_BBYMIN":null,"MAX_BBYMIN":null,"MIN_BBYMAX":null,"MAX_BBYMAX":null,"MEAN_BBXC":null,"MEAN_BBYC":null,"COMPARE":null,"GN_ASCII":null,"FEATURE_CL":null,"FEATURE_CO":null,"ADMIN1_COD":null,"GN_POP":null,"ELEVATION":null,"GTOPO30":null,"TIMEZONE":null,"GEONAMESNO":null,"UN_FID":null,"UN_ADM0":null,"UN_LAT":null,"UN_LONG":null,"POP1950":null,"POP1955":null,"POP1960":null,"POP1965":null,"POP1970":null,"POP1975":null,"POP1980":null,"POP1985":null,"POP1990":null,"POP1995":null,"POP2000":null,"POP2005":null,"POP2010":null,"POP2015":null,"POP2020":null,"POP2025":null,"POP2050":null,"CITYALT":null,"min_zoom":null,"wikidataid":null,"wof_id":null,"CAPALT":null,"name_en":null,"name_de":null,"name_es":null,"name_fr":null,"name_pt":null,"name_ru":null,"name_zh":null,"label":null,"name_ar":null,"name_bn":null,"name_el":null,"name_hi":null,"name_hu":null,"name_id":null,"name_it":null,"name_ja":null,"name_ko":null,"name_nl":null,"name_pl":null,"name_sv":null,"name_tr":null,"name_vi":null,"wdid_score":null,"ne_id":null,"GlobalID":"${pointGlobalID}","created_user":"${username}","created_date":1702339858000,"last_edited_user":"${username}","last_edited_date":1702339858000},"geometry":{"x":-12994151.880899999,"y":3983674.1925999969,"spatialReference":{"wkid":102100,"latestWkid":3857,"xyTolerance":0.001,"zTolerance":0.001,"mTolerance":0.001,"falseX":-20037700,"falseY":-30241100,"xyUnits":10000,"falseZ":0,"zUnits":1,"falseM":0,"mUnits":1}}}]}]     As mentioned earlier, if the insert utilized the feature template, the attributes would have actual values for the initial edit instead of "null".  The point insert operation in this test plan is "simple" in that it uses a fixed geometry of  "x":-12994151.880899999,"y":3983674.1925999969. The test plan could be made to utilize a CSV file with different values to make this part more dynamic. If the applyEdits is successful, a new edit moment is returned to be used on subsequent feature queries. Note: The value of the new historicMoment key in the query requests before applyEdits is called is typically different than the value after. The value of the property (e.g., historicMoment) is updated through the Regular Expression Extractor element. After an edit operation, there are often a few feature queries which use the edit moment, version name and ObjectID that was just inserted.  Update Point Attributes Transaction Since the editing session has already been started and the objectID(s) already reserved, the Update Point Attribute does not have as many moving parts as Add Point Spatially. It contains a handful of queries and one applyEdits request (to perform the update). The update performed by the applyEdits is relatively simple as its just changing a few attributes. However, more columns could be changed in addition to the geometry. The value for the edits key:     [{"id":1,"updates":[{"attributes":{"OBJECTID":${pointObjectID},"SCALERANK":550,"NATSCALE":425,"LABELRANK":231,"NAME":"Otisburg","GlobalID":"${pointGlobalID}"}}]}]     Polygon Transactions The same editing process (e.g., insert followed by an update) is repeated for the polygon layer. While the Add Polygon Spatially contains a new reserveObjectIDs request (since the insert is taking place against a different layer) there is not an additional startEditing request against the named version GUID because the long transaction is still active.  The value for the edits key:      [{"id":7,"adds":[{"attributes":{"OBJECTID":${polygonObjectID},"scalerank":null,"featurecla":null,"area_sqkm":null,"min_zoom":null,"GlobalID":"${polygonGlobalID}","created_user":"${username}","created_date":1702340363000,"last_edited_user":"${username}","last_edited_date":1702340363000,"Shape__Area":293911790.76777756,"Shape__Length":85235.280315167271},"geometry":{"rings":[[[-12973934.7456,3982326.3835999966],[-12974309.137,3979331.2524999976],[-12977753.537900001,3978732.2261999995],[-12982021.5997,3977908.565200001],[-12984193.069800001,3974913.4340000004],[-12991456.2629,3974464.1643000022],[-12996697.7424,3976336.121299997],[-13002688.004700001,3978956.8611000031],[-13003736.3006,3984872.245099999],[-13005758.0142,3991910.8033000007],[-13004485.0834,3994381.7864999995],[-13002762.883000001,3990413.2377000004],[-12996922.3772,3986893.9585999995],[-12986589.174800001,3987193.4717999995],[-12983369.4088,3985995.4192999974],[-12981197.9387,3984348.0971999988],[-12973934.7456,3982326.3835999966]]],"spatialReference":{"wkid":102100,"latestWkid":3857,"xyTolerance":0.001,"zTolerance":0.001,"mTolerance":0.001,"falseX":-20037700,"falseY":-30241100,"xyUnits":10000,"falseZ":0,"zUnits":1,"falseM":0,"mUnits":1}}}]}]     The polygon layer update is similar to the point update in that it modifies several column attributes of the data. Save Edits (to the Named Version) Transaction At this point in the Test Plan, the edits are done and the long transaction can be completed. This happens by a stopEditing request being issued against the named version. When the commit is successful, the branch moment value is updated and the edits performed will persist in the named version. Reconcile With Default Transaction If the changes made against the named version need to be visible in Default, the reconcile operation must  take place first. As mentioned earlier, reconcile helps resolve data conflicts, that might exist between the two versions. Since a write could be taking place during this process, a startEditing request is issued to the named version GUID resource just prior to the reconcile.  From an HTTP request point of view, reconcile is called against the GUID of the named version (which was created from this iteration of the test). The key/value pair request parameters are fairly straight-forward with the only dynamic pieces being the sessionID (and of course the token). A stopEditing request is issued (also against the named version GUID) once reconcile has finished. Once the reconcile has completed successfully, a new branch moment is returned and captured. Note: This test focuses on "write operations" returning an updated edit moment. While this is true, the edit moment change can also be reflected through the responses from VersionManagementServer/versions/[GUID]  requests. Many times, this value is identical to the moment returned by the write operation. Sometimes it is different but still very close. For simplicity, this test plan did not reflect this potentially different moment change with an additional regular expression extractor element (for these VersionManagementServer/versions/[GUID] requests). Post To Default Transaction Post is the last step to bring the edits in the named version to the default version. Like reconcile, the captured editing session used the defaults options. As a result, the key/value pair request parameters are straight-forward with the only dynamic pieces being the sessionID and the token. There are a few more reading and editing calls than the reconcile operation to the named version and default version, but the operation is also straight-forward. Managing the Named Versions Named versions can build up over time, but if the post operation is the last action in the user's edit session, the named version could be deleted. Note: For simplicity, deleting the named version after the post transaction completed was not added to the test. Zoom Out (to Scale 1 Million) Transaction The edit session concludes with a navigation zoom out to the map scale 1:1,000,000. This was done in ArcGIS Pro to make it easier to see the edits with respect to the rest of the data. The result of this simple operation is a handful of feature layer queries. However, it is important to understand that these queries (based on the values for historicMoment and gdbVersion) are still against the named version and not the default version. Note: Queries against default would (always) use the gdbVersion value of "sde.DEFAULT". However, since Default has been updated in this test plan as a result of the post operation, new queries would need to utilize an updated historicMoment in order to pull down the latest data. The most recent value for default's historicMoment can be found in the response of a VersionManagementServer request (e.g., VersionManagementServer/versions/BD3F4817-9A00-41AC-B0CC-58F78DBAE0A1). Such a request exists in the post transaction, but for simplicity, this was not captured with a regular expression. Final Thoughts  The Apache JMeter Test Plan in this Article represents a basic programmatic approach for handling some of the key components of a branch versioning editing workflow. While the steps of the workflow were straight-forward, the end result was a Test Plan that still contained contains several moving parts. To help with this, the edits themselves were static and hard-coded into the test. This was done to keep things as simple as possible so the Article could focus on critical points and strategies.  However, there is nothing to stop you from recording and creating more complex edits in a new test or to make the edits of your effort dynamic where the values are derived from a CSV file. As a tester and GIS analyst, the sky's always the limit when building a JMeter Test Plan to consume or edit data from ArcGIS Enterprise. To download the Apache JMeter Test Plan used in this Article see: branch_versioning_editing1.zip  Running the Load Test Note: Please coordinate with your GIS team if your Apache JMeter test will be sending requests to a server that might impact other users. A load test should be scheduled to run during non-peak business hours. Note: Before running an editing test against an enterprise geodatabase, ensure that proper backups exist, are accessible, and can be utilized, if needed. While named versions can be easily deleted via the version management system, edits and changes posted to default are more difficult to undo. Typically, restoring the data from a backup to get the system to a clean state is often the last step after the edit testing effort has completed anyways. Historic Moment Testing Details The branch moment is a critical component of branch versioning as it allows the feature layer queries to pull data from a specific moment in time from the geodatabase. However, from an HTTP testing perspective, the response from various functions list this value under different string names. Some resources list it under creationdate or modifiedDate, whiles others use editmoment or just moment. As a tester, that is something to look out for when constructing the regular expression to capture the appropriate value. In the case of create (version), creationdate and modifiedDate are returned  though they often contain the same value. For more information see: Historical moments  Operation Concurrency As a tester and analyst, one of your jobs is to observe how your services perform and scale. This curiosity includes an understanding of the branch versioning operations like: query, create version, applyEdits, save, reconcile, and post. Of course, the write operations (back to the enterprise geodatabase) can be more resource intensive. While functions like query and applyEdits can certainly support many concurrent calls across different named versions, you may find calls like post do not behave in that same manner...especially if they are long running (e.g., the named version contained a large number of changes). While every dataset is different and each editing workflow may have a unique performance characteristic, this is something to keep in mind if the load test utilizes a high number of concurrent test threads with the post operation bringing a large amount of changes back to default. Real-world workflows may not always post the changes from multiple named versions back to default all-at-once. Instead, it is not uncommon that a data steward performs this particular job serially and manually to help ensure the changed data meets the needs of the organization.      Attribution File:c8db9197-fc08-4673-8bd1-6b622c7d4f6e_text.gif, Otisburg -- Superman (1978)    Apache JMeter released under the Apache License 2.0. Apache, Apache JMeter, JMeter, the Apache feather, and the Apache JMeter logo are trademarks of the Apache Software Foundation. ... View more

Performance: Challenges and Strategies ArcGIS Enterprise provides a robust and scalable platform for delivering GIS resources for users through services and web applications. Sometimes however, deployments may experience slow performance from the published resource endpoints.   Since ArcGIS is very versatile, there can be different ways, configurations and options when making these resources available for consumption. Obtaining good performance is not always as easy and straight-forward as just toggling a "fast = true" setting. What might be a handy feature for some administrators could be a configuration, that while helpful, may interfere with the performance for others.    In the real-world, performance is often a function of several items and configuration settings. Having strategies to understand and overcome the most common ones may help put the deployment on the road to achieving faster performance. What is Performance? Performance is a description of how fast (or slow) a server or service operates for a particular function of interest. How long it takes an ArcGIS Server service to complete an operation like query, applyEdit or export and then send the response back to the client that requested it would be an example of performance.   This duration is measured in seconds or milliseconds and is commonly referred to as the response time. Although called "response time", there are several important steps that make up the overall time a client like a web browser or ArcGIS Pro spend waiting for a requested server resource: DNS lookup of the server's hostname SSL handshake between client and server TCP/IP connection between client and server Sending the request to server Server processing the request Receiving response from server For large responses, Time-to-first-byte (or TTFB) can be used instead to measure response time Typically, the bulk of the time is spent at 4.1. This is where the server is working on the response. This Community Article will explorer areas that can impact this portion of the response time.  Why is Performance Important? Simply put: the faster the performance, the lower the response time.  The lower the response time, the more requests the server can support at one time. This higher concurrency of requests translates to greater scalability which ultimately means support for more users.   Performance is measured as a unit of time need to complete a single operation at a time (e.g., 0.238 seconds to execute a feature query request).  Scalability on the other hand, is frequently measured as transactions or operations over time (e.g., requests/sec or operations/hour).   When talking about getting better, faster or "more" performance, it implies achieving lower response times. On the other hand, better scalability implies reaching a higher rate of throughput (e.g., more operations/hour).   Note: Of course, for obtaining better performance or throughput, the operations of interest need to execute appropriately and respond with the expected content (e.g., map image, json or pbf data). Errors messages, for example, can be a fast, simple response whose delivery can reach a high rate of throughput. As testers and analysts, this type of throughput is not what we're after...we are interested in the throughput of successful requests. What is Acceptable Performance? It depends.  The criteria or requirement for classifying an item as having fast (or slow) performance can vary greatly by organization, the published services and expected operations users will be calling. It is not uncommon to have different response time goals for ArcGIS Server functions or user application workflows (several requests grouped together to represent one operation). Any number of seconds is fine for a requirement but keep in mind, it may take more hardware as well as more extensive tuning and strategies (this Article) to achieve aggressive goals. How is Performance Measured? Response time is the key metric for determining if performance is meeting or staying within or under a target requirement. Common strategies for measuring are: Single user interaction Through the web browser or ArcGIS Pro This is the easiest place to start If there is no understanding yet on performance, start here Statistically analyzing large volumes of response times Through log analysis tools, the ArcGIS Server Manager Statistics page, or other observability utilities These approaches have the benefit of analyzing real-world requests that users have already executed against the deployment This is discussed in more depth later in the Article Load Testing Can provide an understanding on performance and scalability  More time consuming to setup Online resources exist for getting started Performance Engineering: Load Testing ArcGIS Enterprise  When analyzing logs or running tests, a common strategy is to leverage statistics for breaking down large amounts of response times. The Average and 90th (or 95th) percentile, the Minimum, and Maximum help provide an understanding of the performance the user may have been experiencing. Capturing Response Times -- Web Browser How response times are captured through single user interaction is a fun topic of discussion. The easiest approach to capture response times of REST requests from a web application is with the browser's "developer tools" functionality. All major browsers offer some view of the requests, responses and times being sent and received. This duration can give an idea of how fast a request or operation (potentially multiple requests) performed. Decisions can then be made if this is acceptable or needs to be improved.   Capturing Response Times -- ArcGIS Pro While ArcGIS Pro also communicates with ArcGIS Enterprise via REST, it does not have a built-in equivalent of the developer tools. For capturing responses times, you'll need a separate HTTP debugger. There are many available, a popular choice is Fiddler. With Fiddler installed to the same machine as ArcGIS Pro, it can be configured to intercept the traffic. Request parameters, response content and times can be similarly captured and examined. Are Goals Required for Improving Performance? Absolutely not. GIS Administrators can always analyze, tune and apply best practices to the system even if no official performance requirements are in place. However, it is highly recommended to understand what performance your system is initially delivering (e.g., these are typically referred to as baseline response time numbers) before adjustments are made. This way you can determine if the applied changes are having a positive effect. Common Performance Challenges and Potential Strategies Service Pool Types and Instances One of the most frequent areas that ArcGIS administrators encounter performance challenges with is setting the appropriate number of instances for dedicated services. But first, let's review the different types. There are 3 service types, each with their own strengths. Dedicated Hosted Shared As a GIS Administrator it is important to be able to identify the instance type for a service. This can be easily viewed within ArcGIS Server Manager, under Manage Services:   Selecting the Appropriate Type Choosing a dedicated service type is ideal when achieving the most performance and scalability control is desired. With this type, the administrator can: Set the maximum number of instances to the number of CPU cores to take full advantage of the available processing capability ArcGIS Server machine (via the maximum) Conserve memory when idle (via the minimum) Adjust for predicable performance by setting the min and max to the same value Dedicated services are ideal for heavily requested services or services where performance is paramount. Hosted services do not utilization ArcSOC instances and auto-scale as needed. However, the ArcGIS capabilities available to it (Hosted) are limited as it is used primarily with feature queries. Shared services are great for accessing items that are requested less frequently. They typically have more ArcGIS capabilities available to them but not all ArcGIS functionally is available (e.g., branch versioning). A shared instance pool is the default type when publishing a service with ArcGIS Pro. Note: It is important to reiterate that if the selected service type is set to dedicated, the (minimum and maximum) number of instances should be evaluated to ensure they are optimal. The default when publishing in ArcGIS Pro is to only use a maximum of 2 (instances). This might be too low and inadeqaute for services where performance/scalability are important.  Focus the Map Optimizing the map is not a new strategy but a relatively easy one to follow for getting better performance from your deployment. When the map is focused on its primary purpose and presentation, the system does not have to do unnecessary work. Remember, the web is a multiuser platform. Making the display of the dynamic data as streamlined as possible is key to good performance (and scalability). With potentially many requests occurring at the same time, the content being shared needs to be as efficient as possible. Map Strategies Choose an Optimal Default Extent If the map is providing data on Los Angeles, the default extent should not be showing all of California If many different map scales are required, use scale dependencies and generalization to limit the detail to when you need most (e.g., the largest scales) Remove unneeded layers Consider removing nice-to-have data layers At the very least, unselect them and have user opt-in to enable Purposely limit what users can do with a service Avoid projecting on the fly Use the same coordinate system for data frame and data Definition queries Ensure indexes are in place if comparison logic is applied to attribute columns  Note: "Focusing the map" applies to maps, apps and services Software Releases A particular version of ArcGIS Enterprise (and its related solutions) can have a handful of patches after its initial base release. These patches can offer performance improvements as well as functionality and security fixes.  It is highly recommended to periodically check the Esri Patches and Updates site or run the "Check for ArcGIS Enterprise Updates" tool. Then, apply the updates at the appropriate time. Resource Contention and Expansion There are times when best practices and strategies for performance are applied but lower response times and higher scalability are still required. Perhaps the current hardware running ArcGIS Server is simply exhausted where the processing power or memory have become the bottleneck for improving the user experience. For such situations, you need to consider expansion and/or upgrading the hardware. Scalability For the ArcGIS Server and ArcGIS Web Adaptor tiers of the deployment, you generally have the following options for improving scalability using hardware:  Scaling up Adding more resources to the existing machine (e.g., additional processing cores) Additional memory can also assist with scaling by allowing the deployment to have more ArcSOC instances running concurrently  Ideal for deployments such as: cloud, virtualization, Kubernetes Scaling out Adding more machines of equal resource capacity Ideal for deployments such as: on-premise, cloud, virtualization, Kubernetes Performance To improve performance with hardware, there is typically just one option: Obtaining faster processing cores  Ideal for deployments such as: cloud, Kubernetes The system's paging configuration can also impact scalability even when ample memory resources have been added to a system. Although this is operating system software setting and not hardware, it can play a crucial part in the running of many concurrent ArcSOC instances. Be sure to set this accordingly to handle a workload with many instances or instances with a large memory footprint. There can be situations where performance is limited and appears to be CPU bound (e.g., a bottleneck due to limited processing power). Further inspection may reveal the "culprit" to be one or more slow queries which were suboptimal to begin with or an expensive operation called too frequently (through a periodic administrative tasks). In such a case, it may be more effective to address to the "bad" queries to improve performance and scalability. Note: These expansion strategies are for overcoming general processing and memory limitations. But, disk and network (bandwidth, latency) resources can also be bottlenecks. For some environments, these can be more complicated to expand upon requiring additional steps to upgrade. A General Approach to Scaling For ArcGIS Server it is can be easier to scale up first, then out. The reason is due to the Site configuration and directories which would need to be migrated to shared storage if the architecture is switched from a single machine to a multi-machine deployment.     How much should you scale up...two servers, five servers? Without details on average response times and the anticipated number of users to support, it’s difficult to provide a concise answer. However, a simplistic, general approach would be to just try doubling the current amount (memory and/or physical processing cores) and observing the impact. For many cases, this probably works well up to 16 CPUs. At that point, adding another machine may be more advantageous.   Note: Your current product license may impact how many physical cores you can use with ArcGIS Server. Check with your Esri Account Manager for more details.   Note: For support with capacity planning or architecture design (which can help provide a detailed understanding and estimate on the required hardware resources for a given set of workflows), contact Todd Jarrard (tjarrard@esri.com) in Esri Professional Services. Observability "Quantifying ArcGIS" is a great strategy...a personal favorite. It defines what resources were being requested and how fast were the responses to fulfill these requests. It is important for obtaining an understanding of general system performance. If system resource utilization can also be captured, the analysis can be further elevated. There are many utilities available for periodically examining your system. Some tools may read the access logs and primarily focus on the statistical request performance made by users. Others may poll Server's statistics page and capture the CPU utilization (of ArcGIS Server or the database) for that duration of time. Which approach is best? If observability is currently not taking place then most likely, any one of them. would be a good addition. They all help provide some type of insight to the performance and health of the deployment.  Once analysis has been conducted for a deployment, reports can be typically generated that highlight which map services might be of interest due to: The observed response times being slower than expected The number of requests issued for the resource Both response time and number of requests For an ArcGIS Site with many services, knowing which ones are statistically slow or are consuming the most resources help focus tuning efforts. With such reports, the GIS administrator has turned data into valuable information and are now better informed at making decisions for improving the user experience. That said, examining logs and statistics is just one (important) slice of the analysis pie. A Challenge with Common Observability Tools Many tools for system observability and monitoring focus the analysis on requests and responses for services. This is a good approach and definitely assists administrators with quantifying ArcGIS, but it can have a limitation. The limitation can appear with an assumption that a slow map service can be "fixed" by simply adding processing cores. More cores might improve some aspects of the situation, but it is recommended that the service be examined (or even reexamined) in more depth before more resources are obtained. This goes back to the "Focus the Map" section, for example: Ensure the data for the service is not being shown at too small of scales Avoid suboptimal queries Detailed query analysis can help show the occurrence of these behaviors that might get masked with general service reporting. However, while the break down of service request parameters and the underlying queries can improve the analysis it can add complexity to the reporting itself (e.g., more time to execute, more views of what to look at, the understanding of the views). Additionally, not all observability tools perform this type of inspection. Some recent efforts that are gaining traction are attempting to tackle this issue. They are based on a bottom up approach of analysis where the starting point is on the underlying database queries themselves through mechanism known as "query datastore". Query datastore analysis is powerful and does not impact the database performance like a trace can but it does require some knowledge of the queries themselves and their purpose. Look to this type of analysis capability in the future to help get the most from your observability tools. Conclusion There is no single item to easily adjust for boosting performance and scalability of an ArcGIS Enterprise Site. However, this Article lists some common strategies that can be applied together for improving it. It is also important to understand these are items that should be periodically revisited and acted upon. User habits change over time as does the popularity of a web application or service. Resources that were assigned to a particular service can be reevaluated or reduced to make room for the next featured item in your Site. ArcGIS performance analysis can be fun but it also a continuous effort for maintaining the best user experience.     Attribution   Resource: File:Grayson_running_the_4x100.jpg Description: English:Grayson running the first leg of the 4x100 at the 2010 Tigered invite Author: Graysonbay Created: 02:02, 29 November 2010 License: This file is licensed under the Creative Commons Attribution 3.0 Unported license   Resource: File:Kurvimeter_1_fcm.jpg Author: Frank C. Müller, Baden-Baden License: This file is licensed under the Creative Commons Attribution-Share Alike 4.0 International license.   Resource: File:My_Opera_Server.jpg Description: A server used for the My Home Author: William Viker, william.viker@gmail.com (c) 2006 License: The copyright holder of this file allows anyone to use it for any purpose, provided that the copyright holder is properly attributed. Redistribution, derivative work, commercial use, and all other use is permitted.   Resource: File:Samsung-1GB-DDR2-Laptop-RAM.jpg Description: A 1 gigabyte stick of DDR2 667 MHz (PC2-5300) laptop RAM, made by Samsung and pulled from a 2007 MacBook laptop. Author: Evan-Amos Created: 1 August 2018 License: Public Domain   ... View more

ArcSOC Availability and Utilization Optimizing the ArcSOC instance availability and utilization for your service is a good strategy for helping users obtain fast response times and lower wait times from their dynamic requests to your Site. It can also benefit server resource utilization like memory as the service is not running a lot of instances that it will never use. But optimizing the minimum and maximum number of instances for your dedicated services is not a one-time job. Usage patterns of your services can change over time so the task of collecting this information is something that you will want and to revisit periodically as a GIS administrator. Before diving into how to observe ArcSOC instance activity statistics, let’s review some of the key details of the two ArcSOC-based service types in ArcGIS Server and how they play into this discussion: Dedicated Shared Note: Wait time is the duration the request spends in a “queue” on the server until an ArcSOC instance is available to start working on it. Dedicated Instance Pool Services Dedicated services (e.g., non-hosted and non-shared services) are a mainstay ArcGIS resource in deployments as many applications depend on the capabilities such as geoprocessing, Branch Version editing, and Utility Network workflows (all which require dedicated services). While such services are very versatile and are a major pillar in ArcGIS because of the functionality they provide, as a GIS administrator you need to periodically examine, adjust, and configure the number of ArcSOC instances (minimum and maximum) to take full advantage of your available resources as your Site and users grow. To learn more about ArcSOC instances see: Understand service instances Limitations of Shared Instance Pool Services Shared service instances are great! They are truly a game changer for helping admins manage the demand for many services with finite resources. However, their restrictions and requirements limit which service capabilities can be used with them.  Geoprocessing, Branch Version editing, and Utility Network, for example, are currently not supported through shared services (or through hosted Services). This leaves dedicated services as the only choice for such functionality. Configured ArcSOC Instance Availability vs Instance Demand For services that are very popular, critical and/or have the requirement of running under the dedicated service type, understanding the optimal instance setting is important for several reasons. If the maximum number of active instances is too high, memory is wasted (as well as cost). The over allocation of ArcSOC instances is an important but unique case as its impact would not show up in the analysis of just response times. Alternatively, having the maximum too low can impact performance (in the form of higher response times and longer wait times) as users might be frequently waiting for an already busy ArcSOC instance to become free. Of course, setting the instance minimum and maximum to different values has a trade-off too. For critical services where performance is paramount, having the user’s request wait while an instance needs to be started can be time consuming and will affect performance. So, for predictable performance on essential services, it is recommended to set the minimum and maximum number of instances to the same value. As listed on Introduction to service instances: Accordingly, it's important for ArcGIS Server administrators to monitor the number of instances their site is running, and to limit running instances when performance is inhibited by memory usage. Configuring the service’s availability (through its instance minimums and maximums) and the impact of those settings from the user’s demand on the service is key to an optimally running Site. There is a mutual relationship between configuring the service’s availability (through the instance minimums and maximums) and the direct affect that has to requests coming in to a dedicated service. While finding the optimal setting is an ongoing task, there are some tools and resources to help administrators tackle the challenge. ArcGIS Server Service Report The ArcGIS Server Service Report (introduced in 10.1) is one of those lesser-known REST Admin API gems. This resource can help monitor a Site by providing a configurable summary of all the services in a folder. It is Generally a fast-performing request (depending on the number of services in the folder being requested). The instance service statistics section of the returned response is extremely valuable as it lists details on the ArcSOC instances (min, max, busy) across the whole deployment (e.g., the ArcGIS Server Site). By periodically polling this endpoint, one can get up-to-the-second insight of the service instance configuration vs the demand…as it happens. With such information, better decisions can be made about optimizing machine and service resources. In turn, this can help improving response times and lower wait times. Note: In ArcGIS Server, instance service statistics information and the Statistics page in Manager are actually different resources though they are providing similar views of the same data. The service statistics provide raw access to the instance values (Site-wide and per machine) as well as more detail. The Statistics page is an interface for creating reports from some of that information. Automating the Service Report Collection with Soccer Any script, program or tool that regularly observes the Service Report endpoint of the folder of interest would suffice. However, if you are looking for a free, existing tool then  Soccer is recommended. (Arc)SOC ScannER or Soccer is a utility for scanning and reading the services' statistics on a specific ArcGIS Server folder. It parses the collected data and writes it out to a CSV file for additional post-capture analysis (e.g., creating charts in a spreadsheet to visual the usage). It takes advantage of the REST Admin's Service Report resource in ArcGIS Server to gather this info. The original goal of soccer was to capture the report endpoint output of a specific folder in ArcGIS Server and save the ArcSOC instance statistics (e.g., Running, Busy, Maximum, etc...) for each service. Currently, soccer is a command-line only utility. It is made available in the .NET 6.0 portable runtime for Windows (win-x64), Linux (linux-x64) and macOS (osx-x64). For simplicity, running soccer only requires 3 inputs (other parameters can be passed in to extend functionality): soccer.exe -s "[https://ArcGISServer/ServerWebAdaptor]" -f [FolderToScan] -t "[PreGeneratedArcGISToken]" For example: soccer.exe -s "https://gisserver.domain.com/server" -f "Gas" -t "APLeyWOcKZp9stZ_C01DQ.." Note: A pre-generated ArcGIS Server token can be obtained from Portal. Typically, the generateToken URL is: https://gisserver.domain.com/portal/sharing/rest/generateToken and the Webapp URL would then be: https://gisserver.domain.com/server/admin. Set the Expiration value to something appropriate for your expected monitoring duration. Standard Out while running from a command-window: Connected to: "https://gisserver.domain.com/server " (Gas) Press Ctrl-C twice to stop... Sleeping 5 seconds... When running, soccer connects to the Service Report endpoint of the ArcGIS Server folder specified, collects the data, writes it to a local CSV file, then sleeps. After the sleep duration has elapsed, it repeats the process. Soccer will keep collecting until the process has been manually stopped (Ctrl-C). Note: To collect on the root ArcGIS Server folder, use either: -f “/” or -f “” Analyzing the CSV File Sample contents as seen from a simple text viewer:     DateTime,Epoch,IntervalSeconds,Host,Folder,ServiceName,Type,Provider,Running,Busy,Maximum,Free,NotCreated,Initializing,Transactions,TotalBusyTime,ServicesCollected,ResponseTimeMilliseconds,ContentLength,ConfiguredState,RealTimeState,Message 5/2/2023 1:12:45 AM,1682989965310,5,gisserver.domain.com,Gas,Gas_Utility_Network,MapServer,ArcObjects11,32,0,32,32,0,0,0,0,2,121.6224,6065,STARTED,STARTED,success 5/2/2023 1:12:45 AM,1682989965310,5,gisserver.domain.com,Gas,Landbase_PostgreSQL,MapServer,ArcObjects11,32,0,32,32,0,0,0,0,2,121.6224,6065,STARTED,STARTED,success 5/2/2023 1:12:50 AM,1682989970469,10,gisserver.domain.com,Gas,Gas_Utility_Network,MapServer,ArcObjects11,32,0,32,32,0,0,0,0,2,117.901,6065,STARTED,STARTED,success 5/2/2023 1:12:50 AM,1682989970469,10,gisserver.domain.com,Gas,Landbase_PostgreSQL,MapServer,ArcObjects11,32,0,32,32,0,0,0,0,2,117.901,6065,STARTED,STARTED,success 5/2/2023 1:12:55 AM,1682989975606,15,gisserver.domain.com,Gas,Gas_Utility_Network,MapServer,ArcObjects11,32,0,32,32,0,0,0,0,2,109.6187,6065,STARTED,STARTED,success 5/2/2023 1:12:55 AM,1682989975606,15,gisserver.domain.com,Gas,Landbase_PostgreSQL,MapServer,ArcObjects11,32,0,32,32,0,0,0,0,2,109.6187,6065,STARTED,STARTED,success 5/2/2023 1:13:00 AM,1682989980733,20,gisserver.domain.com,Gas,Gas_Utility_Network,MapServer,ArcObjects11,32,0,32,32,0,0,0,0,2,125.1686,6065,STARTED,STARTED,success 5/2/2023 1:13:00 AM,1682989980733,20,gisserver.domain.com,Gas,Landbase_PostgreSQL,MapServer,ArcObjects11,32,0,32,32,0,0,0,0,2,125.1686,6065,STARTED,STARTED,success 5/2/2023 1:13:05 AM,1682989985874,25,gisserver.domain.com,Gas,Gas_Utility_Network,MapServer,ArcObjects11,32,0,32,32,0,0,0,0,2,116.3949,6065,STARTED,STARTED,success 5/2/2023 1:13:05 AM,1682989985874,25,gisserver.domain.com,Gas,Landbase_PostgreSQL,MapServer,ArcObjects11,32,0,32,32,0,0,0,0,2,116.3949,6065,STARTED,STARTED,success     Note: By design, columns such as IntervalSeconds and ResponseTimeMilliseconds will show duplicate values if more than one service exists in the folder being observed. The collected data is a typical CSV, but there are some important fields which will be helpful for quickly analyzing the ArcSOC activity from out service(s) of interest: IntervalSeconds ServiceName Running Busy Maximum By opening the CSV file in a spreadsheet, other services residing in the same folder can be easily filtered out through the ServiceName column. The IntervalSeconds, Running, Busy, Maximum values can then be plotted to visualize (e.g., through the Scatter with Smooth Line chart ) and show the instance configuration versus incoming demand. In this case, the Gas_Utility_Network service was set to min/max instance configuration of 32/32. The "polished" chart below: Note: The Maximum and Running represent the ArcSOC service configuration instance maximum and minimum, respectively. In this case, Running has the same values and is plotted “behind” Maximum. Busy represents the number of instances that were active (due to requests from users) across all the machines in the Site. The “goal” of tuning and optimization in this case, would be to avoid the Busy values from constantly reaching the Maximum. If this were to happen, it means there was not enough ArcSOCs available for the service to meet the user demand as the instances were always busy. User requests would then most likely be encountering increased response times and wait times in the process. If the requests wait too long in the system, they timeout (typically after 60 seconds). Having a lot of service request timeouts would negatively impact the user experience. Based on the observations from this monitored duration, the rise and fall of the Busy column values did not come anywhere close to the maximum number of instances that were available…which from one perspective was good. However, this also indicated that there was a measurable number of instances running and taking up memory but not being used…which was not ideal. To optimize system resources going forward, the instance configuration of this service could have been set to a lower value that is closer to the (expected) peak usage (e.g., somewhere between 18 – 24). Use 18 to conserve memory with the potential of encountering several instances where user wait longer for their requests to be fulfilled Use 24 to favor performance over memory usage Final Thoughts Having an “optimized” service setting for the minimum and maximum number of instances does not guarantee that users will never encounter slow performance. User needs and habits change over time, so monitoring this information is something that will need to be done periodically. In the real-world, there are conditions where service wait times can still be encountered in a deployment…even with ample instances available and plenty of system resources. It is not realistic (or possible) to eliminate service wait time completely but is instead more practical to try to reduce it where possible. Optimizing the service instances is something admins can directly control that has an impact on wait times. Understanding and periodically evaluating the ArcSOC instance configuration (for dedicated services) and its relationship to the user demand is key to helping GIS administrators better plan and manager their Site by optimizing performance and efficiently utilizing resources.     ... View more