Skip navigation
All Places > Implementing ArcGIS > Blog > 2019 > April
2019

These eight videos cover the GIS Manager Track sessions from the 2018 Esri International User Conference presented July 11-12, 2018 in San Diego, CA.  They are:

 

  • Enterprise GIS: Strategic Planning for Success
  • Communicating the Value of GIS
  • Architecting the ArcGIS Platform: Best Practices
  • Increase GIS Adoption by Integrating Change Management
  • Governance for GIS
  • Moving Beyond Anecdotal GIS Success: An ROI Conversation
  • Workforce Development Planning: A People Strategy for Organizations
  • Supporting Government Transformation & Innovation

A special thanks to those that helped present these sessions: Clinton Johnson Michael Green Matthew Lewin Wade Kloos Justin Kruizenga Andrew Sharer Eric_Apple

 

https://www.youtube.com/playlist?list=PLaPDDLTCmy4auFPPuXEzGYkQUQi8AG_uh

I'm proud to announce the agenda for the 2019 GIS Managers' Open Summit (GISMOS) at the Esri International User Conference on Tuesday, July 9th.  This is the 10th annual GISMOS and looks to be one of the best.  A huge shout out to the presenters: Eric John Abrams, Brandi Rank and Marvin Davis.

 

If you are headed to the User Conference and want to learn proven, real-world strategies focusing on the people/culture/business side of GIS, then please consider attending & participating in GISMOS.  Attendees need to register, but there is no additional cost.  I hope to see you there!

 

esriuc gis manager gis strategy gis leadership

dkrouk-esristaff

What is Egdbhealth?

Posted by dkrouk-esristaff Employee Apr 26, 2019

 

Note: This blog post is the first in a series of three planned posts about egdbhealth.  The second in the series will address how to use the tool to evaluate the health of an Enterprise Geodatabase.  The third in the series will address using egdbhealth in a system design context.

Introduction

Egdbhealth is a tool for reporting on various characteristics of Enterprise Geodatabases (eGDBes).  It provides descriptive information about the content of the eGDB and it provides evaluative information about the eGDB.  The evaluative information is the primary purpose of the tool; to surface existing or latent problems/challenges with the eGDB.  The tool works with eGDBes backed by Oracle, PostgreSQL, and SQL Server.

 

Installation and Execution

Although it is not an extension to ArcGIS Monitor, egdbhealth is available for download from the ArcGIS Monitor Gallery: https://www.arcgis.com/home/item.html?id=ea4bbf9b46084dc49efae9889832aa22

 

Installation and Pre-Requisites

To install, simply download the .zip archive and extract the files to a directory of your choosing. 

There are some pre-requisites for running egdbhealth.  Those considerations are discussed in the documentation in the .zip archive (egdbhealth_README.docx).  But, at a high level, the pre-requisites are:

 

  1. You need a database client for the type of database (Oracle, PostgreSQL, or SQL Server) to which you are connecting (http://desktop.arcgis.com/en/arcmap/latest/manage-data/databases/database-clients.htm).
  2. If you are using “Operating System Authentication” with SQL Server, and your eGDB is owned by “dbo” (instead of “sde”), you need to run egdbhealth as a Windows user that is dbo.  In other words, you must connect as the “Geodatabase Administrator” (http://desktop.arcgis.com/en/arcmap/latest/manage-data/gdbs-in-sql-server/geodatabase-administrator-sqlserver.htm).  You must connect as the Geodatabase Administrator with Oracle or PostgreSQL, but that usually means connecting as the user called “sde”.
  3. If you are connecting to Oracle or PostgreSQL, you must have enabled the ST_Geometry type library (http://desktop.arcgis.com/en/arcmap/latest/manage-data/databases/add-the-st-geometry-type-to-an-oracle-database.htm or http://desktop.arcgis.com/en/arcmap/latest/manage-data/databases/add-the-st-geometry-type-to-a-postgresql-database.htm).

 

Execution

Double-clicking egdbhealth.exe will launch a command window and a Windows form. 

 

Follow the prompts on the form to fill out the connection information for your database.  Tooltips provide hints about the nature of the information required.  But, in principle, the information is the same as what you would provide to ArcGIS to connect to the database as the “Geodatabase Administrator.

 

Graphical User Interface

 

The “Test” button will attempt to confirm that you can connect, with the required privileges, to the eGDB.  If the test is successful, the “Test” button will become a “Run” button.  Clicking that will close the form and begin executing the tool in the command window.

 

When the tool completes, it will open the “output” subdirectory which will contain five output files:

 

  1. An HTML file containing metadata about the queried eGDB and RDBMS target and how the connection is made.
  2. A “Context” Excel file that contains descriptive information about the eGDB and RDBMS target
  3. An “Expert” Excel file that contains evaluative information, classified as “Critical”, “Warning”, and “Information”.
  4. A png file that depicts the version tree.
  5. A png file that depicts the state tree.

 

Introducing the Output

An example of the output files is shown below:

 

Output files

 

The files will bear the “GDB Friendly Name” that you specified in the form (in this example, “SQL_GIS”). 

 

The Expert Excel File

The Expert Excel file is the main information artifact.  It provides the evaluative information about the target eGDB system.  For example, if the eGDB has not been compressed recently, there will be an evaluation that reports this as a concern.

 

The file has a summary sheet which provides an overview of the evaluations.  The evaluations are categorized by general topic (first column, “Category”) and classified (Critical, Warning, and Informational columns) such that you may prioritize your review of the information. 

 

The Description column briefly explains the purpose of the evaluation.  Where there is a red marker in the Description cell, there is a hover tip that provides yet more information about what is being evaluated.

 

Expert Excel file Overview sheet

 

Some findings have no records to report.  For example, if there is no problem with a given kind of Geodatabase Consistency, there will be no records.  However, many findings will have some number of records of various classifications. In those cases, the Name (second column) will have a hyperlink to the sheet in the workbook that has the detailed finding records.

 

The Context Excel File

The Context Excel file is similar in structure but lacks the expert (Critical-Warning-Informational) classification columns.  The information is more descriptive and less evaluative.  For example, in this file, you can find a listing of all of Geodatabase Domains and the ObjectClasses to which they are related.  That information is neither good nor bad (i.e. not evaluative).  But, it may be useful to know.

 

Context Excel file Overview sheet

 

Records that are highlighted in green have SQL statements that you can run for additional information, as appropriate.  Usually, these SQL statements are too expensive to run on all of the content in the eGDB.  But, someone familiar with the eGDB and the issues it has may have ideas about which queries would be useful to run nonetheless.  For example, the “SqlGeomTypeSizeSql” sheet has a SQL query for each FeatureClass in the eGDB.  If you run one of these queries it will report the sizes of the geometries in one FeatureClass.  This is an expensive enough operation that it would not be appropriate to run it on all of the FeatureClasses by default.  But, if there is a FeatureClass that has a performance problem, it may be useful for you to run the query for that FeatureClass to examine the sizes of its geometries.

 

The PNGes

The PNG files (a version tree diagram and a state tree diagram) are typically of interest if your eGDB has data that has been registered as versioned.

 

The version tree graph illustrates the version tree hierarchy.  Color coding (red-yellow-green) indicates the relative degree of “staleness”, or how long it has been since the version has been edited or reconciled.

 

Version tree

 

The state tree schematic illustrates the depth and structure of the state tree (which is the detailed structure upon which the version tree relies).  The Default version is shown in red and “State Zero” is shown in green.  The further these nodes are separated, the more expensive it is for the database to return information about the Default version (the most commonly used version in most systems).

 

State tree

 

The HTML

The HTML provides some general information about the eGDB, RDBMS, and machine that is the target of the evaluation.  It may only be of passing interest in many cases.

 

HTML metadata

 

Summary

This article as described the purpose of egdbhealth, how to run it, and what its outputs are.  As the outputs of the tool contain quite a bit of technical information, other articles will address how to use the outputs for (a) understanding and improving eGDB health (b) designing GIS systems.

 

I hope you find this helpful, do not hesitate to post your questions here: https://community.esri.com/thread/231451-arcgis-architecture-series-tools-of-an-architect

 

Note: The contents presented above are recommendations that will typically improve performance for many scenarios. However, in some cases, these recommendations may not produce better performance results, in which case, additional performance testing and system configuration modifications may be needed.

jdeweese-esristaff

What is VDI Anyway?

Posted by jdeweese-esristaff Employee Apr 25, 2019

Often the term "VDI" is used to define ArcGIS Desktop/ArcGIS Pro deployed as a virtual application. The challenge is understanding what specific virtualization technology is actually being referenced when using this term since VDI, or "Virtual Desktop Infrastructure", represents just one of several desktop virtualization options. So, the intent of this article is define the options and differentiate what VDI truly means.

 

ArcGIS Desktop has been delivered virtually for over 20 years using what is referred to as "hosted virtual applications" which includes technologies such as Citrix XenApp (recently renamed to Virtual Apps) and Microsoft Remote Desktop Services (RDS). This approach is referred to as "hosted"  because it is being hosted by a singular operating system which users share by initiating individual user sessions. This technology option represents a many-to-one relationship in terms of users and virtual machines. Further, the shared operating system is a server OS, such as Windows 2016 and not a desktop OS, such as Windows 10. Hosted virtual applications provides a means to share a singular server with multiple users and is an attractive option since since each user doesn't require their own dedicated virtual machine. For this approach, system resources are shared including processors, memory, and GPU and there isn't a practical way to assign resources at the individual user session level.

 

A more recent innovation is to provide individual virtual machines to users as "virtual desktops" where each user accesses a remote desktop deployed with a desktop operating system such as Windows 10. This includes technologies such as Citrix XenDesktop (recently renamed to Virtual Desktops) and VMware Horizon. This approach represents the true meaning of "VDI" as it is defined by a one-to-one relationship between users and virtual machines. Though this approach increases per-user deployment costs, it also provides a more isolated deployment in terms of resources since processors, memory, and GPU resources can be assigned accordingly. The ability to manage GPU resources for the virtual desktops has made this approach an attractive option for ArcGIS Pro which requires a GPU.

 

So, the next time you hear the term "VDI" used for delivering ArcGIS to users, know that this implies that each user is being presented with their own individual Windows desktop virtual machine with a set of assigned resources as opposed to users accessing a singular server-based virtual machine and sharing it with multiple users, including sharing the server's assigned system resources. 

Amazon and Esri recently published a whitepaper outlining the steps needed to setup and configure Amazon AppStream 2.0 and ArcGIS Pro. 

 

Through testing, Esri and AWS outline the various classes of AppStream hosts:

 

ArcGIS 2D Workloads – stream.compute.large, stream.memory.large. Compute and Memory optimized instances are perfectly suited for ArcGIS Pro workloads that does not require a GPU.

 

ArcGIS 3D Workloads (Normal) – stream.graphics-design.xlarge. Graphics Design instances are ideal for delivering applications such as ArcGIS Pro that rely on hardware acceleration of DirectX, OpenGL, or OpenCL. Powered by AMD FirePro S7150x2 Server GPUs and equipped with AMD Multi user GPU technology, instances start from 2 vCPU, 7.5 GiB system memory, and 1 GiB graphics memory, to 16 vCPUs, 61 GiB system memory, and 8 GiB graphics memory.

 

ArcGIS 3D Workloads (High res) – stream.graphics-design.2xlarge or stream.graphics-pro.4xlarge. The Graphics Pro instance family offers three different instance types to support the most demanding graphics applications. Powered by NVIDIA Tesla M60 GPUs with 2048 parallel 4 processing cores, there are three Graphics Pro instances types starting from 16 vCPUs, 122 GiB system memory, and 8 GiB graphics memory, to 64 vCPUs, 488 GiB system memory, and 32 GiB graphics memory. These instance types are ideal for graphic workloads that need a massive amount of parallel processing power for 3D rendering, visualization, and video encoding, including applications such as ArcGIS Pro.

 

Please find the full whitepaper here: https://d1.awsstatic.com/product-marketing/AppStream2.0/Amazon%20AppStream%202.0%20ESRI%20ArcGIS%20Pro%20Deployment%20Gu…   

What is System Log Parser?

System Log Parser is an ArcGIS for Server (10.1+) log query and analyzer tool to help you quickly quantify the "GIS" in your deployment. When run, it connects to an ArcGIS for Server instance on port 6080/6443/443 as a publisher (or an administrator), retrieves the logs from a time duration (specified as an input), analyzes the information then produces a spreadsheet version of the data that summarizes the service statistics. The command-line version of System Log Parser (slp.exe) is used by ArcGIS Monitor for data capture.

System Log Parser supports the following service types:

  • Feature Services
  • Geoprocessing Services
  • Network Analyst Services
  • Geocode Services
  • KML Services
  • Stream Services
  • GeoData Services
  • Map Services
  • Workflow Manager Services
  • Geometry Services
  • Image Services

 

  • Globe Services
  • Mobile Services

 

System Log Parser (https://arcg.is/0XLnfb), a free-standing application or Add-on for ArcGIS Monitor, is an effective tool for diagnosing and reviewing infrastructure functionality.

 

Getting Started

 

In this section, we’ll configure ArcGIS Server to collect logs at the level needed for the tool and setup System Log Parser to generate a report (MS Excel).

1.   Ensure the following conditions are met on the machine you’ll be running System Log Parser from:

  1. 64-bit Operating System:
    1. Windows 7 (64 bit), Windows 8.x, Windows 10
    2. Windows Server 2008 64 bit, Windows Server 2012, Windows Server 2016
  2. RAM: 4 GB
  3. Microsoft .NET Framework 4.5 or 4.6
  4. Microsoft Excel 2010 or newer (or appropriate .xlsx viewer).

2.   Set your ArcGIS Server logs to Fine on EACH server you’d like to get metrics on. Complete instructions on how to       change ArcGIS Server log levels can be found here:  Specify Server Log Settings

Note:    I recommend running the logging at FINE for AT LEAST one week prior to running System Log              Parser. This should give you a fairly clear picture of a typical weeks load.

3.   Download and extract System Log Parser here: https://arcg.is/0XLnfb

4.   Extract the .zip file.

Note:    This is BOTH the user interface and the Add-on for ArcGIS Monitor.  We will be focused on the user               interface version for this exercise.

5.   Launch System Log Parser

6.   Browse to the location you extracted System Log Parser

7.   In the System Log Parser for ArcGIS folder, locate and launch SystemLogsGUI.exe

System Log Parser GUI

Note:    You may be prompted that Windows has protected your PC.  If you do get this prompt, please click              More info and then click Run Anyway.


 

Configuring System Log Parser

 

The following outlines the configuration required to setup System Log Parser to analyse a weeks worth of logs.

Note:    The System Log Parser will automatically access logging for all clusters that are part of an ArcGIS              Server Site. If you have multiple ArcGIS Server Sites configured

Click the ArcGIS Server (Web) button to display the following:

Fill out the above form as indicated below:

1.   Enter the Server URL.

  1. The typical syntax with ArcGIS Server 10.2 or higher is: https://<host_name>:<port_number>/arcgis  
  2. The typical syntax with ArcGIS Server 10.1 is: https://<host_name>:<port_number>/ArcGIS
Note:    If your URL structure is different, enter it.

2.   Enter the ArcGIS Server Manager user name with publisher or better permissions. 

3.   Enter the users password

4.   Check this box if you are accessing a Site federated to Portal for ArcGIS

Note:   Consider using a web adapter address for the Server URL:  https://<webadaptor_name>/server
Note:   If accessing over the internet, this assumes that the web adapter was registered with administrative access to ArcGIS Server

5.   Check this box if you use IWA(Integrated Windows Authentication)

6.   If needed, specify a token(advanced option)

7.   Select an End Time (Now)

8.   Select Start Time (1 week)

9.   Select Analysis Type (Complete)

  1. Simple: Provides only the Service Summary page data. 

    Note: This mode will also generate a list of the underlying data source by service and layer in the service. 

  2. WithOverviewCharts: Provides the Service Summary page plus charts of Request Count, Average Request Response Time, and Max Request Response Time.

  3. Complete: Provides Service Summary page plus all data and charts in separate tabs for all services.

  4. ErrorsOnly: Provides a report of just the errors.
  5. VerboseMode: Provides full verbose log analysis (Limited to 12 hours).

10.   Select Report Type (Spreadsheet)

11.   Specify where to output the report (Default is your My Documents location)

 

Click Analyze Logs. Analyze Logs

This process can take a few minutes or longer, this all depends on the number of transactions logged.

Review the System Log Parser report

 

When System Log Parser finishes running, it will open the report in Excel if present.  If you ran this from a machine without Microsoft Excel, move it to a computer with Excel and open.

 

You will note that there is a summary tab, and several tabs listed across the bottom of the spreadsheet.  We'll cover each in further detail below, by tab.

 

Summary

When the Excel report opens, you will see the Summary tab. The below screen grab shows what server this was run against and some summary statistics.

 

Summary

 

Statistics

On the bottom of the Excel report select the Statistics tab to view a table of all services by layer and service types.  this is where we'll spend most of our time.  Please read the rest of this post, then click here.

 

Resources

On the bottom of the Excel report select the Resources tab to view several charts:

  • Top 20 Resources by Count
  • Top 20 Resources by Average Response Time
  • Top 20 Resources by Maximum Response Time

 

Methods

On the bottom of the Excel report select the Methods tab to view several charts:

  • Top 20 Methods by Count
  • Top 20 Methods by Average Response Time
  • Top 20 Methods by Maximum Response Time

 

Queue Time

On the bottom of the Excel report select the Queue Time tab to view any services that had to wait for a ArcSOC to return a result. In an ideal setting the below is the desired value:

 

Queue Time Stats

 

Users

On the bottom of the Excel report select the Users tab to view a chart of the top 20 users by request count.

 

Time

On the bottom of the Excel report select the Time tab to view a chart of requests by day.

 

Throughput per Minute

On the bottom of the Excel report select the Throughput per Minute tab to few a minute by minute breakdown of requests.

Below is a sample of what information can be found on the tab:

 

Throughput Per Minute

 

Elapsed Time of All Resources

On the bottom of the Excel report, select the Elapsed Time of All Resources tab to view chronological listing of all requests from the time period the System Log Parser report was generated.

 

I'd also like to thank Aaron Lopez for his help and continued development of this invaluable tool. 

 

Note: The contents presented above are recommendations that will typically improve performance for many scenarios. However, in some cases, these recommendations may not produce better performance results, in which case, additional performance testing and system configuration modifications may be needed.

 

I hope you find this helpful, do not hesitate to post your questions here: ArcGIS Architecture Series: Tools of an Architect

What is System Log Parser?

System Log Parser is an ArcGIS for Server (10.1+) log query and analyzer tool to help you quickly quantify the "GIS" in your deployment. When run, it connects to an ArcGIS for Server instance on port 6080/6443/443 as a publisher (or an administrator), retrieves the logs from a time duration (specified as an input), analyzes the information then produces a spreadsheet version of the data that summarizes the service statistics. The command-line version of System Log Parser (slp.exe) is used by ArcGIS Monitor for data capture.

 

Note:   This post is a second in a series on System Log Parser, please see ArcGIS Server Tuning and Optimization with System Log Parser to learn how to setup your server for System Log Parser and an overview of the report.

Introduction to Statistics Used In System Log Parser

 

There are several statistical categories you should be familiar with when using System Log Parser. (definitions from Wikipedia)

 

Percentile (P) - a measure used in statistics indicating the value below which a given percentage of observations in a group of observations falls. For example, the 20th percentile is the value (or score) below which 20% of the observations may be found. 

 

Average (avg) -   is a single number taken as representative of a list of numbers. Different concepts of average are used in different contexts. Often "average" refers to the arithmetic mean, the sum of the numbers divided by how many numbers are being averaged. In statistics, mean, median, and mode are all known as measures of central tendency, and in colloquial usage any of these might be called an average value. 

 

Maximum (Max) -   [L]argest value of the function within a given range.

 

Minimum (Min) -   [S]mallest value of the function within a given range.

 

Standard Deviation (Stdev) -   [A] measure that is used to quantify the amount of variation or dispersion of a set of data values. A low standard deviation indicates that the data points tend to be close to the mean (also called the expected value) of the set, while a high standard deviation indicates that the data points are spread out over a wider range of values.

 

Fields of the Statistics Collected

 

Field
Definition
Resource Requested resource or service (Service REST endpoint)
Capability The ArcGIS capability of the resource
Method The function performed by the resource (What was accessed)
CountThe number of requests for this resource
Count Pct Count percentage based on total service requests
Avg The average time (in seconds) spent processing request
MinThe time (in seconds) of the shortest request
P5, P25, P50, P75The percentile grouping of the time (in seconds)
P9595% of all responses occur between 0 seconds and the value displayed in this column per service
P9999% of all responses occur between 0 seconds and the value displayed in this column per service
MaxThe time (in seconds) of the longest request
StdevThe standard deviation of time (in seconds)
SumThe total time (in seconds) spent processing requests per resource
Sum PctThe total time (in seconds) spent processing requests

 

We're going to focus on 2 key statistics, P95 and Max.  As we learned above, P95 signifies the response time for the fastest 95% of all requests and Max signifies the maximum draw time per request per service and method.

 

Identifying Opportunities to Tune Service Performance

 

In the example below, I've sorted P95 and Max values over 1/2 second.  User experience drops the longer your draw-time takes. 

 

I've highlighted any Max draw time over 1/2 second in red and any P95 draw time over 1/2 second in yellow.  These are the services and layers I'd focus on cleaning up, focusing first on getting the P95 value below 1/2 second first. 

In the next section you'll find starting points to tune and optimize your services.

 

Another column worth reviewing is the Sum Pct.  this column factors in the number of requests for each service and the respective average time, then weights that in against all the other services.

 

Sum Pct

 

For example:   

  1. One service may have thousands of more requests than all others but it has fast times (Sum Pct should be low)
  2. Another service may have just a small handful of requests but very slow times (Sum Pct should be high). In this case, this service would be a good candidate to for tuning.

 

Best Practices for Services

 

Below are some links to get you started on service tuning and SOC management.

         

In addition to the above, data source performance should be looked at if adjustments to the service do not help enough. You can look at:

 

I hope you find this helpful, do not hesitate to post your questions here: https://community.esri.com/thread/231451-arcgis-architecture-series-tools-of-an-architect

 

Note: The contents presented above are recommendations that will typically improve performance for many scenarios. However, in some cases, these recommendations may not produce better performance results, in which case, additional performance testing and system configuration modifications may be needed.