Latest Contributions by DannyKrouk

Introduction GIS Enterprise Reporter now has a companion application “er_compare.exe” that allows you to compare two ArcGIS Enterprise systems. This is the fourth article in the GIS Enterprise Reporter series.  If you are not familiar with the series, this is the first article: Introducing the GIS Enterprise Reporter.  Why is Comparison Useful? There are many reasons why you might want to compare two ArcGIS Enterprise systems.  GIS Enterprise Reporter envisions the following motivations: You are setting up a Disaster Recovery Site.  In this case, you may wish to understand that the deployments and configuration are sufficiently compatible to allow you to use the webgisdr tool to synchronize the content. You have a Production and one or more lower environments.  In this case, it may be useful to understand the nature and extent of the differences between the Production environment system and the other(s). How to Compare Comparison is a three-step process: Use GIS Enterprise Reporter to generate outputs (“admin” and/or “content” at the initial release of this tool) for the two systems you wish to compare. Run “er_compare.exe” against those output files. Open the resulting Excel document and review its comparison notes Step #1, the use of the GIS Enterprise Reporter tool, has been described in other blog articles in this series.  Please refer to those if you would like more guidance on how to generate outputs for an ArcGIS Enterprise system. The remainder of this blog article will focus on steps #2 and #3. Run "er_compare.exe" The application can be run by double-clicking on the er_compare.exe executable.  When run in this way, you may briefly see a command window appear.  It will be replaced with a graphical user interface as shown here. Navigate to the two output files that you wish to compare from the two systems; those will be your “File One” and “File Two”.  You can compare the “admin” outputs of two systems to see how the administrative configurations and services inventories compare.  Or, you can compare the “content” outputs of two systems to see the differences/similarities between the Portal content. “File One” is typically the “source” system, the one that you consider to be the standard against which you want to measure the other system.  For example, if you have a primary data center and a stand-by data center, the primary data center would be “File One”.  Or, if you have a Production system and a Pre-Production system, Production would be “File One”. The application will suggest an output file location and name based on your inputs.  But, you may name it and place it wherever you like.  Then, click the “Compare” button. The tool will report its progress in the lower status pane of the application.  When it is complete, it will change the “Compare” button into a “Dismiss” button. Note that you can also run the tool on the command-line.  If you do so, provide three space-delimited arguments for “File One”, “File Two”, and “Output File” and the application will do the same thing as it does from the graphical user interface. Review the Comparison Excel Document The application accepts Excel inputs and creates a new Excel output.  The output Excel will have one sheet for each pair of input file sheets that it attempted to compare.  If the input files do not have the same number or kind of sheets, the output will note the “missing” sheets from “File Two” relative to “File One”.  Note that the application does a logical comparison of sheets based on what they represent, not their literal names.  It uses the GIS Enterprise Reporter “site number” to know, in the case of multiple federated ArcGIS Server Sites, which sites are logically comparable. The first sheet is always “Comparison_Metadata”.  It has the structure show below: An "Admin" Comparison Example The remaining sheets will be named for the input sheets that they compare.  For example, “portalAdmin” compares the Admin API sheet from each Portal.  An example of a comparison output for it appears here The column layout is the same for all sheets.  The columns are: DifferenceSeverity The likely relative importance of the difference. DifferenceComment A brief description of what is found to be different. TypeOfSource This is somewhat redundant with the name of the sheet. RecordOne The information that it found in “File One”. RecordTwo The compared information that it found in “File Two”. ProcessingErrorInformation In the case that the application is not able to make the comparison it intends for some reason, there may be additional information about why in this column.   The portalAdmin example above shows that the tool has identified one “Error” and three “Warnings” (the rows in all sheets are color-coded by these classifications).  The Error shown here is that there are a different number of “redirectURI’s” for the “arcgisonline appId”.  These appId’s are used as part of the ArcGIS Enterprise OAuth security implementation.   Note that the application is not comparing the values.  The values across two systems can be different.  Instead, it is comparing the number of redirectURI’s.  It does not know how many there should be.  But, it assumes that the “source system” (“File One”) has been configured according to your intent and the other system should be a numerical match for the number of redirectURI’s.  The first Warning is reporting on a difference in the number of licensed members between the two systems.  The second system may or may not have “enough” licensed members; that is for you to decide.  All this tool can do is note that there are different numbers. The second and third Warnings are noting that the source system is configured for Windows users whereas the second system is configured for “Built-In” users. These are just examples.  The tool does not report on any comparisons for which it does not find a difference of concern.  In other words, in perfectly matched systems, the comparison sheets will be empty.  The screen capture below shows another example; a comparison of the Server Admin API for matching federated Server Sites between the two ArcGIS Enterprise systems. The first Warning is that the number of platform services between the two ArcGIS Server Sites is different.  It is not easy to see what the difference is in the “RecordOne” and “RecordTwo” cells.  But, if you were to study the JSON there carefully, you would see that the “source” system has a “Message Bus” platform service, in addition to a “Synchronization” and a “Compute Platform”.  The other system has only “Synchronization” and “Compute Platform”.  Likely, this indicates some difference in what has been installed or licensed for this Site between the compared systems. The second Warning is that the “source” system supports HTTP_and_HTTPS whereas the other system supports only HTTPS.  This might be because the “source” system was upgraded from a prior release whereas the other system was a fresh installation.  Or, it might be an indication that the “source” system has the preferred configuration and the other one should be matched to it. The third and fourth Warnings aggregate the member and processor resources across all of the machines in the Sites and report differences.  Again, these differences may be intentional or not, the application cannot know.  But, a common source of problems is an inadvertent difference in machine resources.  Note that the tool sums the memory and cpu from all of the machines in the Site.  So, if you have different numbers of machines, these aggregates will have different values even if the per-machine resources are identical. In the final row, there is an example of an “Information” difference.  The log settings (the “logLevel”) are different between the two systems.  This should not cause a problem for most use cases.  But, it may be relevant to know. The final example here shows the “Information” comparisons of the service “analytics”. In this case, the application is reporting that the numbers of services of various kinds, their instancing, etc. is a match between the compared systems.  A "Content" Comparison Example The sheets in the “content” comparison output are likely easier to understand.  Below is an example which shows the sheets you can expect and an example of one of the sheets. As you might expect, the “Users” sheet will report on any users in “File One” that are not in “File Two” … and vice verse.  The same is true for “Groups” and the other sheets. The InventoryAnalytics detail shown here is comparing the aggregate counts and sums of the content in each system.  In the case of Production and Pre-Production environments, this can be useful to get a sense of “how different” the content is.  And, in the case of a Disaster Recovery content synchronization, this can be useful to confirm that all of the content is the same between the two systems after the synchronization. Understanding Comparison The preceding examples of the sheets and comparison rows are a small sample of what the application does.  You should bear in mind that the application does not compare everything.  It only compares differences that are likely to be meaningful or interesting in some way.  The application does not, for example, compare the names of the machines between systems.  However, it will report if the count of machines is different.  And, for similar reasons, it does not compare the subjects of self-signed certificates when comparing certificates.  Rather, it matches the serial numbers of all the non-self-signed certificates on the assumption that the certificates that have been imported in one system should also be in the other system. You should not expect the application to catch all meaningful differences.  The comparison logic is based upon the real-life experiences of Esri Professional Services in helping customers with “Enterprise Migration”, “Disaster Recovery”, “Content Promotion”, “Lower Environment Usage Protocols”, etc.  So, the body of knowledge changes over time based on experience.  At this time the comparison logic is not documented.  If you have an interest in a particular type of comparison, feel free to send an email inquiry to dkrouk@esri.com.  I can report on the nature of the comparison that is currently implemented in the application.  And, if an additional comparison is appropriate, I can plan a new comparison feature. You should also expect “false negatives”; differences that are not important to you.  You should think of all differences, even “Error” differences as potentially significant, but not necessarily significant.  You must use your judgement to determine whether a difference is meaningful to your situation. The output rows have classifications like “Error”, “Warning”, and “Information” in the normal cases.  However, you may also see classifications like “ProcessingConcern”, “ProcessingWarning”, or “ProcessingError”.  These indicate either insufficient input information for the tool to carry-out its intended comparison or an internal error in executing its comparison logic.  In the former case, this typically indicates a difference between your systems or differences in the parameters used when you ran GIS Enterprise Reporter.  In the latter case, an internal error, the ProcessingErrorInformation column will have information about the internal error.  This is, essentially, a log of sorts that can be used to help me troubleshoot the internal error.  So, when you see this kind of issue, please bring it to my attention so I can troubleshoot: dkrouk@esri.com. What to do with the Comparison The best practice pattern with comparison is to do it at least twice: The first comparison is to discover the differences and decide whether and what to do about them. The second comparison is to confirm that the differences you chose to address were, in fact, successfully addressed. Final Thought Hopefully, this tool helps you in your ArcGIS Enterprise administration work.  If you have questions or ideas for improvement, please be in contact: dkrouk@esri.com.             ... View more

Introduction GIS Enterprise Reporter provides a lot of information about the content and configuration of an ArcGIS Enterprise.  Among the outputs available is an "admin" Excel workbook.  That workbook's sheets contain information available from the Admin API's of each server role in the ArcGIS Enterprise (Portal and all of the federated ArcGIS Server Sites).  For each federated ArcGIS Server Site, there are several sheets that describe aspects of the services inventory. This is the third article in the GIS Enterprise Reporter series.  If you are not familiar with the series, this is the first article: Introducing the GIS Enterprise Reporter.  Generating the "Admin" Excel The default execution parameters of GIS Enterprise Reporter will generate the "Admin" Excel output.  The "Admin + Services Inventory" checkbox creates and populates the workbook from the Admin API.  The "With Analytics" checkbox will calculate a variety of aggregate statistics and other information of interest from the raw outputs, placing them in additional sheets in the same Excel workbook.   The workbook will be placed in the output directory and will bear a name indicating the name of the ArcGIS Enterprise followed by "_admin_" and a timestamp as shown here: Understanding the Services Content Within the Admin Excel workbook, each federated ArcGIS Server site will have a series of tabs that begins with a number, the name of the site, and a suffix.  Finding the Basics An example of the main sheets for a federated Site appears below: Each federated Site is given a number by GIS Enterprise Reporter.  In this example, we are looking at site number "1".  And, the site name is dannyk_esri_com (the DNS name by which the Site is known).  The suffixes have the following meanings: serveradm: Information extracted from the Admin API about how the Site is configured. serverdat: If the site is a Hosting Server Site, this sheet will provide a little bit of additional information about the Data Store. serversvc: An exhaustive list of all of the properties of all of the services in the ArcGIS Server Site.  If you need to know any particular fact about a service that ArcGIS Server knows, it can be found here. serversvn: A "normalized" listing of the most commonly used properties of the services in the ArcGIS Server Site.  The sheet is more useful to understand the services inventory over-all. The "Normalized" Services Listing The "normalized" services listing is a good place to start.  Each line is a service and some essential pieces of information about it. The first row has filters that can be used to focus on information of particular interest. The Detail in "serversvc" The "serversvc" sheet has one line per service property.  This makes it less convenient for understanding the big picture.  But, if you want to know the recycleStartTime, the portalProperties, or the referenced datasets, they are available here. Exploring the Analytics If you selected "With Analytics" there will be two additional sheets for each federated ArcGIS Server Site.  Their names include only the site number assigned by GIS Enterprise Reporter and a suffix indicating their content. svc_datasets The svc_datasets sheet is a listing of all of the datasets referenced by the dedicated instance services in the Site (Hosted Services and Shared Instance services do not include this information in the ArcGIS Server Admin API).  As with the other sheets, it is filterable if you wish to focus on particular services or datasets. The dataset information is formatted as it is in ArcGIS Server: datasetname followed by the workspace from where it comes. svc_analytics "svc_analytics" contains a variety of calculated statistics about the services (and the service instances) in the Site. The first few statistics are about the services over-all.  However, most of the statistics are focused on only those services that are running. The "Total Minimum Instances Per Node (started)" is of interest because this is the number of instances that exist as soon as ArcGIS Server starts.  The "maximum" statistic is the upper limit.  However, it is almost never the case that all services have their maximum instantiation at the same time. Statistics are presented by "provider" (indicating something about how the service was published and how it runs) and "type" (indicating something about what it can do).  As noted, these statistics focus on the services that are running, and their min/max instancing. Conclusion Hopefully, these Excel worksheets provide information that help you better understand and manage the services in your ArcGIS Enterprise.  If you have ideas for improved information, please share it with me: Danny Krouk (dkrouk@esri.com). The next article in this series explores how to compare GIS Enterprise Reporter outputs to gain insight into meaningful differences between systems: Comparing ArcGIS Enterprise Systems.               ... View more

The GIS Enterprise Reporter user interface is intended to have few options in the hopes that it is relatively straight-forward to execute.  In principle, you give it a Portal for ArcGIS "Target", with an administrative credential for that Portal, tick some checkboxes, click "Execute", and wait for it to complete.     There are some things that benefit from explanation.  This article will offer information on the following topics: Where should I run this application? How should I run this application? What value should I put in the "Target"? What are some common problems (and solutions)? What do the options (checkboxes) mean? Where Should This Application Be Run? GIS Enterprise Reporter can, in principle, be run from anywhere that it can communicate to your Portal for ArcGIS.  However, there are better and worse locations.  The application works by asking Portal about itself and its federated ArcGIS Servers.  This means that Reporter wants to communicate with the federated Servers in the same way that Portal does.  Various aspects of your configuration and network rules may mean that communicating from your personal computer with Server cannot happen through the same means as Portal.  How would you know?  Not easily.  So, how could you protect yourself against this possibility?  The easiest way to do that is to run the application from within your server network.  For example, you could run it on the machine on which Portal runs. How Should this Application Be Run? It is not necessary to run Reporter as a local machine administrator.  However, doing so will usually allow it to gather more information for you.  And, it may help you to overcome some potential execution challenges. If you wish to run the application as a local machine administrator (assuming you are a member of that local machine Administrators group), right click on the enterprise_reporter.exe executable and select "Run as administrator" from the context menu.   You may or may not receive a message from your operating system or security software that discourages you from running this application.  This application is digitally signed by Esri with a chain of authority from a public Certificate Authority.  So, it is unlikely that the message will mention certificate trust.  But, if it does, you may wish to explore importing the trust chain into your trusted repository.  To initiate the "installation" workflow for the certificate, follow the steps show in the screen capture below :   Selecting "Current User" and "Automatically select the certificate store based on the type of certificate" are the appropriate options on the "Install Certificate" wizard panels.  This may relieve you of the warning. Or, if you are a machine administrator, you may have options to execute the application.  To do so, you may have to click buttons that say things like "More Options", "Run Anyway", etc. ... depending on what is issuing the warning and your rights.  For example:   These technical methods are not offered to circumvent the guidance of your organization with respect to third party executables.  Please use the privileges granted to you by your organization responsibly.   The Reporter application only runs on Windows.  However, you can use it to report on an ArcGIS Enterprise on Windows or Linux.  In the event that your Enterprise is on Linux, running Reporter as a local machine administrator does not produce any additional output information in the report documents.  The only reason to run Reporter as local machine administrator in that case is to give you more options in response to an unsigned application warning if you receive one. What Is the Right Value for "Target"? There are three basic options: The FQDN ('portalmachine.domain.com') of the machine The 7443 endpoint for Portal ('https://portalmachine.domain.com:7443/') The WebContextUrl for Portal ('https://proxy.domain.com/portalcontext') The first two options amount to the same thing.  If you enter the FQDN of your Portal machine (either machine in the case of a two-machine "high-availability" Portal), Reporter will fix-it up into option #2 for you.  In addition to being relatively straight-forwards, this is usually the best option for "Target" value as well.  So, it is recommended that you use the FQDN (fully-qualified domain name) of a Portal machine (the machine's "A RECORD"). Of course, this implies that your network will allow you to get to Portal's 7443 port from where you are running Reporter.  This is why the first tip encourages you to run Reporter on your server network because it is on that network that this network pathway will be allowed. The third option is to provide the WebContextUrl (or Web Adaptor-style address) as the target.  This generally works.  However, Reporter only knows about "Portal tier authentication".  So, if this pathway (WebContextUrl) involves "web tier authentication" or requires SAML, OIDC, etc., Reporter will not be able to authenticate.   As you might be able to infer, the values for "Admin user" and "password" must be a user that has administrative rights on your Portal and is authenticated at the Portal tier.  This certainly includes Portal's "Initial Administrator Account".  But, you can also use any other account that is a Portal administrator and can get a token directly from Portal's 7443 REST endpoints. Common Problems and Solutions The application comes with a Microsoft Word document which includes a section on Troubleshooting.  If you are having a problem, you should consult that first.   The most common problems are usually having to do with access to the token REST endpoint.  Those problems are best avoided by running the application from the server network and using the machine FQDN as the target address.  The message in the application will look like this: Exception while acquiring token. Please use a browser to check the url: https://<f.q.d.n>:7443/arcgis/sharing/rest/generateToken and credentials. As you can see, it asks you to validate that the URL target to get a token is reachable from the machine on which you are running Reporter.  If you can get to that endpoint, then try generating a token.  If you can do both of those things, Reporter will run. What Are the Checkbox Options? At a high-level, the checkboxes have the following implications: "Admin + Services Inventory": This touches all of the REST endpoints in the Admin APIs for your Portal and your federated Servers.  It records all of the configuration information of your Enterprise, lists all of its services and their properties, etc.  The information is quite broad and deep. This makes it very valuable.  But, it can be challenge to get a high-level picture.  The specific contents of this file output will be explored in future articles. "Diagram": The diagram provides a high-level picture of the logical structure of your Enterprise.  This is an example from a very simple (development environment) system: "Certificate Analysis": Reporter does its best to connect to each of the HTTPS endpoints in your Enterprise (i.e. Reverse proxies, Web Adaptors, Portal's on 7443, Server's on 6443, etc.), discovering the certificates and reporting on various facts like certificate subjects, expirations, and trust chains.  There are several sheets in the Excel, each sheet corresponding to a different HTTPS access pathway.  For example: "Portal Content Inventory":  This output is created by stepping through Portal's Sharing API.  It lists all of the content in the Portal, along with its "metadata" (content type, owner, etc.).  And, it can provide lists of users, groups, etc.  This output will also be explored in more detail in future articles. "Migrator Artifacts (advanced)":  This is, as the name implies, an advanced option.  Most users will never need this.  This option does two things.  The first is that it writes out the json from the REST endpoints into a subdirectory.  So, if you are someone that likes working with json, you can find it all there.  Second, when it finds sufficient information in Portal to export out "service definitions", it will do so.  This can be helpful if you wish to re-publish services.  If you published your service through ArcGIS Pro, it was very likely to story the service definitions there such that Reporter can access them.   Conclusion Hopefully, this article has provided a useful orientation to how to run the GIS Enterprise Reporter tool.  As mentioned, it comes with a Microsoft Word document that provides additional information.  Should you run into challenges or have questions that are not clearly addressed, please be in touch through the contact information in that document.  You may post comments questions here also, if you wish. Prior Series Article: Introducing the GIS Enterprise Reporter Next Series Article: GIS Enterprise Reporter: Making the Most of Services Outputs      ... View more

A common challenge in administering an ArcGIS Enterprise system is the need document and/or otherwise communicate its design and configuration.  Typically, this is done by someone creating one or more diagrams.  These diagrams sometimes have errors and/or omissions.  And, they easily lose their relevance as the system changes.  The underlying challenge is that the documentation of the system is not generated by the system itself, it reflects the understanding of one person at one moment in time. GIS Enterprise Reporter is designed to help address this challenge by automating the creation of documentation artifacts.  Point the tool at your ArcGIS Enterprise, with administrative credentials, and it will generate documentation that includes: A logical diagram Listings of administrative/configuration settings (from the REST admin API's) Listings of machine resources and installed software (for the machines running the software in the ArcGIS Enterprise system) ArcGIS Server service inventories (inclusive of workspaces and datasets) Portal content inventory Portal users and groups inventory And, information about TLS/SSL certificates (including certificate expirations, required trust chains, etc.) The documentation is output in Microsoft Excel format, making it easy to share, analyze, and mark-up. This tool is intended to be used by ArcGIS Enterprise administrators to understand and document their system deployments.  It works best if it is run under a credential that is also a local machine administrator, but that is not required. Please note that this tool has been created to support Esri Professional Services consulting practices.  As such, it is not supported by Esri Technical Support.  All of the requests issued by the tool are supported ArcGIS REST API operations.   You can download the tool here: https://github.com/EsriPS/gisenterprisereporter.  The .zip contains both the executable and documentation.  The executable is a ".Net Core" executable that should run on any contemporary Windows operating system.  The executable is signed with a certificate that is not issued by a public certificate authority.  So, you may get warnings (or limits) from your operating system related to that.  If so, please follow the guidelines of your organization and the privileges extended to you by working with the appropriate IT security authorities, importing the certificate into a your trust repository, or using elevated privileges to execute the tool. Please be in contact with me, Danny Krouk (dkrouk@esri.com) with questions or enhancement ideas. This article is intended to be the first in a short series of articles about GIS Enterprise Reporter.  Future articles will address topics such as running the tool, interpreting its output, and creating useful analyses from the output. Next Series Article: Running the GIS Enterprise Reporter, Tips & Tricks   ... View more

Introduction ArcGIS Enterprise architectures often have requirements for use of file shares for storage of shared configuration files, content, or backups (https://enterprise.arcgis.com/en/server/latest/install/windows/choosing-a-nas-device.htm). This is particularly true when the deployment pattern involves multiple-machine sites (such as the high-availability configurations). There are uses of file shares across each of the main ArcGIS Enterprise components (Portal for ArcGIS, ArcGIS Server, and ArcGIS Data Store). In the diagram below, the “Shared Content” and “Shared Config-store and Directories” are located on a file share:   File shares come in many types and varieties, ranging from physical hardware devices to virtual file systems or other providers. Shared file systems provide an efficient way of sharing content between multiple components, but they can also introduce challenges related to performance, permissions, or file-level consistency. When there are problems in an ArcGIS Enterprise deployment, and you suspect they may be related to shared storage, it can be difficult to know how to assess your architecture for potential challenges and what to do about it. In practice, solving a file share problem can be challenging, but your chances of success are much better if you establish a specific metric or measurement to use to define, create a baseline for, and test the suspected problem. This article is designed to help you determine whether there is a file share problem in your ArcGIS Enterprise deployment, what symptoms and indicators might point to this type of problem (its signature), and how to investigate the root cause. While similar principles apply to Linux/NFS and Windows/SMB systems, the details of this article focus on Windows/SMB systems. How does ArcGIS Enterprise use a file share? There are several ways in which ArcGIS Enterprise uses a file share, including the following: A registered data repository in an ArcGIS Server site A shared backup location for ArcGIS Data Store backups A location for storage and extraction of WebGISDR backups for disaster recovery workflows A location to store the “config-store” and “server directories” of a multiple-machine ArcGIS Server site A location to store the “content directory” of a highly available ArcGIS Enterprise portal site The latter two are the focus of this article; in these cases, the file share plays a role in how each machine in the multiple-machine site knows what is going on. Metaphorically, the file share acts like part of the “nervous system” of the ArcGIS Enterprise portal or ArcGIS Server site when it is supporting the “config-store”, “server directories”, or “content directory”. So, if there is a problem with the file share, the ArcGIS Server or Portal for ArcGIS site can display a wide variety of intermittent symptoms, such as trouble publishing services or server “instability”. Recognizing and investigating a problem related to a file share Analyzing a potential file share problem may begin as a solo effort, examining the software logs for each ArcGIS software component. However, if you find evidence there that points to issues related to file access, you will need to work with other information sources or software components. This will likely mean working with other people, as the required access privileges and knowledge are rarely concentrated in a single person. This article describes what you can pursue with different sets of privileges and knowledge. We start with the logs in ArcGIS Enterprise for two reasons: first, we assume that you have those privileges—and second, this is where you make the initial determination of whether there is reason to believe there is a file share problem and what that reason is. Foundations Before you begin, you want to set yourself up to be as productive as possible by choosing an appropriate environment, controlling for complexity, and getting the right team put together. Environments You should try to make use of “lower environments” (in other words, not your production environment) if you can observe the problems in those systems. Whatever problem you see in production, use the ArcGIS Server logs to understand its signature. Finding that signature is discussed below. Then, look in your staging, testing, or UAT environment to see if you can find the same signature. Note that you may have to generate some activity in that environment to see the problem (many problems do not present when the system is idle). If you can see the problem in the lower environment, that is the environment in which you should investigate. Since you have more control over the level of activity in that environment, you will be less distracted by unrelated factors. And, because it is more practical to know what is happening, you can form better ideas about potential causes. Finally, when it comes to making changes for troubleshooting, you should always make them in lower environments first, when possible, to avoid unnecessary disruptions to users. Complexity You want to back off as much complexity as possible without changing your basic configuration. Working in a lower environment helps to reduce complexity. But, when you are working with multiple-machine sites, the multiple machines mean that you have multiple places to look for any given issue.  A frequently effective practice is to shut down the redundant machines in the site. For example, if there are three machines in an ArcGIS Server site, turn off (the OS) or stop (in the site) two machines. The remaining machine is still using the file share for the same purposes, and many problems will continue to express themselves. If shutting down the redundant machines causes the problem to disappear, this is an important clue about the nature of the problem. Specifically, it tells you that the problem has to do with concurrent client access, and perhaps not the network. Involve others When troubleshooting issues that span multiple environments, you may run into problems that exceed your access privileges or experience. While privileges can be granted temporarily, experience in those domains is just as valuable. Setting yourself up with a dedicated problem-solving team will ensure that you preemptively have resources on hand when questions come up. A frequent source of dysfunction in this kind of virtual team effort is getting everyone to understand how they can make a meaningful contribution. Network administrators and file share administrators typically know little about Esri software and are not likely incentivized to learn a lot about the applications on their infrastructure. However, they typically have curious minds and like to solve problems. If you present them with an open-ended question (“Is the network having problems?”) or a prematurely broad accusation (“The network is causing us problems”), you are unlikely to get interesting responses. On the other hand, if you ask evidence-driven, specific questions, your odds of a productive response go up. For example, “We see ‘connection time out’ messages in our application logs at times X, Y, and Z. It seems to be happening every 2 to 3 hours. Would you be able to capture traffic for that period of time and help us understand what is happening with the connections?” Evidence-based hypotheses and investigation Whether you are trying to be effective on your own or as part of a virtual team, an evidence-based approach is the gold standard for making progress. Some cornerstones of an evidence-based approach are as follows: Make your initial observations before making any changes to the system, and establish these observations are consistent. Start with specific observations, about a certain workflow, request, or operation. Find repetition or repeatability. You don’t want to chase outliers or false alarms. Document as you go. You want to be able to go back and confirm details of your observations and share your observations with others. Create more than one hypothesized cause for each problem. Your first idea is rarely the answer; create several, pursuing them one at a time. Identify the evidence that could invalidate (or support) a hypothesis and then pursue that evidence. Leverage the expertise of others. One of the best ways to get participation from an expert is to ask them to show off their expertise by explaining the possible meanings of a specific observation. It is a double win: you advance your understanding, and you make them more likely to want to help you going forward. What you can learn as an ArcGIS Enterprise administrator: The signature The logs in ArcGIS Enterprise (ArcGIS Enterprise portal logs and ArcGIS Server logs) are where you identify your problem signature. It is the measurement you will use to determine whether you are having a file share problem and whether a change you make actually resolved it. When an ArcGIS Enterprise component “talks” to a file share, it is reading and writing file objects, often referred to as Input/Output or I/O. And, it is doing so as a specific account (the service account), so you can have permissions problems and file system problems.  Permissions problems are usually quite recognizable in the log messages and relatively straightforward to address. For example, the log message “Cannot write to directory path ''{0}''. Please check that the location is valid and that the ArcGIS Server account has permissions to the location.” (code 6697) describes the cause and provides an idea for the solution. Note that effective permissions will involve both those for the share itself and the files and directories exposed through the share. Share Permissions File and Directory Permissions       Log messages that mention a path related to the file share, I/O, or an IOException likely mean that permissions are not the issue. At the end of this article, there is an appendix with a partial list of log codes and message types that are correlated with file share problems. Correlation is foundational for establishing causation. If you see messages like these, it’s a good indication that you need to look more closely at the file share and/or the network pathway to it, though not irrefutable proof that the file share is at fault.    The details of the log message types may obscure the high-level patterns you are looking for. A file share is a file system on the other side of a network, so if there’s a problem, there can be at least two types of sources: the file share solution itself or the network. The following are examples of messages that indicate a problem accessing files from a file share (some information removed for clarity or privacy): Enterprise Component Level Code Message Notes Server WARNING 7721 “failed to write heartbeat”   Server WARNING 7712 “An error was encountered while synchronizing with the config store”   Server SEVERE 6561 “Failed to return all folder configurations”   Server SEVERE 9000 “Internal Server Error: “Service <name> not found” Note that a request for a service that does not currently exist will generate a message like this also. For this to indicate a problem with a file share, the service must actually exist in the site. Server SEVERE 6605 “Failed to return all services configurations in the folder … (The system cannot find the file specified)”   Server SEVERE 6652 “Unable to read the service … from the configuration store … (The system cannot find the file specified)”   Server SEVERE 6566 “Failed to retrieve the status of the service … (The system cannot find the file specified)”   Server SEVERE 9015 “Error getting list of services. … (The system cannot find the file specified)”   Server SEVERE 6615 “Unable to retrieve 'Permissions' resource information. …” This message is not exclusive to file share issues, but it can be associated with file share issues. Enterprise portal SEVERE 218037 “The Portal site has been initialized and configured but is currently not accessible because the content directory is not available …”     Which of these messages might lead you to focus on the network and which might lead you to focus on the file share? None of these messages involve phrases like “connection timed out,” “connection was forcibly closed,” or “timeout.”  This absence suggests that there was not a problem connecting to the file share; there was a problem after that point. This points you to the file share as the next place you should focus your attention. Conversely, if you do see a phrase like “connection time out”, this is an indication that you should investigate your network—because connecting is what networks do. Your Esri server log investigation should be able to establish 3.5 things: There are or are not messages that are specific to the file share locations supporting the config-store, server directories, or the content directory. The frequency of occurrence of these messages Ideally, if they are correlated with any particular activity (pattern of requests / use of the software) There is or is not evidence in the messages that the problem has to do with connecting (the network).  This is your signature. When you have a signature, you can begin to look at other information sources for events that are coincident in time. What you can learn with a local machine administrator We go to other log sources to look for the root cause or the “why.” In the case of a file share related problem, the Esri system is the victim. The Esri logs won’t likely offer insight into the root cause.  The local machine is the machine (or machines) on which the Esri software runs. It is the client to the file share server. Logs from the local machine allow you to learn whether there is a machine-specific problem or a problem that is detectable by the local machine.  A local machine administrator has access to the Event Viewer logs. These logs contain a wealth of information from various subsystems of the Windows OS. You want to take the timestamps from your Esri server log observations and see if there are interesting records in a few categories of logs from the Event Viewer system. In most circumstances, you want to look at the Windows System, Security, and Application categories. The screen capture below illustrates the log filtering capabilities that allow you to focus on types of messages for time periods of interest for these main Windows logs:   Because you are focused on file shares, there is an additional source of potential information buried a little more deeply in the Event Viewer log tree hierarchy. If you browse the tree structure on the left to Applications and Services Logs > Microsoft > Windows > SMBClient, there are three more logs (Connectivity, Operational, and Security) that relate specifically to SMB (SMB is the protocol for communicating with the file share). The screen capture below illustrates some log content with an error that, while not common, would certainly cause a problem for an Esri server:   Whether you are looking at the main Windows logs or the SMB Client logs, your primary focus should be on messages that are classified as Critical, Warning, or Error. You should filter based on these event levels and time. What can these logs tell you?  Does the local machine detect a problem? If you do not find thematically related errors in the main Windows logs for the timestamps from your Esri server logs, then you have some evidence that the problem is not due to a problem that is client-machine specific. While this may not be enough evidence to completely rule out the possibility, you have taken important steps toward that objective, and you can prioritize your efforts elsewhere. If you do find interesting errors on the local machine, you prioritize your efforts on understanding and trying to resolve those.  Does the SMB protocol detect a problem? If you do not find errors in the SMBClient logs that match the timestamps, you can infer that the problem is not an error as far as the SMB protocol is concerned. For example, if you are investigating messages that indicate a file cannot be found, the SMB protocol is not going to classify that as an error— For all it knows that file doesn’t exist. In that case, the protocol functioned correctly and returned a correct piece of information.  On the other hand, if you see an error such as the one shown above, it makes sense to focus the investigation on the SMB protocol itself. Finding thematically related errors in these logs is often very meaningful to other specialists who are unfamiliar with Esri logging but more likely to understand or trust operating system logging. What you can learn with other specialists Many file share problems will not present in the Event Viewer logs of the client OS. Most of the other sources of information require elevated privileges (beyond Esri server administration or local machine administration), specialized knowledge, or both. So, to make progress in this space, you must employ two important attributes: (a) your winning personality and (b) your commitment to evidence-based investigation. Networking people If you have the Esri application log signatures that suggest a network-related issue or failure, you will want to investigate in this space. Most networking problems that relate to file shares will not express themselves very clearly in classic network monitoring or logging solutions. Network monitoring often focuses on capacity, throughput, and quality of service. That is useful for network planning and administration overall but not necessarily helpful for troubleshooting specific connections. A firewall’s log of denials is worth consulting, but it is not commonly where file share problem causes are found. The answers are typically found by capturing network traffic during a short period of time when the problem occurs or is manually triggered. Capturing an intermittent problem is not as hard as it may seem. One can set up a ring buffer capture to keep capturing to stable inventory of files, overwriting them as time passes. Your organization’s network team, in addition to having the tools and privileges, will likely know how to do that. So, you don’t need to know exactly when the problem will next occur.  In many cases, a ring buffer can persist many hours’ worth of network capture data on a rolling basis. When the problem does occur again, ask the network team to stop the trace. Then, use the timestamps from the Esri server logs to investigate the network capture information. With some solid network foundational knowledge (from the network team or you) and some dedication, a lot can be learned. Even if your team doesn’t have much network analysis knowledge, you can look for anomalies. Either there are or there aren’t abnormal network characteristics at the timestamps in question. If there is something abnormal or suspicious, additional experts can be brought in as needed. Specificity will likely be attractive to these experts. File share people If your Esri server log signatures do not suggest a network issue, you will want to investigate this space. File share solutions, like most IT systems, typically have logs. A review of these logs at the times in question is appropriate. If there are thematically related errors for the timestamps in question, you have a likely cause to investigate further. And, you have your file share administrative team (and their vendor’s support organization) to lead the way. It is also possible that the file share logs do not have related errors. If the accumulated evidence has ruled out the other likely sources, and the file share logs do not indicate a problem, there is a remaining possibility. It is possible that the file share and the Esri software classify problems differently. For example, if a file share is designed or configured to provide eventual consistency, it will not record errors about that. But we know that ArcGIS Server is expecting immediate consistency. So, the file share sees no error, but ArcGIS Server is not getting what it needs. Let’s explore this idea a bit more, as it does come up, using the example of a two-machine ArcGIS Server site with the configuration store and server directories stored on a file share. If machine A writes to a file, ArcGIS Server site machine B should be able to read that new information right away. That is immediate consistency: read after write for any client to the file share. So, if ArcGIS Server is saying, “Hey, I can’t find that file,” or “That file looks different than I thought,” and the file share is saying, “I don’t know of any problems,” what is your resulting hypothesis? Having ruled out other likely causes, your resulting hypothesis is that the file share is not providing “read after write” consistency. No other idea fits the data nearly as well. What do you do with that? This is another investigation with the file share administration team. But, instead of focusing on an error in its logs, it is seeking to understand how multiple clients, looking at the same file, at the same time, see it in exactly the same state, always. No client can see the before state as valid after the file has been changed by another client. And no client can change a file when a different client has an exclusive lock on it. Uh-oh. We said the “L-word” (lock). That brings up one additional term or concept that you may have heard discussed in the past: opportunistic locking of files, also referred to as Oplocks. Is the problem Oplocks? While it is certainly possible that Oplocks are causing a problem for your system, for the problems we are talking about here the odds are against it. But, as evidence-based decision making is the way to make progress, you can use evidence to prove whether it is a cause. It is worth thinking about what Oplocks, and the alternatives, are. Oplocks are opportunistic locks. The client (the Esri system) assumes, optimistically, that the files it knows about on the file share are unchanged and/or safe to change unless it hears from the server (the file share). The opposite of opportunistic locking is pessimistic locking (no locking is not an option). In that case, the client assumes, pessimistically, that other clients could be using the files it knows about on the file share, so it checks before doing anything. Opportunistic locks are a great strategy when most files are not being accessed by more than one client at any given time. Pessimistic locks are a better strategy when files are frequently accessed by more than one client at a time. What do we know about a multiple-machine ArcGIS Server or Portal for ArcGIS site? There are at least two clients accessing the same files at the same time. So, Oplocks is suboptimal for Esri server use cases. But is it the cause of your problem? You don’t know that. Since you have your problem signature from your Esri logs, you can change the Oplocks settings and see if it changes the problem signature (the presence of the message and its frequency, given the same workload). These parameters can be changed in the client OS (the machines on which the Esri software runs) or the file share solution, depending on your configuration. Here is how to do it if you are a local machine administrator for the client OS. Open a PowerShell command window as “as Administrator” and note the current settings: Get-SmbClientConfiguration Then, you can change them. The following commands will disable all client-side caching (including Oplocks): Set-SmbClientConfiguration -OplocksDisabled 1 Set-SmbClientConfiguration -UseOpportunisticLocking 0 Set-SmbClientConfiguration -DirectoryCacheLifetime 0 Set-SmbClientConfiguration -FileNotFoundCacheLifetime 0 Set-SmbClientConfiguration -FileInfoCacheLifetime 0   You can then inspect these settings with this: Get-SmbClientConfiguration Note that the changes will take effect the next time the client establishes a new connection to the file share. Restarting the Esri server software would cause that to happen right away. Once you have changed your settings, return to the Esri logs to see if the signature, the error message information and frequency for the same workload, has been addressed. If it has, you can celebrate. If it is not, then what? Is the problem something else? Yes—if you are at this point, the problem is something else. You have disproven all reasonable hypotheses thus far. You are left with a diagnosis by exclusion. That diagnosis is that the file share solution (whether by design or not) is not providing immediate consistency.  In this circumstance, it can be useful to have corroborating evidence. A very good way to do that is to compare with a different file share solution. While a file share from a Windows virtual machine may not be a solution that you or your organization want to adopt permanently, it can be very useful to an investigation. Esri does not have empirical evidence of file shares from a single Windows machine producing immediate consistency problems. There is no strong theoretical basis for it. And, if you turn off Oplocks as described above, you neutralize the weak theoretical bases also. The Windows machine file share would need to be configured for the same SMB parameters as the original file share.  PowerShell’s Set-SmbServerConfiguration command can be used to match most parameters that the file share administration team would indicate from their solution. In any event, if you point your Esri server(s) at the Windows machine file share and the problem signature disappears, your diagnosis by exclusion has been corroborated. From there, the file share solution team can decide whether they want to try to match the immediate consistency characteristic or indicate that they do not want to provide such a service. So, you either have a solution or an answer.  In conclusion Investigating a suspected file share problem is one of the more challenging troubleshooting endeavors in the ArcGIS Enterprise administration space. This article has not hoped to make you, the individual reader, a successful solo practitioner in this space; rather, the objective is to give you some foundational information and a process. If you carefully execute this process, you can bring in many different specialists to help you reach the solution. In addition to involving the related domain experts from your own organization, you can bring Esri Technical Support and/or Professional Services to participate. With time, your careful execution and quality team participation can correctly diagnose problems in this space. Appendix: Log messages that are correlated with file share issues The information below is a partial list of error codes and messages that may indicate a file share issue in your ArcGIS Enterprise system. Warning and severe log messages At the default log level, the following codes and messages usually indicate a problem with a file share (in a multiple-machine site). It is often useful to look at the messages immediately before and after. When you do that, you want to use the machine, process, and thread fields to recognize related messages. Since there are many machines, processes, and threads, the immediately neighboring messages, by time, may be from some other process. Enterprise Component Level Code Message Notes Server WARNING 7721 “failed to write heartbeat”   Server WARNING 7712 “An error was encountered while synchronizing with the config store”   Server SEVERE 6561 “Failed to return all folder configurations”   Server SEVERE 9000 “Internal Server Error: “Service <name> not found” Note that a request for a service that does not currently exist will generate a message like this also. For this to indicate a problem with a file share, the service must actually exist in the site. Server SEVERE 6605 “Failed to return all services configurations in the folder … (The system cannot find the file specified)”   Server SEVERE 6652 “Unable to read the service … from the configuration store … (The system cannot find the file specified)”   Server SEVERE 6566 “Failed to retrieve the status of the service … (The system cannot find the file specified)”   Server SEVERE 9015 “Error getting list of services. … (The system cannot find the file specified)”   Server SEVERE 6615 “Unable to retrieve 'Permissions' resource information. …” This message is not exclusive to file share issues, but it can be associated with file share issues. Enterprise portal SEVERE 218037 “The Portal site has been initialized and configured but is currently not accessible because the content directory is not available …”     As should be evident from this table, most of the messages mention something about a file or directory that cannot be found. Debug level At the DEBUG logging level, there are more messages available that can provide more evidence. Thus, if you are looking for a potential file share problem, it may be useful to turn on the DEBUG log level for a period of time. These debug-level messages would mostly be useful to support (or not) hypotheses formed from other log levels or sources and add potentially relevant detail. By themselves, they would not be an actionable indication of a file share problem; there are other plausible interpretations. SITE_IN_USE=ArcGIS Server Site is currently being configured by another administrative operation. Please try again later. - (6850) MACHINE_BEING_CONFIGURED=Server machine ''{0}'' is currently being configured by another administrative operation. Please try again later. - (6851) CONFIG_STORE_BEING_ACCESSED=Another administrative operation is currently accessing the store. Please try again later. - (6852) CLUSTER_BEING_CONFIGURED=Cluster ''{0}'' is currently being configured by another administrative operation. Please try again later. - (6853) CLUSTER_MACHINE_BEING_CONFIGURED=Cluster ''{0}'' - Machine ''{1}'' is currently being configured by another administrative operation. Please try again later. - (7506) CLUSTER_RESOURCE_BEING_CONFIGURED=Cluster ''{0}'' resource is currently being configured by another administrative operation. Please try again later. - (7507) METRIC_BEING_CONFIGURED=Metric ''{0}'' is currently being configured by another administrative operation. Please try again later. - (6614) SERVICE_BEING_CONFIGURED=Service ''{0}''.''{1}'' in folder ''{2}'' is currently being configured by another administrative operation. Please try again later. - (6854) SERVICE_CONFIG_BEING_WRITTEN=Another administrative operation is currently writing the service configuration. Please try again later. - (6859) FOLDER_IN_USE=folder ''{0}'' is currently used by another administrative operation. Please try again later. - (6877) Some of these messages may also appear on DEBUG code 9999.  ... View more

A brief presentation describing the consulting service offering called "GIS Health Check" ... View more