Implementing ArcGIS Blog

cancel
Showing results for 
Search instead for 
Did you mean: 

Latest Activity

(149 Posts)
New Contributor III

In my previous article, Architecture is a Funny Word, I introduced Esri’s Global Architecture Team. I talked a bit about the work we do (and don’t do) and how we can support you in your efforts to design, build, and implement a GIS that makes a real difference at your organization. You might want to check that blog out first, since it sets up some of the context for this one. However, if you think context is dull and overrated, then by all means carry on, dear reader.

While everyone uses both sides of their brain, it is said that ‘right-brained’ people tend to be creative, imaginative and innovative thinkers. They like to employ creative concepts to understand complex ideas, are often focused more on the ‘big picture’ rather than specific details. ‘Left-brained’ people are said to be detail-oriented, analytical, and logical thinkers. They like to approach problems methodically and with concrete evidence and facts, and know that people don’t trip over mountains, but over molehills.

It seems to me that GIS practitioners are a diverse mix of right and left-brainers. Which makes sense, since GIS is as much about analyzing data and identifying patterns as it is about communicating concepts and ideas with beautiful, compelling visuals. So, I figured it could be valuable (and fun) to try and explain architecture in a way that would appeal to a larger group of thinkers.

The Right Brain
For my right-brained readers, I offer you an architecture analogy: Architecture is often compared to the architecture of the physical world of buildings, bridges, or even city planning. This analogy illustrates some key facets of the discipline- describing ‘what should be’ in terms of design, and how it should be adjusted going forward. However, this analogy only describes some of what architects do, and sometimes the ‘how’ and ‘why’ are more interesting, especially to you right-brainers, than the ‘what’. So why don’t we take a new analogy for a spin?

The fundamental idea of “yin-yang” is that seemingly opposite or contrary forces are actually complementary and interrelated. If you’ve ever sat in a room with technologists and business people trying to solve a problem together, you might see how those 'seemingly opposite forces' could represent business and technology. Technologists often feel separated from the business objectives of their organization, and business people sometimes see technology as a frustrating, separate field from their work. Of course, these two practices are completely interrelated and dependent on each other: the business dictates the use of technology, and technology guides and enables what’s possible for the business to achieve.

One of the key tenets of an architect’s duty is to make this relationship strong and clear by aligning the capabilities of technology to the needs of the business. In other words, we strive to highlight and strengthen the interconnectedness of these sometimes seemingly opposing forces. We do this by starting with a vision everyone can agree on and collectively work toward. Almost as importantly, we also need to make that alignment easier to see and understand, especially for leadership and other key stakeholders. This means communicating creatively, often with visuals, to explain complex ideas. Anyone who has had to act as the bridge between business and technology knows that bridge’s foundation relies more heavily on the art of building relationships than the science of technicalities.

The Left Brain

Alright, left-brained readers, it’s your time to shine. Let’s get serious about the methodical, concrete work architects on the Global Architecture team can do with you. Our work revolves around three key questions:

  1. Where is your organization today?
  2. Where do you need to be tomorrow?
  3. What’s the best path to get there?

Notice how I didn’t say:

  1. What does your GIS do today?
  2. What do you want to do with GIS tomorrow?
  3. Here’s what you need to buy or install to do it

Rather, we take a business-first approach to ensure that your investment in GIS is directly impacting outcomes for your organization and its stakeholders. We start by understanding the business challenges impeding your organization’s goals, and design new or augmented business workflows that resolve those challenges. Then, we identify what technology components, information, data, and skills are needed to enable those workflows. This process draws clear connections between technology solutions and business outcomes for your executive sponsors (or potential executive sponsors).

This work can’t be done in isolation. An architect documents perspectives from both the business and technology side before defining how to meet their needs in an efficient, sustainable and adaptable way. This means that architects speak the languages of both technologists and business managers. One way we do this is with diagrams. Kind of how GIS practitioners communicate with maps.

While technical architects communicate with system diagrams, my team and I communicate with conceptual and logical architecture diagrams. These might take the shape of a matrix illustrating where technology is not being effectively leveraged in your organization, a flow diagram outlining a new or augmented business workflow, or a roadmap that details how to get from where you are now to a place where you can overcome the challenges impeding your goals.

Let’s Connect

As Esri’s User Conference is just around the corner, I suspect many of you are getting excited to talk about the future for GIS at your organization. The Global Architecture Team is excited to work with you. Come chat with us. If you have any questions or would like to talk about any of these ideas further, email me at sscher@esri.com. You can also reach my team at global_archiecture@esri.com.

“Nothing is as dangerous in architecture as dealing with separated problems” – Alvar Aalto

more
6 0 272
New Contributor III

Like many of my colleagues on the Global Architecture Team at Esri, I spend a lot of time explaining who we are and what we do, and I think I know why. architecture is a funny word. It means a lot of different things to a lot of different people, which makes sense because depending on your domain, your experience with architects will differ greatly.

You might think of data architects, who design how data is collected, stored and arranged. Perhaps you’re more familiar with application architects, who are concerned with how applications are used and how they work together. Many people think of system architecture, and quickly find themselves in a cold sweat after picturing a horrifyingly complex system diagram. All this preamble to say: The Global Architecture Team doesn’t really do any of that.

What We Do

Our work is more closely aligned to the discipline of enterprise architecture. The role of the enterprise architect is to align the perspectives of business, applications, data, and technology to the organization’s overall strategy, while making sure the design follows organizational governance policies and standards. Now, we obviously we work at Esri, so we aren’t going to architect your entire business-technology landscape (sorry). Rather, we use enterprise architecture principles and frameworks and apply them specifically to the domain of GIS. One of those principles you’ll hear us talk about a lot is to take a ‘top-down, business first approach’.

This means starting with a vision for how GIS needs to support your business. This isn’t a list of maps or apps, and it definitely isn’t a data catalog. Rather, it’s a statement of what your organization will ‘look’ like when it has implemented and is operationalizing GIS effectively. Then we need to understand what business processes need to be designed or augmented with GIS. More simply, we figure out how the work your staff does needs to change to achieve the vision. Only once we have this level of understanding do we start aligning technology components, information products and data needed to enable those new or enhanced processes. Ultimately, we work with you to design the ‘why’ and then the ‘who’, ‘what’, ‘where’, and ‘how’ of GIS in your organization. My colleagues on the Global Architecture team and I are here to work with you to design the most impactful GIS that’s feasible in terms of sustainability, cost, and potential risk.

Let’s Connect

As Esri’s (first-ever virtual) Users Conference(esriuc2020) rapidly approaches, we know many of our customers are eager to connect and learn how they can do more. The Global Architecture team has supported organizations large and small, and in nearly every industry in their efforts to transform the way their organization works with geography and GIS. If you found this blog interesting and are looking for more on how our team can help amplify your success, you can reach me at sscher@esri.com or my team at global_architecture@esri.com.


“An idealist believes the short run doesn’t count. A cynic believes the long run doesn’t matter. A realist believes that what is done or left undone in the short run determines the long run.” - Sydney Harris

more
4 0 224
Esri Contributor

Introduction

Organizations often require a certain level of system uptime for their ArcGIS Enterprise deployments, such as 99 percent of the time or higher. For these organizations, implementing a strategy to ensure high availability is crucial.

High Availability (HA), while related to Disaster Recovery (DR), is a separate concept. Generally, HA is focused on avoiding downtime for service delivery, whereas DR is focused on retaining the data and resources needed to restore a system to a previous acceptable state after a disaster.

This post will focus on best practices to configuring local load balancer (Local Traffic Manager, LTM) in a single location for high availability, and will not include considerations to configuring Global Traffic Manager (GTM) for automatic failover between different locations.

To achieve high availability, you must reduce single points of failure through duplication and load balancing.

Load balancers act as a reverse proxy and distribute traffic to back-end servers. Third-party load balancer is required in a highly available ArcGIS Enterprise deployment to improve the capacity and reliability of the software. They handle client traffic to your portal and server sites, as well as internal traffic between the software components.

ArcGIS Web Adaptor

Though ArcGIS Web Adaptor is considered a load balancer, it’s inadequate to serve as the lone load balancer in a highly available deployment, since ArcGIS Web Adaptor also requires redundancy to achieve high availability.

ArcGIS Web Adaptor is an optional component, as load balancers can forward requests directly to your portal and server sites, yet it is a recommended component. The advantages of using web adaptors are:

  • It provides an easy way to configure a single URL for the system
  • It allows you to choose context names for the different system components, e.g. portal, server, mapping, etc.
  • It is integrated natively with other ArcGIS Enterprise software components, the portal and server sites, and will automatically handle health checks and configuration tasks, e.g. adding new machine to a server site

Web Context URL

The web context URL is the public URL for the portal. Since any item in portal has a URL - file, layer, map, and app, the portal's WebContextURL property helps it construct the correct URLs on all resources it sends to the end user.

External Access and DNS

ArcGIS Enterprise portal supports only one DNS for public portal URL (the web context URL), and currently there is no supported way to change the web context URL without redoing administrative tasks, e.g. federating server sites with your portal. If your ArcGIS Enterprise requires external access, e.g. to allow access to mobile users, contractors, partners, or agencies, without VPN, or if you anticipate that you will need to allow external access in the future, you must use an externally resolvable DNS name for the portal’s web context URL, e.g. https://gis.company.com/portal.

To secure external access to ArcGIS Enterprise, it is common to host a load balancer in a DMZ, and implement a Split Domain Name System (Split DNS), i.e. internal access to ArcGIS Enterprise DNS (e.g. gis.company.com) will be resolved to an internal load balancer IP, so internal users will stay behind the firewall, and external access to ArcGIS Enterprise DNS will be resolved to an external (DMZ) load balancer IP.

Load balancing configuration with external access

URLs used in federation

Several different URLs are used in a highly available ArcGIS Enterprise deployment.

Services URL

This is the URL used by users and client applications to access ArcGIS Server sites. It’s the URL for the load balancer that handles ArcGIS Server traffic and passes requests either to the server site’s Web Adaptor or directly to the server machines.

Administrative URL

This URL is used by administrators, and internally by the portal, to access an ArcGIS Server site when performing administrative operations. This URL is also used for publishing GIS services with reference to a registered data store, e.g. SQL Server enterprise geodatabase, to a federated server site. This must direct to a load balancer; if the administrative URL points to a single machine in the server site and that machine is offline, federation will not work. This can be the same URL as the services URL or can be a second load balancer (VIP) for each federated server site admin URL via port 6443. Configuring a dedicated VIP for each federated server site admin URL via port 6443 will require to open this port for administrators and publishers, and you can disable administrative access through the web adaptor, thus providing additional security controls for the organization.

I recommend using the same URL as the services URL since it simplifies configuration. ArcGIS Server administrative access will be controlled by ArcGIS Enterprise authentication and user roles, similarly to administrative access to portal, e.g. ArcGIS Portal Directory (portaladmin) and Organization Settings. To use web adaptor URL for administrative URL, you must enable administrative access in the server web adaptor.

Private portal URL

This is an internal URL used by your server sites to communicate with the portal. This must also direct to a load balancer and should be defined prior to federating. If you federate your server sites prior to setting the privatePortalURL, follow step 8 and 9 in the topic Configure an existing deployment for high availability to update the URL within your deployment. Similar to the administrative URL, this can be the same as the public URL for the portal (portal’s web context URL), or it can be a second load balancer (VIP) via port 7443.

For configuration simplicity, I would recommend using the public URL for the portal for the private portal URL. If you choose to use a dedicated load balancer VIP via port 7443 for private portal URL, you should configure the load balancer to check the health of the portal machines.

Load Balancer Configuration

Health Check Settings 

The most important capability to use is a Health Check. As described in portal’s health check documentation:

“The health check reports if the responding Portal for ArcGIS machine is able to receive and process requests. For example, before creating the portal, the health check URL reports the site is unavailable because it can't take requests at that time.”

ArcGIS Enterprise portal and server have health checks.

When you use ArcGIS Web Adaptors, the web adaptors will take care of performing the health checks against the portal and the servers. In this case you can configure the load balancer with basic TCP/443 health check or a static page health check, with standard health check settings for timeout, failure trigger, polling interval, and healthy threshold.

If you configure the load balancer to access portal and/or the servers directly, e.g. if you don’t include web adaptors in your architecture, or if you use dedicated load balancer URL for private portal URL (port 7443) or federated server administrative URL (port 6443), you should configure the load balancer to check the health of the portal and server machines.

There are some important considerations about health check settings on the load balancer. Most organizations using a load balancer use a static page as their health check (e.g. index.html) to determine whether the web server is healthy. This is a static file that only requires a disk fetch.  Also, most web servers tend to bottleneck with I/O rather than CPU. 

However, Esri’s ArcGIS Server is different in that our health check requires a very small amount of CPU because our check is more than just a disk fetch, as the software needs to determine whether certain processes are functional.

With health checks there is a timeout value on the poll. Most load balancer administrators set the timeout value very low because disk fetches are usually very fast (though they often leave some buffer for poor network latency).

When using a low timeout value with ArcGIS Server and when using a multi-machine setup, there is a chance one of the machines may exceed the low timeout value and a healthy machine will be removed.

Esri recommends a higher timeout value, ideally at least 5 seconds. Depends on the system, you might need to increase this value even more. You should monitor your environment and adjust this value accordingly. This may seem like a high number to Network Administrators since a health check on a simple page usually takes less than 10ms normally (plus network latency).

However, it is critical for the load balancers to distinguish between when a machine is just slow vs. when a machine is truly dead and non-responsive.  If portal or server is truly down and not listening on a port at all, most load balancers will detect this even sooner than 5 seconds, so this timeout does not impact "normal" failures where a machine goes away.

The second consideration is the failure trigger - how many times does the poll need to fail before removing the machine from the load balancer. 

As a rule, Network Administrators set the failure trigger to be more than 1 failure because they do not want a single network glitch to take down the system.  As noted above, a small spike in CPU usage (which is common on ArcGIS Server systems) can cause a timeout, and it is not desirable to bring down a machine because of a single CPU spike on a single machine. 

Esri has done significant internal testing with load balancers and found that a setting of 5 failures dramatically reduced the number of false positives while still detecting true outages.  We found that a value of 3 was still highly susceptible to false positives.

The third consideration is the polling interval.  Esri has found that 30 second polling intervals, coupled with 5 failures was a sweet spot of detecting true failures and ignoring false positives. 

With this combination, the expected value (using a statistics term) or mean time to detection is 1 minute and 15 seconds of downtime before detection, with a worse case of 2 minutes and 30 seconds.  It is possible to aim for a lower mean time to detection, but the trade-off is receiving false positives.

If the lower mean time to detection is preferred, it may be needed to increase capacity so there are sufficient resources to suffer a true failure on a machine and a false positive without overloading the remaining machines.

The final setting regarding health checks is the healthy threshold for the load balancer to start sending requests again.  Esri does not have a recommendation for this and we have not observed many differences in this number, but we typically see 3 healthy consecutive polls before rejoining.

Throttling 

Throttling settings are worth considering.  ArcGIS Server is CPU-bound which means most of the request time is spent using the CPU rather than waiting for I/O. 

This means if there are 8 cores, ArcGIS Server can handle a little more than 8 simultaneous requests from a practical standpoint. If ArcGIS Server is busy, it starts queueing requests until there are hundreds of requests in line, and after that threshold it refuses connections. When there is a long backlog of requests it will result in a long wait time, but eventually the request gets processed.

ArcGIS Server has settings to control this behavior, so there are less of these abandoned requests being worked on, but it is also a best practice to control this at the load balancer level via its ability to throttle. 

Esri does not have a numerical recommendation because it depends a lot on the specific architecture and the types of requests coming in, but it is typical to throttle at a significant lower value than Network Admins would typically do for a Web Server.

This is beyond the control of the Network Administrator, but the client applications should be written to handle a throttling event and do re-tries in increasing time intervals (e.g. first time you can immediately re-try, second time you wait a second, third re-try wait 5 seconds, etc.).

Sticky Sessions 

Esri does not recommend sticky sessions except in very rare circumstances. Sticky sessions can theoretically overload a machine.  Esri has conducted load tests using sticky sessions to see if we found results that would overload our GIS Server, and this did not occur. We have also not received any customer complaints. That said, since our software is stateless we do not see the value in using sticky sessions. 

Layer 4 vs. Layer 7 

The final setting to mention is whether the load balancer does a "level 7" approach or a "layer 4" approach.  This can be a debate among Network Administrators, but below is a concise summary of the situation describing the differences and advantages of each.

A layer 7 load balancer understands http and https, it therefore decrypts https content and then re-encrypts it.  Because it understands http and https it can cache content and save requests going to the backend server. 

A layer 4 load balancer views all the traffic as TCP packets and has no idea what the packets mean; they could be for ftp, https, smtp, but this does not matter to a layer 4 load balancer.  As a result, it doesn't need to understand the http payload and can be faster.

ArcGIS Server can work with either approach and there is no recommendation on this setting for Network Admins, but there are some pieces of information that an Admin will want to be aware of. 

ArcGIS Server payloads can be much larger than HTML pages, CSS, and JS content (the actual amount would be useful but is often dependent on the data the customer uses).  This means there is more CPU load on a layer 7 load balancer decrypting and encrypting on a per-request basis. 

Also, since much of ArcGIS Server data is dynamic and frequently changing, by default cache headers prohibit caching on the client and load balancer. If the data doesn't change much and the customer wants to use a layer 7 load balancer, they can change and control these cache settings. 

Summary of Recommendations

Health Check

  • If you configure your load balancers with ArcGIS Web Adaptors, you can configure the load balancer with basic TCP/443 health check or a static page health check against the web servers
  • If you configure the load balancer to access portal and/or the servers directly (via ports 6443 and 7443), use HTTPS health check endpoint:

Throttling

  • Use a throttle setting at a significant lower value than typical web servers

Sticky Sessions

  • Do not use sticky sessions

Certificates

ArcGIS Enterprise components come pre-configured with a self-signed server certificates, which allows the software to be initially tested and to help you quickly verify that your installation was successful. However, in almost all cases, an organization should request a certificate from a trusted certificate authority (CA) and configure the software to use it. The certificate can be signed by a corporate (internal) or commercial CA. Commercial CA (known-CA) must be used for externally resolvable DNS, e.g. load balancer VIP DNS; internal domain certificates can be used for internal servers. 
For ArcGIS Enterprise system with externally resolvable DNS, if the load balancer’s SSL method is SSL-passthrough (the load balancer does not decrypt and re-encrypt https content), it does not require certificate, and a commercial CA certificate must be installed on the mapped servers (e.g. web servers where the web adaptors are installed). If the load balancer’s SSL method is SSL re-encryption, a commercial CA certificate must be installed on the load balancer.

more
14 3 2,988
Esri Contributor

Since the release of ArcGIS Enterprise 10.5.1, distributed collaboration has been a powerful way to connect and integrate your web GIS across a network of participants.  Distributed collaboration has made it possible to organize and share content between entities and organizations including departments, other governments, private businesses and more.

 

During the time since the release of distributed collaboration, the implementation of ArcGIS Enterprise “on premises” at customer sites has also grown.  And for good reason!  Deploying ArcGIS Enterprise at your site gives you complete control over your deployment while providing the core capabilities of organization-wide mapping, analysis, data management, sharing and collaboration.

 

While working with ArcGIS Enterprise customers I have discovered a common misconception regarding the ability of an internal-only ArcGIS Enterprise to collaborate with ArcGIS Online.  A common deployment pattern for ArcGIS Enterprise is to not deploy the system as public-facing at your site.  The system is deployed for internal customers only.  The general public cannot connect to this system over the public Internet and items, services, and data are all behind the organization’s firewall.  The misconception is that this internal-only ArcGIS Enterprise cannot sync (send and receive) items such as web maps, apps, and feature layers with ArcGIS Online.  It can!

 

In such a scenario, even though ArcGIS Online is considered the host of the collaboration, all communication is initiated by ArcGIS Enterprise from behind your firewall.  To enable this, organization firewall rules must be configured to support outbound communication on port 443.  A few other technical details include:

 

The following URLs must be whitelisted/reachable by the ArcGIS Enterprise machine:

 

The Portal for ArcGIS service account that runs the Portal for ArcGIS service needs access to the Internet.

 

Once the above requirements are met you may proceed to set up your collaboration.

 

It might seem counterintuitive that your internal ArcGIS Enterprise can sync with ArcGIS Online and even receive items.  It is not only possible, but can be a very effective way to extend the reach and use of your web GIS. 

more
4 0 684
New Contributor II

Many organizations have installed and configured their deployments of ArcGIS Enterprise in the cloud and are required to incorporate disaster recovery plans to ensure the least amount of downtime in the event of a failure or catastrophe. There are various options out there for organizations to choose from to plan for a disaster recovery scenario. For the purpose of this blog post, I will walk through the steps for replicating a primary Enterprise deployment specifically in AWS to a warm, geographically redundant standby site using the WebGIS DR Utility.

Learn more on disaster recovery and replication in ArcGIS Enterprise

Much of this workflow has been covered extensively by my colleagues for on-premise deployments in another blog post Migrate to a new machine in ArcGIS Enterprise using the WebGIS DR tool. The general workflow for our scenario is very similar to the on-premise deployments with some added steps utilizing AWS services* – Application Load Balancer (ALB), Route 53, and S3 buckets – as well as the exclusion of using etc\host file entries in EC2 instances. Please read the blog post above to understand the overall workflow and prerequisites before proceeding with this workflow, as it will contain information not covered in this post.

* I will assume readers will have some prior experience with AWS and its services mentioned above.

Scenario

Let’s say we want to have a replicated and geographically redundant multi-machine deployment with the following components setup in AWS in the US East and US West regions: 

  • Portal for ArcGIS 
  • ArcGIS Hosting Server 
  • ArcGIS Data Store 
  • ArcGIS Geocode Server

Architectural design of base enterprise deployment with additional server

Route 53 Hosted Zones

Route 53 is a scalable Domain Name System (DNS) that utilizes a combination of public and private hosted zones, which are containers for records about how you want to route traffic for a specific domain. Public hosted zones are used to route traffic on the internet, while private hosted zones are used to route traffic within an Amazon VPC. The following workflow will be utilizing both types of hosted zones with overlapping namespaces. Therefore, in our scenario when we are logged into an EC2 instance in a VPC that is associated with a private hosted zone:

  • The Resolver will evaluate whether the name of the private hosted zone matches the domain name in the request. If there is no match, the Resolver will forward the request to a public DNS resolver.
  • If there is a match in the domain name of the request, the hosted zone is searched for a record that matches the domain name. If there is no record in the matching private hosted zone, the Resolver does not forward the request to a public DNS resolver, but will return a non-existent domain (NXDOMAIN) to the client.

That last bullet is very important! If you have other applications in the same VPC and you setup a private hosted zone, please ensure to add records for those applications so that those application remain completely operational.

Before Deployment

Before proceeding with deploying ArcGIS Enterprise, we need to complete the following tasks using the AWS services mentioned above in order to maintain a consistent DNS across the primary and standby sites:

 

  1. Create an ALB in both regions that will manage traffic for the ArcGIS Enterprise deployments. A few things to note: 
    1. Ensure to attach the appropriate certificate from the public domain registered in Route 53.
    2. A target group must be created when configuring a new load balancer. Create a target group for Portal at this time. It does not need to be registered to any targets with this target group.
  2. In Route 53 under your Public Hosted Zone:
    1. Create two identical Record Sets for each of the load balancers using CNAME types. One thing to note here: 
      1. Having multiple Record Sets using the Weighted Routing Policy is not a requirement; it is more a matter of preference. It is possible to have just one Record Set and replace the ALB value when the switch is needed.
    2. Set the time to live (TTL) to the recommended 60 seconds, which will "minimize the amount of time it takes for traffic to stop being routed to your failed endpoint."
    3. Set the Routing Policy to Weighted:
      1. Assign a weight of 100 to the primary site.
      2. Assign a weight of 0 to the standby site.                     Public record set in Route 52 with Weighted Routing PolicyPublic record set in Route 52 with Weighted Routing Policy
  3. Create a Private Hosted Zone for each region using the same Domain Name as the Public Hosted Zone in Route 53.
    1. Attach the private hosted zone to the VPC assigned to your ALB during time of creation.List of hosted zones in Route 53
  4. In Route 53 under each of the Private Hosted Zones:
    1. Create an identical Record Set to match the one created in the Public Hosted Zone.
    2. TTL can remain as the default value of 300 seconds.
    3. The Routing Policy can remain the default value of Simple.Private record set in Route 52 with Simple Routing Policy Private record set in Route 52 with Simple Routing Policy

The last two steps of this pre-deployment process are vital to the success of the WebGIS DR utility. Having the two Private Hosted Zones with identical Domain Names and Record Sets that match the Record Sets in the Public Hosted Zone ensures the ability to configure identical ArcGIS Enterprise deployments. Additionally, the deployments will remain in an operational state to perform consistent backups and restores on the appropriate systems without issue due to being on separate regions and VPCs.

ArcGIS Enterprise Deployment

It is finally time to deploy the components of our architecture to EC2 instances (repeat for each region):

  1. To setup the base deployment (Portal for ArcGIS, ArcGIS Hosting Server, and ArcGIS Data Store), follow steps 1-20 from our documentation on deploying Portal for ArcGIS on AWS, which utilizes Esri’s Amazon Machine Images (AMIs).
    1. Make sure to assign the EC2 instances to the same VPC and Security Group as the ALB.
  2. Repeat steps 11-19 in the linked documentation from the previous step to setup the additional ArcGIS Geocode Server.
  3. Create the two remaining target groups for the ArcGIS Hosting Server and Geocode Server (the Portal target group was created during the ALB creation in the pre-deployment steps).
  4. Register each instance with its appropriate target group.
  5. Configure the appropriate forwarding rules in the ALB to match the target groups. If everything is setup correctly, we should be able to successfully access portal through the DNS alias (Name property) created in the Record Sets.ArcGIS Portal homepage with arrow pointing to URL
  6. Follow steps 21-23 in the linked documentation above using the DNS alias from the Record Sets. For example, my portal’s system properties would have the following entries: Portal system properties
    1. And federating my hosting server (or any others) would look like so (the admin URL may vary depending on the level of administrative access setting placed on the Web Adaptor):ArcGIS Server federation in Portal organizational settings.
  7. Repeat step 22 for the ArcGIS Geocode Server.

We now have two identical and fully functional ArcGIS Enterprise deployments in each region. The deployment that has a Weighted Policy of 100 in the Public Hosted Zone will be accessible over the internet and act as the primary site, while the other deployment (with Weighted Policy of 0) in the Public Hosted Zone can only be accessible within its own VPC and act as the standby site.

Active and standby ArcGIS Enterprise architectural designs

Replication

Now that we have our primary and standby sites, we have to setup two replicating S3 buckets in the appropriate regions to have a fully replicated and geographically redundant deployment prepared for a disaster recovery scenario.

In Amazon S3, create two buckets with easily recognizable naming conventions, like the following:

Amazon S3 buckets

In the second step of the creation process for each bucket under Configure Options, check the box under Versioning. This is a requirement to enable Cross-Region Replication. Once the buckets have been created, select the bucket that will be utilized by the primary site and navigate to Replication under the Management tab and select Get Started. We can leave the defaults selected for all settings throughout this process. We just need to ensure we select our designated standby site bucket for the destination.

Configuring the replication rule in Amazon S3 bucket

Amazon S3 provides a selectable option (seen in the screenshot above) called S3 Replication Time Control, which will replicate 99.99% of new objects within 15 minutes. This may be a valuable option for organizations with large-scale ArcGIS Enterprise deployments with lots of content who need minimal down time. In my tests, I have found replication to be fast without that option selected, granted my backups ran around 5-6 GBs, which amounted to ~300 hosted services and a Geocode service (and its data) copied to ArcGIS Server.

Disaster Recovery

We are now able to create backups of the primary site and restore the standby site from said backups.

  1. Create a backup with the WebGIS DR tool of your primary site on the instance where Portal is installed. Make sure to point to the appropriate S3 bucket (should be the one in the same region as the primary site) in the properties file. The backup will be replicated to the S3 bucket in the other region.
  2. Restore the replicated backup with the DR tool on the standby instance where Portal is installed. Be mindful of the appropriate S3 bucket in the properties file.

This is the final architecture and workflow:

Full workflow/architecture outlined with S3 bucket replication

In the event of a disaster, we now have the ability to “failover” to our warm, standby deployment by just toggling the values of the Weighted Policies of the Record Sets in our Public Hosted Zone with downtime dependent upon the TTL of the Route 53 record set.Additionally, depending on the length of downtime for the original hot site, you may need to modify how your backups and restores are handled on the two environments, so that any new items created in the new hot site can be maintained when your original data center is restored.

* It should be heavily emphasized that this workflow with the WebGIS DR tool is not considered a highly available ArcGIS Enterprise deployment. The “failover” may be quick in this scenario, but the standby deployment will only have content available from the last WebGIS DR backup/restore. Please see our documentation on configuring a highly available ArcGIS Enterprise.

ArcGIS Enterprise failover workflow/architecture

Since both deployments are also identical, there should be no issues with services integrated into other business systems. Most importantly, this entire workflow – running WebGIS DR backups and restores, DNS toggling, as well as detecting who is acting as the primary site at any given moment – can be completely scripted and automated, ensuring that mission critical systems remain operational.

Learn more about preventing data loss and downtime with ArcGIS Enterprise.

more
12 6 2,566
Esri Contributor

At the Architecture Practice, we are getting a lot of questions about shared instance pools.  

Before 10.7.0, the solution for reducing the amount of RAM was to set low-utilization so that the minimum number of instances in that service’s dedicated pool to zero. By doing so, you allow ArcGIS Server to not run any ArcSOCs for the service if it hasn’t received any requests in a while. 

This “min-zero” solution eliminates the resource usage for services that are going unused. Because you can still set the maximum number of instances, you can accommodate services that receive infrequent bursts of traffic.  The next time the service gets a request, an ArcSOC powers up to handle it at the cost of the startup time of the ArcSOC.  Also, this service could then sit idle for a more extended period, consuming the started SOC until the service hits the set idle timeout.

At 10.7.0, Esri announced support for shared instance pools. Every ArcGIS Server Site now comes with a shared instance pool, containing four ArcSOC processes by default. This number can increase to accommodate more services. The shared instance pool utilizes all of the SOCs assigned to it, so you should only increase the pool as you need to.

Once a compatible map service has published to your ArcGIS Server Site, you can designate it to use the shared pool. Any service added to the shared instance pool will no longer have its dedicated pool; it will dip into the shared pool and use a SOC or two as needed. Once it’s done handling a request, that ArcSOC is free to be used by any other service in the shared pool.

The following restrictions limit what services can use the shared instance pool:

  • Only map services published from ArcGIS Pro can be configured to use the shared instance pool. Other service types, such as geoprocessing services, are not supported.
  • Only specific capabilities of map services—feature access, WFS, WMS, and KML—can be enabled. Turn off all other capabilities before continuing.
  • Services that have custom server object extensions (SOEs) or server object interceptors (SOIs) cannot use shared instances.
  • Services published from ArcMap cannot use shared instances.
  • Cached map services published from ArcGIS Pro that meet the above requirements can use shared instances.

For further information, please see:

Introducing shared instances in ArcGIS Server 10.7 

Configure service instance settings—ArcGIS Server Administration (Windows) | ArcGIS Enterprise 

more
3 0 493
Esri Contributor

Interest has been expressed within the Esri community with having Esri provide a set of icons for building presentations, particularly those involving conceptual IT architectures. Esri recently released an initial set of "Esri Presentation Icons" which can be used to develop conceptual architecture diagrams to provide a unified view of GIS deployments. Attached is the initial set of icons that are being provided for customer usage.

The intended use of this icon set is to enhance PowerPoint presentations by illustrating core GIS concepts. These are not intended to be used for schematic drawings or highly detailed workflows.

more
21 11 4,325
Esri Contributor

Introduction

ArcGIS Monitor is designed to help you analyze and optimize the health of your ArcGIS implementation throughout the life cycle of your enterprise GIS. ArcGIS Monitor maximizes your GIS investment by providing timely and insightful system metrics on the status, availability, usage, system performance, and resource usage of your enterprise GIS. Alerts and analysis tools provide system administrators with real-time notifications to facilitate rapid resolution when measurements are outside defined system thresholds. Reports with statistics can be used to visualize historical data and enhance communications among GIS, IT, business owners, and senior management.

The ArcGIS Monitor Server application allows you to configure and export reports for your collections as Microsoft Excel (.xlsx) files. The ArcGIS Monitor Excel Report provides overall, dashboard-like view of your monitored GIS deployment, all in a single Excel file with the ease to navigate, sort and filter the data in a simple way.

For information about configuring and running the tool, please refer to ArcGIS Monitor documentation.

The Report Summary provides a view of all configured categories, e.g. Web, ArcGIS, Infrastructure and Site, Counter Type and Name, e.g. Web Requests Response Time, ArcGIS Services Summary, etc. You can navigate from this page to view counter details page by clicking on the desired link under the Name column.

Glossary of Report Summary page indications:

■  Indicates to investigate high utilization/load.

Indicates to investigate sporadic utilization spikes.

●  Indicates low utilization.

Configure and export reports

When you configure how to export the report, it is important to filter the report time span so it will include only busy time days and hours, for example, if the system is used mainly during business hours you should exclude Saturday and Sunday in Set Working Days and choose only business hours (e.g. 9 AM to 5 PM) in Set Working Days. For the purpose of system design, peak time usage and utilization is much more important than total usage.

Information Objectives for System Design

The Esri system design practice focuses on planning the hardware, software, and network characteristics for the future state of systems based on new or changing requirements.

The current health of an existing system will not necessarily have a strong relationship to a future system that has different requirements.  However, depending on the design objectives, information about the current system can be relevant.

For example, in the case of a planned migration from an on-premises system to a cloud platform, it would be quite useful to describe the current system such that it can be faithfully rendered on a cloud platform.  Or, capacity requirements driving a design may be derived from current system state, e.g. current services inventory, current system throughput, current resources utilization, plus the anticipated services and user growth over a defined term, e.g. two years.

 

Machine Resources and Utilization

It can be useful for system design to understand the current machine resources that support the system.  For example, if you are migrating a system to a cloud platform, the number of processor cores that the system has on premises has some relevance to the number you might deploy on the cloud.

Machines

Clicking on the Infrastructure Summary link in the Report Overview will lead you to the Infrastructure Summary details page. The page will list all monitored machines, with the following details:

  • Logical cores count
  • Physical cores count
  • Processor type
  • Total RAM
  • Virtual memory

Machines Utilization

The characteristics of the machines, and the configuration of the instances, offers incomplete insight into the degree to which machines resources are utilized and what resources are truly needed for the current workload, as a baseline for your system design.

 

Statistics Fields in Machines Utilization

Field

Definition

Min

Minimum percent utilization

Avg 

Average percent utilization

P5, P25, P50, P75

The percentile grouping of resource utilization

P95

The ninety-fifth percentile. Ninety-five percent of the time resource utilization value is lower than this value

P99

The ninety-ninth percentile. Ninety-nine percent of the time resource utilization value is lower than this value

Max

Maximum percent utilization

  

CPU

Clicking on the Infrastructure CPU Utilization link in the Report Overview will lead you to the CPU utilization details page. The page will list all monitored machines, with CPU utilization statistics.

We're going to focus on the P95 percentile. As we learned above, P95 signifies the CPU utilization for the top 5% busiest time. When P95 CPU utilization exceeds 90% it suggests that the machine is overloaded. In this case you should plan how to reduce the load on the machine by distributing the load or by adding more resources. This page will also help you to identify candidate machines with high CPU utilization, even if it’s below 90%, that might require additional resources or load distribution due to the anticipated user growth in your system design.

Current machines CPU utilization can also help you in validating your capacity calculations by comparing capacity calculation results for current usage with actual CPU utilization statistics in order to validate your capacity models before calculating capacity for the anticipated user growth.

Memory

Clicking on the Infrastructure Memory Physical Utilization link in the Report Overview will lead you to the physical memory utilization details page. The page will list all monitored machines, with memory utilization statistics.

For ArcGIS Enterprise system with default services configuration we would usually expect to see small changes in memory utilization, with some exceptions, e.g. geoprocessing services, services configured with higher number of max instances, etc. As with CPU utilization, we're going to focus on the P95 percentile. When P95 memory utilization exceeds 80% it suggests that the machine requires more memory. In this case you should plan how to reduce memory pressure on the machine. There are different ways to do that depending on the machine role, for example:

  • Portal – add more memory to the machine
  • Hosting Server - add more memory to the machine or add more machines to the site
  • Federated Server – use shared instances for less used map/feature services, add more memory to the machine, add more machines to the site, distribute services between sites (workload separation)


This page will also help you to identify candidate machines with high memory utilization, even if it’s below 80%, that might require you to plan for memory pressure alleviation due to the anticipated growth in usage or in the number of services in your system design.

Disk

Disk Utilization can help you identify current machines with potentially slow I/O and if storage upgrades are required.

Disk Space can give you the baseline for disk size requirements for the machines (i.e. not including shared storage) in your system design and identify if disk size has to be increased on existing machines if available disk space is low.

 

Network

Network Utilization can give you the baseline of current network usage for your system design.

 

Process

I recommend configuring Process counters in ArcGIS Monitor to monitor ArcSOC processes in federated ArcGIS Server machines.

Infrastructure Process Count page provides number of total ArcSOC process running on the machine, i.e. the number of service instances. This will help to identify ArcGIS Server usage patterns – is number of service instances steady or volatile? Does the number of service instances during peak time exceed 200? If so, it can threaten the stability of the site, and action must be taken:


1. Tune services and reduce number max instances per service. ArcGIS Services Requests/sec and Instances information (details below) can help with tuning services with the right number of instances.


2. Configure less used map and feature services to use shared instances. ArcGIS Services Count and Requests/sec (details below) can help with identifying candidate services for shared instances configuration.


3. Configure Windows registry to allow more service instances (See this technical article for more information and specific steps: https://support.esri.com/technical-article/000001218)

Process count can also provide baseline for number of services in your system design, to prepare for anticipated growth in number of services and plan services configuration.

ArcGIS Services

It can be useful for system design to understand the current ArcGIS Server services inventory, usage and performance. 

 

ArcGIS Services Summary

ArcGIS Services Summary provides ArcGIS Server services inventory including services configuration, e.g. started/stopped, types of services, etc., as a baseline for services configuration in your system design.


ArcGIS Services Count and Requests per Second

ArcGIS Services Count and Requests per Second provides baseline of current system throughput for your system design, as well as ArcGIS Server services usage information, e.g. most used services, less used services and unused services, for designing services configuration and help tuning services.

ArcGIS Services Instances

ArcGIS Services Instances information is not important for system design but can help with tuning services, e.g. number of min and max service instances for federated services.

ArcGIS Services Response Time

ArcGIS Services Response Time information can be used for capacity planning in your system design, if you are creating custom workflows in the capacity planner.

This information can also be used for optimizing current system by identifying slow-performing services. In the example above, I’ve sorted P95 elapsed time from largest to smallest, and highlighted any elapsed time over 1/2 second in orange. These are the services and layers I'd focus on optimizing, getting the P95 value below 1/2 second if possible.

 Note: The contents presented above are recommendations that will typically improve performance for many scenarios. However, in some cases, these recommendations may not produce better performance results, in which case, additional performance testing and system configuration modifications may be needed.

 

I hope you find this helpful, do not hesitate to post your questions here: ArcGIS Architecture Series: Tools of an Architect

more
7 0 2,297
New Contributor III

In this entry, we will be looking at what a deployment looks like from the infrastructure as code (IaC) perspective with Terraform as well as the configuration management side with PowerShell DSC (Desired State Configuration). Both play important roles in automating ArcGIS Enterprise deployments, so let's jump in.

This deployment will follow a single machine model as described in the ArcGIS Enterprise documentation. It will consist of the following.

  • Portal for ArcGIS 10.7.1

  • ArcGIS Server 10.7.1 (Set as Hosting Server)

    • Services Directory is disabled

  • ArcGIS Data Store 10.7.1 (Relational Storage)

  • Two (2) Web Adaptors within IIS

    • Portal context: portal

    • Server context: hosted

    • Self-Signed certificate matching the public DNS

Additional Configurations

  • WebGISDR is configured to perform weekly full backups that are stored within Azure Blob Storage via Task Scheduler

  • Virtual machine is configured for nightly backups to an Azure Recovery Services Vault

  • RDP (3389) access is restricted via Network Security Group to the public IP of the box in which Terraform is ran from.

  • Internet access (80, 443) is configured for ArcGIS Enterprise via Network Security Group

  • Azure Anti-malware is configured for the virtual machine

The complete code and configurations can be found attached below. You will need to provide your own ArcGIS Enterprise licenses however.

Note:   This is post two (2) in a series on engineering with ArcGIS.

Infrastructure Deployment

If you are not already familiar with Terraform and how it can be used to efficiently handle the lifecycle of your infrastructure, I would recommend taking the time to read through the first entry in this series which can be found here. The Terraform code in that first entry will be used as the basis for the work that will be done in this posting.

As discussed in the first entry, Terraform is a tool designed to help manage the lifecycle of your infrastructure. Instead of rehashing the benefits of Terraform this time however, we will jump straight into the code and review what is being done. As mentioned above, the template from the first entry in this series is used again here with additional code added to perform specific actions needed for configuring ArcGIS Enterprise. Let's take a look at those additions.

These additions handle the creation of two blob containers that will be used for uploading deployment resources ("artifacts") and a empty container ("webgisdr") that will be used when configuring webgisdr backups along with the uploading of the license files, the PowerShell DSC archive and lastly, the web adaptor installer.

resource "azurerm_storage_container" "artifacts" {
name = "${var.deployInfo["projectName"]}${var.deployInfo["environment"]}-deployment"
resource_group_name = "${azurerm_resource_group.rg.name}"
storage_account_name = "${azurerm_storage_account.storage.name}"
container_access_type = "private"
}

resource "azurerm_storage_container" "webgisdr" {
name = "webgisdr"
resource_group_name = "${azurerm_resource_group.rg.name}"
storage_account_name = "${azurerm_storage_account.storage.name}"
container_access_type = "private"
}

resource "azurerm_storage_blob" "serverLicense" {
name = "${var.deployInfo["serverLicenseFileName"]}"
resource_group_name = "${azurerm_resource_group.rg.name}"
storage_account_name = "${azurerm_storage_account.storage.name}"
storage_container_name = "${azurerm_storage_container.artifacts.name}"
type = "block"
source = "./${var.deployInfo["serverLicenseFileName"]}"
}

resource "azurerm_storage_blob" "portalLicense" {
name = "${var.deployInfo["portalLicenseFileName"]}"
resource_group_name = "${azurerm_resource_group.rg.name}"
storage_account_name = "${azurerm_storage_account.storage.name}"
storage_container_name = "${azurerm_storage_container.artifacts.name}"
type = "block"
source = "./${var.deployInfo["portalLicenseFileName"]}"
}

resource "azurerm_storage_blob" "dscResources" {
name = "dsc.zip"
resource_group_name = "${azurerm_resource_group.rg.name}"
storage_account_name = "${azurerm_storage_account.storage.name}"
storage_container_name = "${azurerm_storage_container.artifacts.name}"
type = "block"
source = "./dsc.zip"
}

resource "azurerm_storage_blob" "webAdaptorInstaller" {
name = "${var.deployInfo["marketplaceImageVersion"]}-iiswebadaptor.exe"
resource_group_name = "${azurerm_resource_group.rg.name}"
storage_account_name = "${azurerm_storage_account.storage.name}"
storage_container_name = "${azurerm_storage_container.artifacts.name}"
type = "block"
source = "./${var.deployInfo["marketplaceImageVersion"]}-iiswebadaptor.exe"
}
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

This addition handles the generation of a short-lived SAS token from the storage account that is then used during the configuration management portion to actually grab the needed files from storage securely. In this situation, we could simplify the deployment by marking our containers as public and not requiring a token but that is not recommended.

data "azurerm_storage_account_sas" "token" {
connection_string = "${azurerm_storage_account.storage.primary_connection_string}"
https_only = true
start = "${timestamp()}"
expiry = "${timeadd(timestamp(), "5h")}"

resource_types {
service = false
container = false
object = true
}

services {
blob = true
queue = false
table = false
file = false
}

permissions {
read = true
write = true
delete = true
list = true
add = true
create = true
update = true
process = true
}
}
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

The final change is the addition of an extension to the virtual machine that will handle the configuration management task using PowerShell DSC. Instead of reviewing this in-depth here, just know that the data that is getting passed under the settings and protected_settings json will be passed to PowerShell DSC as parameters for use as needed by the configuration file.

resource "azurerm_virtual_machine_extension" "arcgisEnterprise-dsc" {
name = "dsc"
location = "${azurerm_resource_group.rg.location}"
resource_group_name = "${azurerm_resource_group.rg.name}"
virtual_machine_name = "${element(azurerm_virtual_machine.arcgisEnterprise.*.name, count.index)}"
publisher = "Microsoft.Powershell"
type = "DSC"
type_handler_version = "2.9"
auto_upgrade_minor_version = true
count = "${var.arcgisEnterpriseSpecs["count"]}"

settings = <<SETTINGS
{
"configuration": {
"url": "${azurerm_storage_blob.dscResources.url}${data.azurerm_storage_account_sas.token.sas}",
"function": "enterprise",
"script": "enterprise.ps1"
},
"configurationArguments": {
"webAdaptorUrl": "${azurerm_storage_blob.webAdaptorInstaller.url}${data.azurerm_storage_account_sas.token.sas}",
"serverLicenseUrl": "${azurerm_storage_blob.serverLicense.url}${data.azurerm_storage_account_sas.token.sas}",
"portalLicenseUrl": "${azurerm_storage_blob.portalLicense.url}${data.azurerm_storage_account_sas.token.sas}",
"externalDNS": "${azurerm_public_ip.arcgisEnterprise.fqdn}",
"arcgisVersion" : "${var.deployInfo["marketplaceImageVersion"]}",
"BlobStorageAccountName": "${azurerm_storage_account.storage.name}",
"BlobContainerName": "${azurerm_storage_container.webgisdr.name}",
"BlobStorageKey": "${azurerm_storage_account.storage.primary_access_key}"
}
}
SETTINGS
protected_settings = <<PROTECTED_SETTINGS
{
"configurationArguments": {
"serviceAccountCredential": {
"username": "${var.deployInfo["serviceAccountUsername"]}",
"password": "${var.deployInfo["serviceAccountPassword"]}"
},
"arcgisAdminCredential": {
"username": "${var.deployInfo["arcgisAdminUsername"]}",
"password": "${var.deployInfo["arcgisAdminPassword"]}"
}
}
}
PROTECTED_SETTINGS
}‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Configuration Management

As was touched on above, we are utilizing PowerShell DSC (Desired State Configuration) to handle the configuration of ArcGIS Enterprise as well as a few other tasks on the instance. To simplify things, I have included v2.1 of the ArcGIS module within the archive but the public repo can be found here. The ArcGIS module provides a means with which to interact with ArcGIS Enterprise in a controlled manner by provided various "resources" that perform specific tasks. One of the major benefits of PowerShell DSC is that it is idempotent. This means that we can continually run our configuration and nothing will be modified if the system matches our code. This provides administrators the ability to push changes and updates without altering existing resources as well as detecting configuration drift over time. 

To highlight the use of one of these resources, let's take a quick look at the ArcGIS_Portal resource which is designed to  configure a new site without having to manually do so through the typical browser based workflow. In this deployment, our ArcGIS_Portal resource looks exactly like the below code. The resource specifies the parameters that we must be provided to successfully configure the portal site and will error out if all required parameters are not provided. 

ArcGIS_Portal arcgisPortal {
PortalEndPoint = (Get-FQDN $env:COMPUTERNAME)
PortalContext = 'portal'
ExternalDNSName = $externalDNS
Ensure = 'Present'
PortalAdministrator = $arcgisAdminCredential
AdminEMail = 'example@esri.com'
AdminSecurityQuestionIndex = '12'
AdminSecurityAnswer = 'none'
ContentDirectoryLocation = $portalContentLocation
LicenseFilePath = (Join-Path $(Get-Location).Path (Get-FileNameFromUrl $portalLicenseUrl))
DependsOn = $Depends
}
$Depends += '[ArcGIS_Portal]arcgisPortal'‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Because of the scope of what is being done within the configuration script here, we will not be doing a deep dive. This will come in a later article.

Putting it together

With the changes to the Terraform template as well as a very high level overview of PowerShell DSC and its purpose, we can deploy the environment using the same commands mentioned in the first entry in the series. Within the terminal of your choosing, navigate into the extracted archive that contains your licenses, template and DSC archive, and start by initializing Terraform with the following command.

terraform init

Next, you can run the following to start the deployment process. Keep in mind, we are not only deploying the infrastructure but configuring ArcGIS Enterprise so the time for completion will vary. When it completes, it will output the public facing url to access your ArcGIS Enterprise portal.

terraform apply


Summary

As you should quickly be able to see, removing the manual configuration aspects of software as well as the deployment of infrastructure, a large portion of problems can be mitigated by moving toward IaC and Configuration Management. There are many solutions out there to handle both aspects and these are just two options. Explore what works for you and start moving toward a more DevOps centric approach.

I hope you find this helpful, do not hesitate to post your questions here: Engineering ArcGIS Series: Tools of an Engineer 

 

Note: The contents presented above are examples and should be reviewed and modified as needed for each specific environment.

more
3 0 1,905
Esri Contributor

It’s a long-time dilemma for many organizations whether to devote resources to custom software application development or go for the commercial off-the-shelf (COTS) solution? In this blog post, we will highlight some main fundamental strategies that will help any GIS department choose what will best fit its user needs.


In short, the answer to this question also links us back to what the requirements are and what current infrastructure is. An application implementation strategy is an approach to delivering capabilities that meet your business needs with technology.

Gathering Requirements is crucial for the success of your application whether it's COTS or custom. You should closely work with the sponsor, stakeholders, users, and IT focusing on the requirements, not the solution. Types of the requirements you will have to gather are:

  • Business
High-level vision statements (ex. Share information with the public)
  • Functional
What the application should do

(from a user perspective)

  • Non-functional
How the application does it (usability, security, performance, etc.)

Following the requirements, there are many other factors to consider when deciding the best way to deliver new capabilities through apps. These factors include resourcing, initial development effort, ongoing app maintenance, user training, and technical support. In addition, users now expect frequent updates to their apps, which increases demand for resources to develop and maintain custom apps. As a result, it’s best to select the approach that delivers the capabilities you need with the least cost and effort. An ideal strategy will minimize cost and optimize the use of development resources.

By applying a “configure first” philosophy that prioritizes commercial off-the-shelf (COTS) apps and least-effort design patterns, you can reduce the cost and effort needed to deploy and maintain applications for your users. Organizations that adopt a configure-first philosophy start by configuring COTS apps, then extend and customize apps only when needed. Using this least-effort approach in your application implementation strategy lets you deliver capabilities faster and reserve your development resources for more complex tasks.

Depending on your specific requirements, you can:

  • Configure COTS apps to meet your business needs. ArcGIS provides many configurable COTS apps that support key workflows out of the box. Using COTS apps requires the least effort and the lowest ongoing cost. ArcGIS COTS Apps (Web App Builder, Story Maps, Operational Dashboards, Collector, GeoForm, and Survey 123)

  • Extend existing apps, either by modifying templates or by creating widgets for COTS apps. Esri offers app templates at solutions.arcgis.com and github.com/esri that provide focused solutions for specific problems; you can modify the source code for these templates to add discrete capabilities. In addition, several ArcGIS COTS apps use modular frameworks that let you create custom widgets and plug them into the apps. Extending existing apps lets you develop only the additional functionality you need, saving money and effort.

  • Customize apps using ArcGIS APIs and SDKs. These APIs and SDKs provide objects like the Identity Manager to manage credentials within custom apps that expose parts of the ArcGIS platform (such as secure web maps). Because you don’t have to code those parts yourself, you can build business-focused apps to take advantage of ArcGIS COTS capabilities, reducing the overhead for app development and maintenance. Check out developers.esri.com for more information on the wide selection of APIs and SDKs available.

In conclusion, to establish an effective application implementation strategy for your organization, look deep into your requirements and available resources, and try to follow these 3 simple guidelines:

  1. Adopt a configure-first philosophy, configuring COTS apps when possible to deliver the capabilities you need.
  2. If you have a requirement that cannot be met with configuration alone, extend existing apps with discrete capabilities and widgets.
  3. When you need capabilities that you can’t provide by configuring and extending existing apps, customize apps using ArcGIS APIs and SDKs.

more
4 0 1,084