Implementing ArcGIS Blog

cancel
Showing results for 
Search instead for 
Did you mean: 

Latest Activity

(149 Posts)
New Contributor II

Many organizations have installed and configured their deployments of ArcGIS Enterprise in the cloud and are required to incorporate disaster recovery plans to ensure the least amount of downtime in the event of a failure or catastrophe. There are various options out there for organizations to choose from to plan for a disaster recovery scenario. For the purpose of this blog post, I will walk through the steps for replicating a primary Enterprise deployment specifically in AWS to a warm, geographically redundant standby site using the WebGIS DR Utility.

Learn more on disaster recovery and replication in ArcGIS Enterprise

Much of this workflow has been covered extensively by my colleagues for on-premise deployments in another blog post Migrate to a new machine in ArcGIS Enterprise using the WebGIS DR tool. The general workflow for our scenario is very similar to the on-premise deployments with some added steps utilizing AWS services* – Application Load Balancer (ALB), Route 53, and S3 buckets – as well as the exclusion of using etc\host file entries in EC2 instances. Please read the blog post above to understand the overall workflow and prerequisites before proceeding with this workflow, as it will contain information not covered in this post.

* I will assume readers will have some prior experience with AWS and its services mentioned above.

Scenario

Let’s say we want to have a replicated and geographically redundant multi-machine deployment with the following components setup in AWS in the US East and US West regions: 

  • Portal for ArcGIS 
  • ArcGIS Hosting Server 
  • ArcGIS Data Store 
  • ArcGIS Geocode Server

Architectural design of base enterprise deployment with additional server

Route 53 Hosted Zones

Route 53 is a scalable Domain Name System (DNS) that utilizes a combination of public and private hosted zones, which are containers for records about how you want to route traffic for a specific domain. Public hosted zones are used to route traffic on the internet, while private hosted zones are used to route traffic within an Amazon VPC. The following workflow will be utilizing both types of hosted zones with overlapping namespaces. Therefore, in our scenario when we are logged into an EC2 instance in a VPC that is associated with a private hosted zone:

  • The Resolver will evaluate whether the name of the private hosted zone matches the domain name in the request. If there is no match, the Resolver will forward the request to a public DNS resolver.
  • If there is a match in the domain name of the request, the hosted zone is searched for a record that matches the domain name. If there is no record in the matching private hosted zone, the Resolver does not forward the request to a public DNS resolver, but will return a non-existent domain (NXDOMAIN) to the client.

That last bullet is very important! If you have other applications in the same VPC and you setup a private hosted zone, please ensure to add records for those applications so that those application remain completely operational.

Before Deployment

Before proceeding with deploying ArcGIS Enterprise, we need to complete the following tasks using the AWS services mentioned above in order to maintain a consistent DNS across the primary and standby sites:

 

  1. Create an ALB in both regions that will manage traffic for the ArcGIS Enterprise deployments. A few things to note: 
    1. Ensure to attach the appropriate certificate from the public domain registered in Route 53.
    2. A target group must be created when configuring a new load balancer. Create a target group for Portal at this time. It does not need to be registered to any targets with this target group.
  2. In Route 53 under your Public Hosted Zone:
    1. Create two identical Record Sets for each of the load balancers using CNAME types. One thing to note here: 
      1. Having multiple Record Sets using the Weighted Routing Policy is not a requirement; it is more a matter of preference. It is possible to have just one Record Set and replace the ALB value when the switch is needed.
    2. Set the time to live (TTL) to the recommended 60 seconds, which will "minimize the amount of time it takes for traffic to stop being routed to your failed endpoint."
    3. Set the Routing Policy to Weighted:
      1. Assign a weight of 100 to the primary site.
      2. Assign a weight of 0 to the standby site.                     Public record set in Route 52 with Weighted Routing PolicyPublic record set in Route 52 with Weighted Routing Policy
  3. Create a Private Hosted Zone for each region using the same Domain Name as the Public Hosted Zone in Route 53.
    1. Attach the private hosted zone to the VPC assigned to your ALB during time of creation.List of hosted zones in Route 53
  4. In Route 53 under each of the Private Hosted Zones:
    1. Create an identical Record Set to match the one created in the Public Hosted Zone.
    2. TTL can remain as the default value of 300 seconds.
    3. The Routing Policy can remain the default value of Simple.Private record set in Route 52 with Simple Routing Policy Private record set in Route 52 with Simple Routing Policy

The last two steps of this pre-deployment process are vital to the success of the WebGIS DR utility. Having the two Private Hosted Zones with identical Domain Names and Record Sets that match the Record Sets in the Public Hosted Zone ensures the ability to configure identical ArcGIS Enterprise deployments. Additionally, the deployments will remain in an operational state to perform consistent backups and restores on the appropriate systems without issue due to being on separate regions and VPCs.

ArcGIS Enterprise Deployment

It is finally time to deploy the components of our architecture to EC2 instances (repeat for each region):

  1. To setup the base deployment (Portal for ArcGIS, ArcGIS Hosting Server, and ArcGIS Data Store), follow steps 1-20 from our documentation on deploying Portal for ArcGIS on AWS, which utilizes Esri’s Amazon Machine Images (AMIs).
    1. Make sure to assign the EC2 instances to the same VPC and Security Group as the ALB.
  2. Repeat steps 11-19 in the linked documentation from the previous step to setup the additional ArcGIS Geocode Server.
  3. Create the two remaining target groups for the ArcGIS Hosting Server and Geocode Server (the Portal target group was created during the ALB creation in the pre-deployment steps).
  4. Register each instance with its appropriate target group.
  5. Configure the appropriate forwarding rules in the ALB to match the target groups. If everything is setup correctly, we should be able to successfully access portal through the DNS alias (Name property) created in the Record Sets.ArcGIS Portal homepage with arrow pointing to URL
  6. Follow steps 21-23 in the linked documentation above using the DNS alias from the Record Sets. For example, my portal’s system properties would have the following entries: Portal system properties
    1. And federating my hosting server (or any others) would look like so (the admin URL may vary depending on the level of administrative access setting placed on the Web Adaptor):ArcGIS Server federation in Portal organizational settings.
  7. Repeat step 22 for the ArcGIS Geocode Server.

We now have two identical and fully functional ArcGIS Enterprise deployments in each region. The deployment that has a Weighted Policy of 100 in the Public Hosted Zone will be accessible over the internet and act as the primary site, while the other deployment (with Weighted Policy of 0) in the Public Hosted Zone can only be accessible within its own VPC and act as the standby site.

Active and standby ArcGIS Enterprise architectural designs

Replication

Now that we have our primary and standby sites, we have to setup two replicating S3 buckets in the appropriate regions to have a fully replicated and geographically redundant deployment prepared for a disaster recovery scenario.

In Amazon S3, create two buckets with easily recognizable naming conventions, like the following:

Amazon S3 buckets

In the second step of the creation process for each bucket under Configure Options, check the box under Versioning. This is a requirement to enable Cross-Region Replication. Once the buckets have been created, select the bucket that will be utilized by the primary site and navigate to Replication under the Management tab and select Get Started. We can leave the defaults selected for all settings throughout this process. We just need to ensure we select our designated standby site bucket for the destination.

Configuring the replication rule in Amazon S3 bucket

Amazon S3 provides a selectable option (seen in the screenshot above) called S3 Replication Time Control, which will replicate 99.99% of new objects within 15 minutes. This may be a valuable option for organizations with large-scale ArcGIS Enterprise deployments with lots of content who need minimal down time. In my tests, I have found replication to be fast without that option selected, granted my backups ran around 5-6 GBs, which amounted to ~300 hosted services and a Geocode service (and its data) copied to ArcGIS Server.

Disaster Recovery

We are now able to create backups of the primary site and restore the standby site from said backups.

  1. Create a backup with the WebGIS DR tool of your primary site on the instance where Portal is installed. Make sure to point to the appropriate S3 bucket (should be the one in the same region as the primary site) in the properties file. The backup will be replicated to the S3 bucket in the other region.
  2. Restore the replicated backup with the DR tool on the standby instance where Portal is installed. Be mindful of the appropriate S3 bucket in the properties file.

This is the final architecture and workflow:

Full workflow/architecture outlined with S3 bucket replication

In the event of a disaster, we now have the ability to “failover” to our warm, standby deployment by just toggling the values of the Weighted Policies of the Record Sets in our Public Hosted Zone with downtime dependent upon the TTL of the Route 53 record set.Additionally, depending on the length of downtime for the original hot site, you may need to modify how your backups and restores are handled on the two environments, so that any new items created in the new hot site can be maintained when your original data center is restored.

* It should be heavily emphasized that this workflow with the WebGIS DR tool is not considered a highly available ArcGIS Enterprise deployment. The “failover” may be quick in this scenario, but the standby deployment will only have content available from the last WebGIS DR backup/restore. Please see our documentation on configuring a highly available ArcGIS Enterprise.

ArcGIS Enterprise failover workflow/architecture

Since both deployments are also identical, there should be no issues with services integrated into other business systems. Most importantly, this entire workflow – running WebGIS DR backups and restores, DNS toggling, as well as detecting who is acting as the primary site at any given moment – can be completely scripted and automated, ensuring that mission critical systems remain operational.

Learn more about preventing data loss and downtime with ArcGIS Enterprise.

more
12 6 2,566
New Contributor III

In this entry, we will be looking at what a deployment looks like from the infrastructure as code (IaC) perspective with Terraform as well as the configuration management side with PowerShell DSC (Desired State Configuration). Both play important roles in automating ArcGIS Enterprise deployments, so let's jump in.

This deployment will follow a single machine model as described in the ArcGIS Enterprise documentation. It will consist of the following.

  • Portal for ArcGIS 10.7.1

  • ArcGIS Server 10.7.1 (Set as Hosting Server)

    • Services Directory is disabled

  • ArcGIS Data Store 10.7.1 (Relational Storage)

  • Two (2) Web Adaptors within IIS

    • Portal context: portal

    • Server context: hosted

    • Self-Signed certificate matching the public DNS

Additional Configurations

  • WebGISDR is configured to perform weekly full backups that are stored within Azure Blob Storage via Task Scheduler

  • Virtual machine is configured for nightly backups to an Azure Recovery Services Vault

  • RDP (3389) access is restricted via Network Security Group to the public IP of the box in which Terraform is ran from.

  • Internet access (80, 443) is configured for ArcGIS Enterprise via Network Security Group

  • Azure Anti-malware is configured for the virtual machine

The complete code and configurations can be found attached below. You will need to provide your own ArcGIS Enterprise licenses however.

Note:   This is post two (2) in a series on engineering with ArcGIS.

Infrastructure Deployment

If you are not already familiar with Terraform and how it can be used to efficiently handle the lifecycle of your infrastructure, I would recommend taking the time to read through the first entry in this series which can be found here. The Terraform code in that first entry will be used as the basis for the work that will be done in this posting.

As discussed in the first entry, Terraform is a tool designed to help manage the lifecycle of your infrastructure. Instead of rehashing the benefits of Terraform this time however, we will jump straight into the code and review what is being done. As mentioned above, the template from the first entry in this series is used again here with additional code added to perform specific actions needed for configuring ArcGIS Enterprise. Let's take a look at those additions.

These additions handle the creation of two blob containers that will be used for uploading deployment resources ("artifacts") and a empty container ("webgisdr") that will be used when configuring webgisdr backups along with the uploading of the license files, the PowerShell DSC archive and lastly, the web adaptor installer.

resource "azurerm_storage_container" "artifacts" {
name = "${var.deployInfo["projectName"]}${var.deployInfo["environment"]}-deployment"
resource_group_name = "${azurerm_resource_group.rg.name}"
storage_account_name = "${azurerm_storage_account.storage.name}"
container_access_type = "private"
}

resource "azurerm_storage_container" "webgisdr" {
name = "webgisdr"
resource_group_name = "${azurerm_resource_group.rg.name}"
storage_account_name = "${azurerm_storage_account.storage.name}"
container_access_type = "private"
}

resource "azurerm_storage_blob" "serverLicense" {
name = "${var.deployInfo["serverLicenseFileName"]}"
resource_group_name = "${azurerm_resource_group.rg.name}"
storage_account_name = "${azurerm_storage_account.storage.name}"
storage_container_name = "${azurerm_storage_container.artifacts.name}"
type = "block"
source = "./${var.deployInfo["serverLicenseFileName"]}"
}

resource "azurerm_storage_blob" "portalLicense" {
name = "${var.deployInfo["portalLicenseFileName"]}"
resource_group_name = "${azurerm_resource_group.rg.name}"
storage_account_name = "${azurerm_storage_account.storage.name}"
storage_container_name = "${azurerm_storage_container.artifacts.name}"
type = "block"
source = "./${var.deployInfo["portalLicenseFileName"]}"
}

resource "azurerm_storage_blob" "dscResources" {
name = "dsc.zip"
resource_group_name = "${azurerm_resource_group.rg.name}"
storage_account_name = "${azurerm_storage_account.storage.name}"
storage_container_name = "${azurerm_storage_container.artifacts.name}"
type = "block"
source = "./dsc.zip"
}

resource "azurerm_storage_blob" "webAdaptorInstaller" {
name = "${var.deployInfo["marketplaceImageVersion"]}-iiswebadaptor.exe"
resource_group_name = "${azurerm_resource_group.rg.name}"
storage_account_name = "${azurerm_storage_account.storage.name}"
storage_container_name = "${azurerm_storage_container.artifacts.name}"
type = "block"
source = "./${var.deployInfo["marketplaceImageVersion"]}-iiswebadaptor.exe"
}
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

This addition handles the generation of a short-lived SAS token from the storage account that is then used during the configuration management portion to actually grab the needed files from storage securely. In this situation, we could simplify the deployment by marking our containers as public and not requiring a token but that is not recommended.

data "azurerm_storage_account_sas" "token" {
connection_string = "${azurerm_storage_account.storage.primary_connection_string}"
https_only = true
start = "${timestamp()}"
expiry = "${timeadd(timestamp(), "5h")}"

resource_types {
service = false
container = false
object = true
}

services {
blob = true
queue = false
table = false
file = false
}

permissions {
read = true
write = true
delete = true
list = true
add = true
create = true
update = true
process = true
}
}
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

The final change is the addition of an extension to the virtual machine that will handle the configuration management task using PowerShell DSC. Instead of reviewing this in-depth here, just know that the data that is getting passed under the settings and protected_settings json will be passed to PowerShell DSC as parameters for use as needed by the configuration file.

resource "azurerm_virtual_machine_extension" "arcgisEnterprise-dsc" {
name = "dsc"
location = "${azurerm_resource_group.rg.location}"
resource_group_name = "${azurerm_resource_group.rg.name}"
virtual_machine_name = "${element(azurerm_virtual_machine.arcgisEnterprise.*.name, count.index)}"
publisher = "Microsoft.Powershell"
type = "DSC"
type_handler_version = "2.9"
auto_upgrade_minor_version = true
count = "${var.arcgisEnterpriseSpecs["count"]}"

settings = <<SETTINGS
{
"configuration": {
"url": "${azurerm_storage_blob.dscResources.url}${data.azurerm_storage_account_sas.token.sas}",
"function": "enterprise",
"script": "enterprise.ps1"
},
"configurationArguments": {
"webAdaptorUrl": "${azurerm_storage_blob.webAdaptorInstaller.url}${data.azurerm_storage_account_sas.token.sas}",
"serverLicenseUrl": "${azurerm_storage_blob.serverLicense.url}${data.azurerm_storage_account_sas.token.sas}",
"portalLicenseUrl": "${azurerm_storage_blob.portalLicense.url}${data.azurerm_storage_account_sas.token.sas}",
"externalDNS": "${azurerm_public_ip.arcgisEnterprise.fqdn}",
"arcgisVersion" : "${var.deployInfo["marketplaceImageVersion"]}",
"BlobStorageAccountName": "${azurerm_storage_account.storage.name}",
"BlobContainerName": "${azurerm_storage_container.webgisdr.name}",
"BlobStorageKey": "${azurerm_storage_account.storage.primary_access_key}"
}
}
SETTINGS
protected_settings = <<PROTECTED_SETTINGS
{
"configurationArguments": {
"serviceAccountCredential": {
"username": "${var.deployInfo["serviceAccountUsername"]}",
"password": "${var.deployInfo["serviceAccountPassword"]}"
},
"arcgisAdminCredential": {
"username": "${var.deployInfo["arcgisAdminUsername"]}",
"password": "${var.deployInfo["arcgisAdminPassword"]}"
}
}
}
PROTECTED_SETTINGS
}‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Configuration Management

As was touched on above, we are utilizing PowerShell DSC (Desired State Configuration) to handle the configuration of ArcGIS Enterprise as well as a few other tasks on the instance. To simplify things, I have included v2.1 of the ArcGIS module within the archive but the public repo can be found here. The ArcGIS module provides a means with which to interact with ArcGIS Enterprise in a controlled manner by provided various "resources" that perform specific tasks. One of the major benefits of PowerShell DSC is that it is idempotent. This means that we can continually run our configuration and nothing will be modified if the system matches our code. This provides administrators the ability to push changes and updates without altering existing resources as well as detecting configuration drift over time. 

To highlight the use of one of these resources, let's take a quick look at the ArcGIS_Portal resource which is designed to  configure a new site without having to manually do so through the typical browser based workflow. In this deployment, our ArcGIS_Portal resource looks exactly like the below code. The resource specifies the parameters that we must be provided to successfully configure the portal site and will error out if all required parameters are not provided. 

ArcGIS_Portal arcgisPortal {
PortalEndPoint = (Get-FQDN $env:COMPUTERNAME)
PortalContext = 'portal'
ExternalDNSName = $externalDNS
Ensure = 'Present'
PortalAdministrator = $arcgisAdminCredential
AdminEMail = 'example@esri.com'
AdminSecurityQuestionIndex = '12'
AdminSecurityAnswer = 'none'
ContentDirectoryLocation = $portalContentLocation
LicenseFilePath = (Join-Path $(Get-Location).Path (Get-FileNameFromUrl $portalLicenseUrl))
DependsOn = $Depends
}
$Depends += '[ArcGIS_Portal]arcgisPortal'‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Because of the scope of what is being done within the configuration script here, we will not be doing a deep dive. This will come in a later article.

Putting it together

With the changes to the Terraform template as well as a very high level overview of PowerShell DSC and its purpose, we can deploy the environment using the same commands mentioned in the first entry in the series. Within the terminal of your choosing, navigate into the extracted archive that contains your licenses, template and DSC archive, and start by initializing Terraform with the following command.

terraform init

Next, you can run the following to start the deployment process. Keep in mind, we are not only deploying the infrastructure but configuring ArcGIS Enterprise so the time for completion will vary. When it completes, it will output the public facing url to access your ArcGIS Enterprise portal.

terraform apply


Summary

As you should quickly be able to see, removing the manual configuration aspects of software as well as the deployment of infrastructure, a large portion of problems can be mitigated by moving toward IaC and Configuration Management. There are many solutions out there to handle both aspects and these are just two options. Explore what works for you and start moving toward a more DevOps centric approach.

I hope you find this helpful, do not hesitate to post your questions here: Engineering ArcGIS Series: Tools of an Engineer 

 

Note: The contents presented above are examples and should be reviewed and modified as needed for each specific environment.

more
3 0 1,905
New Contributor III

I work with a lot of Esri users to help them successfully implement ArcGIS within their organization. Customers reach out to us because they are often challenged in a few key areas:

  • Seeking ways to reduce the time it takes to install and configure our products
  • Getting started and building proficiency
  • Maximizing effectiveness and productivity of the implementation (which could include data, maps, apps, etc.)

These are common challenges as our users are often supporting many roles within their organization (analysts, managers, system administrators, etc.) and often have limited bandwidth to roll out the latest version of our products.

I find our users can experience a much quicker time to value and improved productivity through a services package. In these services packages, we often focus on a few best practices:

  • Considering the needs of all stakeholders
  • Supporting quick wins through understanding customer needs and configuring maps and apps to meet those needs
  • Building a better user experience

Please find attached a flyer that can be used a resource to read and understand more about some of our product and subject matter expert specific services packages (ArcGIS GeoEvent Server, ArcGIS Monitor, Insights for ArcGIS, etc.)

more
1 0 155
New Contributor III

What is Infrastructure as Code?

Taken directly from Microsoft, 

"Infrastructure as Code (IaC) is the management of infrastructure (networks, virtual machines, load balancers, and connection topology) in a descriptive model".

In the simplest terms, Infrastructure as Code (IaC) is a methodology to begin treating cloud infrastructure the same as application source code. No longer are changes made directly to the infrastructure, but to the source code for a given environment or deployment. This change in behavior will lead to environments that are easily reproducible, quickly deployable and accurate.

Note:   This post is one (1) in a series on engineering with ArcGIS.

What is Terraform?

Terraform is a tool for building, changing, and versioning infrastructure safely and efficiently. Terraform can manage components such as virtual machines, networking, storage, firewall rules and many others. To define what needs to be deployed or changed, terraform uses what is called a terraform configuration which can be made up of one or more individual files within the same folder. Terraform files utilize the file extension

.tf

as well as HCL (HashiCorp Configuration Language) as the configuration language. HCL is a structured configuration language that is both human and machine friendly for use with command-line tools, but specifically targeted towards devops tools, servers, etc. 

In complex deployments, administrators may find they can utilize a different configuration file for each aspect of their environment as such,

/environment_a

   /production

      network.tf

      machines.tf

      storage.tf

      security.tf

   /staging

      network.tf

      machines.tf

      storage.tf

      security.tf

whereas with simple deployments, a single file may be sufficient, such as

/environment_a

   main.tf


To actually create and manage infrastructure, terraform has a number of constructs to allow users to define Infrastructure as Code but the most important two are Providers and Resources. 

Resources

Resources are the mechanism that tell terraform how the infrastructure should be deployed and configured. Each cloud provider will have its own list of Resources that users will have access to. An example would look something like this which creates an azure resource group and a virtual network within.

resource "azurerm_resource_group" "rg" {
   name = "prd"
   location = "westus2"

}

resource "azurerm_virtual_network" "vnet" {
   name = "prd-vnet"
   location = "westus2"
   address_space = ["10.0.0.0/16"]
   resource_group_name = "prd"
}

Providers

Providers are the mechanism for defining what cloud provider or on-premise Resources are available for use. Each provider offers a set resources, and defines for each resource which arguments it accepts, which attributes it exports, and how changes to resources of that type are actually applied to remote APIs. Most of the available providers correspond to one cloud or on-premises infrastructure platform, and offer resource types that correspond to each of the features of that platform. In simpler terms, if the goal is to define infrastructure on Azure, an Azure based provider must be used before Resources can be defined.

 

A provider example for Azure would look something like this,

provider "azurerm" {
   subscription_id = "this-is-not-a-real-subscription-id"
   client_id = "this-is-not-a-real-client-id"
   client_secret = "this-is-not-a-real-client-secret"
   tenant_id = "this-is-not-a-real-tenant-id"
}

Terraform has multiple methods for authenticating to a given cloud provider and in this example, a Service Principal is being utilized.

State

Lastly, terraform makes use of a State File that keeps track of the infrastructure that has been deployed and configured. This state file is what allows terraform to run checks against the last recorded state of an environment compared to the current run and provide users the delta so validation can be done before making changes. This aspect is very important in that it allows terraform to be idempotent, which is a key aspect of IaC.


Infrastructure Life-cycle

For the purposes of this introduction, we will be using the attached terraform template as the basis for the following examples. This template will be posted to GitHub in the near future for future articles to utilize and build off. It is designed to deploy the following within an Azure subscription.

Before it can be deployed, the deployInfo variable group (Map) will need to be populated. When creating the Service Principal, the following documentation (Service Principal creation) can be reference for assistance. Terraform will also need to be available locally. The steps to ensure it is can be found here.

  • Resource Group
  • Virtual Network
  • Subnet
  • Network Security Group
    • Rule to allow 80 and 443 traffic from the internet into the virtual network
    • Rule to allow RDP access (3389) into the virtual network from the public IP of the person who deploys the template. This is accomplished by querying the web during deployment and retrieving your public IP. This workflow is not recommended in production environments and is only being used for example purposes.
    • Rule to block all other internet traffic into the virtual network.
  • Storage account
  • Availability Set
  • Public IP
  • Network Interface
  • Virtual Machine
    • Windows Defender Extension
    • Recovery Services Vault Extension
  • Key Vault
  • Recovery Services Vault
    • Backup Policy

Creation

Once terraform is available locally and the deployInfo variable map has been completed, the first step in deploying infrastructure is to initialize terraform. This can be accomplished by navigating to the directory in which you have saved the above template with the .tf file extension and running the following command, which will prepare various local settings and data that will be used by subsequent commands.

terraform init

The output from initializing terraform will resemble the following.

With terraform successfully initialized, the next step in the process is to have terraform review the template and determine what changes need to take place. This step will have terraform compare the template with the current state file and produce an output showing the deltas as follows. To do so, use the following command.

terraform plan

Output has been truncated.

As this is the first run of terraform, terraform is only able to see resources it needs to add (create). Later in this post, we will walk through updating existing resources.

With terraform successfully prepared to deploy our infrastructure, we can being the deployment by using the following command which will start the process of creating the resources within Azure and provide the following output when complete.

terraform apply

Updates

As the deployment is utilized, it may be determined that the current infrastructure is not adequately sized and resources need to be increased. As our resources are defined as code, so is the virtual machine sizing. Within the "arcgisEnterpriseSpecs" variable map, a variable is defined with the name "size" which is the Azure machine sizing that is used by terraform when deploying resources.

If this variable is changed to "Standard_D8s_v3" and terraform plan is ran again, it will detect a delta between what is defined in code and what is actually deployed within Azure and present this in the output as such.

Once this delta is planned and the changes are ready to be pushed to the actual infrastructure within Azure, simply running terraform apply again will begin the process.

Decommission

The last phase of a given deployments life-cycle is decommissioning. Once infrastructure has reached the end of it life, it must be terminated and removed and thankfully, terraform provides an easy method to do so with the following command.

terraform destroy

The output of destroying the infrastructure will resemble the following.

In conclusion, as teams aim to become more agile and move at a much faster pace, moving to modern methodologies such as infrastructure as code (IaC) is a great first step and should not be viewed as unnecessary but a crucial step in the right direction.

I hope you find this helpful, do not hesitate to post your questions here: https://community.esri.com/community/implementing-arcgis/blog/2019/07/03/engineering-arcgis-series-t... 

 

Note: The contents presented above are examples and should be reviewed and modified as needed for each specific environment.

more
5 0 1,649
Esri Contributor

The Managed Cloud Services team in Professional Services is pleased to announce a new series that will be highlighting various tools and best practices for implementing ArcGIS Enterprise using modern methodologies.

System implementation, configuration and management of the deployment is a fun challenge, similar to Tetris. As an ArcGIS Enterprise or ArcGIS Server administrator, you are likely tasked with standing up new systems, ensuring they are configured correctly and maintaining them over time. Traditionally, these tasks were done manually where an administrator would work through procuring a new virtual machine, install the required software, work through the configuration steps needed and then ensure users were able to access it. Over time, the needs of the users may change and as such, the administrator would need to further modify the system and its software to meet those needs.

Here at Managed Services, we are faced with managing 100's of customer implementation. This series will cover best practices we developed overtime. Each entry will cover a specific aspect of automating the deployment, configuration & life-cycle management (updates, monitoring, scaling, etc) of both infrastructure and the ArcGIS suite. 

automation, devops, implementation , engineering‌, cloud‌

more
11 3 2,264
Esri Regular Contributor

If you're headed to the 2019 Esri International User Conference and are interested in sessions for GIS Managers, here is a link to the GIS Manager Track:

https://userconference2019.schedule.esri.com/schedule?filters=1964269831

more
0 0 160
Esri Regular Contributor

Are you a GIS manager, leader or other executive headed to the 2019 Esri International User Conference (UC)?  I know it can be a challenge creating your personal agenda for the world's largest GIS conference, so I created this flier to assist.  It covers suggested events and activities you should consider when deciding how to spend your valuable time at UC.  I hope you have a productive UC experience, and I hope to see you there!

UPDATED 6/24/2019 - Added GIS Manager Track and Get Advice from Esri Services section.

UPDATED 6/13/2019 - Corrected the name of the Implementing ArcGIS area in the Expo to “Guiding your Geospatial Journey”

FYI there are other Esri UC fliers here: https://community.esri.com/community/events/user-conference/content?filterID=contentstatus%5Bpublish...

more
3 0 1,364
Esri Contributor

Introduction

To cloud or not to cloud is the question that many organizations are currently facing. While on-premise data center technology is not necessarily on the verge of extinction, cloud computing is an option with many benefits including scalability, agility and cost efficiency.

The ArcGIS platform is supported on both on-premises or in a cloud environment like Microsoft Azure or Amazon Web Services (AWS). By leveraging these components, you can expose GIS content and capabilities as web services and consume those services in your apps. This enables users to access and apply useful GIS resources in their work.

Moving to the cloud is more than just upgrading your servers and software. It represents a radical change in the way that you manage technology. This shift gives you a unique opportunity to align technology with your overall business vision, which in turn can help you grow your business, improve productivity, and reduce costs.

Before starting, it is a must to document your current infrastructure, this includes compiling a list of your servers, storage systems, network components, off the shelf software, bespoke software, and subscriptions to gain a full picture of your current technology. This information is critical for helping you determine the best path forward.

Analyzing your current infrastructure against your user workflows is the next step.  From assessing your desktop workflows to assessing your integrations, your security policies, SLAs, data storage requirements, authorization, and authentication providers all these points will play a huge factor in designing your cloud infrastructure. We will discuss these topics briefly in this article and will go into the specifics of each separately in future blogs posts.

 

General Database Considerations

Moving your Geodatabase to the cloud is a big step, you need to consider where your clients accessing the data exist, are they running from the cloud or on-premise, and what is the type of work is being performed (read-only, frequent edits, etc.).

 

General Network Bandwidth and Speed

Moving all the infrastructure to the cloud will need a resilient and reliable Internet connection, most importantly with low latency. Low bandwidth can be a serious problem when you factor in a large number of employees sharing the same network. Sometimes cloud solutions don’t offer enough bandwidth, especially when it comes to uploading larger files such as raw satellite imagery. However, latency is the primary concern as network latency is the key factor in determining cloud architecture decisions, for example, because of the latency it is likely not practical to keep desktops on premise and move the database into the cloud.

 

Assessing ArcGIS Desktop Workflows

First and Foremost, do you need to provide this functionality through ArcGIS Pro or ArcGIS Desktop, or can we have it replaced via a web-based app? That would save us setting up a dedicated machine or application instance on the cloud.

Is the data you are processing 2D or 3D? ArcGIS Pro required a GPU and the resourcces are even more demanding if processing 3D data. GPU enabled machines on the cloud cost considerably more than machines without a GPU. Do you have advanced Geoprocessing workflows that require hefty computing power, do these workflows run overnight? Check out https://pro.arcgis.com/en/pro-app/get-started/virtualization-overview.htm for more information about running ArcGIS Pro on the cloud.

 

ArcGIS Server Sites and Portal for ArcGIS Sites

The least difficult method for starting to use the cloud for hosting your own software is perform a lift-and-shift migration, where you move software currently running on-premises to the cloud host. The behavior of the software is precisely the same after moving into the cloud however now it is running with the benefits of underlying cloud infrastructure, such as having affordable and reliable virtual machines that don’t require high upfront purchasing costs and little ongoing maintenance of the infrastructure.

Using specialized deployment tools can make it easier to install and configure the software on certain cloud platforms. You can also create your own machine images to host Esri software in cloud environments. These deployment tools don’t just install and configure the software; they also provision and set up the underlying infrastructure, including the virtual machines, load balancers, networking, and storage.

 

Bespoke GIS Applications

Whether you have Geocentric applications, Geo-enabled applications, or Composite applications you will need to reassess the architecture of these applications and ensure the compatibility with the new cloud environment. check out developers.arcgis.com for information about our wide coverage of SDKs and APIs.

No single application integration pattern fits all situations. You can use the application pattern that best combines capabilities from ArcGIS and your business system to deliver the greatest impact.

 

Integration to Other Systems

Application integration lets you deliver solutions that combine data and tools from different systems—including your GIS as well as business systems like permitting, licensing, and asset management systems. With integrated solutions, you can improve cross-functional business processes and provide decision-makers with integrated views of your organization’s information. Insuring that these systems integrate prevents information duplication and enables having the right data available when you need it. You need the option of deploying your integration technology so that it works in both environments. To decide whether your integration technology will reside in the cloud or in a rack in your data center, you have to start with the specific requirements of your business.

 

Security

Security is not only an important decision to avoid becoming a statistic, but also one that may be government regulated. Depending on your industry, market, and location, you may have to abide by an array of rules determining how you use and store sensitive data. Cloud security has come a long way, though, and is arguably as secure as most private data centers. In either case, security and regulations of your industry are something to think about when considering cloud and on-premise. With either option, security is granularity configurable to meet your security needs. Consider Esri best practices for configuring secure ArcGIS Server and Portal for ArcGIS environments as well as your organization's needs and policies when designing your ArcGIS Enterprise site security.

 

High Availability and Redundancy

Designing the HA environments in Azure or AWS are somewhat the same design principles as of on-premises however more functionality is provided by cloud providers such as scalability on demand. You can configure your site so that ArcGIS Server machines are added in response to certain triggers, such as CPU usage. New servers can be created in a matter of minutes, allowing your site to gracefully respond to abrupt spikes in traffic. When you no longer need the instances, you can destroy them and incur no further infrastructure charges for them. The installation is much easier as now there are images already provided for faster installation times.

 

Backups and Disaster Recovery

Cloud computing has led to a new way of preparing for IT disasters by providing secondary environments for backing up and restoring data and failing over business applications. These disaster recovery services are cost-effective and straightforward to set up. ArcGIS Server and Portal for ArcGIS include utilities that you can use to create backups and restore your sites on Azure and AWS.

 

Access Management Providers

When implementing a cloud-based application it can be a challenge to ensure that information security and access management controls are applied to the cloud application.

Organizations should ensure that the single sign-on (SSO) access management is implemented in the cloud. In particular, organizations need to understand how employees can seamlessly access the various cloud-based applications using their existing access management protocol.

Portal for ArcGIS is compliant with SAML 2.0 and integrates with identity providers that support SAML 2 Web Single Sign-On. The advantage of setting up SAML is that you do not need to create additional logins for users to access your ArcGIS Enterprise portal; instead, they use the login that is already set up in an enterprise identity store. For example, Microsoft Azure Active Directory is a SAML-compliant identity provider. You can configure it as your identity provider for enterprise logins in Portal for ArcGIS on-premises and in the cloud.

I hope you find this helpful, do not hesitate to post your questions here: - Arcgis Architecture Series : Moving to the Cloud

more
9 0 3,469
New Contributor III

Every organization strives to keep its systems running smoothly, but for some, that objective is mission critical.

Learn about the technical assistance level that aligns with your mission critical needs.  This is accomplished through assigned Technical Account Managers, 1-hour initial response to support requests, and 24/7/365 support.  No matter what your GIS needs may be Premium Support is designed to provide next level support for organizations of all sizes and industry.

 

Perhaps you’re not sure if Premium Support is right for your organization.  We can review your support history and help you better understand if a higher-level support would unlock your organizations potential. 

 

Using the link below you can learn more about the program and submit your request for additional information at the bottom.

 

https://support.esri.com/en/other-resources/supportservices#premium/usadomestic

more
3 0 425
New Contributor II

For many of our customers, installation and implementation of ArcGIS Monitor is a straightforward and quick process. Once the minimum requirements are met, most Monitor installations flow smoothly.

However, when advanced firewall and security practices are in place, these installation and configuration of ArcGIS Monitor can be much more complicated. For optimal success in highly secure environments, ask IT support staff to join in installation activities.

When the ArcGIS Monitor can’t quickly make a connection with other systems in the Enterprise ask IT to monitor the network traffic and see if any internal ports are blocking traffic. This may be an iterative process as you install the software, but without System and Process collectors, ArcGIS Monitor can't fully measure ArcGIS Enterprise Health. 

Onsite recently, in addition to opening ports 6443 and 7443 for ArcGIS Server and Portal connections, we had to request permission for ArcGIS Monitor to operate on ports 135, 49153 and 49154 on the ArcGIS Server, Portal and SQL machines in the deployment. Once these ports were opened, we could begin collecting on Memory, Network and Processing utilization. 

Collaboration between GIS Admins and IT is crucial for understanding security rules and limitations when implementing a product like ArcGIS Monitor.

more
4 0 386