ArcGIS Enterprise on AWS (Configuring Enterprise with AWS Load Balancers & using WebGISDr Tool)

[cdickerson]

Hi All,

I have a dilemma I am facing with our ArcGIS Enterprise configuration on AWS. First, I will describe our system to give context to the issue. Note our system is fully up and running and we are not experiencing any issues at the moment.  Most of what I will be discussing are issues or questions that came up when planning for system outages and recovery plan, and deploying a development environment for testing system upgrades.  Any questions for this thread will be listed at the end of the tread, I reference these questions with *Q# for context in the discussion.

System Config

We have a GIS Enterprise system (ArcGIS 10.6.1) fully in the AWS Cloud. In our system we have two servers that drive the use of our multi-machine deployment of ArcGIS Enterprise. First, a GIS Server with ArcGIS Server, Portal for ArcGIS, and Data store software along with both Web Adaptors and second, Image Server with ArcGIS Server software. The last time our SSL Certificate expired, in the attempt to save a little bit of money we stood up a Application Load Balancer (ALB) in front of our servers that handled secure routing of traffic to our servers.  Load Balancers in AWS can host the SSL certificates but these certificates cannot be installed on the instances (servers) themselves. Through the use of an ALB, our DNSs switched from being associated with an elastic IP to the ALB (*Q1). This changed the way that ArcGIS Server and Protal allowed us to configure the Web Adaptors. Instead of being able to configure the Web Adaptors with the DNS (https://gisserver.domain.com:6443),  it changed how our systems communicate with each other and we had to configure the Web Adaptors to use the private IP (https://10.x.x.x:6443) and federate our servers in Portal the same way (*Q2). Through these nuances ...

Development Environment Deployment

From the system detailed above, we took images of our two servers in AWS and deployed them in the separate Virtual Private Cloud (VPC) for testing. We replicated security groups, launched a load balancer for those servers, and ensured all the components of what is needed to operate our GIS system were present and account for. These servers had their own variants of the DNSs that we were looking to reconfigure for the dev system. We ran into some hurdles with ArcGIS Server and ended up uninstalling and reinstalling the software (preserving the config-store and directories folders). We were able to create a new GIS site for server manager but unable to recreate the site from an existing one using the preserved folders.  We were unsure if the issue was because of the DNS change or differences in the admin passwords I chose to use (*Q3). We ended up doing the same thing form Portal and Data store and we now have a replicated system with no content but it is successfully running ArcGIS Enterprise. Additionally, we were able to reconfigure our Image server without losing content (I do believe the same admin account password was preserved).

WebGISDr Tool

Both within our production and development environments we were able to use the WebGISDr Tool to export full backups of our Enterprise to AWS.  Within development we were able to import a backup successfully after exporting it. We however have been unable to migrate data/content from our production environment to dev with the tool.  I have been told that as long as my DNS's match I should be able to import a back up generated from our production environment and use it in development (*Q4). Other than some minor differences (admin account password being different) in our prod and dev systems the one attempt at doing this led to the tool failing once it got to the data store. Looking to understand if this failure is something others have seen or if it is a nuance of our particular configuration or of our system stemming back to the way we have our ALB configured. 

That said, here are my questions:

* Q1 - Most traditional deployment models have the SSL Certificate installed on the servers themselves. Has anyone experienced any issues managing their SSL Certificates through Load Balancers?

*Q2 - In AWS you have no control over the what the primary private IP will be for a server but you have the ability to assign a secondary private IP. Does any one know if ArcGIS Enterprise can be configured using secondary private IPs? Has anyone ever tried, and could there be unforeseen consequences to doing this?

*Q3 - Has anyone recovered the content that was published to ArcGIS Server through replacing the config-store and directories folders? Was it for the same system where the DNS and the admin accounts matched?

*Q4 - Has anyone ever successfully imported a backup with the WebGISDr tool, to a different server? Were there any nuances you encountered that you needed to do to get it to work?

Thank you in advance.

[JonathanQuinn]

Q1: I think a big part about managing certificates for load balancers is the type of load balancing you're doing, layer 4 vs layer 7. If you're using layer 4, then the certificate you use needs to have SANs for the LB as well as all backend machines, since you're not decrpyting the trarffic at the LB. On the other hand, a layer 7 LB must be able to inspect the traffic to know where to send the request. The certificate on the LB can be different from the certificates used by the backend machines, which can remain as the self-signed certificate, (or any domain certificate you want to use).

Q2: Yes, ArcGIS Enterprise can function in an environment where there are multiple NICs, but the changes described below have to be done prior to configuring the environment:

Configure ArcGIS Server in advanced scenarios—Deploy | ArcGIS Enterprise 

Installing Portal for ArcGIS—Portal for ArcGIS (10.7 and 10.7.1) | ArcGIS Enterprise 

Create a data store—ArcGIS Data Store (Windows) Installation Guide | ArcGIS Enterprise 

If you already have an existing deployment, then you may have trouble adding that second private IP and configuring Enterprise to use it.

Q3: I don't suggest copying files/folders around as a backup/recovery or disaster recovery plan. That workflow isn't technically support by Support Services so any issues that may come up after doing so may not be fixable. Which is a good lead in to:

Q4: The documentation on disaster recovery, (taking a backup from one environment and restoring to another), using the WebGIS DR tool is here. The documentation on what must be the same between primary and standby deployments are documented here. Did you use the AWS CF templates to create the site? If so, those add etc\hosts entries to the machines, so the public URL resolves to the local IP when you're on the machine. This makes it easy to set up a new environment because you don't need to worry about traffic getting sent to your primary environment during a restore.

[cdickerson]

Thanks you John, I appreciate the response. I believe we are using layer 7 on our LB since it routing traffic based on URL. Q2 was really getting at the response to see if I can use secondary IP to configure a development environment that matches our production environment. I believe my predecessor did use Cloud Formation Templates to launch the original array of servers but I can't say that with 100% certainty. I found out the hard way about how servers resolve the traffic when I originally did not put my dev system in a different VPC.