Pushing patches to ArcGIS Server AWS CloudFormation Auto-Scaled Deployments

Question

Hello. Has anyone out there had any experience pushing patches for ArcGIS Server or ArcGIS Enterprise out to deployments that use AWS CloudFormation with auto-scaling groups?

My ArcGIS Server site has new machines being added and terminated daily using an AWS auto-scaling group. I'm unclear on how to push a security patch out to my ArcGIS Server 10.7.1 template. I read from an ESRI presentation last year that the AWS "Run Command" could be used from AWS Systems Manager. I have used this to install AWS components on my EC2 instances, but not to change my ArcGIS Server configuration. Any one with experience doing this? Thanks

NCCGIA · Answer

My workflow is slightly different. I get really nervous when talking about fiddling with machines in the site or the site file server. We've had occasions when the machines wouldn't come back and join the site correctly and end up with a corrupted site (i.e., the pink "contact your administrator" screen). Thank goodness we use WebGISdr and could restore, but in the meantime everything is down and turns into a headache.

What we do is:

* create the new AMI
* associate it with a new launch template version
* set the auto-scaling group to use the new launch template
* set the autoscaling group minimum and desired instances to increase by the number of instances you currently use (e.g., if you currently have 2 instances active, set it to 4)
* let the new instances (using the new launch template) spin up and join the site 
* enable "termination protection" on the newly deployed instances
* change the autoscaling group min and desired back to the original values (e.g., 2)
* let the "old" instances terminate

With our routine, we avoid the anxiety of messing with file server(s), potential site corruption by having zero machines in the site (ever!), etc.

DavidHoy · Answer

Hi Michael,

you may have already received this answer from Support, but I will post here in case others are interested.

In an AWS Autoscaling group new EC2 instances are launched and added to your ArcGIS Server site as a result of a Cloudwatch monitored load threshhold being passed or the administrator altering the Max Instances setting for the group.

When this happens, a new instance is deployed using the AMI associated with the "Launch Configuration" for the group. The new instance is associated as a target with the Load Balancer and then runs any "UserData" steps. If you created the Server using the Esri supplied CloudFormation templates, the UserData includes running the Chef recipes that add the additional ArcGIS Server machine to an existing site. The AMI used is the Esri supplied image. These images are generally only supplied for new version releases.

So, you will need to create your own AMI, by launching a new temporary instance from the Esri supplied AMI and manually installing the latest ArcGIS Server patches on this image (you may also decide to install Windows Updates). Once patched, save as a new AMI in your own S3 bucket. (and shutdown your "gold" instance).

The next few steps come from a support ticket

This is assuming you have a "file server" as created by the Esri supplied HA ArcGIS Server CF templates. This was also created using the same original AMI (this includes ArcGIS Server pre-installed), and is a cunning trick to temporarily keep the ArcGIS Site "live" while removing and recreating the servers in the Autoscaling Group from the "new" AMI.

Hope this is useful - there is a bit of manual AWS config work to be done there.