Adding new machines to Enterprise quickly in AWS using warm pools

Question

Our current Enterprise deployment in AWS takes about 20-25 minutes to spin up an additional machine in our auto scaling group and have it fully added to the site. We are tinkering with using a Warm Pool to add new machines and put them in a stopped state so that we can respond more dynamically to rapid increases in system load. Has anyone considered this or already successfully implemented?

georisk · Answer

Hi,

We have a similar problem and interested on your approach. Can you provide details on how you did the step # 4?

Thanks!

NitinChauhan · Answer

Hi Everyone.

Can somebody share how you have automated the join site for the new EC2 being launched as a part of ASG. I believe you have created a custom AMI to the stage of join site. How is it being taken forward after that?

NCCGIA · Answer

We've deployed a solution that seems to show promise in testing. Our time to add a new machine to a site has gone from ~20min to ~4min. Our process:

* Set the max number of instances for the auto-scaling group (i.e., 6)
* Create an AWS warm pool and set the max count equal to the value entered in step #1
* Use the AWS CLI to set the "instance reuse" policy for the warm pool so that instances are not potentially terminated. Setting this policy is not supported in the console. (If using Windows, must use AWS Cloudshell. Command does not work if using the CLI in the CMD window).* Example: * aws autoscaling put-warm-pool --auto-scaling-group-name agsSuper-ServerStack-AutoScalingGroup-1K7QS9RIGCV3K --pool-state Stopped --instance-reuse-policy '{\"ReuseOnScaleIn\": true}'

* Use AGS REST Admin to change the "remove from site" setting so that machines are not removed
* Confirm that all instances have joined the site
* Decrease the desired number of instances in the auto-scaling group to the number that we want to be used during normal traffic (i.e., 2)

The unused instances will be returned to the warm pool but will still appear in ArcGIS Manager. When it needs to scale out again, the machines simply change state (e.g. stopped to start) and are part of the AGS site as soon as AGS gets a heartbeat. Doing this avoids the 20min spin up time to create the new instance and allow the Chef and any "user data" scripts to run. Because we've "pre-configured" everything they are ready to go and just need to change state. Doing this also avoids potential IP address issues in Enterprise when attempting to use auto-scaling and warm pools (more info).  The only potential issue we've identified with our approach is that you could get into IP address conflicts if the combined number of instances you have active and have configured for the warm pool are less than the maximum number of instances you've set for the auto-scaling group. As scaling events occur, there could be IP address issues between what AWS assigns to new instances (i.e., not coming from the warm pool but actual new instances) and what Enterprise thinks is in use. It's a bit deep and convoluted to explain, but the takeaway is that by: 1) configuring ArcGIS to not release the IP addresses in the site and 2) we've configured the warm pool max count to be equal to the max count of the auto-scaling group, we mitigate that concern.