Chef Test Kitchen for ArcGIS Enterprise
There has been a growing trend with ArcGIS Enterprise and its deployment schemes. This trend has been automation in the form of Chef and PowerShell DSC. These automation's allow us to deploy these complex setups into Windows and Linux environments with little manual involvement and ease. We can use these same automation's to deploy ArcGIS Enterprise into Cloud hosted systems as well such as Amazon Web Services and Microsoft Azure. The complexity of the deployments of the ArcGIS Enterprise can range from a Base Enterprise Deployment (single machine hosting Enterprise) to more advanced deployments such as High-Available ArcGIS Enterprise (multiple machines hosting pieces of the Enterprise with the intention of combating downtime of your Enterprise).
Automations can allow us to design a template of what and how we want to deploy software, and pending on our resources to deploy to, we can deploy multiple Enterprise setups with a few clicks and commands. There are some tools out there already to help with this in Cloud settings such as the Amazon Web Services CloudFormation (which utilizes Chef) and Azure CloudBuilder (which utilizes PowerShell DSC). There are solutions as well though, that allow you to design your templates a bit more freely, these solutions are Chef and PowerShell. There are workflows for ArcGIS Enterprise automation with these automation software packages:
-https://esri.github.io/arcgis-cookbook/
-https://esri.github.io/arcgis-powershell-dsc/
Our focus will shift to a specific automation software, Chef. Automation with Chef (PowerShell can be roped in as well) comes with a question/scenario that some organizations will be faced with:
- “How do I test Chef deployments of ArcGIS Enterprise or ArcGIS Desktop without actually deploying it onto a virtual machine or my desktop? “
- “I don’t want to use up my organization’s resources with some automation script that might not work.”
- “I want an easy way to do a simple test of my Chef script, I don’t want to have to spin up a whole Virtual machine to test something so simple”.
This article has the proposed answer to these types of questions and scenarios, and it comes in the form of Chef Test Kitchen. Chef Test Kitchen is essentially a Chef utility that can allow you to test Chef scripts/deployments on local virtualization (hypervisors) such as VMware, Hyper-V or VirtualBox. For our article here, we will be primarily focusing on utilization Chef Test Kitchen with VirtualBox with the driver plugin Vagrant (which will provide us our vagrant boxes to upload into VirtualBox).
One of the primary focuses of this article is to provide simplified steps to getting VirtualBox, Vagrant, and Chef all setup and running for you to begin deploying/testing Chef scripts in timely fashion.
Prerequisites:
We will need the following software installed on our desktop to be able to start utilizing Chef Test Kitchen.
NOTE: Follow the defaults for the installations. Keep in mind, VirtualBox will be utilizing Disk Space on your hosting system. If your default drive (C) is low on space, install VirtualBox on a non-default drive with plenty of space.
ChefDK: https://downloads.chef.io/chefdk (4.2.0 version)
VirtualBox: https://www.virtualbox.org/wiki/Downloads (Windows Platform)
-VirtualBox does not work well with Hyper-V. If you have it enabled on your desktop, you can disable it by accessing Windows Features "Turn Windows features on or off" and disable Hyper-V.
Vagrant: https://www.vagrantup.com/downloads.html (Windows 64bit)
Chef Cookbook 3.4.0: https://github.com/Esri/arcgis-cookbook/releases
-Be sure to download the .zip package of the 3.4.0 cookbook.
-Once downloaded, unzip it and place it in folder called C:\Chef (or your non-default drive :\Chef)
ArcGIS Enterprise 10.7.1 Installers and Licenses.
Valid CA .pfx Certificate (be sure to utilize a Fully Qualified Domain Name. The deployment will utilize a host file so you can make it whatever you would like, example: testing.esri.com)
Getting Chef Test Kitchen setup to deploy:
Now that we have our prerequisites squared away and setup, we can now focus on getting our Chef Test Kitchen.yml file configured and making sure our planned Enterprise deployment is proper.
The kitchen.yml file is what Chef Test Kitchen will utilize while deploying your Virtual machine on VirtualBox and it will also deploy your ArcGIS Enterprise. The kitchen.yml file came with the Chef Cookbook we downloaded and unzipped and is located within the arcgis-enterprise cookbook.
-Typical pathing C:\Chef\cookbook\arcgis-enterprise.
1.) When we open this file, we will see information pertaining to our Virtual machine specifications:
-The only specification we would want to update would be the "memory" as a server system hosting an ArcGIS Enterprise would need more memory than what the default presents. For my testing I set it to 12288 (12GB). I attempted 8192 (8GB) as well, but this was met with failure for ArcGIS Enterprise deployments
REMINDER: THIS RAM USAGE IS GOING TO BE COMING FROM YOUR HOSTING SYSTEM. IF YOU HAVE 16 GB OF RAM ON YOUR SYSTEM, SETTING 12GB OF RAM WILL LEAVE YOU WITH 4GB.
2.) We can leave the provisioner defaults as they are.
3.) Platform information we will need to update.
-The default kitchen,yml template will have many Operating Systems listed, but for our test, we will remove all of these besides Windows-2016.
UPDATE: Newly configured vagrant box with Windows 2019 - nickolaitc/Windows2019
-You can create your own VirtualBoxes. This article can help you with this: https://community.esri.com/people/TNickolai-esristaff/blog/2020/02/19/creating-your-own-vagrant-box-...
-We will be utilizing one of my own Vagrant Boxes that I have designed to allow for a proper ArcGIS Enterprise deployment
4.) With suites - driver, we can make a few changes.
- “network”, this will allow us to enable forward porting which will mean that from out own systems hosting VirtualBox, we can access Enterprise via the ports designated.
Example: ["forwarded_port", {guest: 7443, host: 17443}] means on our own hosting systems web browser, to access ArcGIS Portal internal URL (https://localhost:7443/arcgis/home) we would actually need to type https://localhost:17443/arcgis/home and the port forward will push us to the VirtualBox machines port 7443.
-By default, only 6080, 6443, and 7443 are designated. You will need to manually add 443 such as below. You can designate another “host” port, but 1443 is an uncommon port which no other software on my system utilizes.
- “synced_folders” are essentially moving your ArcGIS Enterprise installers/licenses to a VirtualBox repository. I would suggest for the sake of ease, move the ArcGIS Enterprise install archives to one folder, move your licenses to another folder, and your certificate to another folder.
-The left set of folder paths are coming from my hosting system, the right set of folders are synced folders that will get created on the Virtual machines C drive.
5.) We need to update the run_list as the default template is meant to deploy and destroy. Update the run_list to match the following:
6.) Now we need to configure our deployment parameters in which Chef will utilize to install/configure our ArcGIS Enterprise to our Virtual machine.
-The bellow syntax is copied directly from my own kitchen,yml file. Make your own customizations to the necessary fields within your own kitchen.yml such as domain_name, installers, license names, URL’s, usernames and passwords.
attributes:
chef_client:
config:
log_level: ":debug"
arcgis:
version: "10.7.1"
run_as_user: "arcgis"
run_as_password: "Testing123456789!"
configure_windows_firewall: false
hosts:
trevor.test.com: ""
iis:
domain_name: "trevor.test.com"
keystore_file: "C:\\certificates\\kitchentest.pfx"
keystore_password: "test"
server:
admin_username: "siteadmin"
admin_password: "siteadmin"
setup_archive: "C:\\setups\\ArcGIS_Server_Windows_1071_169677.exe"
authorization_file: "C:\\license\\Server_Ent_Adv.ecp"
wa_url: "https://trevor.test.com/server""
url: "https://trevor.test.com:6443/arcgis""
domain_name: "trevor.test.com"
keystore_file: "C:\\certificates\\kitchentest.pfx"
keystore_password: "test"
data_store:
setup_archive: "C:\\setups\\ArcGIS_DataStore_Windows_1071_169689.exe"
types: "relational"
portal:
setup_archive: "C:\\setups\\Portal_for_ArcGIS_Windows_1071_169688.exe"
user_license_type_id: 'creatorUT'
authorization_file: "C:\\license\\AllUTs_AllAddOnApps.json"
admin_username: "siteadmin"
admin_password: "siteadmin1"
admin_email: "tnickolai@test.com"
security_question: "What city were you born in?"
security_question_answer: "test"
wa_url: "https://trevor.test.com/portal""
url: "https://trevor.test.com:7443/arcgis""
domain_name: "trevor.test.com"
keystore_file: "C:\\certificates\\kitchentest.pfx"
keystore_password: "test"
web_adaptor:
setup_archive: "C:\\setups\\Web_Adaptor_for_Microsoft_IIS_1071_169690.exe"
admin_access: true
6.) At this point our kitchen.yml file should be prepared to be utilized. We will want to open our command prompt as administrator (right click on command prompt and run-as administrator).
-We will then want to change our directory to point to our Chef Cookbook which is holding the kitchen.yml file: cd C:\Chef\cookbook\arcgis-enterprise.
-We can now utilize our Chef Test Kitchen commands. We will first want to complete the command: kitchen list. This command will be a simple test to see if the kitchen is successfully being accessed.
-If we get a response looking like this:
-If we get a proper response, we can now move onto creating our Virtual machine with Chef Test Kitchen. We would do this by inputting the following command: kitchen create webgis-windows-2016
-We should see the command prompt providing us details on the building of the Virtual machine. If we encounter any errors, further troubleshooting on the VirtualBox or kitchen.yml file will be needed.
SIDENOTE: If we encounter an error pertaining to the kitchen.yml file not being a proper YML file, this typically is due to syntax/spacing issues.
7.) We should have a Virtual machine with Windows 2016, being hosted by VirtualBox.
-The process can take a few minutes as Chef Test Kitchen is reaching out to Vagrant to import the Vagrant Box we are attempting to utilize (nickolaitc/Windows2016)
-You should be able to open the VirtualBox application and see a Windows Virtual machine.
8.) Our final step is to complete the Chef Test Kitchen converge.
-In command prompt (in the arcgis-enterprise directory) run the following command: kitchen converge webgis-windows-2016
-This will now utilize the Enterprise deployment we had configured in step 6.
-With the run_list setup (step 5) the way it is in the example, we will see that the system is validated first, then IIS is setup/configured, ArcGIS Server will be installed/configured, ArcGIS Server's Web Adaptor will then be installed/configured, Portal for ArcGIS will be installed/configured, ArcGIS Datastore will be installed/configured, Portal for ArcGIS Web Adaptor will be installed/configured, and lastly the Portal for ArcGIS and ArcGIS Server will be federated.
9.) Our end product will be a Virtual machine running Windows 2016 and hosting a full Base Enterprise Deployment.
When you want to access the Virtual machine, open VirtualBox, double click on the Virtual machine we have listed. This should launch a window for you and connect.
-For logging into the system, the Administrator password is: Administrator
We have our Chef Test Kitchen deployment completed at this point and we would be able to login to Virtual machine and check out are Enterprise deployment. We would want to keep in mind that this deployment isn’t for the means of making a production environment, but you could use this environment to test publishing, web application creations, etc. Chef Test Kitchen could go beyond your local system and even be utilized to deploy a testing script into an EC2 instance hosted by Amazon Web Services, however, that is a topic for another document at a later time!
I hope you have found this article pertaining to Chef Test Kitchen insightful and helpful to begin your own Chef script testing with our ArcGIS software!
-Please keep in mind this article is meant to provide understanding and workflows to utilize Chef with our ArcGIS software. If there are questions or issues encountered, please feel free to log a support issue with https://github.com/Esri/arcgis-cookbook/issues and we can provide any support we can!