Best practice to migrate ArcGIS Server seamlessly while upgrading / migrating from physical Windows Server to newer VM Server?

[RexRobichaux3]

Hello GeoNet, 

 We are beginning to plan an impending upgrade to upgrade our physical Windows Server 2008 R2 boxes to Windows Server 2016 VMs (About time right?). Currently, we have ArcGIS Server 10.4.1 installed (no Portal currently) which hosts a variety of services for consumption in a custom web application developed by a consultant. We have this environment on dev (ArcGIS Developer sub), Test (Enterprise staging license), and Prod (Enterprise prod license) tiers that will all be migrated. Additionally, server names will change between as-is and to-be environments.

The long-term roadmap will be to migrate the current servers from 2008R2 < 2016, stay at 10.4.1 for the upgrade, and then begin to plan our 10.7.1 ArcGIS Enterprise upgrade to include ArcGIS Portal deployment however I'm most concerned with the 10.4.1 Server migration at this time.

At this point, I've struggled to find definitive best practice guidance on the prescribed method to migrate a single-server ArcGIS site when the Windows OS is undergoing a similar upgrade. I've found the following recources, but all seem to have an achilles heel or issue when looking at our deployment / requirements:

• Join Site method: Migrate to a new machine in ArcGIS Enterprise  
  • I believe I read this can only work when all ArcGIS Servers are ont he same level of OS- is this correct?
• WebGIS DR method: Migrate to a new machine in ArcGIS Enterprise using the WebGIS DR tool 
  • Will this work with a standalone ArcGIS Server deployment with no Enterprise / Portal?
• Fresh install on the new Windows Server VM, and migrate over all contents and directories (anyone have a good workflow and list of required directories to achieve this, and would this work with a changing server name?)
• Any other methods I've missed? I found this somewhat similar topic and thinking it might apply to our situation but wanted to confirm: Migrate to a new Server 

 Sorry for all of the questions, and I appriciate any insight Enterprise / ArcGIS Server gurus Jonathan Quinn or others may have to offer on this!

Many thanks!

-Rex

[ScottFierro2]

Rex,

I think you will find there are several ways to accomplish this and it becomes more about preference, level of trust and speed. We manage around 70 servers covering 6 major projects and supplemental pieces like ArcGIS Monitor. Coming from a government perspective where rigid documentation for auditors is required, our level of trust is moderate and our speed can slide to the right a bit to ensure extra thoroughness. As a result we never upgrade in place for OS or ESRI version but for various reasons we upgrade about every 18-24 months keeping all servers within 1 release of each other (i.e. we went from 10.3/10.3.1 to 10.5/10.5.1 and are finalizing our 10.7/10.7.1 migrations now).

What's good about our lift and shift, build it over from scratch each time method is the isolation and constant re-evaluation of all aspects of the environments from database, to servers, to network, etc. keeps us continuously evaluating how we do things and looking for efficiencies or better (new) best practices. What's bad is it does initially have a learning curve and add time to projects because the repetition of routine things like setting static routes or firewall rules, etc. isn't there. Once it is there it still likely takes a little longer than some other options but not much.

As for your questions:

1) Join Site - I can't say I've ever tried to use the Join Site option to leverage a network shared store for hot swapping servers at different OS levels. We've had servers go down with issues and spun up brand new one's to hot swap into an ArcServer site but the machines always had the same OS

2) WebGIS DR - This is a valid but as discussed more complicated process although it's one I don't think you'd be up against. I believe the real sticking point behind a lot of this is the need to maintain a consistent Web Context URL for your ends users between the old and new environments. This is specific to Portal configurations and without Portal in your existing environment this is likely overkill. Based on your networking expertise or available network staff the routing of traffic from your old environment to the new one should be about as easy as flipping alight switch. If you have an F5 BigIP or something similar this is a breeze and if not it can be done via DNS entries. Either way you can establish your entire new environment behind anything you want and then simply redirect traffic in 15 minutes during off hours. If your current environment is behind https://abc.myorgdomain/ then you simply build everything under https://abcnew.myorgdomian/ and can perform all your migration and testing of the new environment behind that "new" route then when it's time to switch the "new" route gets deleted and the https://abc.myorgdomain/ gets re-directed to send traffic to your fresh migrated environment.

3) Fresh install - ties into what I said we do and sort of bottom of the #2 response. New server names won't matter it's really just about managing your traffic. By content and directories I am not sure if you are referring to ESRI software components (i.e. config-store & directories) or your own application and other file needs. As a best practice for us, all our VM's have a designated storage drive (we use D:/ internally or E:/ in Azure IaaS boxes). That drive is where we store all our apps, Oracle and SQL client installs, direct ArcSever logs, and re-direct our IIS websites to. It ensures separation of system files (C:/) and our projects specific needs/installs (storage drive) across any server making it easier to troubleshoot and have people cross-trained to support. For your own files you are sort of on your own to identify what goes or gets dropped but depending on how your vendor built that app you would want to determine or reach back out to them and ask whether you have any critical file structures or hard coded entries which must be preserved, goes not just for files but URL's. As for the ESRI pieces there's likely options for migrating the ArcServer directories and config-store folders since your initial move is staying at the same 10.4.1 version. Never had a need to do this mirrored lift and shift before so can't point you to a solution there but know there'd be steps for sure due to entries in the config-store. This is where we bite the bullet on time building from scratch. Anything we publish gets stored on a network drive in it's original MXD, Toolbox, etc. and from that we create and also store the .sd drafts for each. This allows for rapid re-publishing if we decide to delete a service for any reason but then when we do these migrations we can quickly and easily update things to use new databases (we generally push SQL version upgrades during migrations) with new SDE versions, and publish everything from scratch into the new environment. Eliminates the need to carry over the ArcServer files but also helps identify any issues with configurations, networking, publishing, etc.

4) Not seen that forum post before or fact it was a year ago I've just forgotten it. So this really speaks to an area of dissent and opinion around Portal, in particular federation. I won't go into my various gripes about it as they are long but leave it at, in environments our size it's only going to be used where/when it absolutely must be used and stay contained within that particular projects/environments space. We have several project environments doing exactly what you are doing with regards to a custom app and I'd say it's completely acceptable to do using an un-federated environment. I do like the scripted idea Andrew laid out and may take a look more into it although we often make changes to some parts of this so goes back to our slow and methodical steps and documentation allow for consistency and full review.

[AngusHooper1]

Great post Scott.

In my experience, the hiccup with these workflows is if your environment is fundamentally leveraging the federation & hosting configurations. This is particularly apparent when you have a system that has been in production for some time and you are dealing with old design niggles. As it currently stands, I cannot see how you could possibly migrate a GIS that has a hosting server from one environment to another without an outage. You would need to set the hosting GIS to be in read-only mode as you run a chance of orphaning objects/data edits during the migration proces (e.g. webgisdr). As your migration mechanism (e.g. webgisdr) creates a point-in-time backup of the GIS, no further changes can be made to the live GIS. If you only have a GIS services and registered database requirement, then you are probably fine as these can be static for a short period of time.

With regards to old design niggles, something that is quite frustrating is how old server URL's can be baked into almost everything in the GIS. You could potentially get around this with DNS redirect logic, but I would suggest that this is a bandaid and you'll have to comprehensively edit items in the GIS via the python API to point to new URL's. In an ideal world, all your servers have been sitting behind a DNS or load balancer which would help you with the 'swap out' model of Join Site. This is potentially where my team will be moving to, but the first item to tick off is remediating legacy content that references FQDN's. You'll need to make sure that the federated server admin url is pointing through a DNS/load balancer for similar reasons.

Any comments/advice for the above? Happy to be proven wrong. It's a topic that many in the GIS community are trying to solve it seems.

[ScottFierro2]

Angus, I agree that hosted configs are by far the hardest but federated servers also don't make it easy. You are correct you would need an outage period and we perform those sorts of activities on off hours for those reasons. I was looking at the possible options Rex Robichaux‌ originally listed and considering what/how they would play into what he was trying to do. Since he doesn't currently have Portal it's just an ArcServer migration and both hosting and federation aren't at play, so how did those options fit his needs. You are correct that in those scenarios you are performing the same sort of live/transactional data steps that DBA's perform and as a result you must account for that.

Exactly, I've been behind load balancers since the 10.0 days but think they would have made some of my early frustrations with ArcServer in the 9.x days better. I'd highly suggest getting to that point if you are able but agree you will likely have your work cut out trying to re-route things. Currently, we are behind the F5 BigIP load balancer and as I referenced before, it makes life night and day easier and why we tend to rebuild a new each time so we can leave current systems running for usage while we fully vet all aspects of the new environment and then flip the switch when ready. As you mentioned this isn't as straightforward once hosting is involved. We've avoided hosting for several reasons mostly tied to authoritative data and data governance concerns but also because for the few activities we'd address with that sort of solution we leverage our ArcGIS Online environment. Then for Portal we will embed an entire portal server as part of a single environment silo, i.e. projects a and c are independent, unfederated vertical stacks of arcservers and web adaptors with project b having portal, federated arcservers and web adaptor but noting from a or c will ever touch project b's portal. We have such an array of needs that trying to utilize a single organization wide Portal that all our arcserver environments must go through would be crippling because you'd be pushing everything to upgrade together. Our decisions for this also tie into this conundrum because Portal uses the singular Web Context URI which means you must embed all your projects behind it and rely on web adaptor naming to address the ease of the user end having a simple, logical, easy to remember URL to get at the various things they need. Another concern for us is that organization wide usage of Portal induces a single point of failure and thus localizing portal to each projects needs ensures only system 1 goes down and not all of them.

Hosting aside, I have been considering some upcoming testing of federation to see how far we can stretch things by using iRules and URI's against Portal. The idea being that we establish an outward facing URL on the F5 load balancer and then use the iRules to establish 2 separate Portals with independent set Web Context URL's that we can hot swap behind the load balancer. This in theory is feasible and we've done it with other systems just never tried it with Portal and becomes a question around how the applications (Portal/ArcServer) handle the traffic loop across the URI's. Again, if we can get that done then in theory you could migrate machines with a minimal flip of the switch through redirects for all non-hosted environments.

[JonathanQuinn]

Great feedback from Scott Fierro‌. I'll comment on the questions:

1) Join Site does work between operating system versions. It briefly states that in the bug, but we can certainly call it out better:

Perhaps you’re experiencing hardware performance issues, or you want to run a newer operating system.

2) The DR tool requires Portal, and as Scott Fierro‌ indicates, the approach is more complex than the first option. The use of the etc\hosts file can get around some of the requirements of the tool, (for example that the front-end URLs are the same). That way you don't have to worry about DNS and how traffic is routed, nor how to migrate content between deployments that have different URLs. This is described in the Etc\hosts file section:

Etc\hosts file

A quick word on the use of the etc\hosts file, located under C:\Windows\System32\drivers\etc on Windows and the \etc folder on Linux: it’s used to resolve hostnames by entering IP addresses and associating the IP addresses to hostnames.

For example, if my machine name has an IP address of 10.0.0.1 and is resolved to enterprise.domain.com via DNS, I can add an entry to the etc\hosts file on the machine to resolve the machine’s IP address to a different hostname:

10.0.0.1 alias.domain.com

If I ping alias.domain.com in a command prompt the etc\hosts file is resolving alias.domain.com to 10.0.0.1, which is enterprise.domain.com. If I had a web server running on the machine, I can reach the web server via alias.domain.com or enterprise.domain.com.

This approach is slightly more complicated than migrating using the Join Site workflow as the steps are dependent on your deployment type and requires modifying the etc\hosts file to manage hostname resolution.

 This approach is useful if you need to maintain availability during the migration as you can stand up that separate environment without causing downtime for your production environment.

Honestly, I wouldn't suggest setting up a new system with new URLs and migrating content over. Assuming you'll be using the same URLs are previously, this is how I'd foresee this going:

1)  Set up your new environment using new URLs

2) Use the Python API, specifically the ability to migrate content between environments, to move stuff from your old environment to your new environment. Hosted services would need to be republished.

3) Since there's no easy way to update the URLs of existing content, you'll need to set up another new environment that uses the same original URLs.

4) Move everything back to the original environment to use the correct URLs

If you don't intend on using the same URLs, (which is unlikely as you're simply looking to migrate to new machines), then you wouldn't have to worry about 3 and 4.

[RexRobichaux3]

Wow- lots of great conversation and input here. Thank you all for contributing. If I'm following this properly, it sounds like with our current (relatively simplistic AGS deployment), it might be worth trying the Join Site route? If that doesn't work out, it sounds like Jonathan's suggestion (last 4 steps) might be worth trying- did I interpret that correctly?

I've not ever setup a multi-machine site (even temporarily), so I'm assuming the secondary / newer ArcGIS Server that gets added to the site shares the same Server Enterprise License as the original machine? Just want to ensure we have our licensing ducks in a row as well ahead of time. Thanks again everyone! 

[ScottFierro2]

I think Join Site can work for your scenario and wouldn't foresee it causing you issues. Again, for trust reasons I'd suggest after joining the site and before pulling your original server out of the site that you go directly against the new server to do some checks that all services are in place, gp's are working, can run queries, etc. After confirming your new server is performing as expected then drop the old one out of the site.

I believe you would need a second license for the new server because you will need the license during installation of your new server where it will create the new keycodes file (C:\Program Files\ESRI\License10.7\sysgen) local to the new machine. While the licenses are CPU driven, I can say I've never tried, but I don't believe a single license that supports 4 cores could be used on your old server against 2 cores and then also used on your new server against 2 cores. Something that Jonathan Quinn could maybe answer but I know many of the ESRI staff who will wave the white flag when it comes to licensing issues and point you towards your local rep.

[JonathanQuinn]

I definitely stay away from licensing if I can, but I'm pretty sure a 4 core license can be used on a 4 core machine or two 2 core machines.

Ultimately, talking to your account manager is the best way to go. Personally, I think migration workflows (joinSite or the DR tool), should be treated like a failover environment. You can acquire a failover license at no extra cost, as no requests are going to that failover environment. This is somewhat similar, as long as you don't run in that temporary migration status for long, I think you should be able to use the same license but again, contact your account manager for further clarification.

[BrianMacPherson]

Hi Rex,

I recently completed a GIS server migration from 2008 to 2016 under similar circumstances to yours.

We had no published content and restoring the site was not a concern, so I decided to go with a clean 10.6.1 install of everything on a new VM.

It was pretty straight forward for the most part, and I documented the process in the attached notes if you're interested.

Brian

[JayateerthDeshpande]

Some great Discussions here indeed! With the Sunset of Win2008r2, almost everyone facing these questions! Will post my experiences, soon, once we complete our migration.