Timing of VM backups for ArcGIS Enterprise

Question

Hi Jonathan Quinn

We have an ArcGIS Enterprise 10.6.1. environment. I have powershell scripts that do a full webgisdr backup on Sundays and incremental backups every other day.

The VM servers are also backed up everyday using Commvault as part of our organisations enterprise backup policy.

My first restore strategy would be to restore from the webgisdr tool given the interdependency between Portal, Server and the Data Store. However, if the webgisdr  restore process failed for some reason I would then try and restore from VM. I understand that a restore from VM can be problematic due to the timing of the backups . I notice you wrote in a previous GeoNet question

"Some organizations rely on VM backups, but they have to be timed correctly with file server backups so you capture the data of the entire deployment at a point in time. If you take backups at different times, you'll have differences in the dependencies between Portal, Server, and Data Store which will cause issues when you attempt to restore the backups"

So I contacted our backup provider to check whether or not VM backups all occur at the same time. The response I got was that the backups are scheduled to all start at the same time however because of the way Commvault performs resource management it cant be guaranteed that all of the backups will start at the same time. This is done so as to not flood the VM environment. Essentially they are queued and run through in sequence.

My question is how close together do the backups need to be performed. Is within the same hour ok ? (this is out of business hours and no updates to the system should be taking place). Or do the backups all need to be done at the precisely the same instant?

Thanks,

Clinton

Todd_Metzler · Answer

Suggest syncing your scheduled webgisdr backup closely after your service recycle schedule has completed and before the commvault backup.

Suggested backup plan flow: Snapshot VM, recycle services, archive webgisdr, snapshot VM again, Commvault backup.

Now, with only one data point on a physical machine deployment of ArcGIS Enterprise 10.4.1 base deployment a few years ago, a catastrophic hardware failure killed our server.  After fixing the hardware, we did reach the point of system restore from commvault.  That didn't work.  Never identified root cause but our theory was the commvault changed file backup logic didn't identify an ArcGIS Enterprise critical file(s) as changed and that file(s) wasn't in the restored backup.  Again only one data point.  Not enough for an informed decision.

ClintonBallandis1 · Answer

Hi Jonathan,

Thanks for the reply.

To answer your question no the webgisdr backups haven't failed. I just wanted to clarify the VM snapshot timing encase I had to ever resort to it .

Clinton

JonathanQuinn · Answer

If no one is using the system, then your backup schedule, or rather the scheduling of VM snapshots, won't be a problem. Even if someone created an item during that time and it wasn't captured within a snapshot of one of the components, the deployment would still work after you restore the snapshots. You'll just have to delete/recreate that content.

Just to be clear, you're not saying your restore attempt failed, correct? In the event that it did, then you'd resort to VM snapshots?