Webgisdr gets slower every day

[Scott_Tansley]

I have multiple clients that use WebGISDR with consistently good outcomes.  However, I have one that is struggling and I can see no valid reason for this.

We're on a change freeze for ArcGIS Enterprise Content, and there really isn't a lot of content in there at this point.  The backup files are always ~2.5GB which is pretty standard for a newish deployment.

The scenario is this.  I restart the VM to make it nice and fresh.  The webgisdr runs on a schedule at 2AM and takes about 25 minutes.  There are no performance problems during the day.  On Day 2 it runs at 2AM and takes 50 minutes.  On Day 3 it takes nearly 2 hours.  On Day 4 it takes nearly 4 hours and on Day 5 it takes nearly 8 hours.  Remember, no change in content during this five-day period.

The webgisdr logs are showing that both the portal and server exports are taking longer each day, the datastore and packaging completion times remain consistent.  

Transaction logs are being cleared every day.  There is no growth in storage on the E:.  

The client uses a local physical host, which is then virtualised down using VMWare as the hypervisor.  The machines are thinly provisioned for disk, CPU, RAM (8vCPUs and 64GB RAM).  There is a single host with the base deployment on it, and no federated servers.  The ArcGIS application and content is on the E: of the machine.  Webgisdr is configured to write to the local F drive (also SSD).  The components write to F:\Temp and then these files are packaged to F:\compile.  There are no network/UNC paths, and everything is local.  The output file that is created on the F: is then moved to a folder share using a different separate script on a separate scheduled task.  This keeps all operations using local disk only.

Using monitoring software, I see no changes in CPU/RAM characteristics during the backup window.  It just takes longer each day.  CPU/RAM are generally sitting at 20-30% for the duration.

If I run the process manually, then the process monitor is showing disk reads/writes in the Kb not the Mb.  It's shockingly slow.  Even file operations outside of GIS are slow.

When running the installer (MSI) the install was pretty quick.  However, patching (MSP files) take 4-5 times longer than a typical client.  

I've asked for full provisioning and for the malware to be temporarily disabled.  That can't be done for operational reasons.

My gut is saying this is not an Esri issue, but I'm looking to see if anyone else in the community has a similar story or even a fix?

[A_Wyn_Jones]

In your webgisdr properties - which backup mode are you using?

Using "Full" without "Incrementals" will build up a log, although saying that - everytime you run in "Full" it should reset this log.

If you're not already, can you try backup_mode = backup. 

[Scott_Tansley]

Apologies for the delay, we'll put that down to time zones.

Thanks for this response, that was a detail that I'd missed.  Most of my client environments were built at 10.7.1 or 10.8.1 and have been upgraded annually, with webgisdr continuing to work.  Because it formed part of my deployment routines I've maintained the same settings for all clients.  And yes, I've just use 'full' each day, with those logs being blown away each time.  I will retrospectively change that as I engage with each client going forward.

I've made the suggested change and 'just' kicked off the tool.  I'll report back, but just wanted to say thank you for the taking the time to respond.

[Scott_Tansley]

I'm going to keep the 'backup' flag up my sleeve for all future changes in client environments, but it's had no bearing on performance.  I'm sat watching the server operate at a few KB/sec on resource monitor while webgisdr runs.  I'm starting to think it may be in the Hypervisor.

Thanks for pointing me in the right direction on that one.  

I'll continue to try and determine the underlying issues.

[A_Wyn_Jones]

@Scott_Tansley no problem! this sentence is telling;

"When running the installer (MSI) the install was pretty quick.  However, patching (MSP files) take 4-5 times longer than a typical client. "

I have seen this before with aggressively tuned AntiVirus software, even windows defender can cause similar issues if tuned in such a way. 

I can only refer you to the AV guidance here:

https://trust.arcgis.com/en/customer-documents/ArcGIS_Enterprise_AV_Guidance.pdf

and if you look under "Virus tool types", it states that in certain circumstances that you may choose to relax the AV. 

"You must balance your performance expectations against your security needs"

Here's another article on patching best practices:

https://support.esri.com/en-us/knowledge-base/faq-what-are-the-best-practices-when-installing-an-arc...

Thousands of files are being transferred and packaged during the WebGISDr export - an active antivirus with real time monitoring would be a reasonable explaination for the performance hit you're seeing. Would be interesting to compare with your other clients.