Chris, hoping I'm not too late here in giving you a little more details on our system. We essentially, did the same thing as you folks. Had an On Prem Server (no Portal/Enterprise) at 10.3.1. Was incredibly stable and responsive even with about 30 - 35 arcsoc.exe services running at baseline default. Moved and upgraded to 10.6.1 Full Enterprise deployment within the Azure platform. I kept the same baseline services running and found terrible if not unmanageable performance as this thread has exposed.
What tipped me off was the print service not working from a custom app. It would fail after 30 seconds or so and the pop up which should have been a pdf was a 400 or 500 IIS error saying something about permissions accessing the file etc. Then popping over to Server Manager I would see all the services in the stopping stage until I restarted the server service from windows services.
The first thing I did that would start to relive said behavior was to reduce the number of services running from my Server. These are NOT the hosted services that I spun up from Pro by copying data to the Data store of Portal i.e. Hosted Service. These were the services that had source data in the registered gdb and had 2 - 4 services running at all times. after reviewing usage of services, I had clearly over provisioned the always running services and brought them all down to 1 always running and allowing up to 3 to spin up. This alone has stabilized our system.
Some other strange occurrences with our system that I'm working on: What confused me during our troubleshooting was that memory and CPU were never being slammed. The system was pretty much always showing enough resources that we should have been able to have lots of services running, more than we have. Turns out when our system was first deployed in Azure our consultant has added a 2 machine setup using the availability set in Azure with a load balancer. They had then removed this 2nd machine from the GIS Enterprise and Azure. HOWEVER there is still network traffic going to this old server (no longer exists) when a service using enterprise gdb resisted data is called. This does not happen when we call HOSTED services. Clearly there is something in our Azure config that is putting extra load (even just network) on the system which is causing services to fail and print behavior to fail when there are more resources.
That's my book, hope its an ok read. Please free to contact me via inbox or whatever and we can hook up outside of geonet for a more in depth discussion. Hoping I can help out. Boy was i stressed when this was happening on my new deployment/platform.
And maybe someone reading has insight from the Azure side for me?
transparent.png "Kathleen_Crombez"){kind=avatar}