geoPortal down after successfully publishing map service

Question

I successfully published map service containing 60K points. It does not contain attachments. Then use it for viewing in 1 device only for 5 minutes.

Suddenly, the server crashed. The CPU hits 98%.

I have a server with 2 physical cores (4 logical), 16 GB RAM and 500 GB HDD running on AWS. It host Arcgis Server and Portal on the same machine

I have a total of 2 feature services running with dedicated instances (8 maximum instances in total)  and 6 map services running with shared instances. So,  8 (2+6) services are in use with 8 in total of maximum instances.

Also I have around 10 hosted map services but not in use.

I have also tested by using Shared instances for Feature Service that is heavily used. I only see 7 instances are created (with 128 MB Heap memory) in the statistics page. But, eventually after 5 days, this configuration lead to crash as well.

Questions :

1.  By looking at the feature Service that was created using Shared Instance, can I conclude that the maximum number of instances that the system is created is maximum 7 ?

2. Do 10 hosted map services create burden on the system eventhough not being used ?

3. Why ordinary Map Service crashed ? Is it because it doesn't have any serving instance (the number of instances created on the server already maxed out) ?

4. What is possibly down, Arcgis Server or Arcgis Portal ? or both are down ?

Thanks for any clues

ryanuthoff · Answer

I suspect you've already discovered the root cause of the issue. Your CPU maxes out directly before your server crashes. My recommendation is that you increase the CPU on the machine. In my experience with working with Esri support, that is one of the first things they check and if they see that the resources are maxed out, then they will tell you to increase the resources. If you're not able to increase the resources, then I'd remove or scale back as much as your existing services as possible.

yockee · Answer

The server constantly crashed out after I build tiling service twice (first tiling is only 3GB, second one is 1,5GB) ,  then I ditch out the tiling service and using map service hoping that the server does not crash. But, it crashed again after being used for a couple of minutes for 1 client only. Prior to 3 straight crashes, the CPU hit 98% mark.
So I am scared in crashing the server again cos its in the production. LoL

ryanuthoff · Answer

Does it consistently crash every time after using the map service for a couple of minutes? It could have just been a coincidence.