geoPortal down after successfully publishing map service

[yockee]

I successfully published map service containing 60K points. It does not contain attachments. Then use it for viewing in 1 device only for 5 minutes.

Suddenly, the server crashed. The CPU hits 98%.

I have a server with 2 physical cores (4 logical), 16 GB RAM and 500 GB HDD running on AWS. It host Arcgis Server and Portal on the same machine

I have a total of 2 feature services running with dedicated instances (8 maximum instances in total)  and 6 map services running with shared instances. So,  8 (2+6) services are in use with 8 in total of maximum instances.

Also I have around 10 hosted map services but not in use. 

I have also tested by using Shared instances for Feature Service that is heavily used. I only see 7 instances are created (with 128 MB Heap memory) in the statistics page. But, eventually after 5 days, this configuration lead to crash as well.

Questions :

1.  By looking at the feature Service that was created using Shared Instance, can I conclude that the maximum number of instances that the system is created is maximum 7 ?

2. Do 10 hosted map services create burden on the system eventhough not being used ?

3. Why ordinary Map Service crashed ? Is it because it doesn't have any serving instance (the number of instances created on the server already maxed out) ?

4. What is possibly down, Arcgis Server or Arcgis Portal ? or both are down ? 

Thanks for any clues

[RyanUthoff]

Every ArcGIS Enterprise environment is different, so it's hard to answer your questions without being familiar with your environment. What could cause the server to crash for you, might not cause it to crash for someone else. It depends on resources of the machine as well as the services that are published to the server. These questions would be best answered by your GIS Server Administrator or your IT department. However, I will try to answer your questions.

I don't understand your first question. But each instance of a feature service is ran using an "ArcSOC.exe" Windows process. In a command prompt, you can type in "tasklist | find "ArcSOC.exe" /c" to find the number of feature services actively running on your server.

Hosted feature services are typically lightweight, so I wouldn't expect 10 map services to create a burden on the system.

I don't understand your third question. But if your entire server crashed, then your ordinary map services will crash as well. Maxing out the number of instances a map service can use typically will not cause the service to crash. The end user will just have to wait until one of the instances becomes available.

It's hard to answer what is down without being in your environment. However, if Portal is still accessible but the map services are not accessible, then that means only your ArcGIS Server is down. If Portal is not accessible, then that means Portal is down.

Ultimately, it sounds like you are maxing out the resources on your machine, especially if the server is crashing when the CPU is maxed out. I've been told by Esri that they consider a machine "at capacity" when resources (CPU/RAM) hit 80%. If you're at 98%, you most certainly need to scale up the resources on your server. Alternatively, you can scale down the individual map services on your server in an attempt to decrease the resource usage.

[yockee]

for the third question, I meant that after successfully publishing the map service and using it for a couple of minutes, the server crashed. What is causing the crash ? I always think that the map service is very lightweight.

[RyanUthoff]

Does it consistently crash every time after using the map service for a couple of minutes? It could have just been a coincidence.

[yockee]

The server constantly crashed out after I build tiling service twice (first tiling is only 3GB, second one is 1,5GB) ,  then I ditch out the tiling service and using map service hoping that the server does not crash. But, it crashed again after being used for a couple of minutes for 1 client only. Prior to 3 straight crashes, the CPU hit 98% mark.
So I am scared in crashing the server again cos its in the production. LoL

[RyanUthoff]

I suspect you've already discovered the root cause of the issue. Your CPU maxes out directly before your server crashes. My recommendation is that you increase the CPU on the machine. In my experience with working with Esri support, that is one of the first things they check and if they see that the resources are maxed out, then they will tell you to increase the resources. If you're not able to increase the resources, then I'd remove or scale back as much as your existing services as possible.

[TanuHoque]

@yockee 

I'd strongly recommend you to reach out to Esri Support to help you with this. I can't agree more to what @RyanUthoff said -- without knowing your environment etc., trying to troubleshoot be like shooting dart in the dark. Support analysts are equipped with tools needed to help you.

Thank you.