Arcgis Server Service takes more than 5hours to come up

[NileshValwaikar]

Hello All,

we are having Arcgis enterprise 11.0 setup done by ESRI team on AWS. Its a Multi Machine Single Site. Its been more than a year this setup was done and we have continously experienced performance issue but the issue that we have observered in past 1 month is that if we reboot the hosting server or restart its arcserver service than it takes more than 5 hours for the server to resume normal task.  We also have an Image server which resumes within 5mins on reboot or service restart.

Upon checking the hosting server logs i found only the below entries continously...

<Msg time="2023-11-27T20:40:12,221" type="SEVERE" code="6598" source="Admin" process="12966" thread="1" methodName="" machine="OYSARCSERVER.OYSTERMAP.COM" user="" elapsed="" requestID="a49a4600-49a0-4b00-bfe1-fea778ab0a98">Failed to get the status of the server machine 'OYSARCSERVER.OYSTERMAP.COM'. Could not connect to the ArcGIS component at URL 'https://oysarcserver.oystermap.com:6443/arcgis/admin/local/isAppServerStarted'. The ArcGIS component on that machine may not be running or the machine may not be reachable at this time.Error: connect timed out</Msg>

This is only the 2nd time in this month that we have faced this issue of long time for service to resume.

When checked the service status in terminal it shows the below...

arcgisserver.service - ArcGIS Server Service
Loaded: loaded (/etc/systemd/system/arcgisserver.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Mon 2023-11-27 21:37:07 IST; 58min ago
Process: 14817 ExecStart=/home/arcgis/server/startserver.sh (code=exited, status=0/SUCCESS)
Process: 14934 ExecStop=/home/arcgis/server/stopserver.sh (code=exited, status=0/SUCCESS)

Nov 27 21:36:50 oysarcserver.oystermap.com systemd[1]: Starting ArcGIS Server Service...
Nov 27 21:36:51 oysarcserver.oystermap.com startserver.sh[14820]: Attempting to start ArcGIS Server...
Nov 27 21:36:51 oysarcserver.oystermap.com startserver.sh[14820]: *** Notice: There are still server processes running. Please run
Nov 27 21:36:51 oysarcserver.oystermap.com startserver.sh[14820]: stopserver.sh to kill these processes and run startserver.sh again.
Nov 27 21:36:51 oysarcserver.oystermap.com startserver.sh[14820]: [ OK ]
Nov 27 21:37:07 oysarcserver.oystermap.com stopserver.sh[14937]: Attempting to stop ArcGIS Server...
Nov 27 21:37:07 oysarcserver.oystermap.com stopserver.sh[14937]: [ OK ]
Nov 27 21:37:07 oysarcserver.oystermap.com systemd[1]: arcgisserver.service: Succeeded.
Nov 27 21:37:07 oysarcserver.oystermap.com systemd[1]: Started ArcGIS Server Service.

After multiple tries of Reboot or Service restart i somehow manage to get the Service started msg but the arcsoc process does not show up... 

After Half Hour First ArcSOC process was visible and even after 1.5 hours the count of ArcSOC process is still just 50 whereas in normal conditions the count is 164 as we have large amount or Published Services. Last time when this same issue occurred it took 5.5 hours for the server to resume. 

if anyone of you have faced this kind of issue or know any possible root cause and how to remedy it than it would be of great help to me.

Thanks in Advance

NileshV

[gis_grow]

Hello Nilesh have you try to delete the service and publish it from scratch? You could publish the same service with other name and try the restart time with this new service.

Could you share the capability of the server (core, ram, etc) and you have the possibility monitor this element during  restart?

[Nilesh_Valwaikar]

Today again we have faced the same issue. The issue is not with a particular service as such. We also have an Image server which also has almost 125 arcsoc processes running, these all comes up within 2 minutes of service retstart.

whereas after arc server service restart the first arcsoc process takes upto 10 to 15 mins to come up and than almost 4 to 5 hours  for all the arcsoc (which is almost 200+) to come up.

also one more thing i found is that i always had to run the below command to start turning up the arcsoc processes.

https://FQDN:6443/arcgis/admin/local/start

does anyone has any thoughts on this peculiar behaviour of ArcServer services..

[Nilesh_Valwaikar]

Also i would like to add some more information...

please see the below snapshot, the moment i have any of the arcsoc coming up, i also get few items in the status of arcserver services stating as below...

├─5115 /home/arcgis/server/framework/runtime/ArcGIS/bin/arcsoc -xy:-UsePerfData -xy:-CreateMinidumpOnCrash -Xmx64M -xy:+UseParalelGC -Dservice=Viewshed.GPServer -Djava.class.

when my count of arcsoc increases these line items in  the service status also increases. I am not able to understand the root cause of these issues as there is nothing much in the server log too as the log contains only the below errors...

<Msg time="2024-12-09T20:49:03,438" type="SEVERE" code="6598" source="Admin" process="4390" thread="1" methodName="" machine="OYSARCSERVER.OYSTERMAP.COM" user="" elapsed="" requestID="a0a6f7b9-e1ce-40eb-b9c4-7b754770f71b">Failed to get the status of the server machine 'OYSARCSERVER.OYSTERMAP.COM'. Could not connect to the ArcGIS component at URL 'https://oysarcserver.oystermap.com:6443/arcgis/admin/local/isAppServerStarted'. The ArcGIS component on that machine may not be running or the machine may not be reachable at this time.Error: connect timed out</Msg>