Hello All,
we are having Arcgis enterprise 11.0 setup done by ESRI team on AWS. Its a Multi Machine Single Site. Its been more than a year this setup was done and we have continously experienced performance issue but the issue that we have observered in past 1 month is that if we reboot the hosting server or restart its arcserver service than it takes more than 5 hours for the server to resume normal task. We also have an Image server which resumes within 5mins on reboot or service restart.
Upon checking the hosting server logs i found only the below entries continously...
<Msg time="2023-11-27T20:40:12,221" type="SEVERE" code="6598" source="Admin" process="12966" thread="1" methodName="" machine="OYSARCSERVER.OYSTERMAP.COM" user="" elapsed="" requestID="a49a4600-49a0-4b00-bfe1-fea778ab0a98">Failed to get the status of the server machine 'OYSARCSERVER.OYSTERMAP.COM'. Could not connect to the ArcGIS component at URL 'https://oysarcserver.oystermap.com:6443/arcgis/admin/local/isAppServerStarted'. The ArcGIS component on that machine may not be running or the machine may not be reachable at this time.Error: connect timed out</Msg>
This is only the 2nd time in this month that we have faced this issue of long time for service to resume.
When checked the service status in terminal it shows the below...
arcgisserver.service - ArcGIS Server Service
Loaded: loaded (/etc/systemd/system/arcgisserver.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Mon 2023-11-27 21:37:07 IST; 58min ago
Process: 14817 ExecStart=/home/arcgis/server/startserver.sh (code=exited, status=0/SUCCESS)
Process: 14934 ExecStop=/home/arcgis/server/stopserver.sh (code=exited, status=0/SUCCESS)
Nov 27 21:36:50 oysarcserver.oystermap.com systemd[1]: Starting ArcGIS Server Service...
Nov 27 21:36:51 oysarcserver.oystermap.com startserver.sh[14820]: Attempting to start ArcGIS Server...
Nov 27 21:36:51 oysarcserver.oystermap.com startserver.sh[14820]: *** Notice: There are still server processes running. Please run
Nov 27 21:36:51 oysarcserver.oystermap.com startserver.sh[14820]: stopserver.sh to kill these processes and run startserver.sh again.
Nov 27 21:36:51 oysarcserver.oystermap.com startserver.sh[14820]: [ OK ]
Nov 27 21:37:07 oysarcserver.oystermap.com stopserver.sh[14937]: Attempting to stop ArcGIS Server...
Nov 27 21:37:07 oysarcserver.oystermap.com stopserver.sh[14937]: [ OK ]
Nov 27 21:37:07 oysarcserver.oystermap.com systemd[1]: arcgisserver.service: Succeeded.
Nov 27 21:37:07 oysarcserver.oystermap.com systemd[1]: Started ArcGIS Server Service.
After multiple tries of Reboot or Service restart i somehow manage to get the Service started msg but the arcsoc process does not show up...
After Half Hour First ArcSOC process was visible and even after 1.5 hours the count of ArcSOC process is still just 50 whereas in normal conditions the count is 164 as we have large amount or Published Services. Last time when this same issue occurred it took 5.5 hours for the server to resume.
if anyone of you have faced this kind of issue or know any possible root cause and how to remedy it than it would be of great help to me.
Thanks in Advance
NileshV
