ArcGIS Server 11.3: intermittently not avalable

NicolasGIS · ‎10-22-2024

Hello,

Since we upgraded from ArcGIS Server 11.1 to 11.3, I am facing intermittently an issue with ArcGIS Server: the site is temporarily not available.

Here is a typical output of our monitoring probing both:

/arcgis/rest/info/healthCheck
:6443/arcgis/rest/info/healthCheck

If I connect on the VM and try to access manager:

https://localhost:6443/arcgis/manager

I sometimes get a response, sometimes not. Checking the process, everything seems to be in order: ArcGIS Server service started, plently of ArcSOCs processes running. Having a look at the "classic" logs, I dont see anything.

If I check the tomcat logs in "\framework\runtime\tomcat\logs", in "catalina.0", I have the following:

Oct 22, 2024 3:09:15 PM org.apache.tomcat.util.net.Nio2Endpoint$Nio2Acceptor failed
SEVERE: Socket accept failed
java.io.IOException: The specified network name is no longer available
at java.base/sun.nio.ch.Iocp.translateErrorToIOException(Unknown Source)
at java.base/sun.nio.ch.Iocp$EventHandlerTask.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Oct 22, 2024 3:47:56 PM org.apache.tomcat.util.net.Nio2Endpoint$Nio2Acceptor failed
SEVERE: Socket accept failed
java.io.IOException: The specified network name is no longer available
at java.base/sun.nio.ch.Iocp.translateErrorToIOException(Unknown Source)
at java.base/sun.nio.ch.Iocp$EventHandlerTask.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Oct 22, 2024 3:51:02 PM org.apache.tomcat.util.net.Nio2Endpoint$Nio2Acceptor failed
SEVERE: Socket accept failed
java.io.IOException: The specified network name is no longer available
at java.base/sun.nio.ch.Iocp.translateErrorToIOException(Unknown Source)
at java.base/sun.nio.ch.Iocp$EventHandlerTask.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Oct 22, 2024 3:56:59 PM org.apache.tomcat.util.net.Nio2Endpoint$Nio2Acceptor failed
SEVERE: Socket accept failed
java.io.IOException: The specified network name is no longer available
at java.base/sun.nio.ch.Iocp.translateErrorToIOException(Unknown Source)
at java.base/sun.nio.ch.Iocp$EventHandlerTask.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Oct 22, 2024 3:59:01 PM org.apache.tomcat.util.net.Nio2Endpoint$Nio2Acceptor failed
SEVERE: Socket accept failed
java.io.IOException: The specified network name is no longer available
at java.base/sun.nio.ch.Iocp.translateErrorToIOException(Unknown Source)
at java.base/sun.nio.ch.Iocp$EventHandlerTask.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Oct 22, 2024 4:02:57 PM org.apache.tomcat.util.net.Nio2Endpoint$Nio2Acceptor failed
SEVERE: Socket accept failed
java.io.IOException: The specified network name is no longer available
at java.base/sun.nio.ch.Iocp.translateErrorToIOException(Unknown Source)
at java.base/sun.nio.ch.Iocp$EventHandlerTask.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Oct 22, 2024 4:06:01 PM org.apache.tomcat.util.net.Nio2Endpoint$Nio2Acceptor failed
SEVERE: Socket accept failed
java.io.IOException: The specified network name is no longer available
at java.base/sun.nio.ch.Iocp.translateErrorToIOException(Unknown Source)
at java.base/sun.nio.ch.Iocp$EventHandlerTask.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Oct 22, 2024 4:16:03 PM org.apache.tomcat.util.net.Nio2Endpoint$Nio2Acceptor failed
SEVERE: Socket accept failed
java.io.IOException: The specified network name is no longer available
at java.base/sun.nio.ch.Iocp.translateErrorToIOException(Unknown Source)
at java.base/sun.nio.ch.Iocp$EventHandlerTask.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Oct 22, 2024 4:19:03 PM org.apache.tomcat.util.net.Nio2Endpoint$Nio2Acceptor failed
SEVERE: Socket accept failed
java.io.IOException: The specified network name is no longer available
at java.base/sun.nio.ch.Iocp.translateErrorToIOException(Unknown Source)
at java.base/sun.nio.ch.Iocp$EventHandlerTask.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Oct 22, 2024 4:23:09 PM org.apache.tomcat.util.net.Nio2Endpoint$Nio2Acceptor failed
SEVERE: Socket accept failed
java.io.IOException: The specified network name is no longer available
at java.base/sun.nio.ch.Iocp.translateErrorToIOException(Unknown Source)
at java.base/sun.nio.ch.Iocp$EventHandlerTask.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Oct 22, 2024 4:44:02 PM org.apache.tomcat.util.net.Nio2Endpoint$Nio2Acceptor failed
SEVERE: Socket accept failed
java.io.IOException: The specified network name is no longer available
at java.base/sun.nio.ch.Iocp.translateErrorToIOException(Unknown Source)
at java.base/sun.nio.ch.Iocp$EventHandlerTask.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)

As you can see, the error time matches the unavailability detected by the monitoring. Any idea, what could be the issue ?

The only solution I found is to restart the ArcGIS Server service and then it is back to normal.

Thanks

AhmadAwada1 · ‎11-12-2024

Hi Nicolas, did u figured out the issue?? I am facing the same and it is killing 😐

NicolasGIS · ‎11-12-2024

Hi @AhmadAwada1,

I am still working on it but the infrastructure has been more stable (no issue in last 10 days) since I deleted a broken “data store” on ArcGIS Server which was pointing to a broken SMB file system path. I found it by analysing windows event viewer which logged SMB error.
Will keep you updated if the issue arises again.

NicolasGIS · ‎11-12-2024

Hi @AhmadAwada1,

Just an update: the issue occured again... so the incorrect registered data store was not the culprit.

Out of curiosity, what version are you running ? Do you observe the same intermittent phenomena ? Same errors in the logs ?

Thanks

NicolasGIS · ‎11-12-2024

Just an update: to me it seems more related to the activity of the GIS Server site.

Today was interesting: intermittently down from 10 to 12 and then from 2 to 4:30. We can clearly see the lunch break in my organization !

/cc @CedricDespierreCorporon

AhmadAwada1 · ‎11-12-2024

i am experiencing the same with version 11.1. In my case there is a lot of workload on the gis server (200+ arcsoc.exe). In your case, too much workload? how many ArcSOCs?

NicolasGIS · ‎11-14-2024

Well in my case it is about 100 ArcSocs per VM (2) but they have lots of RAM (60 GB).

When inspecting the activity manually, VMs do not seem overloaded CPU or RAM wise so I am not sure it is the issue... just trying to correlate the fact that during lunch time (less activity), there was no issue.

Do you have a monitoring of your CPU or RAM by any chance ? I personally don't. I wonder if it could not correlate with a CPU activity spikes.

NicolasGIS · ‎11-19-2024

Hello @AhmadAwada1 ,

Out of curiosity, do you also have this kind of error 'localhost.0.log' in C:\Program Files\Arcgis\Server\framework\runtime\tomcat\logs:

WARNING: The web application [arcgis#rest] appears to have started a thread named [Thread-216] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@17.0.10/jdk.internal.misc.Unsafe.park(Native Method)
 java.base@17.0.10/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
 java.base@17.0.10/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
 java.base@17.0.10/java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source)
 com.esri.arcgis.discovery.logging.Logger$c.run(Logger$c.java:555)

I have plenty of those after reboot when my site was unavailable.

The issue start occuring on my side after upgrade from 11.1 to 11.3. What did it start for you ?

Thanks !

AhmadAwada1 · ‎11-19-2024

Hi Nicolas,

Yes i have this error. Did you ever try to check the database health? I believe the issue in my case was a database bottleneck and when i increased the DB resources and optimized my expensive queries, the errors disappeared and the system become stable. Let me know if you need further help.

NicolasGIS · ‎11-19-2024

Hi Ahmad,

Thanks for your quick reply ! Much appreciated.

Could you please expand on what did you increase on the DB so that I can cross check with DBA ?

But in my case, 90% of my services are served from a local file geodatabase at the root of the server (C:\).

Not sure much can be optimized but will investigate this lead.

Thanks !