topic Re: ArcGIS Server 11.3: intermittently not avalable in ArcGIS Enterprise Questions

ArcGIS Server 11.3: intermittently not avalable

NicolasGIS — Tue, 22 Oct 2024 15:06:00 GMT

Hello,

Since we upgraded from ArcGIS Server 11.1 to 11.3, I am facing intermittently an issue with ArcGIS Server: the site is temporarily not available.

Here is a typical output of our monitoring probing both:

/arcgis/rest/info/healthCheck
:6443/arcgis/rest/info/healthCheck

If I connect on the VM and try to access manager:

https://localhost:6443/arcgis/manager

I sometimes get a response, sometimes not. Checking the process, everything seems to be in order: ArcGIS Server service started, plently of ArcSOCs processes running. Having a look at the "classic" logs, I dont see anything.

If I check the tomcat logs in "\framework\runtime\tomcat\logs", in "catalina.0", I have the following:

Oct 22, 2024 3:09:15 PM org.apache.tomcat.util.net.Nio2Endpoint$Nio2Acceptor failed
SEVERE: Socket accept failed
java.io.IOException: The specified network name is no longer available
at java.base/sun.nio.ch.Iocp.translateErrorToIOException(Unknown Source)
at java.base/sun.nio.ch.Iocp$EventHandlerTask.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Oct 22, 2024 3:47:56 PM org.apache.tomcat.util.net.Nio2Endpoint$Nio2Acceptor failed
SEVERE: Socket accept failed
java.io.IOException: The specified network name is no longer available
at java.base/sun.nio.ch.Iocp.translateErrorToIOException(Unknown Source)
at java.base/sun.nio.ch.Iocp$EventHandlerTask.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Oct 22, 2024 3:51:02 PM org.apache.tomcat.util.net.Nio2Endpoint$Nio2Acceptor failed
SEVERE: Socket accept failed
java.io.IOException: The specified network name is no longer available
at java.base/sun.nio.ch.Iocp.translateErrorToIOException(Unknown Source)
at java.base/sun.nio.ch.Iocp$EventHandlerTask.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Oct 22, 2024 3:56:59 PM org.apache.tomcat.util.net.Nio2Endpoint$Nio2Acceptor failed
SEVERE: Socket accept failed
java.io.IOException: The specified network name is no longer available
at java.base/sun.nio.ch.Iocp.translateErrorToIOException(Unknown Source)
at java.base/sun.nio.ch.Iocp$EventHandlerTask.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Oct 22, 2024 3:59:01 PM org.apache.tomcat.util.net.Nio2Endpoint$Nio2Acceptor failed
SEVERE: Socket accept failed
java.io.IOException: The specified network name is no longer available
at java.base/sun.nio.ch.Iocp.translateErrorToIOException(Unknown Source)
at java.base/sun.nio.ch.Iocp$EventHandlerTask.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Oct 22, 2024 4:02:57 PM org.apache.tomcat.util.net.Nio2Endpoint$Nio2Acceptor failed
SEVERE: Socket accept failed
java.io.IOException: The specified network name is no longer available
at java.base/sun.nio.ch.Iocp.translateErrorToIOException(Unknown Source)
at java.base/sun.nio.ch.Iocp$EventHandlerTask.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Oct 22, 2024 4:06:01 PM org.apache.tomcat.util.net.Nio2Endpoint$Nio2Acceptor failed
SEVERE: Socket accept failed
java.io.IOException: The specified network name is no longer available
at java.base/sun.nio.ch.Iocp.translateErrorToIOException(Unknown Source)
at java.base/sun.nio.ch.Iocp$EventHandlerTask.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Oct 22, 2024 4:16:03 PM org.apache.tomcat.util.net.Nio2Endpoint$Nio2Acceptor failed
SEVERE: Socket accept failed
java.io.IOException: The specified network name is no longer available
at java.base/sun.nio.ch.Iocp.translateErrorToIOException(Unknown Source)
at java.base/sun.nio.ch.Iocp$EventHandlerTask.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Oct 22, 2024 4:19:03 PM org.apache.tomcat.util.net.Nio2Endpoint$Nio2Acceptor failed
SEVERE: Socket accept failed
java.io.IOException: The specified network name is no longer available
at java.base/sun.nio.ch.Iocp.translateErrorToIOException(Unknown Source)
at java.base/sun.nio.ch.Iocp$EventHandlerTask.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Oct 22, 2024 4:23:09 PM org.apache.tomcat.util.net.Nio2Endpoint$Nio2Acceptor failed
SEVERE: Socket accept failed
java.io.IOException: The specified network name is no longer available
at java.base/sun.nio.ch.Iocp.translateErrorToIOException(Unknown Source)
at java.base/sun.nio.ch.Iocp$EventHandlerTask.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Oct 22, 2024 4:44:02 PM org.apache.tomcat.util.net.Nio2Endpoint$Nio2Acceptor failed
SEVERE: Socket accept failed
java.io.IOException: The specified network name is no longer available
at java.base/sun.nio.ch.Iocp.translateErrorToIOException(Unknown Source)
at java.base/sun.nio.ch.Iocp$EventHandlerTask.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)

As you can see, the error time matches the unavailability detected by the monitoring. Any idea, what could be the issue ?

The only solution I found is to restart the ArcGIS Server service and then it is back to normal.

Thanks

Re: ArcGIS Server 11.3: intermittently not avalable

AhmadAwada1 — Tue, 12 Nov 2024 13:10:55 GMT

Hi Nicolas, did u figured out the issue?? I am facing the same and it is killing 😐

Re: ArcGIS Server 11.3: intermittently not avalable

NicolasGIS — Tue, 12 Nov 2024 13:26:50 GMT

Hi @AhmadAwada1,

I am still working on it but the infrastructure has been more stable (no issue in last 10 days) since I deleted a broken “data store” on ArcGIS Server which was pointing to a broken SMB file system path. I found it by analysing windows event viewer which logged SMB error.
Will keep you updated if the issue arises again.

Re: ArcGIS Server 11.3: intermittently not avalable

NicolasGIS — Tue, 12 Nov 2024 15:31:04 GMT

Hi @AhmadAwada1,

Just an update: the issue occured again... so the incorrect registered data store was not the culprit.

Out of curiosity, what version are you running ? Do you observe the same intermittent phenomena ? Same errors in the logs ?

Thanks

Re: ArcGIS Server 11.3: intermittently not avalable

NicolasGIS — Tue, 12 Nov 2024 15:56:15 GMT

Just an update: to me it seems more related to the activity of the GIS Server site.

Today was interesting: intermittently down from 10 to 12 and then from 2 to 4:30. We can clearly see the lunch break in my organization !

/cc @CedricDespierreCorporon

Re: ArcGIS Server 11.3: intermittently not avalable

AhmadAwada1 — Tue, 12 Nov 2024 16:30:37 GMT

i am experiencing the same with version 11.1. In my case there is a lot of workload on the gis server (200+ arcsoc.exe). In your case, too much workload? how many ArcSOCs?

Re: ArcGIS Server 11.3: intermittently not avalable

NicolasGIS — Fri, 15 Nov 2024 06:09:19 GMT

Well in my case it is about 100 ArcSocs per VM (2) but they have lots of RAM (60 GB).

When inspecting the activity manually, VMs do not seem overloaded CPU or RAM wise so I am not sure it is the issue... just trying to correlate the fact that during lunch time (less activity), there was no issue.

Do you have a monitoring of your CPU or RAM by any chance ? I personally don't. I wonder if it could not correlate with a CPU activity spikes.

Re: ArcGIS Server 11.3: intermittently not avalable

NicolasGIS — Tue, 19 Nov 2024 08:21:54 GMT

Hello @AhmadAwada1 ,

Out of curiosity, do you also have this kind of error 'localhost.0.log' in C:\Program Files\Arcgis\Server\framework\runtime\tomcat\logs:

WARNING: The web application [arcgis#rest] appears to have started a thread named [Thread-216] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: java.base@17.0.10/jdk.internal.misc.Unsafe.park(Native Method) java.base@17.0.10/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) java.base@17.0.10/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source) java.base@17.0.10/java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source) com.esri.arcgis.discovery.logging.Logger$c.run(Logger$c.java:555)

I have plenty of those after reboot when my site was unavailable.

The issue start occuring on my side after upgrade from 11.1 to 11.3. What did it start for you ?

Thanks !

Re: ArcGIS Server 11.3: intermittently not avalable

AhmadAwada1 — Tue, 19 Nov 2024 08:38:13 GMT

Hi Nicolas,

Yes i have this error. Did you ever try to check the database health? I believe the issue in my case was a database bottleneck and when i increased the DB resources and optimized my expensive queries, the errors disappeared and the system become stable. Let me know if you need further help.

Re: ArcGIS Server 11.3: intermittently not avalable

NicolasGIS — Tue, 19 Nov 2024 08:46:19 GMT

Hi Ahmad,

Thanks for your quick reply ! Much appreciated.

Could you please expand on what did you increase on the DB so that I can cross check with DBA ?

But in my case, 90% of my services are served from a local file geodatabase at the root of the server (C:\).

Not sure much can be optimized but will investigate this lead.

Thanks !

Re: ArcGIS Server 11.3: intermittently not avalable

AhmadAwada1 — Tue, 19 Nov 2024 08:50:18 GMT

in your file gdb, do you have any tables with huge number of records? in the remaining 10% that you are serving from RDBMS (for ex SQL server), you need to check with ur DBA. In case there is a lot of hits on the services served from SQL server, then you can still experience the issue

Re: ArcGIS Server 11.3: intermittently not avalable

KlintMaliakal3 — Sat, 30 Nov 2024 00:22:52 GMT

Hey @NicolasGIS and @AhmadAwada1,

Can you run the following?:

Total IPv4 connections

netstat -an | findstr /R /C:"TCP[ ][ ]*[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*:6443" | find " " /c

Total IPv6 connections:

netstat -an | findstr /R /C:"TCP[ ][ ]*\[[a-f0-9:%%]*\]:6443" | find " " /c

What's the total number of those connections on each machine in the site?

Can you check if this webServerMaxRequestThreads property has any improvement to the behavior when the value is greater than the total number of connections?

https://<machine name>:6443/arcgis/admin/system/properties/update

Server properties | ArcGIS REST APIs | ArcGIS Developers

webServerMaxRequestThreads

Introduced at 10.9.1. This property allows users to change the maximum number of threads that are available at one time to process requests sent to ArcGIS Server. The maximum number of threads for ArcGIS Server is 150 by default.

Note: Always test in a non-prod environment first.

Kind Regards,
- Klint Maliakal

Re: ArcGIS Server 11.3: intermittently not avalable

WoutervanderHulst — Tue, 10 Dec 2024 15:02:57 GMT

I will also try this change. We have the same symptoms and been investigating the cause for quite some time. We have no insight yet about the connections at the time of issues but we expect that we are limited by default 150 during office hours.

Re: ArcGIS Server 11.3: intermittently not avalable

WoutervanderHulst — Wed, 11 Dec 2024 07:36:04 GMT

Had some trouble applying this settings for all machines in a multi machine site.

The setting was initially only applied to 1 of our 3 machine at the moment it was changed in the Rest interface: https://<machine name1>:6443/arcgis/admin/system/properties/update

The other 2 machines didn't recieve an update for the maxtread update in the tomcat server.xml.

I tried to force an update with an call to the "Synchronize with site" for each of these servers. But then also the setting was not applied to the other 2 machines. I tried a restart of the ArcGIS Server service, but still on 2 machines it was the default setting.

In the end I updated the setting for each machine on rhe Rest Interface. The setting was already visible in the Rest Interface for each machine but wasn't set in Tomcat.

https://<machine name2>:6443/arcgis/admin/system/properties/update and https://<machine name3>:6443/arcgis/admin/system/properties/update

Hereafter each of the machine restarted and the setting was applied.

On another multi site machine also only one of the two machine now has the adjusted setting, although I changed the setting last evening.

The first results of our monitoring shows that at startup of the server already more than 150 TCP connections were active and we already had a spike in te morning to 200 TCP connections. However it is still to early to draw any conclusions yet about improved performance

Re: ArcGIS Server 11.3: intermittently not avalable

NicolasGIS — Wed, 11 Dec 2024 20:33:22 GMT

On our side, in the end we kept having this SMB error that we thought would be solved by removing the incorrect registered data store.

We found out using "process monitor" that the SMB error was coming from ArcSoc process when starting a new soc instance. We searched everywhere for this path on ArcGIS Manager and Admin interface: could not find any mention of it.

Then we checked the ArcGIS Project used for publishing and found out that this path was referenced as broken in the "Folders" list section as on the image below. We republished each service. SMB errors went away. No issue since then (10 days). To be followed.

Broken path in ArcGIS Pro folders project

Conclusions:

- I did not think that these paths matter in the ArcGIS Pro project for publishing

- The issue is still unclear to me: SMB error could occur without having necessarily a crash.

- Maybe because almost all our services (50) were published with this broken path, if several of these broken services start spinning up ArcSOC processes, it could lead to it ?!

Re: ArcGIS Server 11.3: intermittently not avalable

WoutervanderHulst — Mon, 16 Dec 2024 10:24:15 GMT

This didn't solve the problem even seems to made te downtimes worse.