Community
Select to view content in your preferred language

ArcGIS Server 11.3: intermittently not avalable

2294
15
10-22-2024 08:05 AM
NicolasGIS
by
Frequent Contributor
‎10-22-2024 08:05 AM

Hello,

Since we upgraded from ArcGIS Server 11.1 to 11.3, I am facing intermittently an issue with ArcGIS Server: the site is temporarily not available.

Here is a typical output of our monitoring probing both:

  •  /arcgis/rest/info/healthCheck
  •  :6443/arcgis/rest/info/healthCheck

NicolasGIS_1-1729609009375.png

If I connect on the VM and try to access manager:

https://localhost:6443/arcgis/manager

I sometimes get a response, sometimes not. Checking the process, everything seems to be in order: ArcGIS Server service started, plently of ArcSOCs processes running. Having a look at the "classic" logs, I dont see anything.

If I check the tomcat logs in "\framework\runtime\tomcat\logs", in "catalina.0", I have the following:

Oct 22, 2024 3:09:15 PM org.apache.tomcat.util.net.Nio2Endpoint$Nio2Acceptor failed
SEVERE: Socket accept failed
java.io.IOException: The specified network name is no longer available
at java.base/sun.nio.ch.Iocp.translateErrorToIOException(Unknown Source)
at java.base/sun.nio.ch.Iocp$EventHandlerTask.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)

Oct 22, 2024 3:47:56 PM org.apache.tomcat.util.net.Nio2Endpoint$Nio2Acceptor failed
SEVERE: Socket accept failed
java.io.IOException: The specified network name is no longer available
at java.base/sun.nio.ch.Iocp.translateErrorToIOException(Unknown Source)
at java.base/sun.nio.ch.Iocp$EventHandlerTask.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)

Oct 22, 2024 3:51:02 PM org.apache.tomcat.util.net.Nio2Endpoint$Nio2Acceptor failed
SEVERE: Socket accept failed
java.io.IOException: The specified network name is no longer available
at java.base/sun.nio.ch.Iocp.translateErrorToIOException(Unknown Source)
at java.base/sun.nio.ch.Iocp$EventHandlerTask.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)

Oct 22, 2024 3:56:59 PM org.apache.tomcat.util.net.Nio2Endpoint$Nio2Acceptor failed
SEVERE: Socket accept failed
java.io.IOException: The specified network name is no longer available
at java.base/sun.nio.ch.Iocp.translateErrorToIOException(Unknown Source)
at java.base/sun.nio.ch.Iocp$EventHandlerTask.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)

Oct 22, 2024 3:59:01 PM org.apache.tomcat.util.net.Nio2Endpoint$Nio2Acceptor failed
SEVERE: Socket accept failed
java.io.IOException: The specified network name is no longer available
at java.base/sun.nio.ch.Iocp.translateErrorToIOException(Unknown Source)
at java.base/sun.nio.ch.Iocp$EventHandlerTask.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)

Oct 22, 2024 4:02:57 PM org.apache.tomcat.util.net.Nio2Endpoint$Nio2Acceptor failed
SEVERE: Socket accept failed
java.io.IOException: The specified network name is no longer available
at java.base/sun.nio.ch.Iocp.translateErrorToIOException(Unknown Source)
at java.base/sun.nio.ch.Iocp$EventHandlerTask.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)

Oct 22, 2024 4:06:01 PM org.apache.tomcat.util.net.Nio2Endpoint$Nio2Acceptor failed
SEVERE: Socket accept failed
java.io.IOException: The specified network name is no longer available
at java.base/sun.nio.ch.Iocp.translateErrorToIOException(Unknown Source)
at java.base/sun.nio.ch.Iocp$EventHandlerTask.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)

Oct 22, 2024 4:16:03 PM org.apache.tomcat.util.net.Nio2Endpoint$Nio2Acceptor failed
SEVERE: Socket accept failed
java.io.IOException: The specified network name is no longer available
at java.base/sun.nio.ch.Iocp.translateErrorToIOException(Unknown Source)
at java.base/sun.nio.ch.Iocp$EventHandlerTask.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)

Oct 22, 2024 4:19:03 PM org.apache.tomcat.util.net.Nio2Endpoint$Nio2Acceptor failed
SEVERE: Socket accept failed
java.io.IOException: The specified network name is no longer available
at java.base/sun.nio.ch.Iocp.translateErrorToIOException(Unknown Source)
at java.base/sun.nio.ch.Iocp$EventHandlerTask.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)

Oct 22, 2024 4:23:09 PM org.apache.tomcat.util.net.Nio2Endpoint$Nio2Acceptor failed
SEVERE: Socket accept failed
java.io.IOException: The specified network name is no longer available
at java.base/sun.nio.ch.Iocp.translateErrorToIOException(Unknown Source)
at java.base/sun.nio.ch.Iocp$EventHandlerTask.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)

Oct 22, 2024 4:44:02 PM org.apache.tomcat.util.net.Nio2Endpoint$Nio2Acceptor failed
SEVERE: Socket accept failed
java.io.IOException: The specified network name is no longer available
at java.base/sun.nio.ch.Iocp.translateErrorToIOException(Unknown Source)
at java.base/sun.nio.ch.Iocp$EventHandlerTask.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)

As you can see, the error time matches the unavailability detected by the monitoring. Any idea, what could be the issue ?

The only solution I found is to restart the ArcGIS Server service and then it is back to normal.

Thanks

0 Kudos
15 Replies
AhmadAwada1
by
Emerging Contributor
‎11-19-2024 12:50 AM

in your file gdb, do you have any tables with huge number of records? in the remaining 10% that you are serving from RDBMS (for ex SQL server), you need to check with ur DBA. In case there is a lot of hits on the services served from SQL server, then you can still experience the issue

0 Kudos
KlintMaliakal3
by Esri Contributor
Esri Contributor
‎11-29-2024 04:20 PM

Hey @NicolasGIS and @AhmadAwada1,

Can you run the following?:

Total IPv4 connections

netstat -an | findstr /R /C:"TCP[ ][ ]*[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*:6443" | find " " /c


Total IPv6 connections:

netstat -an | findstr /R /C:"TCP[ ][ ]*\[[a-f0-9:%%]*\]:6443" | find " " /c

 

What's the total number of those connections on each machine in the site?

Can you check if this webServerMaxRequestThreads property has any improvement to the behavior when the value is greater than the total number of connections?

https://<machine name>:6443/arcgis/admin/system/properties/update

Server properties | ArcGIS REST APIs | ArcGIS Developers

webServerMaxRequestThreads

Introduced at 10.9.1. This property allows users to change the maximum number of threads that are available at one time to process requests sent to ArcGIS Server. The maximum number of threads for ArcGIS Server is 150 by default.

 

Note: Always test in a non-prod environment first.

Kind Regards,
- Klint Maliakal

0 Kudos
WoutervanderHulst
by
Occasional Contributor
‎12-10-2024 07:02 AM

I will also try this change. We have the same symptoms and been investigating the cause for quite some time. We have no insight yet about the connections at the time of issues but we expect that we are limited by default 150 during office hours.

0 Kudos
WoutervanderHulst
by
Occasional Contributor
‎12-10-2024 11:36 PM

Had some trouble applying this settings for all machines in a multi machine site.

The setting was initially only applied to 1 of our 3 machine at the moment it was changed in the Rest interface: https://<machine name1>:6443/arcgis/admin/system/properties/update

The other 2 machines didn't recieve an update for the maxtread update in the tomcat server.xml.

I tried to force an update with an call to the "Synchronize with site" for each of these servers. But then also the setting was not applied to the other 2 machines. I tried a restart of the ArcGIS Server service, but still on 2 machines it was the default setting.

In the end I updated the setting for each machine on rhe Rest Interface. The setting was already visible in the Rest Interface for each machine but wasn't set in Tomcat.

https://<machine name2>:6443/arcgis/admin/system/properties/update and https://<machine name3>:6443/arcgis/admin/system/properties/update

Hereafter each of the machine restarted and the setting was applied.

On another multi site machine also only one of the two machine now has the adjusted setting, although I changed the setting last evening.

The first results of our monitoring shows that at startup of the server already more than 150 TCP connections were active and we already had a spike in te morning to 200 TCP connections. However it is still to early to draw any conclusions yet about improved performance

0 Kudos
WoutervanderHulst
by
Occasional Contributor
‎12-16-2024 02:24 AM

This didn't solve the problem even seems to made te downtimes worse.

0 Kudos
NicolasGIS
by
Frequent Contributor
‎12-11-2024 12:33 PM

On our side, in the end we kept having this SMB error that we thought would be solved by removing the incorrect registered data store.

We found out using "process monitor" that the SMB error was coming from ArcSoc process when starting a new soc instance. We searched everywhere for this path on ArcGIS Manager and Admin interface: could not find any mention of it.

Then we checked the ArcGIS Project used for publishing and found out that this path was referenced as broken in the "Folders" list section as on the image below. We republished each service. SMB errors went away. No issue since then (10 days). To be followed.

Broken path in ArcGIS Pro folders projectBroken path in ArcGIS Pro folders project

Conclusions:

- I did not think that these paths matter in the ArcGIS Pro project for publishing

- The issue is still unclear to me: SMB error could occur without having necessarily a crash.

- Maybe because almost all our services (50) were published with this broken path, if several of these broken services start spinning up ArcSOC processes, it could lead to it ?!

0 Kudos