Slow Performance/Time Out Issue for ArcGIS Portal & Server

[BidhanAdhikari]

Hi,

We currently have a federated ArcGIS Enterprise deployed from DEV - PROD. We noticed timeouts/slow performance to load Portal homepage. When this occurred it impacted all three environments and we aren't sure what could have caused that issues. The data lives in NAS for ArcGIS server directors, config and portal content. When investigating we found that there weren't any issues with NAS at the time. Below are the logs from ArcGIS Portal during the issue:

URL 'https://geopvis.prodlb.travp.net/arcgis/rest/services/Claim/US_CLM_AUTO_PROP_TERRITORY/MapServer/exp...' is not accessible: Timeout. Read timed out.

URL 'https://geopvis.prodlb.travp.net/arcgis/rest/services/Claim/US_CLM_AUTO_PROP_TERRITORY/MapServer/exp...' is not accessible: Error. java.io.EOFException: Unexpected EOF read on the socket.

URL 'https://geopvis.prodlb.travp.net/arcgis/rest/services/Claim/US_CLM_AUTO_PROP_TERRITORY/MapServer/leg...' is not accessible: Timeout. Read timed out.

URL 'https://geopvis.prodlb.travp.net/arcgis/admin/data/findItems?managed=true&f=json&token=Sz-nmbTxxS_uz....' is not accessible: Timeout. Read timed out.

URL 'https://geopgpr.prodlb.travp.net/arcgis/admin/data/findItems?managed=true&f=json&token=Sz-nmbTxxS_uz....' is not accessible: Timeout. Read timed out.

java.lang.RuntimeException: java.lang.Exception: Exception: Read timed out at com.esri.gw.publish.AgsServicePublisher.getServersManagedEgdb(AgsServicePublisher.java:743) at com.esri.gw.account.GWAccounts.a(GWAccounts.java:2346) at com.esri.gw.account.GWAccounts.getAccountProperties(GWAccounts.java:2460) at com.esri.gw.account.AccountPropertiesHandler.getAccountProperties(AccountPropertiesHandler.java:60) at com.esri.gw.account.AccountsRequestHandler.G(AccountsRequestHandler.java:4769) at com.esri.gw.account.AccountsRequestHandler.service(AccountsRequestHandler.java:1430) at com.esri.gw.DispatchServlet.service(DispatchServlet.java:161) at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:126) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.esri.gw.filters.PortalFilter.doFilter(PortalFilter.java:80) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:176) at org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:145) at org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:92) at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:394) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.esri.gw.filters.JSONFilter.doFilter(JSONFilter.java:132) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.esri.gw.filters.GZipFilter.doFilter(GZipFilter.java:55) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.esri.gw.filters.SecurityFilter.doFilter(SecurityFilter.java:95) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.esri.gw.filters.ErrorFilter.doFilter(ErrorFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.esri.gw.filters.TraceFilter.doFilter(TraceFilter.java:104) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.esri.gw.filters.ProxyRequestCaptureFilter.doFilter(ProxyRequestCaptureFilter.java:126) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.esri.gw.filters.CharEncodingFilter.doFilter(CharEncodingFilter.java:32) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:200) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:607) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) at com.esri.arcgis.portal.util.TomcatValve.invoke(TomcatValve.java:43) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1415) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.lang.Exception: Exception: Read timed out at com.esri.gw.util.PooledHttpClientUtil.a(PooledHttpClientUtil.java:222) at com.esri.gw.util.PooledHttpClientUtil.postPropsToURL(PooledHttpClientUtil.java:180) at com.esri.gw.publish.AgsServicePublisher.sendAgsRestRequest(AgsServicePublisher.java:2628) at com.esri.gw.publish.AgsServicePublisher.sendAgsRestRequest(AgsServicePublisher.java:2607) at com.esri.gw.publish.AgsServicePublisher.getServersManagedEgdb(AgsServicePublisher.java:726) ... 62 more

From ArcGIS Server:

The operation timed out (less) (WinINet Error while using HTTPS security, 12002), URL = https://ttgpvis1.prod.travp.net:6443/arcgis/admin/generateToken

Any thoughts? This only occurred for 15 minutes and self corrected after. I checked the transaction as well and that wasn't any surprising since ArcGIS Server has handled similar transactions before. 

[JayateerthDeshpande]

We are also facing same issue however key differences are:

1. ArcGIS Enterprise 10.9.1, all available patches except patches for Utility Network are applied

2. Entire System is on Azure VMs and Config store / ArcGIS /Portal Directories on Azure File share.

Is there a solution anyone found out? Or is it that the HA system on Windows environment is known to experirience such issues when ArcGIS Enterprise is configured in HA mode.

[GianniCampanile2]

Hi,

we have the same problem and trying to solve it my modifying the keepalive parameter of the tomcat server but with no success up to now.

Did you manage to solve this issue ?

Thanks

Regards

Gianni