Slow Performance/Time Out Issue for ArcGIS Portal & Server

BidhanAdhikari · ‎02-16-2022

Hi,

We currently have a federated ArcGIS Enterprise deployed from DEV - PROD. We noticed timeouts/slow performance to load Portal homepage. When this occurred it impacted all three environments and we aren't sure what could have caused that issues. The data lives in NAS for ArcGIS server directors, config and portal content. When investigating we found that there weren't any issues with NAS at the time. Below are the logs from ArcGIS Portal during the issue:

URL 'https://geopvis.prodlb.travp.net/arcgis/rest/services/Claim/US_CLM_AUTO_PROP_TERRITORY/MapServer/exp...' is not accessible: Timeout. Read timed out.

URL 'https://geopvis.prodlb.travp.net/arcgis/rest/services/Claim/US_CLM_AUTO_PROP_TERRITORY/MapServer/exp...' is not accessible: Error. java.io.EOFException: Unexpected EOF read on the socket.

URL 'https://geopvis.prodlb.travp.net/arcgis/rest/services/Claim/US_CLM_AUTO_PROP_TERRITORY/MapServer/leg...' is not accessible: Timeout. Read timed out.

URL 'https://geopvis.prodlb.travp.net/arcgis/admin/data/findItems?managed=true&f=json&token=Sz-nmbTxxS_uz....' is not accessible: Timeout. Read timed out.

URL 'https://geopgpr.prodlb.travp.net/arcgis/admin/data/findItems?managed=true&f=json&token=Sz-nmbTxxS_uz....' is not accessible: Timeout. Read timed out.

java.lang.RuntimeException: java.lang.Exception: Exception: Read timed out at com.esri.gw.publish.AgsServicePublisher.getServersManagedEgdb(AgsServicePublisher.java:743) at com.esri.gw.account.GWAccounts.a(GWAccounts.java:2346) at com.esri.gw.account.GWAccounts.getAccountProperties(GWAccounts.java:2460) at com.esri.gw.account.AccountPropertiesHandler.getAccountProperties(AccountPropertiesHandler.java:60) at com.esri.gw.account.AccountsRequestHandler.G(AccountsRequestHandler.java:4769) at com.esri.gw.account.AccountsRequestHandler.service(AccountsRequestHandler.java:1430) at com.esri.gw.DispatchServlet.service(DispatchServlet.java:161) at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:126) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.esri.gw.filters.PortalFilter.doFilter(PortalFilter.java:80) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:176) at org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:145) at org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:92) at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:394) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.esri.gw.filters.JSONFilter.doFilter(JSONFilter.java:132) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.esri.gw.filters.GZipFilter.doFilter(GZipFilter.java:55) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.esri.gw.filters.SecurityFilter.doFilter(SecurityFilter.java:95) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.esri.gw.filters.ErrorFilter.doFilter(ErrorFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.esri.gw.filters.TraceFilter.doFilter(TraceFilter.java:104) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.esri.gw.filters.ProxyRequestCaptureFilter.doFilter(ProxyRequestCaptureFilter.java:126) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.esri.gw.filters.CharEncodingFilter.doFilter(CharEncodingFilter.java:32) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:200) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:607) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) at com.esri.arcgis.portal.util.TomcatValve.invoke(TomcatValve.java:43) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1415) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.lang.Exception: Exception: Read timed out at com.esri.gw.util.PooledHttpClientUtil.a(PooledHttpClientUtil.java:222) at com.esri.gw.util.PooledHttpClientUtil.postPropsToURL(PooledHttpClientUtil.java:180) at com.esri.gw.publish.AgsServicePublisher.sendAgsRestRequest(AgsServicePublisher.java:2628) at com.esri.gw.publish.AgsServicePublisher.sendAgsRestRequest(AgsServicePublisher.java:2607) at com.esri.gw.publish.AgsServicePublisher.getServersManagedEgdb(AgsServicePublisher.java:726) ... 62 more

From ArcGIS Server:

The operation timed out (less) (WinINet Error while using HTTPS security, 12002), URL = https://ttgpvis1.prod.travp.net:6443/arcgis/admin/generateToken

Any thoughts? This only occurred for 15 minutes and self corrected after. I checked the transaction as well and that wasn't any surprising since ArcGIS Server has handled similar transactions before.