Hi,
We currently have a federated ArcGIS Enterprise deployed from DEV - PROD. We noticed timeouts/slow performance to load Portal homepage. When this occurred it impacted all three environments and we aren't sure what could have caused that issues. The data lives in NAS for ArcGIS server directors, config and portal content. When investigating we found that there weren't any issues with NAS at the time. Below are the logs from ArcGIS Portal during the issue:
URL 'https://geopvis.prodlb.travp.net/arcgis/rest/services/Claim/US_CLM_AUTO_PROP_TERRITORY/MapServer/exp...' is not accessible: Timeout. Read timed out.
URL 'https://geopvis.prodlb.travp.net/arcgis/rest/services/Claim/US_CLM_AUTO_PROP_TERRITORY/MapServer/exp...' is not accessible: Error. java.io.EOFException: Unexpected EOF read on the socket.
URL 'https://geopvis.prodlb.travp.net/arcgis/rest/services/Claim/US_CLM_AUTO_PROP_TERRITORY/MapServer/leg...' is not accessible: Timeout. Read timed out.
URL 'https://geopvis.prodlb.travp.net/arcgis/admin/data/findItems?managed=true&f=json&token=Sz-nmbTxxS_uz....' is not accessible: Timeout. Read timed out.
URL 'https://geopgpr.prodlb.travp.net/arcgis/admin/data/findItems?managed=true&f=json&token=Sz-nmbTxxS_uz....' is not accessible: Timeout. Read timed out.
java.lang.RuntimeException: java.lang.Exception: Exception: Read timed out at com.esri.gw.publish.AgsServicePublisher.getServersManagedEgdb(AgsServicePublisher.java:743) at com.esri.gw.account.GWAccounts.a(GWAccounts.java:2346) at com.esri.gw.account.GWAccounts.getAccountProperties(GWAccounts.java:2460) at com.esri.gw.account.AccountPropertiesHandler.getAccountProperties(AccountPropertiesHandler.java:60) at com.esri.gw.account.AccountsRequestHandler.G(AccountsRequestHandler.java:4769) at com.esri.gw.account.AccountsRequestHandler.service(AccountsRequestHandler.java:1430) at com.esri.gw.DispatchServlet.service(DispatchServlet.java:161) at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:126) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.esri.gw.filters.PortalFilter.doFilter(PortalFilter.java:80) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:176) at org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:145) at org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:92) at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:394) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.esri.gw.filters.JSONFilter.doFilter(JSONFilter.java:132) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.esri.gw.filters.GZipFilter.doFilter(GZipFilter.java:55) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.esri.gw.filters.SecurityFilter.doFilter(SecurityFilter.java:95) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.esri.gw.filters.ErrorFilter.doFilter(ErrorFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.esri.gw.filters.TraceFilter.doFilter(TraceFilter.java:104) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.esri.gw.filters.ProxyRequestCaptureFilter.doFilter(ProxyRequestCaptureFilter.java:126) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.esri.gw.filters.CharEncodingFilter.doFilter(CharEncodingFilter.java:32) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:200) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:607) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) at com.esri.arcgis.portal.util.TomcatValve.invoke(TomcatValve.java:43) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1415) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.lang.Exception: Exception: Read timed out at com.esri.gw.util.PooledHttpClientUtil.a(PooledHttpClientUtil.java:222) at com.esri.gw.util.PooledHttpClientUtil.postPropsToURL(PooledHttpClientUtil.java:180) at com.esri.gw.publish.AgsServicePublisher.sendAgsRestRequest(AgsServicePublisher.java:2628) at com.esri.gw.publish.AgsServicePublisher.sendAgsRestRequest(AgsServicePublisher.java:2607) at com.esri.gw.publish.AgsServicePublisher.getServersManagedEgdb(AgsServicePublisher.java:726) ... 62 more
From ArcGIS Server:
Any thoughts? This only occurred for 15 minutes and self corrected after. I checked the transaction as well and that wasn't any surprising since ArcGIS Server has handled similar transactions before.
