Slow Performance/Time Out Issue for ArcGIS Portal & Server

BidhanAdhikari — Wed, 16 Feb 2022 16:54:15 GMT

Hi,

We currently have a federated ArcGIS Enterprise deployed from DEV - PROD. We noticed timeouts/slow performance to load Portal homepage. When this occurred it impacted all three environments and we aren't sure what could have caused that issues. The data lives in NAS for ArcGIS server directors, config and portal content. When investigating we found that there weren't any issues with NAS at the time. Below are the logs from ArcGIS Portal during the issue:

URL 'https://geopvis.prodlb.travp.net/arcgis/rest/services/Claim/US_CLM_AUTO_PROP_TERRITORY/MapServer/export' is not accessible: Timeout. Read timed out.

URL 'https://geopvis.prodlb.travp.net/arcgis/rest/services/Claim/US_CLM_AUTO_PROP_TERRITORY/MapServer/export' is not accessible: Error. java.io.EOFException: Unexpected EOF read on the socket.

URL 'https://geopvis.prodlb.travp.net/arcgis/rest/services/Claim/US_CLM_AUTO_PROP_TERRITORY/MapServer/legend' is not accessible: Timeout. Read timed out.

URL 'https://geopvis.prodlb.travp.net/arcgis/admin/data/findItems?managed=true&f=json&token=Sz-nmbTxxS_uzUpLP-LTb0kduyUVp5BlZDoFf0R9hYLie0EoIwbAbxMXa1CmTeHAKeKt4mczVa6LToJNoOROZr0HSJHL02vHujSkmuADZHn2I5ILTjp9jrdDINWimtLgOu3BEtqrSpFeylnZBL7rD3DixDval_Sy6ImfxUc9wfM.' is not accessible: Timeout. Read timed out.

URL 'https://geopgpr.prodlb.travp.net/arcgis/admin/data/findItems?managed=true&f=json&token=Sz-nmbTxxS_uzUpLP-LTb0kduyUVp5BlZDoFf0R9hYLie0EoIwbAbxMXa1CmTeHAKeKt4mczVa6LToJNoOROZr0HSJHL02vHujSkmuADZHn2I5ILTjp9jrdDINWimtLgOu3BEtqrSpFeylnZBL7rD3DixDval_Sy6ImfxUc9wfM.' is not accessible: Timeout. Read timed out.

java.lang.RuntimeException: java.lang.Exception: Exception: Read timed out at com.esri.gw.publish.AgsServicePublisher.getServersManagedEgdb(AgsServicePublisher.java:743) at com.esri.gw.account.GWAccounts.a(GWAccounts.java:2346) at com.esri.gw.account.GWAccounts.getAccountProperties(GWAccounts.java:2460) at com.esri.gw.account.AccountPropertiesHandler.getAccountProperties(AccountPropertiesHandler.java:60) at com.esri.gw.account.AccountsRequestHandler.G(AccountsRequestHandler.java:4769) at com.esri.gw.account.AccountsRequestHandler.service(AccountsRequestHandler.java:1430) at com.esri.gw.DispatchServlet.service(DispatchServlet.java:161) at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:126) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.esri.gw.filters.PortalFilter.doFilter(PortalFilter.java:80) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:176) at org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:145) at org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:92) at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:394) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.esri.gw.filters.JSONFilter.doFilter(JSONFilter.java:132) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.esri.gw.filters.GZipFilter.doFilter(GZipFilter.java:55) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.esri.gw.filters.SecurityFilter.doFilter(SecurityFilter.java:95) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.esri.gw.filters.ErrorFilter.doFilter(ErrorFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.esri.gw.filters.TraceFilter.doFilter(TraceFilter.java:104) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.esri.gw.filters.ProxyRequestCaptureFilter.doFilter(ProxyRequestCaptureFilter.java:126) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.esri.gw.filters.CharEncodingFilter.doFilter(CharEncodingFilter.java:32) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:200) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:607) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) at com.esri.arcgis.portal.util.TomcatValve.invoke(TomcatValve.java:43) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1415) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.lang.Exception: Exception: Read timed out at com.esri.gw.util.PooledHttpClientUtil.a(PooledHttpClientUtil.java:222) at com.esri.gw.util.PooledHttpClientUtil.postPropsToURL(PooledHttpClientUtil.java:180) at com.esri.gw.publish.AgsServicePublisher.sendAgsRestRequest(AgsServicePublisher.java:2628) at com.esri.gw.publish.AgsServicePublisher.sendAgsRestRequest(AgsServicePublisher.java:2607) at com.esri.gw.publish.AgsServicePublisher.getServersManagedEgdb(AgsServicePublisher.java:726) ... 62 more

From ArcGIS Server:

The operation timed out (less) (WinINet Error while using HTTPS security, 12002), URL = https://ttgpvis1.prod.travp.net:6443/arcgis/admin/generateToken

Any thoughts? This only occurred for 15 minutes and self corrected after. I checked the transaction as well and that wasn't any surprising since ArcGIS Server has handled similar transactions before.

Re: Slow Performance/Time Out Issue for ArcGIS Portal & Server

JayateerthDeshpande — Wed, 27 Sep 2023 05:58:18 GMT

We are also facing same issue however key differences are:

1. ArcGIS Enterprise 10.9.1, all available patches except patches for Utility Network are applied

2. Entire System is on Azure VMs and Config store / ArcGIS /Portal Directories on Azure File share.

Is there a solution anyone found out? Or is it that the HA system on Windows environment is known to experirience such issues when ArcGIS Enterprise is configured in HA mode.

Re: Slow Performance/Time Out Issue for ArcGIS Portal & Server

GianniCampanile2 — Wed, 21 Feb 2024 14:27:48 GMT

Hi,

we have the same problem and trying to solve it my modifying the keepalive parameter of the tomcat server but with no success up to now.

Did you manage to solve this issue ?

Thanks

Regards

Gianni

Re: Slow Performance/Time Out Issue for ArcGIS Portal & Server

AbhishekTandon — Sat, 24 Aug 2024 02:57:16 GMT

Hi,

did anyone found solution for this?

topic Re: Slow Performance/Time Out Issue for ArcGIS Portal & Server in ArcGIS Enterprise Questions

Slow Performance/Time Out Issue for ArcGIS Portal & Server

Re: Slow Performance/Time Out Issue for ArcGIS Portal & Server

Re: Slow Performance/Time Out Issue for ArcGIS Portal & Server

Re: Slow Performance/Time Out Issue for ArcGIS Portal & Server