Upgrade 10.8.1 issues

NickXu — Sat, 26 Sep 2020 04:59:40 GMT

we have one load balancer, 2 portals, 2 servers, 2 data stores and all portal content and server directories are on a shared file server. Recently, we updated our sandbox from 10.6.1 to 10.8.1, before and after upgradation, we verified federation and data store, server status and all works.

however, we noticed that when I open a rest service folder, sometimes, it directs to the login page. the server log file shows severe errors:

Failed to return all services configurations in the folder 'Locator'. Could not connect to the ArcGIS component at URL 'https://sandbox.ourdomain.ca:7443/arcgis/sharing/rest/search'. The ArcGIS component on that machine may not be running or the machine may not be reachable at this time.Error:

I checked the server covgis33 log file and the error:

<Msg time="2020-09-25T21:21:36,62" type="SEVERE" code="6605" source="Admin" process="17392" thread="1" methodName="" machine="COVGIS33.DS.CITY.ourCity.BC.CA" user="" elapsed="" requestID="0fce8b36-e385-472f-942a-de447ab2eb09">Failed to return all services configurations in the folder 'Locator'. Could not connect to the ArcGIS component at URL 'https://sandbox.ourDomain.ca:7443/arcgis/sharing/rest/search'. The ArcGIS component on that machine may not be running or the machine may not be reachable at this time.Error: </Msg>
<Msg time="2020-09-25T21:21:36,63" type="DEBUG" code="9999" source="Rest" process="17392" thread="1" methodName="" machine="COVGIS33.DS.CITY.ourCity.BC.CA" user="" elapsed="" requestID="0fce8b36-e385-472f-942a-de447ab2eb09">Time (ms): 28 (/arcgis/rest/services/Locator)</Msg>
<Msg time="2020-09-25T21:21:36,62" type="DEBUG" code="9999" source="Server" process="17392" thread="26" methodName="" machine="COVGIS33.DS.CITY.ourCity.BC.CA" user="" elapsed="" requestID="0fce8b36-e385-472f-942a-de447ab2eb09">Failed to return all services configurations in the folder 'Locator'. Could not connect to the ArcGIS component at URL 'https://sandbox.ourdomain.ca:7443/arcgis/sharing/rest/search'. The ArcGIS component on that machine may not be running or the machine may not be reachable at this time.Error: </Msg>
<Msg time="2020-09-25T21:21:36,62" type="DEBUG" code="9999" source="Admin" process="17392" thread="1" methodName="" machine="COVGIS33.DS.CITY.ourcity.BC.CA" user="" elapsed="" requestID="0fce8b36-e385-472f-942a-de447ab2eb09">java.io.IOException: Could not connect to the ArcGIS component at URL 'https://sandbox.ourdomain.ca:7443/arcgis/sharing/rest/search'. The ArcGIS component on that machine may not be running or the machine may not be reachable at this time.Error:
at com.esri.arcgis.discovery.admin.util.HttpUtil.handleConnectionIOException(HttpUtil.java:721)
at com.esri.arcgis.discovery.admin.util.PooledHttpClientUtil.postToURL(PooledHttpClientUtil.java:253)
at com.esri.arcgis.discovery.admin.util.PooledHttpClientUtil.postPropsToURL(PooledHttpClientUtil.java:289)
at com.esri.arcgis.discovery.admin.security.portal.ArcGISPortal.search(ArcGISPortal.java:2075)
at com.esri.arcgis.discovery.admin.security.portal.ArcGISPortal.checkAccessOnItems(ArcGISPortal.java:2009)
at com.esri.arcgis.discovery.admin.services.ServiceManager.getAllServiceCfgsInFederatedMode(ServiceManager.java:3071)
at com.esri.arcgis.discovery.security.manager.WebSecurityManager.showFolder(WebSecurityManager.java:572)
at com.esri.discovery.app.security.SecurityPreHandler.authorizeServices(SecurityPreHandler.java:432)
at com.esri.discovery.app.security.SecurityPreHandler.servicesSecurity(SecurityPreHandler.java:154)
at com.esri.discovery.app.security.SecurityPreHandler.preHandle(SecurityPreHandler.java:85)
at com.esri.rf.RServlet.service(RServlet.java:113)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:126)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:666)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:367)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
at org.apache.tomcat.util.net.Nio2Endpoint$SocketProcessor.doRun(Nio2Endpoint.java:1684)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at org.apache.tomcat.util.net.AbstractEndpoint.processSocket(AbstractEndpoint.java:1104)
at org.apache.tomcat.util.net.Nio2Endpoint$Nio2SocketWrapper$2.completed(Nio2Endpoint.java:599)
at org.apache.tomcat.util.net.Nio2Endpoint$Nio2SocketWrapper$2.completed(Nio2Endpoint.java:577)
at org.apache.tomcat.util.net.SecureNio2Channel$1.completed(SecureNio2Channel.java:969)
at org.apache.tomcat.util.net.SecureNio2Channel$1.completed(SecureNio2Channel.java:898)
at java.base/sun.nio.ch.Invoker.invokeUnchecked(Unknown Source)
at java.base/sun.nio.ch.Invoker$2.run(Unknown Source)
at java.base/sun.nio.ch.AsynchronousChannelGroupImpl$1.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Unknown Source)
</Msg>

I have created a esri ticket: Esri Case #02645437

Thanks!

Nick

Re: Upgrade 10.8.1 issues

IngridMans — Mon, 23 Nov 2020 15:19:28 GMT

I have no answers, but I'd like to chime in I'm seeing some similar error messages. We have a single base deployment on with separate machines running ArcGIS Server, Portal, and Data Store, respectively. In our Server Manager Logs, there are very many for our various services (including Locators and map image 7 feature services) with:

"Failed to return the service configuration for '__/__.MapServer'. Could not connect to the ArcGIS component at URL [Portal machine]:7443/arcgis/sharing/rest/content/items/[item id]. The ArcGIS component on that machine may not be reachable at this time.Error: Read timed out

As an end user viewing our various map applications, everything *looks* okay, but these messages don't instill confidence. Given that the error is coming from the Server machine, but referencing being unable to connect to the Portal machine, what more can I do to investigate this?

I'll also note another similarity with the OP about being redirected to the login page. While working in Server Manager, it has started redirecting me to login more frequently, even if I've only been signed in for a minute or two. We use a single-sign-on authentication through Okta.

Re: Upgrade 10.8.1 issues

MichailMarinakis1 — Wed, 23 Jun 2021 10:40:46 GMT

We have the same issue in our environment. We suspect that it happened, after temporarily we lost connection to the shared folder for arcgisportal/content. The connection to the folder is back and everything seems fine, when it points to portal A but the issue is reproducible when it redirects to portal B.

@JonathanQuinn , is this issue something being investigated?

we haven't tried yet to restart of the portal services.

Re: Upgrade 10.8.1 issues

JonathanQuinn — Wed, 23 Jun 2021 15:58:16 GMT

The problem is that the privatePortalURL property is not honored after the upgrade.

BUG-000139829 The privatePortalURL property is not honored after upgrading Portal to 10.8.1

There's no HTTP exception when Server makes a request to Portal via the privatePortalURL, so that's why the Error is blank:

java.io.IOException: Could not connect to the ArcGIS component at URL 'https://sandbox.ourdomain.ca:7443/arcgis/sharing/rest/search'. The ArcGIS component on that machine may not be running or the machine may not be reachable at this time.Error:

You can fix it by going into Portaladmin and set the system properties again.

Re: Upgrade 10.8.1 issues

MichailMarinakis1 — Wed, 23 Jun 2021 16:00:11 GMT

Perfect! Thanks for the quick reply!

topic Upgrade 10.8.1 issues in ArcGIS Enterprise Questions

Upgrade 10.8.1 issues

Re: Upgrade 10.8.1 issues

Re: Upgrade 10.8.1 issues

Re: Upgrade 10.8.1 issues

Re: Upgrade 10.8.1 issues