Upgrade 10.8.1 issues

[NickXu]

we have one load balancer, 2 portals, 2 servers, 2 data stores and all portal content and server directories are on a shared file server. Recently, we updated our sandbox from 10.6.1 to 10.8.1, before and after upgradation, we verified federation and data store, server status  and all works.

however, we noticed that when I open a rest service folder, sometimes, it directs to the login page. the server log file shows severe errors:

Failed to return all services configurations in the folder 'Locator'. Could not connect to the ArcGIS component at URL 'https://sandbox.ourdomain.ca:7443/arcgis/sharing/rest/search'. The ArcGIS component on that machine may not be running or the machine may not be reachable at this time.Error:

I checked the server covgis33 log file and the error:

<Msg time="2020-09-25T21:21:36,62" type="SEVERE" code="6605" source="Admin" process="17392" thread="1" methodName="" machine="COVGIS33.DS.CITY.ourCity.BC.CA" user="" elapsed="" requestID="0fce8b36-e385-472f-942a-de447ab2eb09">Failed to return all services configurations in the folder 'Locator'. Could not connect to the ArcGIS component at URL 'https://sandbox.ourDomain.ca:7443/arcgis/sharing/rest/search'. The ArcGIS component on that machine may not be running or the machine may not be reachable at this time.Error: </Msg>
<Msg time="2020-09-25T21:21:36,63" type="DEBUG" code="9999" source="Rest" process="17392" thread="1" methodName="" machine="COVGIS33.DS.CITY.ourCity.BC.CA" user="" elapsed="" requestID="0fce8b36-e385-472f-942a-de447ab2eb09">Time (ms): 28 (/arcgis/rest/services/Locator)</Msg>
<Msg time="2020-09-25T21:21:36,62" type="DEBUG" code="9999" source="Server" process="17392" thread="26" methodName="" machine="COVGIS33.DS.CITY.ourCity.BC.CA" user="" elapsed="" requestID="0fce8b36-e385-472f-942a-de447ab2eb09">Failed to return all services configurations in the folder 'Locator'. Could not connect to the ArcGIS component at URL 'https://sandbox.ourdomain.ca:7443/arcgis/sharing/rest/search'. The ArcGIS component on that machine may not be running or the machine may not be reachable at this time.Error: </Msg>
<Msg time="2020-09-25T21:21:36,62" type="DEBUG" code="9999" source="Admin" process="17392" thread="1" methodName="" machine="COVGIS33.DS.CITY.ourcity.BC.CA" user="" elapsed="" requestID="0fce8b36-e385-472f-942a-de447ab2eb09">java.io.IOException: Could not connect to the ArcGIS component at URL 'https://sandbox.ourdomain.ca:7443/arcgis/sharing/rest/search'. The ArcGIS component on that machine may not be running or the machine may not be reachable at this time.Error:
at com.esri.arcgis.discovery.admin.util.HttpUtil.handleConnectionIOException(HttpUtil.java:721)
at com.esri.arcgis.discovery.admin.util.PooledHttpClientUtil.postToURL(PooledHttpClientUtil.java:253)
at com.esri.arcgis.discovery.admin.util.PooledHttpClientUtil.postPropsToURL(PooledHttpClientUtil.java:289)
at com.esri.arcgis.discovery.admin.security.portal.ArcGISPortal.search(ArcGISPortal.java:2075)
at com.esri.arcgis.discovery.admin.security.portal.ArcGISPortal.checkAccessOnItems(ArcGISPortal.java:2009)
at com.esri.arcgis.discovery.admin.services.ServiceManager.getAllServiceCfgsInFederatedMode(ServiceManager.java:3071)
at com.esri.arcgis.discovery.security.manager.WebSecurityManager.showFolder(WebSecurityManager.java:572)
at com.esri.discovery.app.security.SecurityPreHandler.authorizeServices(SecurityPreHandler.java:432)
at com.esri.discovery.app.security.SecurityPreHandler.servicesSecurity(SecurityPreHandler.java:154)
at com.esri.discovery.app.security.SecurityPreHandler.preHandle(SecurityPreHandler.java:85)
at com.esri.rf.RServlet.service(RServlet.java:113)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:126)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:666)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:367)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
at org.apache.tomcat.util.net.Nio2Endpoint$SocketProcessor.doRun(Nio2Endpoint.java:1684)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at org.apache.tomcat.util.net.AbstractEndpoint.processSocket(AbstractEndpoint.java:1104)
at org.apache.tomcat.util.net.Nio2Endpoint$Nio2SocketWrapper$2.completed(Nio2Endpoint.java:599)
at org.apache.tomcat.util.net.Nio2Endpoint$Nio2SocketWrapper$2.completed(Nio2Endpoint.java:577)
at org.apache.tomcat.util.net.SecureNio2Channel$1.completed(SecureNio2Channel.java:969)
at org.apache.tomcat.util.net.SecureNio2Channel$1.completed(SecureNio2Channel.java:898)
at java.base/sun.nio.ch.Invoker.invokeUnchecked(Unknown Source)
at java.base/sun.nio.ch.Invoker$2.run(Unknown Source)
at java.base/sun.nio.ch.AsynchronousChannelGroupImpl$1.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Unknown Source)
</Msg>

I have created a esri ticket: Esri Case #02645437

Thanks!

Nick

[IngridMans]

I have no answers, but I'd like to chime in I'm seeing some similar error messages. We have a single base deployment on with separate machines running ArcGIS Server, Portal, and Data Store, respectively. In our Server Manager Logs, there are very many for our various services (including Locators and map image 7 feature services) with:

"Failed to return the service configuration for '__/__.MapServer'. Could not connect to the ArcGIS component at URL [Portal machine]:7443/arcgis/sharing/rest/content/items/[item id].  The ArcGIS component on that machine may not be reachable at this time.Error: Read timed out

As an end user viewing our various map applications, everything *looks* okay, but these messages don't instill confidence. Given that the error is coming from the Server machine, but referencing being unable to connect to the Portal machine, what more can I do to investigate this? 

I'll also note another similarity with the OP about being redirected to the login page. While working in Server Manager, it has started redirecting me to login more frequently, even if I've only been signed in for a minute or two. We use a single-sign-on authentication through Okta. 

[MichailMarinakis1]

We have the same issue in our environment. We suspect that it happened, after temporarily we lost connection to the shared folder for arcgisportal/content. The connection to the folder is back and everything seems fine, when it points to portal A but the issue is reproducible when it redirects to portal B.

@JonathanQuinn , is this issue something being investigated? 

we haven't tried yet to restart of the portal services.

[JonathanQuinn]

The problem is that the privatePortalURL property is not honored after the upgrade. 

BUG-000139829 The privatePortalURL property is not honored after upgrading Portal to 10.8.1

There's no HTTP exception when Server makes a request to Portal via the privatePortalURL, so that's why the Error is blank:

java.io.IOException: Could not connect to the ArcGIS component at URL 'https://sandbox.ourdomain.ca:7443/arcgis/sharing/rest/search'. The ArcGIS component on that machine may not be running or the machine may not be reachable at this time.Error:

You can fix it by going into Portaladmin and set the system properties again.

[MichailMarinakis1]

Perfect! Thanks for the quick reply!