topic Re: AWS 10.8.1 HA Portal not restarting in ArcGIS Enterprise Questions

AWS 10.8.1 HA Portal not restarting

ShanonLoughton — Sat, 11 Dec 2021 07:59:06 GMT

In AWS Ive deployed ArcGIS Enterprise 10.8.1 using Windows in HA and using ELB, both templates available here.
It works on first deployment and successful CloudFormation create-stack.
However when stopping/restarting Portal service on a given instance, or rebooting the Portal instances (either of them), they fail to restart Portal service, citing internal database not started properly, and symptoms like passwords rejected at /portaladmin login.
Here's one instance's set of logs:
Windows EventViewer

PostgreSQL : The description for Event ID 0 from source PostgreSQL cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer. (repeated, initially Information class, but further back is Error class)
Portal: The description for Event ID 5 from source Portal for ArcGIS cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer. (repeated, initially Information class, but further back is Error class)

"C:\Program Files\ArcGIS\Portal\framework\service\logs\service-error-0.log"

Mon Aug 24 13:01:13.237 UTC 2020 Exception in thread "I/O dispatcher 17" 
Mon Aug 24 13:01:13.237 UTC 2020 java.lang.NoSuchMethodError: org.apache.http.ConnectionClosedException: method 'void <init>()' not found
Mon Aug 24 13:01:13.237 UTC 2020 at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.endOfInput(HttpAsyncRequestExecutor.java:356)
Mon Aug 24 13:01:13.237 UTC 2020 at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:261)
Mon Aug 24 13:01:13.237 UTC 2020 at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
Mon Aug 24 13:01:13.237 UTC 2020 at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
Mon Aug 24 13:01:13.237 UTC 2020 at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
Mon Aug 24 13:01:13.237 UTC 2020 at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
Mon Aug 24 13:01:13.237 UTC 2020 at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
Mon Aug 24 13:01:13.237 UTC 2020 at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
Mon Aug 24 13:01:13.237 UTC 2020 at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
Mon Aug 24 13:01:13.237 UTC 2020 at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
Mon Aug 24 13:01:13.237 UTC 2020 at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
Mon Aug 24 13:01:13.237 UTC 2020 at java.base/java.lang.Thread.run(Unknown Source)

"C:\arcgisportal\logs\10.72.4.42\portal\portal-20200824.123347-12940-0.0.log"

<Msg time="2020-08-24T12:34:34,908" type="SEVERE" code="218010" source="Portal Admin" process="12940" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID="">The portal has been initialized and configured but is not accessible. The internal portal database does not appear to be running or accepting connections. Restart the portal machine or machines and if the problem persists, contact Esri technical support (U.S.) or your distributor (customers outside the U.S.).</Msg>
<Msg time="2020-08-24T12:34:34,908" type="WARNING" code="218012" source="Portal Admin" process="12940" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID="">Error at check and update urls. The portal is currently unavailable. Please contact your portal administrator.</Msg>
<Msg time="2020-08-24T12:35:12,375" type="SEVERE" code="218010" source="Portal Admin" process="12940" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID="">The portal has been initialized and configured but is not accessible. The internal portal database does not appear to be running or accepting connections. Restart the portal machine or machines and if the problem persists, contact Esri technical support (U.S.) or your distributor (customers outside the U.S.).</Msg>
<Msg time="2020-08-24T12:35:18,695" type="SEVERE" code="218010" source="Portal Admin" process="12940" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID="">The portal has been initialized and configured but is not accessible. The internal portal database does not appear to be running or accepting connections. Restart the portal machine or machines and if the problem persists, contact Esri technical support (U.S.) or your distributor (customers outside the U.S.).</Msg>
<Msg time="2020-08-24T12:35:22,903" type="WARNING" code="218014" source="Portal" process="12940" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID="">Starting Index Service.</Msg>
<Msg time="2020-08-24T12:36:28,437" type="SEVERE" code="218010" source="Portal Admin" process="12940" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID="">The portal has been initialized and configured but is not accessible. The internal portal database does not appear to be running or accepting connections. Restart the portal machine or machines and if the problem persists, contact Esri technical support (U.S.) or your distributor (customers outside the U.S.).</Msg>
<Msg time="2020-08-24T12:36:49,886" type="SEVERE" code="218010" source="Portal Admin" process="12940" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID="">The portal has been initialized and configured but is not accessible. The internal portal database does not appear to be running or accepting connections. Restart the portal machine or machines and if the problem persists, contact Esri technical support (U.S.) or your distributor (customers outside the U.S.).</Msg>
<Msg time="2020-08-24T12:37:44,483" type="SEVERE" code="218010" source="Portal Admin" process="12940" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID="">The portal has been initialized and configured but is not accessible. The internal portal database does not appear to be running or accepting connections. Restart the portal machine or machines and if the problem persists, contact Esri technical support (U.S.) or your distributor (customers outside the U.S.).</Msg>
<Msg time="2020-08-24T12:38:09,93" type="SEVERE" code="218010" source="Portal Admin" process="12940" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID="">The portal has been initialized and configured but is not accessible. The internal portal database does not appear to be running or accepting connections. Restart the portal machine or machines and if the problem persists, contact Esri technical support (U.S.) or your distributor (customers outside the U.S.).</Msg>
<Msg time="2020-08-24T12:38:41,750" type="WARNING" code="218015" source="Portal" process="12940" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID="">Started Index Service.</Msg>
<Msg time="2020-08-24T12:38:41,828" type="WARNING" code="217060" source="Portal" process="12940" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID="">The database server was found to be stopped. Re-starting it.</Msg>
<Msg time="2020-08-24T12:38:51,974" type="WARNING" code="217064" source="Portal" process="12940" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID="">The web server was found to be stopped. Re-starting it.</Msg>
<Msg time="2020-08-24T12:43:09,108" type="WARNING" code="219999" source="Portal Admin" process="5948" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID=""> Failed to generate a token for user 'portaladmin'. java.lang.Exception: Server returned HTTP status code 404. Requested URL: https://10.72.4.42:7443/arcgis/sharing/generateToken</Msg>
<Msg time="2020-08-24T12:43:12,517" type="WARNING" code="219999" source="Portal Admin" process="5948" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID=""> Failed to generate a token for user 'portaladmin'. java.lang.Exception: Server returned HTTP status code 404. Requested URL: https://10.72.4.42:7443/arcgis/sharing/generateToken</Msg>
<Msg time="2020-08-24T12:43:15,819" type="WARNING" code="219999" source="Portal Admin" process="5948" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID=""> Failed to generate a token for user 'portaladmin'. java.lang.Exception: Server returned HTTP status code 404. Requested URL: https://10.72.4.42:7443/arcgis/sharing/generateToken</Msg>
<Msg time="2020-08-24T12:43:20,950" type="WARNING" code="219999" source="Portal Admin" process="5948" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID=""> Failed to generate a token for user 'portaladmin'. java.lang.Exception: Server returned HTTP status code 404. Requested URL: https://10.72.4.42:7443/arcgis/sharing/generateToken</Msg>
<Msg time="2020-08-24T12:43:31,158" type="WARNING" code="219999" source="Portal Admin" process="5948" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID=""> Failed to generate a token for user 'portaladmin'. java.lang.Exception: Server returned HTTP status code 404. Requested URL: https://10.72.4.42:7443/arcgis/sharing/generateToken</Msg>
<Msg time="2020-08-24T13:01:13,481" type="WARNING" code="217068" source="Portal" process="12940" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID="">Error in the process monitor watchdog thread. sleep interrupted</Msg>

"C:\arcgisportal\logs\database\pg_log\postgresql-2020-08-24_060707.log"

2020-08-24 05:38:42 PDT: [6272]: LOG: database system was shut down at 2020-08-24 05:32:36 PDT
2020-08-24 05:38:42 PDT: [5824]: LOG: database system is ready to accept connections
2020-08-24 05:38:42 PDT: [10060]: LOG: incomplete startup packet
2020-08-24 05:38:42 PDT: [10180]: FATAL: no pg_hba.conf entry for host "::1", user "arcgis", database "arcgis", SSL off
2020-08-24 05:39:12 PDT: [12016]: LOG: incomplete startup packet
2020-08-24 05:40:12 PDT: [12196]: LOG: incomplete startup packet
2020-08-24 05:41:12 PDT: [12120]: LOG: incomplete startup packet
2020-08-24 05:42:12 PDT: [11232]: LOG: incomplete startup packet (repeated)
2020-08-24 05:42:33 PDT: [11348]: LOG: could not receive data from client: An existing connection was forcibly closed by the remote host. (repeated)

Thanks for any help.

Re: AWS 10.8.1 HA Portal not restarting

NicolasGIS — Wed, 02 Sep 2020 07:44:03 GMT

Hello,

I have exactly the same issue.

My setup is the following:

1 external load balancer load balancing traffic of port 443, 6443 and 7443.

2 VMS with Portal for ArcGIS 10.8.1, ArcGIS Server 10.8.1, Relational Datastore 10.8.1 and 2 * ArcGIS WebAdaptor (1 for Portal and 1 for ArcGIS Server).

The setup was configured using ArcGIS DSC and tt was working fine.

Then I switched off one VM, and it was still working fine: happy to have HA . But then I started it again and now I have the very same symptoms:

- When trying to log in to portaladmin, I receive an error "You are not authorized to use this resource.".

- Portal database does not want to start with the same error as you : "FATAL: no pg_hba.conf entry for host "::1", user "myuser", database "arcgis", SSL off"

I had a look at the pg_hba.conf and it is true that there is no entry for service account "myuser" as there are only entries for portal primary administrator "siteadmin".

It is the third times this happen to me: I recreate VM, configure HA with DSC and everything works fine. Then I switch off one VM and then issues start occuring and my portal is messed up.

There is definetly an issue somewhere but I can't find yet the logic !

Did you find a solution ?

Jonathan Quinn‌, any idea ?

Thanks !

Re: AWS 10.8.1 HA Portal not restarting

JonathanQuinn — Wed, 02 Sep 2020 18:39:02 GMT

FATAL: no pg_hba.conf entry for host "::1", user "myuser", database "arcgis", SSL off

This error is not indicative of the database not starting; Portal checks whether the database is up and in some places, doesn't pass in the right username. The process then assumes the check will use the account used to run the Portal for ArcGIS service which returns the error.

Are you testing in a cloud deployment, or on premises? Can you access the Sharing API, ex https://machine.domain.com:7443/arcgis/sharing/rest?

Re: AWS 10.8.1 HA Portal not restarting

JonathanQuinn — Wed, 02 Sep 2020 18:40:17 GMT

Shanon Loughton‌ can you take a look at the localhost logs under <install directory>\framework\runtime\tomcat\logs? They'll show why the Sharing API is not initializing and returning a 404.

Re: AWS 10.8.1 HA Portal not restarting

NicolasGIS — Wed, 02 Sep 2020 21:02:16 GMT

Many thanks Jonathan Quinn‌ for having a look at this issue so quickly and for your explanation.

I am testing an on premises deployment (based on Openstack).

I cannot access https://machine.domain.com:7443/arcgis/sharing/rest:

It returns 404 on both machines.

I also had a look at localhost logs in <install directory>\framework\runtime\tomcat\logs:

"""

org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class [org.springframework.web.context.ContextLoaderListener]
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'accountsRequestHandler' defined in class path resource [spring/gw-dispatcher.xml]: Cannot resolve reference to bean 'gwAccounts' while setting bean property 'accounts'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'gwAccounts' defined in class path resource [spring/gw-config.xml]: Cannot resolve reference to bean 'gwDatabase' while setting bean property 'database'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'gwDatabase' defined in class path resource [spring/gw-config.xml]: Cannot resolve reference to bean 'memoryCache' while setting bean property 'cache'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'memoryCache' defined in class path resource [spring/gw-config-beans-portal.xml]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.esri.gw.cache.PortalDistributedHashImpl]: Constructor threw exception; nested exception is java.lang.IllegalStateException: Node failed to start!
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:378)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:110)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1602)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1354)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:572)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:495)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:317)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:315)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:759)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:869)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550)
at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:409)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:291)
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:103)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4685)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5146)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:717)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:690)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:705)
at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:631)
at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1831)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
at java.base/java.util.concurrent.AbstractExecutorService.submit(Unknown Source)
at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:526)
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:425)
at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1576)
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:309)
at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423)
at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:366)
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:936)
at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:841)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1384)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1374)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
at java.base/java.util.concurrent.AbstractExecutorService.submit(Unknown Source)
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:909)
at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:262)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.StandardService.startInternal(StandardService.java:421)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:930)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.startup.Catalina.start(Catalina.java:633)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:343)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:474)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'gwAccounts' defined in class path resource [spring/gw-config.xml]: Cannot resolve reference to bean 'gwDatabase' while setting bean property 'database'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'gwDatabase' defined in class path resource [spring/gw-config.xml]: Cannot resolve reference to bean 'memoryCache' while setting bean property 'cache'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'memoryCache' defined in class path resource [spring/gw-config-beans-portal.xml]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.esri.gw.cache.PortalDistributedHashImpl]: Constructor threw exception; nested exception is java.lang.IllegalStateException: Node failed to start!
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:378)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:110)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1602)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1354)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:572)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:495)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:317)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:315)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:367)
... 56 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'gwDatabase' defined in class path resource [spring/gw-config.xml]: Cannot resolve reference to bean 'memoryCache' while setting bean property 'cache'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'memoryCache' defined in class path resource [spring/gw-config-beans-portal.xml]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.esri.gw.cache.PortalDistributedHashImpl]: Constructor threw exception; nested exception is java.lang.IllegalStateException: Node failed to start!
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:378)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:110)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1602)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1354)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:572)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:495)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:317)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:315)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:367)
... 66 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'memoryCache' defined in class path resource [spring/gw-config-beans-portal.xml]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.esri.gw.cache.PortalDistributedHashImpl]: Constructor threw exception; nested exception is java.lang.IllegalStateException: Node failed to start!
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:282)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1267)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1124)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:535)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:495)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:317)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:315)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:367)
... 76 more
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.esri.gw.cache.PortalDistributedHashImpl]: Constructor threw exception; nested exception is java.lang.IllegalStateException: Node failed to start!
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:182)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:117)
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:275)
... 85 more
Caused by: java.lang.IllegalStateException: Node failed to start!
at com.hazelcast.instance.HazelcastInstanceImpl.<init>(HazelcastInstanceImpl.java:133)
at com.hazelcast.instance.HazelcastInstanceFactory.constructHazelcastInstance(HazelcastInstanceFactory.java:202)
at com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:181)
at com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:131)
at com.hazelcast.core.Hazelcast.newHazelcastInstance(Hazelcast.java:57)
at com.esri.gw.cache.HazelCastUtil.getHazelcastInstance(HazelCastUtil.java:46)
at com.esri.gw.cache.DistributedMap.a(DistributedMap.java:92)
at com.esri.gw.cache.DistributedMap.<init>(DistributedMap.java:38)
at com.esri.gw.cache.PortalDistributedHashImpl.<init>(PortalDistributedHashImpl.java:44)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source)
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:170)
... 87 more

Thanks !

Re: AWS 10.8.1 HA Portal not restarting

JonathanQuinn — Thu, 03 Sep 2020 17:03:01 GMT

It sounds like an internal memory cache component couldn't start up, which listens on a port between 5701 and 5703. Do you have anything running on those ports?

Re: AWS 10.8.1 HA Portal not restarting

NicolasGIS — Thu, 03 Sep 2020 17:44:58 GMT

On the server from which this log is extracted, there is nothing running on those ports.

On the other server, there is one javaw process listening on port 5701

Note that on the latter server, the error is slightly different:

SEVERE [main] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class [org.springframework.web.context.ContextLoaderListener]
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'accountsRequestHandler' defined in class path resource [spring/gw-dispatcher.xml]: Cannot resolve reference to bean 'gwAccounts' while setting bean property 'accounts'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'gwAccounts' defined in class path resource [spring/gw-config.xml]: Cannot resolve reference to bean 'gwCommunity' while setting bean property 'community'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'gwCommunity' defined in class path resource [spring/gw-config.xml]: Cannot resolve reference to bean 'securityPermissions' while setting bean property 'securityPermissions'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'securityPermissions' defined in class path resource [spring/gw-config.xml]: Cannot resolve reference to bean 'gwContent' while setting bean property 'content'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'gwContent' defined in class path resource [spring/gw-config.xml]: Cannot resolve reference to bean 'gwSearch' while setting bean property 'search'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'gwSearch' defined in class path resource [spring/gw-config.xml]: Cannot resolve reference to bean 'searchIndexRemoteServiceProxy' while setting bean property 'index'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'searchIndexRemoteServiceProxy' defined in class path resource [spring/index-clients_es_portal.xml]: Cannot resolve reference to bean 'searchIndexServiceDispatcher' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'searchIndexServiceDispatcher' defined in class path resource [spring/index-clients_es_portal.xml]: Cannot resolve reference to bean 'searchIndexServiceProxyEs' while setting bean property 'indexingServices' with key [0]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'searchIndexServiceProxyEs' defined in class path resource [spring/index-clients_es_portal.xml]: Invocation of init method failed; nested exception is java.net.ConnectException: Connection refused: no further information
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:378)

Thanks

Re: AWS 10.8.1 HA Portal not restarting

StefKoroblitsas — Thu, 24 Sep 2020 08:30:54 GMT

Nicolas GIS‌ Hi, did you find a resolution? I'm getting similar logs, we have support looking at it but its taking a long time to resolve

Cheers,

Stef

Re: AWS 10.8.1 HA Portal not restarting

NicolasGIS — Thu, 24 Sep 2020 08:42:29 GMT

Hi Stef Koroblitsas,

Unfortunately not ! I have been struggling for weeks trying to identify a pattern but I can't figure out what is the issue.

Please let me know if anything come out from your support case (I was about to open one as well as I am in a dead end...) !

Any idea Jonathan Quinn‌ ? Seems like I am not the only one !

Re: AWS 10.8.1 HA Portal not restarting

JonathanQuinn — Thu, 01 Oct 2020 00:28:11 GMT

You'll need to check why the java.exe process is not starting. You can look at the logs under <install directory>\framework\runtime\ds\usr. They may tell you why it won't start.

Re: AWS 10.8.1 HA Portal not restarting

NicolasGIS — Thu, 22 Oct 2020 11:57:15 GMT

I believe it is BUG-000134458:

After rebooting the standby Portal in HA on Windows (OS), the portal Postgres database does not reboot properly to allow successful connections in portal/sharing/rest or portaladmin.

As I had in portal folder loads of DBs !

Re: AWS 10.8.1 HA Portal not restarting

JonathanQuinn — Mon, 26 Oct 2020 19:04:44 GMT

I'm a bit doubtful that the problem is as simple as stopping and starting the standby. We go through a considerable amount of regression testing and this problem has not been discovered internally. The fact that you had many db folders means that the standby is unable to rejoin; that behavior is not described in the bug. Anyway, we'll need to have a close look at that issue.

Re: AWS 10.8.1 HA Portal not restarting

NicolasGIS — Wed, 28 Oct 2020 14:15:18 GMT

In the folder you mentionned:

<install directory>\framework\runtime\ds\usr\logs\MYVM01,

there is only an 'elasticlog' folder which I believe is only about tileCache datastore.

Re: AWS 10.8.1 HA Portal not restarting

NicolasGIS — Wed, 28 Oct 2020 14:49:27 GMT

I found some error in Event viewer related to the service account running ArcGIS Enterprise:

"The machine-default permission settings do not grant Local Activation permission for the COM Server application with CLSID
{0358B920-0AC7-461F-98F4-58E32CD89148}
and APPID
{3EB3C877-1F16-487C-9050-104DBCD66683}
to the user MYCOMPANY\MYUSER SID (S-1-5-21-1526224874-1540688658-1361462980-7571414) from address LocalHost (Using LRPC) running in the application container Unavailable SID (Unavailable). This security permission can be modified using the Component Services administrative tool."

CLSID {0358B920-0AC7-461F-98F4-58E32CD89148} refers to Wininet Cache task object

APPID {3EB3C877-1F16-487C-9050-104DBCD66683} refers to WinInetCacheServer

Could it explain why memory cache component couldn't start up ?

Cheers

Re: AWS 10.8.1 HA Portal not restarting

MichailMarinakis1 — Wed, 25 Nov 2020 12:45:13 GMT

Hi!

Same behavior for us with on premise HA portal.

I have tried different patterns. stopping the standby restarting the primary one but no luck.

Any feedback can be useful.

Cheers,

Michail

Re: AWS 10.8.1 HA Portal not restarting

MichailMarinakis1 — Wed, 25 Nov 2020 15:33:58 GMT

We have noticed something. In the HA portal, the standby machine that is complaining with "You are not authorized to use this resource." when admin is trying to log in, has lost the SSL Certificates configuration! When we joined this machine to the portal site, we had set correctly the root and server certificate.

If we try to update this configuration, then we receive this

As a workaround, we have tried to:

stop the standby machine,
restart the primary,
start the standby again

Sometimes this pattern works, sometimes not...

Another point is that the following link: https://standbymachine.domain.com:7443/arcgis/sharing/rest/generateToken returns a 404 error message. Everything works fine for the primary machine.

Any ideas will be very helpful!

Re: AWS 10.8.1 HA Portal not restarting

Anonymous User — Mon, 01 Feb 2021 18:01:57 GMT

I forgot about this post during the Geonet backend changeover... looks like Im in dual user account land too..

The original environment is gone but a similar one I spun up showed similar symptoms of Portal in HA not restart-able. I have not checked if exactly the same situation, however it was resolved by:

Ensuring the Standby machine is stopped first.
And then Primary machine is stopped and started.
Check that Primary machine works functions and is up for a while (>10 mins).
Then start Standby machine and let it presumably sync/catchup/revert whatever.

Prior to the above resolution I also checked <install directory>\framework\runtime\tomcat\logs - there were a variety of Java errors, SEVERE ones were about:

Unauthorized access, No valid token (presumably my attempts)
One or more listeners failed to start
Could not join cluster shutting down now.

Logs attached, 10.0.0.1 is Primary, 10.0.0.2 is Standby.

@ShanonLoughton

Re: AWS 10.8.1 HA Portal not restarting

Trevor_Hart — Fri, 08 Jan 2021 00:49:36 GMT

This is a very interesting discussion. We have had this as an ongoing issue for over a year now without a reliable fix.

We restart as per @Anonymous User suggestion and this generally fixes it.

One thing to add here is that Postgres is actually available as I am able to log into the database using another database client.

@Anonymous User your logs show this error;

1-Dec-2020 01:50:28.894 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [arcgis#portaladmin] registered the JDBC driver [org.postgresql.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.

This makes me think that its not the database but rather the (lack of) JDBC driver thats causes the problem.

Re: AWS 10.8.1 HA Portal not restarting

WilliamCraft — Mon, 25 Jan 2021 18:44:33 GMT

I'm having the exact same issue/symptoms with my Azure Esri HA deployment. Everything seems to work fine until I restart the Portal service on either of the two Portal machines. There doesn't appear to be a workaround in my case. I've tried this with two entirely clean installations and both times it has happened to me. Once I restart the Portal, I begin getting 400s and 500s on the Portal Sharing REST application which prevents Portal from behaving as expected. The logs show all of the things mentioned above in others' posts. @JonathanQuinn you stated previously "I'm a bit doubtful that the problem is as simple as stopping and starting the standby" but from many of our perspectives restarting the Portal breaks the configuration. It sounds like this issue occurs for on-premise and cloud deployments of HA.

Re: AWS 10.8.1 HA Portal not restarting

Trevor_Hart — Mon, 25 Jan 2021 18:49:01 GMT

@WilliamCraft weve been told by Esri that this has been replicated and is fixed in 10.9.0 - there is a bug number; BUG-000134458

Besides trying to stop/start the servers, killing the javaw.exe process associated with Portal seems to resolve the issue - sometimes I need to do this on both servers (I think it depends which was the primary and which was the standby).