AWS 10.8.1 HA Portal not restarting

[ShanonLoughton]

Hi

In AWS Ive deployed ArcGIS Enterprise 10.8.1 using Windows in HA and using ELB, both templates available here.
It works on first deployment and successful CloudFormation create-stack.
However when stopping/restarting Portal service on a given instance, or rebooting the Portal instances (either of them), they fail to restart Portal service, citing internal database not started properly, and symptoms like passwords rejected at /portaladmin login.
Here's one instance's set of logs:
Windows EventViewer

PostgreSQL : The description for Event ID 0 from source PostgreSQL cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer. (repeated, initially Information class, but further back is Error class)
Portal: The description for Event ID 5 from source Portal for ArcGIS cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer. (repeated, initially Information class, but further back is Error class)

"C:\Program Files\ArcGIS\Portal\framework\service\logs\service-error-0.log"

Mon Aug 24 13:01:13.237 UTC 2020 Exception in thread "I/O dispatcher 17" 
Mon Aug 24 13:01:13.237 UTC 2020 java.lang.NoSuchMethodError: org.apache.http.ConnectionClosedException: method 'void <init>()' not found
Mon Aug 24 13:01:13.237 UTC 2020 at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.endOfInput(HttpAsyncRequestExecutor.java:356)
Mon Aug 24 13:01:13.237 UTC 2020 at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:261)
Mon Aug 24 13:01:13.237 UTC 2020 at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
Mon Aug 24 13:01:13.237 UTC 2020 at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
Mon Aug 24 13:01:13.237 UTC 2020 at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
Mon Aug 24 13:01:13.237 UTC 2020 at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
Mon Aug 24 13:01:13.237 UTC 2020 at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
Mon Aug 24 13:01:13.237 UTC 2020 at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
Mon Aug 24 13:01:13.237 UTC 2020 at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
Mon Aug 24 13:01:13.237 UTC 2020 at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
Mon Aug 24 13:01:13.237 UTC 2020 at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
Mon Aug 24 13:01:13.237 UTC 2020 at java.base/java.lang.Thread.run(Unknown Source)

"C:\arcgisportal\logs\10.72.4.42\portal\portal-20200824.123347-12940-0.0.log"

<Msg time="2020-08-24T12:34:34,908" type="SEVERE" code="218010" source="Portal Admin" process="12940" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID="">The portal has been initialized and configured but is not accessible. The internal portal database does not appear to be running or accepting connections. Restart the portal machine or machines and if the problem persists, contact Esri technical support (U.S.) or your distributor (customers outside the U.S.).</Msg>
<Msg time="2020-08-24T12:34:34,908" type="WARNING" code="218012" source="Portal Admin" process="12940" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID="">Error at check and update urls. The portal is currently unavailable. Please contact your portal administrator.</Msg>
<Msg time="2020-08-24T12:35:12,375" type="SEVERE" code="218010" source="Portal Admin" process="12940" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID="">The portal has been initialized and configured but is not accessible. The internal portal database does not appear to be running or accepting connections. Restart the portal machine or machines and if the problem persists, contact Esri technical support (U.S.) or your distributor (customers outside the U.S.).</Msg>
<Msg time="2020-08-24T12:35:18,695" type="SEVERE" code="218010" source="Portal Admin" process="12940" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID="">The portal has been initialized and configured but is not accessible. The internal portal database does not appear to be running or accepting connections. Restart the portal machine or machines and if the problem persists, contact Esri technical support (U.S.) or your distributor (customers outside the U.S.).</Msg>
<Msg time="2020-08-24T12:35:22,903" type="WARNING" code="218014" source="Portal" process="12940" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID="">Starting Index Service.</Msg>
<Msg time="2020-08-24T12:36:28,437" type="SEVERE" code="218010" source="Portal Admin" process="12940" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID="">The portal has been initialized and configured but is not accessible. The internal portal database does not appear to be running or accepting connections. Restart the portal machine or machines and if the problem persists, contact Esri technical support (U.S.) or your distributor (customers outside the U.S.).</Msg>
<Msg time="2020-08-24T12:36:49,886" type="SEVERE" code="218010" source="Portal Admin" process="12940" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID="">The portal has been initialized and configured but is not accessible. The internal portal database does not appear to be running or accepting connections. Restart the portal machine or machines and if the problem persists, contact Esri technical support (U.S.) or your distributor (customers outside the U.S.).</Msg>
<Msg time="2020-08-24T12:37:44,483" type="SEVERE" code="218010" source="Portal Admin" process="12940" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID="">The portal has been initialized and configured but is not accessible. The internal portal database does not appear to be running or accepting connections. Restart the portal machine or machines and if the problem persists, contact Esri technical support (U.S.) or your distributor (customers outside the U.S.).</Msg>
<Msg time="2020-08-24T12:38:09,93" type="SEVERE" code="218010" source="Portal Admin" process="12940" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID="">The portal has been initialized and configured but is not accessible. The internal portal database does not appear to be running or accepting connections. Restart the portal machine or machines and if the problem persists, contact Esri technical support (U.S.) or your distributor (customers outside the U.S.).</Msg>
<Msg time="2020-08-24T12:38:41,750" type="WARNING" code="218015" source="Portal" process="12940" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID="">Started Index Service.</Msg>
<Msg time="2020-08-24T12:38:41,828" type="WARNING" code="217060" source="Portal" process="12940" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID="">The database server was found to be stopped. Re-starting it.</Msg>
<Msg time="2020-08-24T12:38:51,974" type="WARNING" code="217064" source="Portal" process="12940" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID="">The web server was found to be stopped. Re-starting it.</Msg>
<Msg time="2020-08-24T12:43:09,108" type="WARNING" code="219999" source="Portal Admin" process="5948" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID=""> Failed to generate a token for user 'portaladmin'. java.lang.Exception: Server returned HTTP status code 404. Requested URL: https://10.72.4.42:7443/arcgis/sharing/generateToken</Msg>
<Msg time="2020-08-24T12:43:12,517" type="WARNING" code="219999" source="Portal Admin" process="5948" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID=""> Failed to generate a token for user 'portaladmin'. java.lang.Exception: Server returned HTTP status code 404. Requested URL: https://10.72.4.42:7443/arcgis/sharing/generateToken</Msg>
<Msg time="2020-08-24T12:43:15,819" type="WARNING" code="219999" source="Portal Admin" process="5948" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID=""> Failed to generate a token for user 'portaladmin'. java.lang.Exception: Server returned HTTP status code 404. Requested URL: https://10.72.4.42:7443/arcgis/sharing/generateToken</Msg>
<Msg time="2020-08-24T12:43:20,950" type="WARNING" code="219999" source="Portal Admin" process="5948" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID=""> Failed to generate a token for user 'portaladmin'. java.lang.Exception: Server returned HTTP status code 404. Requested URL: https://10.72.4.42:7443/arcgis/sharing/generateToken</Msg>
<Msg time="2020-08-24T12:43:31,158" type="WARNING" code="219999" source="Portal Admin" process="5948" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID=""> Failed to generate a token for user 'portaladmin'. java.lang.Exception: Server returned HTTP status code 404. Requested URL: https://10.72.4.42:7443/arcgis/sharing/generateToken</Msg>
<Msg time="2020-08-24T13:01:13,481" type="WARNING" code="217068" source="Portal" process="12940" thread="1" methodName="" machine="10.72.4.42" user="" elapsed="" requestID="">Error in the process monitor watchdog thread. sleep interrupted</Msg>

"C:\arcgisportal\logs\database\pg_log\postgresql-2020-08-24_060707.log"

2020-08-24 05:38:42 PDT: [6272]: LOG: database system was shut down at 2020-08-24 05:32:36 PDT
2020-08-24 05:38:42 PDT: [5824]: LOG: database system is ready to accept connections
2020-08-24 05:38:42 PDT: [10060]: LOG: incomplete startup packet
2020-08-24 05:38:42 PDT: [10180]: FATAL: no pg_hba.conf entry for host "::1", user "arcgis", database "arcgis", SSL off
2020-08-24 05:39:12 PDT: [12016]: LOG: incomplete startup packet
2020-08-24 05:40:12 PDT: [12196]: LOG: incomplete startup packet
2020-08-24 05:41:12 PDT: [12120]: LOG: incomplete startup packet
2020-08-24 05:42:12 PDT: [11232]: LOG: incomplete startup packet (repeated)
2020-08-24 05:42:33 PDT: [11348]: LOG: could not receive data from client: An existing connection was forcibly closed by the remote host. (repeated)

Thanks for any help.

[NicolasGIS]

Hello,

I have exactly the same issue.

My setup is the following:

1 external load balancer load balancing traffic of port 443, 6443 and 7443.

2 VMS with Portal for ArcGIS 10.8.1, ArcGIS Server 10.8.1, Relational Datastore 10.8.1 and 2 * ArcGIS WebAdaptor (1 for Portal and 1 for ArcGIS Server).

The setup was configured using ArcGIS DSC and tt was working fine.

Then I switched off one VM, and it was still working fine: happy to have HA .  But then I started it again and now I have the very same symptoms:

- When trying to log in to portaladmin, I receive an error "You are not authorized to use this resource.".

- Portal database does not want to start with the same error as you : "FATAL: no pg_hba.conf entry for host "::1", user "myuser", database "arcgis", SSL off"

I had a look at the pg_hba.conf and it is true that there is no entry for service account "myuser" as there are only entries for portal primary administrator "siteadmin".

It is the third times this happen to me: I recreate VM, configure HA with DSC and everything works fine. Then I switch off one VM and then issues start occuring and my portal is messed up.

There is definetly an issue somewhere but I can't find yet the logic !

Did you find a solution ?

Jonathan Quinn‌, any idea ?

Thanks !

[JonathanQuinn]

FATAL: no pg_hba.conf entry for host "::1", user "myuser", database "arcgis", SSL off

This error is not indicative of the database not starting; Portal checks whether the database is up and in some places, doesn't pass in the right username. The process then assumes the check will use the account used to run the Portal for ArcGIS service which returns the error.

Are you testing in a cloud deployment, or on premises? Can you access the Sharing API, ex https://machine.domain.com:7443/arcgis/sharing/rest?

[NicolasGIS]

Many thanks Jonathan Quinn‌ for having a look at this issue so quickly and for your explanation.

I am testing an on premises deployment (based on Openstack).

I cannot access https://machine.domain.com:7443/arcgis/sharing/rest:

It returns 404 on both machines.

I also had a look at localhost logs in <install directory>\framework\runtime\tomcat\logs:

"""

org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class [org.springframework.web.context.ContextLoaderListener]
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'accountsRequestHandler' defined in class path resource [spring/gw-dispatcher.xml]: Cannot resolve reference to bean 'gwAccounts' while setting bean property 'accounts'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'gwAccounts' defined in class path resource [spring/gw-config.xml]: Cannot resolve reference to bean 'gwDatabase' while setting bean property 'database'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'gwDatabase' defined in class path resource [spring/gw-config.xml]: Cannot resolve reference to bean 'memoryCache' while setting bean property 'cache'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'memoryCache' defined in class path resource [spring/gw-config-beans-portal.xml]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.esri.gw.cache.PortalDistributedHashImpl]: Constructor threw exception; nested exception is java.lang.IllegalStateException: Node failed to start!
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:378)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:110)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1602)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1354)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:572)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:495)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:317)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:315)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:759)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:869)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550)
at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:409)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:291)
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:103)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4685)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5146)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:717)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:690)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:705)
at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:631)
at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1831)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
at java.base/java.util.concurrent.AbstractExecutorService.submit(Unknown Source)
at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:526)
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:425)
at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1576)
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:309)
at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423)
at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:366)
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:936)
at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:841)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1384)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1374)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
at java.base/java.util.concurrent.AbstractExecutorService.submit(Unknown Source)
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:909)
at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:262)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.StandardService.startInternal(StandardService.java:421)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:930)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.startup.Catalina.start(Catalina.java:633)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:343)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:474)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'gwAccounts' defined in class path resource [spring/gw-config.xml]: Cannot resolve reference to bean 'gwDatabase' while setting bean property 'database'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'gwDatabase' defined in class path resource [spring/gw-config.xml]: Cannot resolve reference to bean 'memoryCache' while setting bean property 'cache'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'memoryCache' defined in class path resource [spring/gw-config-beans-portal.xml]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.esri.gw.cache.PortalDistributedHashImpl]: Constructor threw exception; nested exception is java.lang.IllegalStateException: Node failed to start!
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:378)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:110)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1602)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1354)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:572)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:495)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:317)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:315)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:367)
... 56 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'gwDatabase' defined in class path resource [spring/gw-config.xml]: Cannot resolve reference to bean 'memoryCache' while setting bean property 'cache'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'memoryCache' defined in class path resource [spring/gw-config-beans-portal.xml]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.esri.gw.cache.PortalDistributedHashImpl]: Constructor threw exception; nested exception is java.lang.IllegalStateException: Node failed to start!
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:378)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:110)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1602)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1354)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:572)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:495)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:317)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:315)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:367)
... 66 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'memoryCache' defined in class path resource [spring/gw-config-beans-portal.xml]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.esri.gw.cache.PortalDistributedHashImpl]: Constructor threw exception; nested exception is java.lang.IllegalStateException: Node failed to start!
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:282)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1267)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1124)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:535)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:495)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:317)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:315)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:367)
... 76 more
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.esri.gw.cache.PortalDistributedHashImpl]: Constructor threw exception; nested exception is java.lang.IllegalStateException: Node failed to start!
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:182)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:117)
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:275)
... 85 more
Caused by: java.lang.IllegalStateException: Node failed to start!
at com.hazelcast.instance.HazelcastInstanceImpl.<init>(HazelcastInstanceImpl.java:133)
at com.hazelcast.instance.HazelcastInstanceFactory.constructHazelcastInstance(HazelcastInstanceFactory.java:202)
at com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:181)
at com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:131)
at com.hazelcast.core.Hazelcast.newHazelcastInstance(Hazelcast.java:57)
at com.esri.gw.cache.HazelCastUtil.getHazelcastInstance(HazelCastUtil.java:46)
at com.esri.gw.cache.DistributedMap.a(DistributedMap.java:92)
at com.esri.gw.cache.DistributedMap.<init>(DistributedMap.java:38)
at com.esri.gw.cache.PortalDistributedHashImpl.<init>(PortalDistributedHashImpl.java:44)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source)
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:170)
... 87 more

Thanks !

[JonathanQuinn]

It sounds like an internal memory cache component couldn't start up, which listens on a port between 5701 and 5703. Do you have anything running on those ports?

[NicolasGIS]

On the server from which this log is extracted, there is nothing running on those ports.

On the other server, there is one javaw process listening on port 5701

Note that on the latter server, the error is slightly different:

SEVERE [main] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class [org.springframework.web.context.ContextLoaderListener]
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'accountsRequestHandler' defined in class path resource [spring/gw-dispatcher.xml]: Cannot resolve reference to bean 'gwAccounts' while setting bean property 'accounts'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'gwAccounts' defined in class path resource [spring/gw-config.xml]: Cannot resolve reference to bean 'gwCommunity' while setting bean property 'community'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'gwCommunity' defined in class path resource [spring/gw-config.xml]: Cannot resolve reference to bean 'securityPermissions' while setting bean property 'securityPermissions'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'securityPermissions' defined in class path resource [spring/gw-config.xml]: Cannot resolve reference to bean 'gwContent' while setting bean property 'content'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'gwContent' defined in class path resource [spring/gw-config.xml]: Cannot resolve reference to bean 'gwSearch' while setting bean property 'search'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'gwSearch' defined in class path resource [spring/gw-config.xml]: Cannot resolve reference to bean 'searchIndexRemoteServiceProxy' while setting bean property 'index'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'searchIndexRemoteServiceProxy' defined in class path resource [spring/index-clients_es_portal.xml]: Cannot resolve reference to bean 'searchIndexServiceDispatcher' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'searchIndexServiceDispatcher' defined in class path resource [spring/index-clients_es_portal.xml]: Cannot resolve reference to bean 'searchIndexServiceProxyEs' while setting bean property 'indexingServices' with key [0]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'searchIndexServiceProxyEs' defined in class path resource [spring/index-clients_es_portal.xml]: Invocation of init method failed; nested exception is java.net.ConnectException: Connection refused: no further information
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:378)

Thanks

[StefKoroblitsas]

Nicolas GIS‌ Hi, did you find a resolution? I'm getting similar logs, we have support looking at it but its taking a long time to resolve

Cheers,

Stef

[NicolasGIS]

Hi Stef Koroblitsas,

Unfortunately not ! I have been struggling for weeks trying to identify a pattern but I can't figure out what is the issue.

Please let me know if anything come out from your support case (I was about to open one as well as I am in a dead end...) !

Any idea Jonathan Quinn‌ ? Seems like I am not the only one !

[JonathanQuinn]

You'll need to check why the java.exe process is not starting. You can look at the logs under <install directory>\framework\runtime\ds\usr. They may tell you why it won't start.

[NicolasGIS]

I believe it is BUG-000134458:

After rebooting the standby Portal in HA on Windows (OS), the portal Postgres database does not reboot properly to allow successful connections in portal/sharing/rest or portaladmin.

As I had in portal folder loads of DBs !