Portal not starting - Issue with hazelcast in Portal

Anonymous User · ‎03-12-2021

Hi Guys,

I have ArcGIS Enterprise 10.8.1 installed in Microsoft Azure in multi-machines environment. Currently, I receive the following message when I try to access Portal (message translated).

"Enable to access Portal machines. Please contact your system administrator."

I receive the following messages in catalina log file in C:\Program Files\ArcGIS\Portal\framework\runtime\tomcat\logs :

:37.006 INFOS [main] com.esri.arcgis.portal.tomcat.ArcGISDataSourceFactory.getObjectInstance ArcGISDataSourceFactory.getObjectInstance invoked
12-Mar-2021 07:50:37.016 INFOS [main] com.esri.arcgis.portal.tomcat.ArcGISDataSourceFactory.getObjectInstance Site is created. Using DB admin credentials.
12-Mar-2021 07:50:37.224 INFOS [main] org.apache.jasper.servlet.TldScanner.scanJars Au moins un fichier JAR a été analysé pour trouver des TLDs mais il n'en contenait pas, le mode "debug" du journal peut être activé pour obtenir une liste complète de JAR scannés sans succès; éviter d'analyser des JARs inutilement peut améliorer sensiblement le temps de démarrage et le temps de compilation des JSPs
12-Mar-2021 07:50:41.251 AVERTISSEMENT [main] com.hazelcast.instance.HazelcastInstanceFactory.null Hazelcast is starting in a Java modular environment (Java 9 and newer) but without proper access to required Java packages. Use additional Java arguments to provide Hazelcast access to Java internal API. The internal API access is used to get the best performance results. Arguments to be used:
--add-modules java.se --add-exports java.base/jdk.internal.ref=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.nio=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.management/sun.management=ALL-UNNAMED --add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED
12-Mar-2021 07:50:41.470 AVERTISSEMENT [main] com.hazelcast.instance.AddressPicker.null [LOCAL] [24ea005a-61ce-49ac-aac0-3b0617e3d807] [3.11] You configured your member address as host name. Please be aware of that your dns can be spoofed. Make sure that your dns configurations are correct.
12-Mar-2021 07:50:41.470 INFOS [main] com.hazelcast.instance.AddressPicker.null [LOCAL] [24ea005a-61ce-49ac-aac0-3b0617e3d807] [3.11] Resolving domain name 'ARCGIS-PORTAL.PPPP.LOCAL' to address(es): [172.168.1.11, fe80:0:0:0:2d16:8ce3:c8f4:b7e5%13]
12-Mar-2021 07:50:41.470 INFOS [main] com.hazelcast.instance.AddressPicker.null [LOCAL] [24ea005a-61ce-49ac-aac0-3b0617e3d807] [3.11] Interfaces is disabled, trying to pick one address from TCP-IP config addresses: [ARCGIS-PORTAL.PPPP.LOCAL/172.168.1.11, ARCGIS-PORTAL.PPPP.LOCAL/fe80:0:0:0:2d16:8ce3:c8f4:b7e5%13]
12-Mar-2021 07:50:41.470 INFOS [main] com.hazelcast.instance.AddressPicker.null [LOCAL] [24ea005a-61ce-49ac-aac0-3b0617e3d807] [3.11] Prefer IPv4 stack is true, prefer IPv6 addresses is false
12-Mar-2021 07:50:41.516 INFOS [main] com.hazelcast.instance.AddressPicker.null [LOCAL] [24ea005a-61ce-49ac-aac0-3b0617e3d807] [3.11] Picked [ARCGIS-PORTAL.PPPP.LOCAL]:5701, using socket ServerSocket[addr=/0:0:0:0:0:0:0:0,localport=5701], bind any local is true
12-Mar-2021 07:50:41.595 INFOS [main] com.hazelcast.system.null [ARCGIS-PORTAL.PPPP.LOCAL]:5701 [24ea005a-61ce-49ac-aac0-3b0617e3d807] [3.11] Hazelcast 3.11 (20181023 - 1500bbb) starting at [ARCGIS-PORTAL.PPPP.LOCAL]:5701
12-Mar-2021 07:50:41.595 INFOS [main] com.hazelcast.system.null [ARCGIS-PORTAL.PPPP.LOCAL]:5701 [24ea005a-61ce-49ac-aac0-3b0617e3d807] [3.11] Copyright (c) 2008-2018, Hazelcast, Inc. All Rights Reserved.
12-Mar-2021 07:50:41.595 INFOS [main] com.hazelcast.instance.Node.null [ARCGIS-PORTAL.PPPP.LOCAL]:5701 [24ea005a-61ce-49ac-aac0-3b0617e3d807] [3.11] A non-empty group password is configured for the Hazelcast member. Starting with Hazelcast version 3.8.2, members with the same group name, but with different group passwords (that do not use authentication) form a cluster. The group password configuration will be removed completely in a future release.
12-Mar-2021 07:50:42.876 INFOS [main] com.hazelcast.spi.impl.operationservice.impl.BackpressureRegulator.null [ARCGIS-PORTAL.PPPP.LOCAL]:5701 [24ea005a-61ce-49ac-aac0-3b0617e3d807] [3.11] Backpressure is disabled
12-Mar-2021 07:50:45.298 INFOS [main] com.hazelcast.instance.Node.null [ARCGIS-PORTAL.PPPP.LOCAL]:5701 [24ea005a-61ce-49ac-aac0-3b0617e3d807] [3.11] Creating TcpIpJoiner
12-Mar-2021 07:50:46.221 INFOS [main] com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl.null [ARCGIS-PORTAL.PPPP.LOCAL]:5701 [24ea005a-61ce-49ac-aac0-3b0617e3d807] [3.11] Starting 4 partition threads and 3 generic threads (1 dedicated for priority tasks)
12-Mar-2021 07:50:46.221 INFOS [main] com.hazelcast.internal.diagnostics.Diagnostics.null [ARCGIS-PORTAL.PPPP.LOCAL]:5701 [24ea005a-61ce-49ac-aac0-3b0617e3d807] [3.11] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
12-Mar-2021 07:50:46.241 INFOS [main] com.hazelcast.core.LifecycleService.null [ARCGIS-PORTAL.PPPP.LOCAL]:5701 [24ea005a-61ce-49ac-aac0-3b0617e3d807] [3.11] [ARCGIS-PORTAL.PPPP.LOCAL]:5701 is STARTING
12-Mar-2021 07:50:46.356 INFOS [hz._hzInstance_1_24ea005a-61ce-49ac-aac0-3b0617e3d807.cached.thread-3] com.hazelcast.nio.tcp.TcpIpConnector.null [ARCGIS-PORTAL.PPPP.LOCAL]:5701 [24ea005a-61ce-49ac-aac0-3b0617e3d807] [3.11] Connecting to ARCGIS-PORTAL.PPPP.LOCAL/172.168.1.11:5703, timeout: 0, bind-any: true
12-Mar-2021 07:50:46.356 INFOS [hz._hzInstance_1_24ea005a-61ce-49ac-aac0-3b0617e3d807.cached.thread-2] com.hazelcast.nio.tcp.TcpIpConnector.null [ARCGIS-PORTAL.PPPP.LOCAL]:5701 [24ea005a-61ce-49ac-aac0-3b0617e3d807] [3.11] Connecting to ARCGIS-PORTAL.PPPP.LOCAL/172.168.1.11:5702, timeout: 0, bind-any: true
12-Mar-2021 07:50:47.589 INFOS [hz._hzInstance_1_24ea005a-61ce-49ac-aac0-3b0617e3d807.cached.thread-3] com.hazelcast.nio.tcp.TcpIpConnector.null [ARCGIS-PORTAL.PPPP.LOCAL]:5701 [24ea005a-61ce-49ac-aac0-3b0617e3d807] [3.11] Could not connect to: ARCGIS-PORTAL.PPPP.LOCAL/172.168.1.11:5703. Reason: SocketException[Connection refused: connect to address ARCGIS-PORTAL.PPPP.LOCAL/172.168.1.11:5703]
12-Mar-2021 07:50:47.589 INFOS [hz._hzInstance_1_24ea005a-61ce-49ac-aac0-3b0617e3d807.cached.thread-3] com.hazelcast.cluster.impl.TcpIpJoiner.null [ARCGIS-PORTAL.PPPP.LOCAL]:5701 [24ea005a-61ce-49ac-aac0-3b0617e3d807] [3.11] [ARCGIS-PORTAL.PPPP.LOCAL]:5703 is added to the blacklist.
12-Mar-2021 07:50:47.589 INFOS [hz._hzInstance_1_24ea005a-61ce-49ac-aac0-3b0617e3d807.cached.thread-2] com.hazelcast.nio.tcp.TcpIpConnector.null [ARCGIS-PORTAL.PPPP.LOCAL]:5701 [24ea005a-61ce-49ac-aac0-3b0617e3d807] [3.11] Could not connect to: ARCGIS-PORTAL.PPPP.LOCAL/172.168.1.11:5702. Reason: SocketException[Connection refused: connect to address ARCGIS-PORTAL.PPPP.LOCAL/172.168.1.11:5702]
12-Mar-2021 07:50:47.589 INFOS [hz._hzInstance_1_24ea005a-61ce-49ac-aac0-3b0617e3d807.cached.thread-2] com.hazelcast.cluster.impl.TcpIpJoiner.null [ARCGIS-PORTAL.PPPP.LOCAL]:5701 [24ea005a-61ce-49ac-aac0-3b0617e3d807] [3.11] [ARCGIS-PORTAL.PPPP.LOCAL]:5702 is added to the blacklist.
12-Mar-2021 07:50:48.355 INFOS [main] com.hazelcast.internal.cluster.ClusterService.null [ARCGIS-PORTAL.PPPP.LOCAL]:5701 [24ea005a-61ce-49ac-aac0-3b0617e3d807] [3.11]

Members {size:1, ver:1} [
Member [ARCGIS-PORTAL.PPPP.LOCAL]:5701 - f4365794-3991-4feb-bd2a-c6a719943f32 this
]

12-Mar-2021 07:50:48.449 INFOS [main] com.hazelcast.core.LifecycleService.null [ARCGIS-PORTAL.PPPP.LOCAL]:5701 [24ea005a-61ce-49ac-aac0-3b0617e3d807] [3.11] [ARCGIS-PORTAL.PPPP.LOCAL]:5701 is STARTED
12-Mar-2021 07:50:49.886 INFOS [main] com.hazelcast.internal.partition.impl.PartitionStateManager.null [ARCGIS-PORTAL.PPPP.LOCAL]:5701 [24ea005a-61ce-49ac-aac0-3b0617e3d807] [3.11] Initializing cluster partition table arrangement...
12-Mar-2021 07:50:59.545 GRAVE [main] org.apache.catalina.core.StandardContext.startInternal Un ou plusieurs écouteurs n'ont pas pu démarrer, les détails sont dans le fichier de log du conteneur
12-Mar-2021 07:50:59.545 GRAVE [main] org.apache.catalina.core.StandardContext.startInternal Erreur de démarrage du contexte [/arcgis/sharing] suite aux erreurs précédentes
12-Mar-2021 07:50:59.637 AVERTISSEMENT [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads L'application web [arcgis#sharing] semble avoir démarré un thread nommé [ExpiringMap.timer] mais ne l'a pas arrêté, ce qui va probablement créer une fuite de mémoire; la trace du thread est:
java.base@11.0.6/java.lang.Object.wait(Native Method)
java.base@11.0.6/java.lang.Object.wait(Unknown Source)
java.base@11.0.6/java.util.TimerThread.mainLoop(Unknown Source)
java.base@11.0.6/java.util.TimerThread.run(Unknown Source)
12-Mar-2021 07:50:59.637 AVERTISSEMENT [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads L'application web [arcgis#sharing] semble avoir démarré un thread nommé [hz._hzInstance_1_24ea005a-61ce-49ac-aac0-3b0617e3d807.scheduled.thread-] mais ne l'a pas arrêté, ce qui va probablement créer une fuite de mémoire; la trace du thread est:
java.base@11.0.6/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.6/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
java.base@11.0.6/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
java.base@11.0.6/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
java.base@11.0.6/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
java.base@11.0.6/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.6/java.lang.Thread.run(Unknown Source)
com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:64)
com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:80)
12-Mar-2021 07:50:59.637 AVERTISSEMENT [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads L'application web [arcgis#sharing] semble avoir démarré un thread nommé [hz._hzInstance_1_24ea005a-61ce-49ac-aac0-3b0617e3d807.event-1] mais ne l'a pas arrêté, ce qui va probablement créer une fuite de mémoire; la trace du thread est:
java.base@11.0.6/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.6/java.util.concurrent.locks.LockSupport.park(Unknown Source)
java.base@11.0.6/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
java.base@11.0.6/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
com.hazelcast.util.executor.StripedExecutor$Worker.run(StripedExecutor.java:208)
12-Mar-2021 07:50:59.637 AVERTISSEMENT [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads L'application web [arcgis#sharing] semble avoir démarré un thread nommé [hz._hzInstance_1_24ea005a-61ce-49ac-aac0-3b0617e3d807.event-2] mais ne l'a pas arrêté, ce qui va probablement créer une fuite de mémoire; la trace du thread est:
java.base@11.0.6/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.6/java.util.concurrent.locks.LockSupport.park(Unknown Source)
java.base@11.0.6/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
java.base@11.0.6/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
com.hazelcast.util.executor.StripedExecutor$Worker.run(StripedExecutor.java:208)
12-Mar-2021 07:50:59.637 AVERTISSEMENT [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads L'application web [arcgis#sharing] semble avoir démarré un thread nommé [hz._hzInstance_1_24ea005a-61ce-49ac-aac0-3b0617e3d807.event-3] mais ne l'a pas arrêté, ce qui va probablement créer une fuite de mémoire; la trace du thread est:
java.base@11.0.6/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.6/java.util.concurrent.locks.LockSupport.park(Unknown Source)
java.base@11.0.6/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
java.base@11.0.6/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
com.hazelcast.util.executor.StripedExecutor$Worker.run(StripedExecutor.java:208)

Any idea what's going wrong?

Thanks

@Anonymous User

JonathanQuinn · ‎03-12-2021

Members {size:1, ver:1} [
Member [ARCGIS-PORTAL.PPPP.LOCAL]:5701 - f4365794-3991-4feb-bd2a-c6a719943f32 this
]

That line is an indication the cluster is getting created successfully. You can ignore most of what you see in the tomcat logs.

What do the severe or warning messages in the portal logs say?