GeoEvent Server - the zookeeper cannot be found

[roemhildtg]

I'm getting the generic error message about geo-event manager not being able to connect to the zookeeper so I started poking around in the logs. 

I'm seeing a certificate error:

java.security.cert.CertificateParsingException: signed fields invalid
     at sun.security.x509.X509CertImpl.parse(X509CertImpl.java:1791) [?:?]
     at sun.security.x509.X509CertImpl.<init>(X509CertImpl.java:195) [?:?]
     at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:102) [?:?]
     at java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339) [?:?]
     at com.esri.ges.httpclient.HttpClientService.readSiteCertificates(HttpClientService.java:378) [53:com.esri.ges.framework.httpclient:10.6.1]
     at com.esri.ges.httpclient.HttpClientService.init(HttpClientService.java:429) [53:com.esri.ges.framework.httpclient:10.6.1]
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
     at java.lang.reflect.Method.invoke(Method.java:498) ~[?:?]
     at org.apache.aries.blueprint.utils.ReflectionUtils.invoke(ReflectionUtils.java:299) [245:org.apache.aries.blueprint.core:1.8.0]
     at org.apache.aries.blueprint.container.BeanRecipe.invoke(BeanRecipe.java:980) [245:org.apache.aries.blueprint.core:1.8.0]
     at org.apache.aries.blueprint.container.BeanRecipe.runBeanProcInit(BeanRecipe.java:736) [245:org.apache.aries.blueprint.core:1.8.0]
     at org.apache.aries.blueprint.container.BeanRecipe.internalCreate2(BeanRecipe.java:848) [245:org.apache.aries.blueprint.core:1.8.0]
     at org.apache.aries.blueprint.container.BeanRecipe.internalCreate(BeanRecipe.java:811) [245:org.apache.aries.blueprint.core:1.8.0]
     at org.apache.aries.blueprint.di.AbstractRecipe$1.call(AbstractRecipe.java:79) [245:org.apache.aries.blueprint.core:1.8.0]
     at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:?]
     at org.apache.aries.blueprint.di.AbstractRecipe.create(AbstractRecipe.java:88) [245:org.apache.aries.blueprint.core:1.8.0]
     at org.apache.aries.blueprint.container.BlueprintRepository.createInstances(BlueprintRepository.java:255) [245:org.apache.aries.blueprint.core:1.8.0]
     at org.apache.aries.blueprint.container.BlueprintRepository.createAll(BlueprintRepository.java:186) [245:org.apache.aries.blueprint.core:1.8.0]
     at org.apache.aries.blueprint.container.BlueprintContainerImpl.instantiateEagerComponents(BlueprintContainerImpl.java:725) [245:org.apache.aries.blueprint.core:1.8.0]
     at org.apache.aries.blueprint.container.BlueprintContainerImpl.doRun(BlueprintContainerImpl.java:412) [245:org.apache.aries.blueprint.core:1.8.0]
     at org.apache.aries.blueprint.container.BlueprintContainerImpl.run(BlueprintContainerImpl.java:277) [245:org.apache.aries.blueprint.core:1.8.0]
     at org.apache.aries.blueprint.container.BlueprintExtender.createContainer(BlueprintExtender.java:300) [245:org.apache.aries.blueprint.core:1.8.0]
     at org.apache.aries.blueprint.container.BlueprintExtender.createContainer(BlueprintExtender.java:269) [245:org.apache.aries.blueprint.core:1.8.0]
     at org.apache.aries.blueprint.container.BlueprintExtender.createContainer(BlueprintExtender.java:265) [245:org.apache.aries.blueprint.core:1.8.0]
     at org.apache.aries.blueprint.container.BlueprintExtender.modifiedBundle(BlueprintExtender.java:255) [245:org.apache.aries.blueprint.core:1.8.0]
     at org.apache.aries.blueprint.container.BlueprintExtender.addingBundle(BlueprintExtender.java:218) [245:org.apache.aries.blueprint.core:1.8.0]
     at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.customizerAdding(BundleHookBundleTracker.java:487) [245:org.apache.aries.blueprint.core:1.8.0]
     at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.customizerAdding(BundleHookBundleTracker.java:433) [245:org.apache.aries.blueprint.core:1.8.0]
     at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$AbstractTracked.trackAdding(BundleHookBundleTracker.java:749) [245:org.apache.aries.blueprint.core:1.8.0]
     at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$AbstractTracked.trackInitial(BundleHookBundleTracker.java:674) [245:org.apache.aries.blueprint.core:1.8.0]
     at org.apache.aries.util.tracker.hook.BundleHookBundleTracker.open(BundleHookBundleTracker.java:159) [245:org.apache.aries.blueprint.core:1.8.0]
     at org.apache.aries.util.tracker.RecursiveBundleTracker.open(RecursiveBundleTracker.java:106) [245:org.apache.aries.blueprint.core:1.8.0]
     at org.apache.aries.blueprint.container.BlueprintExtender$2.serviceFound(BlueprintExtender.java:119) [245:org.apache.aries.blueprint.core:1.8.0]
     at org.apache.aries.util.tracker.SingleServiceTracker.update(SingleServiceTracker.java:174) [245:org.apache.aries.blueprint.core:1.8.0]
     at org.apache.aries.util.tracker.SingleServiceTracker.findMatchingReference(SingleServiceTracker.java:132) [245:org.apache.aries.blueprint.core:1.8.0]
     at org.apache.aries.util.tracker.SingleServiceTracker.access$200(SingleServiceTracker.java:32) [245:org.apache.aries.blueprint.core:1.8.0]
     at org.apache.aries.util.tracker.SingleServiceTracker$1.serviceChanged(SingleServiceTracker.java:61) [245:org.apache.aries.blueprint.core:1.8.0]
     at org.apache.felix.framework.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:990) [?:?]
     at org.apache.felix.framework.EventDispatcher.fireEventImmediately(EventDispatcher.java:838) [?:?]
     at org.apache.felix.framework.EventDispatcher.fireServiceEvent(EventDispatcher.java:545) [?:?]
     at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4578) [?:?]
     at org.apache.felix.framework.Felix.registerService(Felix.java:3570) [?:?]
     at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:348) [?:?]
     at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:322) [?:?]
     at org.apache.aries.proxy.impl.ProxyManagerActivator.start(ProxyManagerActivator.java:66) [251:org.apache.aries.proxy:1.1.1]
     at org.apache.felix.framework.util.SecureAction.startActivator(SecureAction.java:697) [?:?]
     at org.apache.felix.framework.Felix.activateBundle(Felix.java:2238) [?:?]
     at org.apache.felix.framework.Felix.startBundle(Felix.java:2144) [?:?]
     at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1371) [?:?]
     at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308) [?:?]
     at java.lang.Thread.run(Thread.java:748) [?:?]
2018-08-07T13:24:31,027 | ERROR | FelixStartLevel  | HttpClientService                | 53 - com.esri.ges.framework.httpclient - 10.6.1 | Failed to read certificate file at D:\arcgisserver\config-store\machines\<machine-ip-removed>\<machine.certificate.com.removed>.pfx.cer: signed fields invalid
2018-08-07T13:24:31,027 | INFO  | FelixStartLevel  | HttpClientService                | 53 - com.esri.ges.framework.httpclient - 10.6.1 | signed fields invalid‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

I've already verified that arcgis server has imported and is using the valid certificate. Looks like Geo-event server isn't though...

What do I do now?

[GregoryChristakos]

Hello Gregg,

I'd be curious to know what error in regards to Zookeeper that you were seeing but to comment on the issue at hand, is GeoEvent Server referencing the original self signed certificate?

If so, it may be worth taking a look at what certificates are present in the location specified (see below). You can start by comparing these certificates to those found in the ArcGIS Server framework to see if there's any sort of mismatch (...\ProgramFiles\ArcGIS\Server\framework\etc\certificates).

D:\arcgisserver\config-store\machines\<machine-ip-removed>\<machine.certificate.com.removed>.pfx.cer

More importantly, I think that the type of advice I can give you is largely contingent on the version of ArcGIS GeoEvent Server that you are working with. There were quite a few changes to Zookeeper over the last several versions so that could play into how this issue gets treated. Furthermore, some of the information you omitted in the logs could be useful for more nuanced troubleshooting.

If you are open to doing so, I invite you to log a case with Esri Support Services so that you and I can investigate this issue. Should you choose to go this route, just be sure to reference me in the case request so that it can make its way to me. I'm happy to be of help.

[roemhildtg]

Hi Gregory, Thanks for the info. I'm working on getting a case opened up with my organization, but will need to wait for the appropriate permissions. The version of geo event server is actually 10.6.1, we've just installed a new setup with portal/server/geo-event.

The situation is a little more tricky as we are running this in aws and are setting up a certificate that we received from lets encrypt.

I will try and provide more details as I am able to.

[roemhildtg]

According to the error logs, the certificate referenced by GeoEvent\data\log\karaf.log is the Lets encrypt certificate.

[roemhildtg]

I found what might appear to be the zookeeper log also:

2018-08-13 11:01:46,410 [myid:] - INFO  [WrapperSimpleAppMain:GatewayManager@138] - --Starting zk--
2018-08-13 11:01:48,676 [myid:] - INFO  [Thread-4:QuorumPeerConfig@117] - Reading configuration from: C:\ProgramData\Esri\GeoEvent-Gateway\zookeeper-data\zookeeper.properties
2018-08-13 11:01:48,676 [myid:] - INFO  [Thread-4:QuorumPeerConfig@317] - clientPort is not set
2018-08-13 11:01:48,676 [myid:] - INFO  [Thread-4:QuorumPeerConfig@331] - secureClientPort is not set
2018-08-13 11:01:48,691 [myid:] - WARN  [Thread-4:QuorumPeerConfig@590] - No server failure will be tolerated. You need at least 3 servers.
2018-08-13 11:01:48,691 [myid:1] - INFO  [Thread-4:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2018-08-13 11:01:48,707 [myid:1] - INFO  [Thread-4:DatadirCleanupManager@79] - autopurge.purgeInterval set to 1
2018-08-13 11:01:48,711 [myid:1] - INFO  [Thread-4:ManagedUtil@46] - Log4j found with jmx enabled.
2018-08-13 11:01:48,713 [myid:1] - INFO  [PurgeTask:DatadirCleanupManager$PurgeTask@138] - Purge task started.
2018-08-13 11:01:48,754 [myid:1] - INFO  [PurgeTask:DatadirCleanupManager$PurgeTask@144] - Purge task completed.
2018-08-13 11:01:48,939 [myid:] - INFO  [Thread-5:Environment@109] - Client environment:zookeeper.version=3.5.3-beta-8ce24f9e675cbefffb8f21a47e06b42864475a60, built on 04/03/2017 16:19 GMT
2018-08-13 11:01:48,939 [myid:] - INFO  [Thread-5:Environment@109] - Client environment:host.name=portal.gis.wsbeng.com
2018-08-13 11:01:48,939 [myid:] - INFO  [Thread-5:Environment@109] - Client environment:java.version=1.8.0_162
2018-08-13 11:01:48,939 [myid:] - INFO  [Thread-5:Environment@109] - Client environment:java.vendor=Oracle Corporation
2018-08-13 11:01:48,939 [myid:] - INFO  [Thread-5:Environment@109] - Client environment:java.home=C:\Program Files\ArcGIS\Server\framework\runtime\jre
2018-08-13 11:01:48,939 [myid:] - INFO  [Thread-5:Environment@109] - Client environment:java.class.path=C:\Program Files\ArcGIS\Server\/GeoEvent/gateway/lib/wrapper/wrapper.jar;C:\Program Files\ArcGIS\Server\/GeoEvent/gateway/lib/gateway-10.6.1.jar;C:\Program Files\ArcGIS\Server\/GeoEvent/gateway/etc;C:\Program Files\ArcGIS\Server\/GeoEvent/system/org/slf4j/slf4j-api/1.7.7/slf4j-api-1.7.7.jar;C:\Program Files\ArcGIS\Server\/GeoEvent/system/org/slf4j/slf4j-log4j12/1.7.7/slf4j-log4j12-1.7.7.jar;C:\Program Files\ArcGIS\Server\/GeoEvent/system/log4j/log4j/1.2.17/log4j-1.2.17.jar;C:\Program Files\ArcGIS\Server\/GeoEvent/system/io/netty/netty/3.7.0.Final/netty-3.7.0.Final.jar;C:\Program Files\ArcGIS\Server\/GeoEvent/system/org/apache/curator/curator-framework/4.0.0/curator-framework-4.0.0.jar;C:\Program Files\ArcGIS\Server\/GeoEvent/system/org/apache/curator/curator-client/4.0.0/curator-client-4.0.0.jar;C:\Program Files\ArcGIS\Server\/GeoEvent/system/org/apache/curator/curator-recipes/4.0.0/curator-recipes-4.0.0.jar;C:\Program Files\ArcGIS\Server\/GeoEvent/system/com/esri/ges/arcgis-discovery/10.6.1/arcgis-discovery-10.6.1.jar;C:\Program Files\ArcGIS\Server\/GeoEvent/system/commons-configuration/commons-configuration/1.9/commons-configuration-1.9.jar;C:\Program Files\ArcGIS\Server\/GeoEvent/system/commons-lang/commons-lang/2.6/commons-lang-2.6.jar;C:\Program Files\ArcGIS\Server\/GeoEvent/system/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar;C:\Program Files\ArcGIS\Server\/GeoEvent/system/commons-collections/commons-collections/3.2.2/commons-collections-3.2.2.jar;C:\Program Files\ArcGIS\Server\/GeoEvent/system/org/apache/logging/log4j/log4j-api/2.6.2/log4j-api-2.6.2.jar;C:\Program Files\ArcGIS\Server\/GeoEvent/system/commons-codec/commons-codec/1.9/commons-codec-1.9.jar;C:\Program Files\ArcGIS\Server\/GeoEvent/system/commons-io/commons-io/2.4/commons-io-2.4.jar;C:\Program Files\ArcGIS\Server\/GeoEvent/system/com/google/guava/guava/18.0/guava-18.0.jar;C:\Program Files\ArcGIS\Server\/GeoEvent/system/org/apache/kafka/kafka-esri_2.11/0.11.0.0/kafka-esri_2.11-0.11.0.0.jar;C:\Program Files\ArcGIS\Server\/GeoEvent/system/net/sf/jopt-simple/jopt-simple/5.0.3/jopt-simple-5.0.3.jar;C:\Program Files\ArcGIS\Server\/GeoEvent/system/com/yammer/metrics/metrics-core/2.2.0/metrics-core-2.2.0.jar;C:\Program Files\ArcGIS\Server\/GeoEvent/system/org/scala-lang/scala-library/2.11.8/scala-library-2.11.8.jar;C:\Program Files\ArcGIS\Server\/GeoEvent/system/com/101tec/zkclient/0.10/zkclient-0.10.jar;C:\Program Files\ArcGIS\Server\/GeoEvent/system/org/scala-lang/modules/scala-parser-combinators_2.11/1.0.6/scala-parser-combinators_2.11-1.0.6.jar;C:\Program Files\ArcGIS\Server\/GeoEvent/system/org/apache/kafka/kafka-clients-esri/0.11.0.0/kafka-clients-esri-0.11.0.0.jar;C:\Program Files\ArcGIS\Server\/GeoEvent/system/net/jpountz/lz4/lz4/1.3.0/lz4-1.3.0.jar;C:\Program Files\ArcGIS\Server\/GeoEvent/system/org/xerial/snappy/snappy-java/1.1.4/snappy-java-1.1.4.jar;C:\Program Files\ArcGIS\Server\/GeoEvent/system/org/apache/commons/commons-exec/1.3/commons-exec-1.3.jar;C:\Program Files\ArcGIS\Server\/GeoEvent/system/commons-cli/commons-cli/1.3/commons-cli-1.3.jar;C:\Program Files\ArcGIS\Server\/GeoEvent/system/org/apache/zookeeper/zookeeper-geoevent/3.5.3-beta/zookeeper-geoevent-3.5.3-beta.jar;C:\Program Files\ArcGIS\Server\/GeoEvent/system/com/esri/ges/framework/httpclient/10.6.1/httpclient-10.6.1.jar;C:\Program Files\ArcGIS\Server\/GeoEvent/system/com/esri/ges/framework/messaging/jms-messaging/10.6.1/jms-messaging-10.6.1.jar;C:\Program Files\ArcGIS\Server\/GeoEvent/system/org/apache/commons/commons-lang3/3.5/commons-lang3-3.5.jar;C:\Program Files\ArcGIS\Server\/GeoEvent/system/com/microsoft/azure/azure-storage/4.2.0/azure-storage-4.2.0.jar;C:\Program Files\ArcGIS\Server\/GeoEvent/system/com/fasterxml/jackson/core/jackson-core/2.8.7/jackson-core-2.8.7.jar;C:\Program Files\ArcGIS\Server\/GeoEvent/system/com/fasterxml/jackson/core/jackson-databind/2.8.7/jackson-databind-2.8.7.jar;C:\Program Files\ArcGIS\Server\/GeoEvent/system/com/fasterxml/jackson/core/jackson-annotations/2.8.7/jackson-annotations-2.8.7.jar
2018-08-13 11:01:48,939 [myid:] - INFO  [Thread-5:Environment@109] - Client environment:java.library.path=C:\Program Files\ArcGIS\Server\/GeoEvent/gateway/lib/wrapper/
2018-08-13 11:01:48,939 [myid:] - INFO  [Thread-5:Environment@109] - Client environment:java.io.tmpdir=C:\Users\<User-Removed>\AppData\Local\Temp\
2018-08-13 11:01:48,939 [myid:] - INFO  [Thread-5:Environment@109] - Client environment:java.compiler=<NA>
2018-08-13 11:01:48,939 [myid:] - INFO  [Thread-5:Environment@109] - Client environment:os.name=Windows Server 2016
2018-08-13 11:01:48,939 [myid:] - INFO  [Thread-5:Environment@109] - Client environment:os.arch=amd64
2018-08-13 11:01:48,939 [myid:] - INFO  [Thread-5:Environment@109] - Client environment:os.version=10.0
2018-08-13 11:01:48,939 [myid:] - INFO  [Thread-5:Environment@109] - Client environment:user.name=SYSTEM
2018-08-13 11:01:48,939 [myid:] - INFO  [Thread-5:Environment@109] - Client environment:user.home=C:\Users\<User-Removed>
2018-08-13 11:01:48,939 [myid:] - INFO  [Thread-5:Environment@109] - Client environment:user.dir=C:\Program Files\ArcGIS\Server\GeoEvent\gateway\bin
2018-08-13 11:01:48,939 [myid:] - INFO  [Thread-5:Environment@109] - Client environment:os.memory.free=421MB
2018-08-13 11:01:48,939 [myid:] - INFO  [Thread-5:Environment@109] - Client environment:os.memory.max=1820MB
2018-08-13 11:01:48,939 [myid:] - INFO  [Thread-5:Environment@109] - Client environment:os.memory.total=491MB
2018-08-13 11:01:49,004 [myid:1] - INFO  [Thread-4:QuorumPeerMain@138] - Starting quorum peer
2018-08-13 11:01:49,065 [myid:] - INFO  [Thread-5:CuratorFrameworkImpl@284] - Starting
2018-08-13 11:01:49,082 [myid:1] - INFO  [Thread-4:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 8 worker threads, and 64 kB direct buffers.
2018-08-13 11:01:49,098 [myid:1] - INFO  [Thread-4:NIOServerCnxnFactory@686] - binding to port /0.0.0.0:4181
2018-08-13 11:01:49,115 [myid:] - INFO  [Thread-5:ZooKeeper@865] - Initiating client connection, connectString=<IP-Removed>:4181 sessionTimeout=30000 watcher=org.apache.curator.ConnectionState@4e8cc629
2018-08-13 11:01:49,121 [myid:1] - WARN  [Thread-4:AdminServerFactory@58] - Unable to load jetty, not starting JettyAdminServer
java.lang.NoClassDefFoundError: org/eclipse/jetty/server/Connector
     at java.lang.Class.forName0(Native Method)
     at java.lang.Class.forName(Class.java:264)
     at org.apache.zookeeper.server.admin.AdminServerFactory.createAdminServer(AdminServerFactory.java:43)
     at org.apache.zookeeper.server.quorum.QuorumPeer.<init>(QuorumPeer.java:725)
     at org.apache.zookeeper.server.quorum.QuorumPeerMain.getQuorumPeer(QuorumPeerMain.java:195)
     at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:157)
     at com.esri.ges.gateway.zookeeper.MainThread.run(MainThread.java:47)
     at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.eclipse.jetty.server.Connector
     at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
     at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
     at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
     ... 8 more
2018-08-13 11:01:49,146 [myid:] - INFO  [Thread-5:ClientCnxnSocket@236] - jute.maxbuffer value is 4194304 Bytes
2018-08-13 11:01:49,149 [myid:1] - INFO  [Thread-4:QuorumPeer@1349] - Local sessions disabled
2018-08-13 11:01:49,149 [myid:1] - INFO  [Thread-4:QuorumPeer@1360] - Local session upgrading disabled
2018-08-13 11:01:49,149 [myid:1] - INFO  [Thread-4:QuorumPeer@1327] - tickTime set to 4000
2018-08-13 11:01:49,149 [myid:1] - INFO  [Thread-4:QuorumPeer@1371] - minSessionTimeout set to 8000
2018-08-13 11:01:49,151 [myid:1] - INFO  [Thread-4:QuorumPeer@1382] - maxSessionTimeout set to 80000
2018-08-13 11:01:49,151 [myid:1] - INFO  [Thread-4:QuorumPeer@1397] - initLimit set to 10
2018-08-13 11:01:49,170 [myid:] - INFO  [Thread-5:CuratorFrameworkImpl@326] - Default schema
2018-08-13 11:01:49,183 [myid:<IP-Removed>:4181] - INFO  [Thread-5-SendThread(<IP-Removed>:4181):ClientCnxn$SendThread@1113] - Opening socket connection to server <IP-Removed>/<IP-Removed>:4181. Will not attempt to authenticate using SASL (unknown error)
2018-08-13 11:01:49,183 [myid:<IP-Removed>:4181] - INFO  [Thread-5-SendThread(<IP-Removed>:4181):ClientCnxn$SendThread@948] - Socket connection established, initiating session, client: /<IP-Removed>:49714, server: <IP-Removed>/<IP-Removed>:4181
2018-08-13 11:01:49,183 [myid:1] - INFO  [Thread-4:FileSnap@83] - Reading snapshot C:\ProgramData\Esri\GeoEvent-Gateway\zookeeper-data\version-2\snapshot.10000017d
2018-08-13 11:01:49,223 [myid:1] - ERROR [Thread-4:QuorumPeer@822] - Unable to load database on disk
java.io.IOException: The current epoch, 1, is older than the last zxid, 8589934621
     at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:804)
     at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:772)
     at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:185)
     at com.esri.ges.gateway.zookeeper.MainThread.run(MainThread.java:47)
     at java.lang.Thread.run(Thread.java:748)
2018-08-13 11:01:49,223 [myid:1] - ERROR [Thread-4:MainThread@52] - Error in Gateway MainThread
java.lang.RuntimeException: Unable to run quorum server 
     at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:823)
     at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:772)
     at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:185)
     at com.esri.ges.gateway.zookeeper.MainThread.run(MainThread.java:47)
     at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: The current epoch, 1, is older than the last zxid, 8589934621
     at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:804)
     ... 4 more
2018-08-13 11:01:53,660 [myid:] - WARN  [WrapperSimpleAppMain:CuratorZookeeperClient@120] - session timeout [10000] is less than connection timeout [30000]
2018-08-13 11:01:53,660 [myid:] - INFO  [WrapperSimpleAppMain:CuratorFrameworkImpl@284] - Starting
2018-08-13 11:01:53,660 [myid:] - INFO  [WrapperSimpleAppMain:ZooKeeper@865] - Initiating client connection, connectString=<IP-Removed>:4181 sessionTimeout=10000 watcher=org.apache.curator.ConnectionState@9741ca5
2018-08-13 11:01:53,660 [myid:] - INFO  [WrapperSimpleAppMain:ClientCnxnSocket@236] - jute.maxbuffer value is 4194304 Bytes
2018-08-13 11:01:53,660 [myid:] - INFO  [WrapperSimpleAppMain:CuratorFrameworkImpl@326] - Default schema
2018-08-13 11:01:53,863 [myid:<IP-Removed>:4181] - INFO  [WrapperSimpleAppMain-SendThread(<IP-Removed>:4181):ClientCnxn$SendThread@1113] - Opening socket connection to server <IP-Removed>/<IP-Removed>:4181. Will not attempt to authenticate using SASL (unknown error)
2018-08-13 11:01:53,863 [myid:<IP-Removed>:4181] - INFO  [WrapperSimpleAppMain-SendThread(<IP-Removed>:4181):ClientCnxn$SendThread@948] - Socket connection established, initiating session, client: /<IP-Removed>:49734, server: <IP-Removed>/<IP-Removed>:4181
2018-08-13 11:02:03,879 [myid:<IP-Removed>:4181] - WARN  [WrapperSimpleAppMain-SendThread(<IP-Removed>:4181):ClientCnxn$SendThread@1181] - Client session timed out, have not heard from server in 10001ms for sessionid 0x0
2018-08-13 11:02:03,879 [myid:<IP-Removed>:4181] - INFO  [WrapperSimpleAppMain-SendThread(<IP-Removed>:4181):ClientCnxn$SendThread@1229] - Client session timed out, have not heard from server in 10001ms for sessionid 0x0, closing socket connection and attempting reconnect
2018-08-13 11:02:04,446 [myid:] - INFO  [Curator-Framework-0:ConnectionStateManager@237] - State change: SUSPENDED
2018-08-13 11:02:04,446 [myid:] - ERROR [Curator-Framework-0:CuratorFrameworkImpl@664] - Background operation retry gave up
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

[roemhildtg · Accepted solution]

Okay. Looks like the correct solution was to go in and delete all of the geo-event server log and data storage folders. Must've been some old config left over from a previous install or something.

I followed the steps here and the server was able to start up. https://community.esri.com/community/gis/enterprise-gis/geoevent/blog/2018/07/25/howto-administrativ... 

[GregoryChristakos]

Hello Gregg,

If I am understand correctly, working through an administrative reset had resolved this particular issue for you? Looking back at the previous post of yours, the problem seemed to be because the certificate was incorrectly specified as .pfx.cer within ArcGIS Server. Had this changed? Were you able to confirm that the certificate had changed after working through the administrative reset?

[roemhildtg]

Hi Gregory,

Yes, the administrative reset worked. The issue was related to the certificate, but it was not dealing with ArcGIS Server or portal. Arcgis server and portal both were configured correctly to use the certificate. It appeared that Geo-event server also had imported the certificate from arcgis server, but it was not using it correctly for some reason.

For instance, I could see the correct certificate being referenced by the log files. But if I visited the geo event server rest pages, it was not serving the correct https certificate because browsers would complain.