GeoAnalytics Server: Zookeeper realtimeState is UNKNOWN

[Jan-Tschada]

We just installed the GeoAnalytics Server on a Windows Server 2019 Standard. Whenever we try to use a GeoAnalytics tool an error "BD_101057: Unable to start a distributed job." is raised.

We already stopped the ArcGIS Services and reseted the Zookeeper "synchronization service" using the REST admin endpoint and started + stopped the hell out of Spark and Zookeeper.

Also, we delete the "local repository path" C:/arcgissserver/local, but with no luck.

The health check of Spark is always fine and if the Zookeeper is stopped the status request is fine, too. When the Zookeeper is started the JSON response shows a realtimeState of "STARTED".

Only when Zookeeper was started/is running the status request is running for some seconds and a realtimeState of "UNKNOWN" is shown in the JSON response.

Stop:

{
"configuredState": "STOPPED",
"details": [],
"status": "success"
}

Start:

{
"configuredState": "STARTED",
"details": [{
"machine": "geoana.ab.cde.fghi-jk.lmn",
"realtimeState": "STARTED"
}]
}

Status:

{ "configuredState": "STARTED", "details": [{ "machine": "geoana.ab.cde.fghi-jk.lmn", "realtimeState": "UNKNOWN" }], "status": "success" }

Is there any chance to narrow it down by using some Windows/System or JVM monitoring tools?

[SarahAmbrose]

Hi Jan Tschada‌, what version of Enterprise are you on? 

Thanks,

Sarah

GeoAnalytics Product Engineer. 

[Jan-Tschada]

We are using ArcGIS Enterprise 10.8, the ArcGIS Server machine is 10.8.12790 having the German Language Pack installed.

[SarahAmbrose]

Thanks Jan, sorry for the delay, we have the UC going on.

At 10.8.1 there is a known issue that zookeeper always say "unknown" - even when it's fine. Since there might be a bit of back and forth on this - can you email the GeoAnalytics product engineer alias directly at geoanalytics-pes@esri.com? 

The first two things I'll want to confirm are:

1. Can you successfully create a big data file share? I want to see if Spark starts up fir this task which is in a different toolbox.

2. What is shown when you check the health of spark?

3. Have you had any tool run, or has it been in this state since you deployed?

Thanks,

Sarah

[Jan-Tschada]

Right now, it seems like the GeoAnalytics Server is running fairly stable. As far as we can judge it, there was some issue with the data store or between the three virtual machines running in our test environment. The test environment is schedule to shutdown every Friday. Every Monday we are starting the data store, portal and GeoAnalytics machine again and running our smoke tests.

If we are seeing this strange behavior again we are going to create some system info logs for your guys.

Thanks in advance.