topic GeoEvent 10.4.1 - ZooKeeper WARN messages in ArcGIS GeoEvent Server Questions

GeoEvent 10.4.1 - ZooKeeper WARN messages

ErikLanhammar — Fri, 01 Sep 2017 14:45:23 GMT

Hello!

Im running GEE 10.4.1 and Im starting to recieve GEE log WARN messages like these:

org.apache.zookeeper.server.persistence. FileTxnLog fsync-ing the write ahead log in SyncThread:1 took 1332ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide.

The ZooKeeper files are getting quite big, and I guess thats the reason for the messages...

Whats a good plan to automate a purge job on these files, keeping them small and GEE happy?!

Rgds,

/Erik

Re: GeoEvent 10.4.1 - ZooKeeper WARN messages

LorenMueller — Wed, 06 Dec 2017 18:15:45 GMT

Hi Erik. I am seeing the same thing and the ZooKeeper logs are up to about 30GB! How did you resolve this?

Thanks,

Loren

Re: GeoEvent 10.4.1 - ZooKeeper WARN messages

ErikLanhammar — Wed, 06 Dec 2017 23:08:07 GMT

Hey Loren,

I havent solved it yet.. I tried to delete the contents of the log files ”manually”. Dont.

I had to reinstall GEE..

So a good way to purge Zoo-keeper logfiles is something Im still looking for.. I will hopefully/eventually post the solution on the forum.

Rgds,

/Erik

Re: GeoEvent 10.4.1 - ZooKeeper WARN messages

ChrisBeyett — Thu, 07 Dec 2017 16:16:56 GMT

Hi Erik,

The Zookeeper log files should be purging themselves once they reach 64mb in size. There is an auto-purge process built into zookeeper that ensures this occurs. Do you have more than 50 map/feature services published on the ArcGIS Server? I've seen this exact behavior happen when more than about 50 services are published and zookeeper simply keeps writing the same files over and over.

- Chris

Re: GeoEvent 10.4.1 - ZooKeeper WARN messages

ErikLanhammar — Thu, 07 Dec 2017 17:24:22 GMT

Hi Chris,

No I only have 10-15 services published in total, including 2 steam services. Log files are +200 MB...

How configurable is ZK? Edit config files for purge file size, file location etc? Should this be done while GEE windows service is stopped?

Rgds,

/Erik

Re: GeoEvent 10.4.1 - ZooKeeper WARN messages

ChrisBeyett — Thu, 07 Dec 2017 20:21:04 GMT

Hi Erik,

Can you confirm a behavior within the GeoEvent config files?

1. Navigate to the following file location and open it in a text editor:

'C:\ProgramFiles\ArcGIS\Server\GeoEvent\etc\ArcGISGeoEvent.cfg'

2. Scoll down to the 16th variable location. It should read as follows:

wrapper.java.additional.16=-Djute.maxbuffer=0xA000003

3. Pay particular attention to the "b" in the maxbuffer variable. Is it upper or lower case?

- Chris

Re: GeoEvent 10.4.1 - ZooKeeper WARN messages

ErikLanhammar — Thu, 07 Dec 2017 20:32:11 GMT

Mvh,

/Erik

Re: GeoEvent 10.4.1 - ZooKeeper WARN messages

LorenMueller — Thu, 07 Dec 2017 20:38:24 GMT

Hello,

I am following this thread as I am having the same issue (with only about 6 custom AGS map services running).

Should the 'b' be lower or upper case? It is lower case in my ArcGISGeoEvent.cfg file.

Re: GeoEvent 10.4.1 - ZooKeeper WARN messages

ChrisBeyett — Fri, 08 Dec 2017 16:16:53 GMT

In my testing I've found that switching the "b" to an upper case "B" results in the log messages purging themselves. Feel free to stop the GeoEvent Windows service and switch the case of the letter.

- Chris

Re: GeoEvent 10.4.1 - ZooKeeper WARN messages

ErikLanhammar — Fri, 08 Dec 2017 17:41:20 GMT

Thanks Chris, will test this.

/Erik

Re: GeoEvent 10.4.1 - ZooKeeper WARN messages

LorenMueller — Fri, 08 Dec 2017 17:47:29 GMT

Below are steps I received from ESRI support. Again, I did not have very many services running, but the fix for this bug seems to be working, and it is the same fix essentially that Chris is suggesting. Note: It is really important to export and save the Configuration first because none of your info (inputs, outputs, services, etc.) is there anymore after restarting the GEE Service in this workflow. (I only applied this fix yesterday so still not sure it will be the fix I am hoping it will be. I will be keeping an eye on the ZK logs though and will report back.)

***

I believe we are encountering the following bug logged against ArcGIS GeoEvent Server 10.4.1 (fixed at 10.5):

BUG-000097658 - The ZooKeeper logs continue to grow when registered ArcGIS for Server connections contain a large number of services

What is happening is that the ZooKeeper logs are unable to properly process the service detail responses from registered ArcGIS Server connections. The number of characters needed to list all of the child items (including folder, service, layer, and field names) of those Servers can exceed the default ZooKeeper data size limit of 1MB. This creates a lock on the log files preventing them from being properly managed and results in the log files continuing to grow.

To workaround this, we can increase the default data size limit with the workflow below:

1. Export the current GeoEvent Configuration: Site > Configuration Store > Export Configuration.
2. Stop the ArcGIS GeoEvent Windows Service.
3. Open the ArcGIS GeoEvent configuration file located in,'C:\ProgramFiles\ArcGIS\Server\GeoEvent\etc\ArcGISGeoEvent.cfg' in a Text editor.
4. Replace the following line:
wrapper.java.additional.16=-Djute.maxbuffer=0xA00000
WITH
wrapper.java.additional.16=-Djute.maxBuffer=0xA00000

- note the capital B in Buffer

5. Navigate to C:\ProgramData\Esri\GeoEvent
6. Delete the ‘zookeeper’ folder.
7. Start the ArcGIS GeoEvent Windows Service.
8. Import the GeoEvent Configuration.

***

Loren

Re: GeoEvent 10.4.1 - ZooKeeper WARN messages

ErikLanhammar — Fri, 08 Dec 2017 18:19:44 GMT

Thanks Loren. One could say the devil is in the details on this one.. I will test and report my results back to you.

/Erik

Re: GeoEvent 10.4.1 - ZooKeeper WARN messages

ErikLanhammar — Sat, 09 Dec 2017 09:28:10 GMT

After following steps 1-8 GEE looks OK and performs well so far. I will monitor ZK log file size for a while and report back.

Now listen up everybody: DO NOT FORGET STEP 1, or you will be…. ☹

Thanks for posting Loren.

Rgds,

/Erik