nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

GeoEvent 10.4.1 - ZooKeeper WARN messages

ErikLanhammar

Hello!

Im running GEE 10.4.1 and Im starting to recieve GEE log WARN messages like these:

org.apache.zookeeper.server.persistence. FileTxnLog fsync-ing the write ahead log in SyncThread:1 took 1332ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide.

The ZooKeeper files are getting quite big, and I guess thats the reason for the messages...

ZooKeeper files

Whats a good plan to automate a purge job on these files, keeping them small and GEE happy?!

Rgds,

/Erik

Find more posts tagged with

Accepted answers

All comments

LorenMueller

Hi Erik. I am seeing the same thing and the ZooKeeper logs are up to about 30GB! How did you resolve this?

Thanks,

Loren

ErikLanhammar

Hey Loren,

I havent solved it yet.. I tried to delete the contents of the log files ”manually”. Dont.

I had to reinstall GEE..

So a good way to purge Zoo-keeper logfiles is something Im still looking for.. I will hopefully/eventually post the solution on the forum.

Rgds,

/Erik

ChrisBeyett

Hi Erik,

The Zookeeper log files should be purging themselves once they reach 64mb in size. There is an auto-purge process built into zookeeper that ensures this occurs. Do you have more than 50 map/feature services published on the ArcGIS Server? I've seen this exact behavior happen when more than about 50 services are published and zookeeper simply keeps writing the same files over and over.

- Chris

ErikLanhammar

Hi Chris,

No I only have 10-15 services published in total, including 2 steam services. Log files are +200 MB...

How configurable is ZK? Edit config files for purge file size, file location etc? Should this be done while GEE windows service is stopped?

Rgds,

/Erik

ChrisBeyett

In my testing I've found that switching the "b" to an upper case "B" results in the log messages purging themselves. Feel free to stop the GeoEvent Windows service and switch the case of the letter.

- Chris

ErikLanhammar

Thanks Chris, will test this.

/Erik

LorenMueller

Below are steps I received from ESRI support. Again, I did not have very many services running, but the fix for this bug seems to be working, and it is the same fix essentially that Chris is suggesting. Note: It is really important to export and save the Configuration first because none of your info (inputs, outputs, services, etc.) is there anymore after restarting the GEE Service in this workflow. (I only applied this fix yesterday so still not sure it will be the fix I am hoping it will be. I will be keeping an eye on the ZK logs though and will report back.)

***

I believe we are encountering the following bug logged against ArcGIS GeoEvent Server 10.4.1 (fixed at 10.5):

BUG-000097658 - The ZooKeeper logs continue to grow when registered ArcGIS for Server connections contain a large number of services

What is happening is that the ZooKeeper logs are unable to properly process the service detail responses from registered ArcGIS Server connections. The number of characters needed to list all of the child items (including folder, service, layer, and field names) of those Servers can exceed the default ZooKeeper data size limit of 1MB. This creates a lock on the log files preventing them from being properly managed and results in the log files continuing to grow.

To workaround this, we can increase the default data size limit with the workflow below:

1. Export the current GeoEvent Configuration: Site > Configuration Store > Export Configuration.
2. Stop the ArcGIS GeoEvent Windows Service.
3. Open the ArcGIS GeoEvent configuration file located in,'C:\ProgramFiles\ArcGIS\Server\GeoEvent\etc\ArcGISGeoEvent.cfg' in a Text editor.
4. Replace the following line:
wrapper.java.additional.16=-Djute.maxbuffer=0xA00000
WITH
wrapper.java.additional.16=-Djute.maxBuffer=0xA00000

- note the capital B in Buffer

5. Navigate to C:\ProgramData\Esri\GeoEvent
6. Delete the ‘zookeeper’ folder.
7. Start the ArcGIS GeoEvent Windows Service.
8. Import the GeoEvent Configuration.

***

Loren

ErikLanhammar

Thanks Loren. One could say the devil is in the details on this one.. I will test and report my results back to you.

/Erik

ErikLanhammar

After following steps 1-8 GEE looks OK and performs well so far. I will monitor ZK log file size for a while and report back.

Now listen up everybody: DO NOT FORGET STEP 1, or you will be…. ☹

Thanks for posting Loren.

Rgds,

/Erik

Join the Conversation

Signed in members can post, follow updates, and more. New here? Register a free account.

Quick Links

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.