Select to view content in your preferred language

GeoEvent 10.4.1 - ZooKeeper WARN messages

2641
12
09-01-2017 07:45 AM
ErikLanhammar
Esri Contributor

Hello!

Im running GEE 10.4.1 and Im starting to recieve GEE log WARN messages like these:

org.apache.zookeeper.server.persistence.   FileTxnLog fsync-ing the write ahead log in SyncThread:1 took 1332ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide.

The ZooKeeper files are getting quite big, and I guess thats the reason for the messages...

ZooKeeper files

Whats a good plan to automate a purge job on these files, keeping them small and GEE happy?!

Rgds,

/Erik

Tags (1)
12 Replies
LorenMueller
Regular Contributor

Below are steps I received from ESRI support. Again, I did not have very many services running, but the fix for this bug seems to be working, and it is the same fix essentially that Chris is suggesting. Note: It is really important to export and save the Configuration first because none of your info (inputs, outputs, services, etc.) is there anymore after restarting the GEE Service in this workflow. (I only applied this fix yesterday so still not sure it will be the fix I am hoping it will be. I will be keeping an eye on the ZK logs though and will report back.)

***

I believe we are encountering the following bug logged against ArcGIS GeoEvent Server 10.4.1 (fixed at 10.5): 

  • BUG-000097658 - The ZooKeeper logs continue to grow when registered ArcGIS for Server connections contain a large number of services

What is happening is that the ZooKeeper logs  are unable to properly process the service detail responses from registered ArcGIS Server connections. The number of characters needed to list all of the child items (including folder, service, layer, and field names) of those Servers can exceed the default ZooKeeper data size limit of 1MB. This creates a lock on the log files preventing them from being properly managed and results in the log files continuing to grow. 

To workaround this, we can increase the default data size limit with the workflow below:

1. Export the current GeoEvent Configuration: Site > Configuration Store > Export Configuration. 
2. Stop the ArcGIS GeoEvent Windows Service. 
3. Open the ArcGIS GeoEvent configuration file located in,'C:\ProgramFiles\ArcGIS\Server\GeoEvent\etc\ArcGISGeoEvent.cfg' in a Text editor. 
4. Replace the following line: 
wrapper.java.additional.16=-Djute.maxbuffer=0xA00000 
WITH 
wrapper.java.additional.16=-Djute.maxBuffer=0xA00000 

- note the capital B in Buffer

5. Navigate to C:\ProgramData\Esri\GeoEvent
6. Delete the ‘zookeeper’ folder. 
7. Start the ArcGIS GeoEvent Windows Service. 
8. Import the GeoEvent Configuration.

***

Loren

ErikLanhammar
Esri Contributor

Thanks Loren. One could say the devil is in the details on this one.. I will test and report my results back to you.

/Erik

0 Kudos
ErikLanhammar
Esri Contributor

After following steps 1-8 GEE looks OK and performs well so far. I will monitor ZK log file size for a while and report back.

Now listen up everybody: DO NOT FORGET STEP 1, or you will be…. ☹

Thanks for posting Loren.

Rgds,

/Erik

0 Kudos