Kafka input connector not working

[Bert]

Hi guys,

We can't get the Kafka input connector to work with our Kafka nodes. I've been in contact with Esri about this but a solution is not in sight. So I hope someone here can help us.

Below is a short recap of our efforts:

• With GeoEvent Server 10.7.1 we could ingest data from our Cloudera HDF Kafka cluster. But only with gzip, lz4 or no compression on kafka topic. With snappy and zstd compression it didn’t work. We also sometimes needed to flush the entire Kafka topic and reboot the GeoEvent Server to make it work.
• When we upgraded to 10.8.1 we where no longer able to ingest data from Kafka. The Kafka cluster is used a lot within our company and as far as we know our GeoEvent Server is the only application having problems with it.
• After upgrading to 10.8.1 we also tried a clean install of GeoEvent Server but that didn’t fix the problem.
• We also use the GeoEvent Server with other connectors(REST) and those work fine
• After starting Kafka input connector in GeoEvent we can see on Kafka side(in Apache Ranger) that GeoEvent server is authenticating correctly and grabbing some data. But no data is coming in on GeoEvent server. After initial connection no more activity can be seen in Ranger. Only after stopping and starting Kafka connector we can see it connecting in Ranger again.
• We don’t see any errors in the logs on our GeoEvent Server
• We installed a separate Kafka client on the same machine as our GeoEvent Server. That client works fine and data is coming in. We used the same ssl/tls certificates as GeoEvent Server for authentication with our Kafka cluster. We also used the same Kafka topics, etc

Regards,

Bert

[JoshuaPust]

Did you ever figure this out @Bert ? I have something similar.

[Bert]

After some investigation we discovered that the Kafka connector in 10.8.1 didn't always load all the needed Java libraries correctly for our Kafka cluster(Cloudera). Esri released a fix for this in 10.9.1.

[JoshuaPust]

@Bert Thanks for the reply! Unfortunately, my SSL handshake failures are only in 10.9.1

[JoshuaPust]

And in 11.1 as well.

[sk_tbc]

Hi @Bert , @JoshuaPust ,

I have been trying to connect one Azure Event hub gzip compressed stream on  but it seems not to be working.Not getting any updates in the count and could see few errors as malformed input message for both json/text.

Do you have any clue if thats a supported format and is working without any issue in v11.2 ? Thank you!