topic Re: Kafka input connector not working in ArcGIS GeoEvent Server Questions

Kafka input connector not working

Bert — Tue, 02 Feb 2021 10:31:53 GMT

Hi guys,

We can't get the Kafka input connector to work with our Kafka nodes. I've been in contact with Esri about this but a solution is not in sight. So I hope someone here can help us.

Below is a short recap of our efforts:

With GeoEvent Server 10.7.1 we could ingest data from our Cloudera HDF Kafka cluster. But only with gzip, lz4 or no compression on kafka topic. With snappy and zstd compression it didn’t work. We also sometimes needed to flush the entire Kafka topic and reboot the GeoEvent Server to make it work.
When we upgraded to 10.8.1 we where no longer able to ingest data from Kafka. The Kafka cluster is used a lot within our company and as far as we know our GeoEvent Server is the only application having problems with it.
After upgrading to 10.8.1 we also tried a clean install of GeoEvent Server but that didn’t fix the problem.
We also use the GeoEvent Server with other connectors(REST) and those work fine
After starting Kafka input connector in GeoEvent we can see on Kafka side(in Apache Ranger) that GeoEvent server is authenticating correctly and grabbing some data. But no data is coming in on GeoEvent server. After initial connection no more activity can be seen in Ranger. Only after stopping and starting Kafka connector we can see it connecting in Ranger again.
We don’t see any errors in the logs on our GeoEvent Server
We installed a separate Kafka client on the same machine as our GeoEvent Server. That client works fine and data is coming in. We used the same ssl/tls certificates as GeoEvent Server for authentication with our Kafka cluster. We also used the same Kafka topics, etc

Regards,

Bert

Re: Kafka input connector not working

JoshuaPust — Thu, 15 Dec 2022 17:55:54 GMT

Did you ever figure this out @Bert ? I have something similar.

Re: Kafka input connector not working

Bert — Mon, 02 Jan 2023 07:22:52 GMT

After some investigation we discovered that the Kafka connector in 10.8.1 didn't always load all the needed Java libraries correctly for our Kafka cluster(Cloudera). Esri released a fix for this in 10.9.1.

Re: Kafka input connector not working

JoshuaPust — Tue, 03 Jan 2023 15:24:48 GMT

@Bert Thanks for the reply! Unfortunately, my SSL handshake failures are only in 10.9.1

Re: Kafka input connector not working

JoshuaPust — Wed, 20 Sep 2023 12:22:59 GMT

And in 11.1 as well.

Re: Kafka input connector not working

sk_tbc — Fri, 16 Feb 2024 23:58:31 GMT

Hi @Bert , @JoshuaPust ,

I have been trying to connect one Azure Event hub gzip compressed stream on but it seems not to be working.Not getting any updates in the count and could see few errors as malformed input message for both json/text.

Do you have any clue if thats a supported format and is working without any issue in v11.2 ? Thank you!

Re: Kafka input connector not working

LorenzMeyer1 — Fri, 07 Mar 2025 08:15:13 GMT

Did you ever find a solution to this issue? Our Kafka connector worked with 11.0 and now we upgraded to 11.3 and I get the SSL handshake error.

Re: Kafka input connector not working

GaryLSheppardJr — Mon, 04 May 2026 17:08:18 GMT

We have the same problem in ArcGIS GeoEvent Server 11.5. We use the same TLS certificate from another Kafka client and it works fine. But in GeoEvent Server, we get an error:

Stream creation failed: Stream (my-kafka-json-in) failed to validate: Credential details are required. com.esri.ges.stream.StreamException: Stream creation failed: Stream (my-kafka-json-in) failed to validate: Credential details are required. at com.esri.ges.manager.stream.internal.StreamManagerImpl.newStream(StreamManagerImpl.java:255) ~[?:?]

Does anyone know how to fix this?