Kafka input connector not working

116
0
02-02-2021 02:31 AM
Bert
by
New Contributor

Hi guys,

 

We can't get the Kafka input connector to work with our Kafka nodes. I've been in contact with Esri about this but a solution is not in sight. So I hope someone here can help us.

Below is a short recap of our efforts:

  • With GeoEvent Server 10.7.1 we could ingest data from our Cloudera HDF Kafka cluster. But only with gzip, lz4 or no compression on kafka topic. With snappy and zstd compression it didn’t work. We also sometimes needed to flush the entire Kafka topic and reboot the GeoEvent Server to make it work.
  • When we upgraded to 10.8.1 we where no longer able to ingest data from Kafka. The Kafka cluster is used a lot within our company and as far as we know our GeoEvent Server is the only application having problems with it.
  • After upgrading to 10.8.1 we also tried a clean install of GeoEvent Server but that didn’t fix the problem.
  • We also use the GeoEvent Server with other connectors(REST) and those work fine
  • After starting Kafka input connector in GeoEvent we can see on Kafka side(in Apache Ranger) that GeoEvent server is authenticating correctly and grabbing some data. But no data is coming in on GeoEvent server. After initial connection no more activity can be seen in Ranger. Only after stopping and starting Kafka connector we can see it connecting in Ranger again.
  • We don’t see any errors in the logs on our GeoEvent Server
  • We installed a separate Kafka client on the same machine as our GeoEvent Server. That client works fine and data is coming in. We used the same ssl/tls certificates as GeoEvent Server for authentication with our Kafka cluster. We also used the same Kafka topics, etc

Regards,

Bert

Tags (1)
0 Kudos
0 Replies