Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add kafka input/output plugins using kafka-node #96

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

whitedr
Copy link

@whitedr whitedr commented Mar 17, 2015

I know more is needed for this (like testing...), but this is the beginnings of a kafka input/output plugin. This is using https://github.com/SOHU-Co/kafka-node/ and the HighLevelProducer/HighLevelConsumer interfaces to produce/consume messages to kafka topics. On the output side, I also created a kafka_river output that is the same as the kafka output but formats the messages published to kafka with an JSON envelope that can be indexed into ElasticSearch with the https://github.com/clippPR/elasticsearch-river-kafka ES river. An example of this JSON envelope structure can be seen here: https://github.com/clippPR/elasticsearch-river-kafka/blob/master/src/main/java/org/elasticsearch/river/kafka/JsonMessageHandler.java#L35

One thing I wasn't completely sure of was whether to have the kafka_river as a separate output type or have it be a filter. I ended up following the example of the output_elasticsearch_zeromq.

@bpaquet
Copy link
Owner

bpaquet commented Sep 22, 2016

Can you add documentation for this great plugin ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants