Consumer with multiple topics

[mbuhot]

Hi and thanks for creating Cafex!

I'd like to have a consumer receive input from two or more topics to perform some stream joining, similar to how a samza job might be structured: (https://samza.apache.org/learn/documentation/0.9/container/state-management.html)

It looks like the underlying Kafka Fetch protocol supports multiple topics (https://github.com/upyun/cafex/blob/master/lib/cafex/protocol/fetch.ex#L10), but it is not surfaced in the Cafex consumer configuration API?

In the meantime, is the best alternative to start a consumer for each topic, and forward the messages to my own GenServer, which can maintain some state and produce the outputs?

Thanks again!

[belltoy]

Yes, I'm afraid you're right.

According to the Kafka Protocol Guild, batching is a very significant performance win. Most of the APIs support batching across multiple topics. Cafex will support it, in one consumer.