Feature Request - Akka Stream Support

[etspaceman]

Akka Actors are a great start for implementing the reactive API. I think the logical next step for an interface here would be to provide Akka Stream implementations. This would particularly benefit the Consumer as you could provide significant features such as back-pressure.

[etspaceman]

Just wanted to put this out here as a reference. Looks like Alpakka is working on a KCL/KPL stream integration:

akka/alpakka#433

CC @markglh

[markglh]

Thanks @etspaceman, I agree. It's hard to get right - the Kafka stream stuff was very problematic when dealing with non-happy path scenarios. Alpakka already has some kinesis stuff but it needs a lot of work:
https://github.com/akka/alpakka/tree/master/kinesis/src

That issue you linked to is more interesting for sure! The consumer will be far harder than the producer for this I imagine

[markglh]

Back pressure is an interesting one, you kind of already have that because until you've acked the whole batch you won't receive another one. You don't have it within the batch though currently that's true.

[markglh]

It's definitely worth a look in here: https://github.com/akka/alpakka/pull/434/files

Specifically at the use of how they're handling blocking/back-pressure in the KCL using semaphores.

The KCL keeps sending batches if you don't block in the Processor like this:
https://github.com/WW-Digital/reactive-kinesis/blob/master/src/main/scala/com/weightwatchers/reactive/kinesis/consumer/ConsumerProcessingManager.scala#L114

I hate this code, we can surely do better!

[markglh]

Source implementation here: #39 (review)

I had a good look through https://github.com/akka/alpakka/pull/434/files. It's a shame they're gonna block the merge due to the KCL licence. Some interesting implementation ideas in there. Couple of things I'm not sure of but there are definitely things we could take into consideration.

In the New Year I'm hoping to swap out the backend for the non-blocking driver. I'm gonna reach out to amazon to see if they're updating the KPL/KCL - if not then it's gonna be a reasonable amount of work. But I think it'll pay off if we get it right.