Subscribers skip messages on module upgrade

[matt2e]

Steps to repro:

• Have a module with a subscriber with no retry policy that is trying to get through a backlog of messages
• Deploy a newer version of the module

What goes wrong:

• The context gets cancelled which fails any active calls
• These failures are not detected any differently than other errors, and so we commit these messages as having been processed
• Then sarama closes the consumer group, which includes sending all the commits to kafka

[stuartwdouglas]

I think this is part of the more general problem of request draining.

[matt2e]

I played around with the code, and i think it is, but also there were some underlying problems with how the consumer code handled context cancellation. This issue ended up also happening each time the consumer group was rebalanced for whatever reason.
Fixed in the PR linked above.