feat: make the kafka topic configurable #353
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Part of #349 (not a fix until we propagate the topic settings up to the control plane and api server)
Problem
Currently we use the name of the "message type" string to create the Kafka topic that'll be used in the message queue. Problem is that different instances of control planes using the same Kafka instances will step into each other, with catastrific effects. Note that this problem is common to other message queue brokers like S3, so eventually this fix will be rolled out for all the supported queues.
Solution
Make the Kafka topic name configurable, so that different control planes will use different topics. This PR only makes the topic configurable, another PR will make use of this new feature and resolve the original issue.
Notes
While working at the fix, I noticed a few things we can improve and already implemented in this PR in a backward compatible manner:
ABC
will make the job.if __name__ == "__main__"
that was used for testing into a proper e2e test. Tests for message queues will most certainly need docker to be avaiable, I'll add the required CI bits (this part is WIP, you'll see it's done when tests are all green).