Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: make the kafka topic configurable #353

Merged
merged 8 commits into from
Nov 9, 2024
Merged

feat: make the kafka topic configurable #353

merged 8 commits into from
Nov 9, 2024

Conversation

masci
Copy link
Member

@masci masci commented Nov 7, 2024

Part of #349 (not a fix until we propagate the topic settings up to the control plane and api server)

Problem

Currently we use the name of the "message type" string to create the Kafka topic that'll be used in the message queue. Problem is that different instances of control planes using the same Kafka instances will step into each other, with catastrific effects. Note that this problem is common to other message queue brokers like S3, so eventually this fix will be rolled out for all the supported queues.

Solution

Make the Kafka topic name configurable, so that different control planes will use different topics. This PR only makes the topic configurable, another PR will make use of this new feature and resolve the original issue.

Notes

While working at the fix, I noticed a few things we can improve and already implemented in this PR in a backward compatible manner:

  • I am not sure the base class for queues should be a pydantic model, that's very limiting when you need to treat the class as a regular class (get and set local instance variables for example). I think a plain abstract class derived from ABC will make the job.
  • There's a lot of duplication in the configuration settings: the class constructor, the config settings model, the static methods... I think we should have a single source of truth, being the config settings model. For Kafka I made the constructor take the config model instance directly.
  • I moved the original code behind the if __name__ == "__main__" that was used for testing into a proper e2e test. Tests for message queues will most certainly need docker to be avaiable, I'll add the required CI bits (this part is WIP, you'll see it's done when tests are all green).

@coveralls
Copy link

coveralls commented Nov 7, 2024

Coverage Status

coverage: 71.969% (+0.9%) from 71.038%
when pulling ac5a92a on massi/349
into 34b9299 on main.

llama_deploy/apiserver/deployment.py Outdated Show resolved Hide resolved
llama_deploy/deploy/deploy.py Outdated Show resolved Hide resolved
@masci masci requested a review from nerdai November 8, 2024 11:49
@logan-markewich logan-markewich merged commit 5f5e76e into main Nov 9, 2024
10 checks passed
@logan-markewich logan-markewich deleted the massi/349 branch November 9, 2024 16:25
@masci masci added the enhancement New feature or request label Nov 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants