Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Calling assign requires valid permissioned group.id even if no rebalance/offset management is required #717

Open
c-campbell-mwam opened this issue Sep 9, 2024 · 0 comments

Comments

@c-campbell-mwam
Copy link

c-campbell-mwam commented Sep 9, 2024

Across the various librdkafka github repositories there are issues relating to the usage of assign without a valid group (librdkafka, older librdkafka, node-rdkafka etc). A quick summary:

  1. librdkafka requires a group.id to be set in order to call assign (unlike the JVM client)
  2. If you correctly configure the consumer, it will not attempt to do anything requiring this group.id in communications (rebalancing, offset committing etc.).
  3. Therefore, the group.id can be set to a random string and just calling assign and seek etc. should be fine.

However in the Rust crate I am getting errors for GroupAuthorizationFailed. Here's some example code:

    let consumer: StreamConsumer = config
        .set("bootstrap.servers", cluster.bootstrap_servers())
        .set("group.id", "random.string.123")
        .set("enable.auto.commit", "false")
        .set("auto.offset.reset", "smallest")
        .set("enable.auto.offset.store", "false")
        .set("client.id", "topic_consumer")
        .create()?;

Even at this point with no assign call, I can see in the debug logs for the consumer that it is already seeing GroupAuthorizationFailed in one of the threads. When I call assign and then recv I get an actual Error. In order to verify the behaviour, I set up consumer group that I am authorised on and the above works (with group.id appropriately modified) and I am able to consume in those circumstances.

I'm guessing this is me messing up the config, or it's a bug in the way the consumer is set up. Would really appreciate any input. It so happens I can work around this for now because of some infrastructure my team has in place, but the application I'm writing is aimed to be deployed outside of this infrastructure in the future and this will block that happening if we can't find a fix.

Version 0.36.2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant