Only first among multiple clusters connecting properly

[swimand]

Describe the bug
Only first cluster successfully connects to kafka/zookeeper/schema-registry instance.

Set up
Deployed through helm onto a kubernetes cluster hosted on GCP containing multiple kafka/zookeeper/schema-registry instances.

Configured using the following values files:

envs:
  config:
    KAFKA_CLUSTERS_0_NAME: "dev"
    KAFKA_CLUSTERS_0_SCHEMAREGISTRY: "http://dev-cp-schema-registry:8081"
    KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: "dev-cp-kafka:9092"
    KAFKA_CLUSTERS_0_ZOOKEEPER: "dev-cp-zookeeper:2181"

    KAFKA_CLUSTERS_1_NAME: "test"
    KAFKA_CLUSTERS_1_SCHEMAREGISTRY: "http://test-cp-schema-registry:8081"
    KAFKA_CLUSTERS_1_BOOTSTRAPSERVERS: "test-cp-kafka:9092"
    KAFKA_CLUSTERS_1_ZOOKEEPER: "test-cp-zookeeper:2181"

Steps to Reproduce
Deploy two seperate kafka instances using the charts from cp-helm-charts, where one is assigned the name test and the other dev.

Once running, deploy the kafka-ui using the values file, defined above:

helm install kafka-ui kafka-ui/kafka-ui -f values.yaml

Expected behavior
All clusters connecting properly.

Additional context

2021-10-29T14:40:04.432121221Z14:40:04.431 [parallel-1] DEBUG com.provectus.kafka.ui.service.MetricsUpdateService - Start getting metrics for kafkaCluster: test
2021-10-29T14:40:04.472536958Z14:40:04.470 [parallel-1] ERROR com.provectus.kafka.ui.service.KafkaService - Failed to collect cluster test info
2021-10-29T14:40:04.472609113Zorg.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:540) ~[kafka-clients-2.8.0.jar!/:?] at org.apache.kafka.clients.admin.Admin.create(Admin.java:134) ~[kafka-clients-2.8.0.jar!/:?] at org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:39) ~[kafka-clients-2.8.0.jar!/:?] at com.provectus.kafka.ui.service.AdminClientServiceImpl.lambda$createAdminClient$2(AdminClientServiceImpl.java:41) ~[classes!/:?] at reactor.core.publisher.MonoSupplier.call(MonoSupplier.java:85) ~[reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE] at reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:126) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE] at reactor.core.publisher.MonoFlatMap.subscribeOrReturn(MonoFlatMap.java:53) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE] at reactor.core.publisher.Mono.subscribe(Mono.java:4090) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE] at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onComplete(FluxSwitchIfEmpty.java:75) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE] at reactor.core.publisher.Operators.complete(Operators.java:132) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE] at reactor.core.publisher.MonoEmpty.subscribe(MonoEmpty.java:45) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE] at reactor.core.publisher.Mono.subscribe(Mono.java:4105) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE] at reactor.core.publisher.FluxFlatMap$FlatMapMain.onNext(FluxFlatMap.java:418) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE] at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:114) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE] at reactor.core.publisher.FluxPublishOn$PublishOnSubscriber.runAsync(FluxPublishOn.java:398) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE] at reactor.core.publisher.FluxPublishOn$PublishOnSubscriber.run(FluxPublishOn.java:484) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE] at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE] at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE] at java.util.concurrent.FutureTask.run(Unknown Source) [?:?] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?] at java.lang.Thread.run(Unknown Source) [?:?] Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:89) ~[kafka-clients-2.8.0.jar!/:?] at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:48) ~[kafka-clients-2.8.0.jar!/:?] at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:494) ~[kafka-clients-2.8.0.jar!/:?] ... 22 more```

[Haarolean]

Hi, thanks for reaching out. We'll check it out.

[Haarolean]

Hey,

seems like there's an issue with your configuration.
From your logs: No resolvable bootstrap urls given in bootstrap.servers.
No clue how to fix this, but we've got some users using kafka-ui with helm (but not quite sure about multiple clusters) and this is the first time I see this kind of a problem.
Since this works via docker-compose, this is probably some misconfiguration issue.
I'd suggest trying reaching out to people on our discord, maybe someone will spot an issue in your configuration.
Also, I'll convert this to a discussion so this issue can get more attention.
Thanks

[IndeedSi]

I tried it out with helm in our k8s cluster. My Kafka UI works fine with multiple clusters.
I have got multiple Kafka brokers started before this test

Here is my full values.yaml

ReplicaCount: 1

......

envs:
  secret: {}
  config:
    KAFKA_CLUSTERS_0_NAME: "dev"
    KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: "si-test1-broker:9092"
    KAFKA_CLUSTERS_0_ZOOKEEPER: "si-test1-zookeeper:2181"

    KAFKA_CLUSTERS_1_NAME: "test"
    KAFKA_CLUSTERS_1_BOOTSTRAPSERVERS: "si-test2-broker:9092"
    KAFKA_CLUSTERS_1_ZOOKEEPER: "si-test2-zookeeper:2181"

podAnnotations: {}

podSecurityContext: {}
  # fsGroup: 2000

securityContext:
  # capabilities:
  #   drop:
  #   - ALL
  # readOnlyRootFilesystem: true
  runAsNonRoot: true
  runAsUser: 1000

...... same as https://github.com/provectus/kafka-ui/blob/master/charts/kafka-ui/values.yaml

And the pod starts without any error.

07:43:11.166 [main] INFO  com.provectus.kafka.ui.KafkaUiApplication - Started KafkaUiApplication in 3.437 seconds (JVM running for 4.834)
07:43:11.530 [kafka-admin-client-thread | adminclient-1] DEBUG com.provectus.kafka.ui.service.ZookeeperService - Start getting Zookeeper metrics for kafkaCluster: dev
07:43:11.532 [kafka-admin-client-thread | adminclient-2] DEBUG com.provectus.kafka.ui.service.ZookeeperService - Start getting Zookeeper metrics for kafkaCluster: test
07:43:40.989 [parallel-1] DEBUG com.provectus.kafka.ui.service.MetricsUpdateService - Start getting metrics for kafkaCluster: dev
07:43:40.990 [parallel-1] DEBUG com.provectus.kafka.ui.service.MetricsUpdateService - Start getting metrics for kafkaCluster: test
07:43:40.998 [kafka-admin-client-thread | adminclient-1] DEBUG com.provectus.kafka.ui.service.ZookeeperService - Start getting Zookeeper metrics for kafkaCluster: dev
07:43:40.998 [kafka-admin-client-thread | adminclient-2] DEBUG com.provectus.kafka.ui.service.ZookeeperService - Start getting Zookeeper metrics for kafkaCluster: test
07:44:10.989 [parallel-1] DEBUG com.provectus.kafka.ui.service.MetricsUpdateService - Start getting metrics for kafkaCluster: dev
07:44:10.989 [parallel-1] DEBUG com.provectus.kafka.ui.service.MetricsUpdateService - Start getting metrics for kafkaCluster: test
07:44:10.997 [kafka-admin-client-thread | adminclient-1] DEBUG com.provectus.kafka.ui.service.ZookeeperService - Start getting Zookeeper metrics for kafkaCluster: dev
07:44:10.997 [kafka-admin-client-thread | adminclient-2] DEBUG com.provectus.kafka.ui.service.ZookeeperService - Start getting Zookeeper metrics for kafkaCluster: test

Everything looks fine with multiple clusters.

[Haarolean]

@IndeedSi thank you very much for checking this out!