Geo Replication using Proxy

[rillo-carrillo]

Environment:
2 Kubernetes Environments
2 Pulsar Clusters v 2.10.0

Have added on each cluster another cluster to replicate messages with the next command:

pulsar-admin clusters create --cluster-config-file /broker/conf/cluster2.yaml cluster2

Content of cluster2.yaml

serviceUrlTls: "https://pulsar-admin-cluster2:8443"
proxyServiceUrl: "pulsar+ssl://pulsar-proxy-cluster2:6651"
brokerClientTlsEnabled: true
brokerClientTrustCertsFilePath: "/volume/broker/certs/ca.crt"

Do you know if this is a valid configuration to add another cluster to the list of local cluster?

I have done the same on cluster2 pointing to cluster1.

Same have given permissions to a tenant to both clusters.

For testing, I have just set replication to just one namespace in cluster 1.

But what I have observed is that the remote proxy is not handling the requests to remote broker's servers.
Rather my local brokers try to communicate with remote brokers using lookup from proxy and not using the proxy.

remote cluster log:

2022-08-23T18:15:09,738+0000 [pulsar-web-41-8] INFO  org.eclipse.jetty.server.RequestLog - 10.24.192.80 - - [23/Aug/2022:18:15:09 +0000] "GET /lookup/v2/topic/persistent/test/testn/testt HTTP/1.1" 200 392 "-" "Pulsar-Java-v2.10.0" 0
2022-08-23T18:15:09,739+0000 [pulsar-external-web-5-8] INFO  org.eclipse.jetty.server.RequestLog - 10.24.192.7 - - [23/Aug/2022:18:15:09 +0000] "GET /lookup/v2/topic/persistent/test/testn/testt HTTP/1.1" 200 392 "-" "Pulsar-Java-v2.10.0" 1

local cluster log:

2022-08-23T18:15:09,735+0000 [broker-client-shared-timer-11-1] INFO org.apache.pulsar.client.impl.ConnectionHandler - [persistent://test/testn/testt] [pulsar.repl.cluster1-->cluster2] Reconnecting after connection was closed
2022-08-23T18:15:09,742+0000 [pulsar-io-7-4] INFO org.apache.pulsar.client.impl.ProducerImpl - [persistent://test/testn/testt] [pulsar.repl.cluster1-->cluster2] Creating producer on cnx [id: 0xf9de2d0d, L:/10.42.13.202:53428 - R:broker-0.brokers.pulsar-dev.svc.cluster.local/10.42.4.195:6651]
2022-08-23T18:15:09,743+0000 [pulsar-io-7-6] WARN org.apache.pulsar.client.impl.ClientCnx - [id: 0xf9de2d0d, L:/10.42.13.202:53428 - R:broker-0.brokers.pulsar-dev.svc.cluster.local/10.42.4.195:6651] Received error from server: Namespace bundle for topic (persistent://test/testn/testt) not served by this instance. Please redo the lookup. Request is denied: namespace=test/testn
2022-08-23T18:15:09,744+0000 [pulsar-io-7-6] ERROR org.apache.pulsar.client.impl.ProducerImpl - [persistent://test/testn/testt] [pulsar.repl.cluster1-->cluster2] Failed to create producer: {"errorMsg":"Namespace bundle for topic (persistent://test/testn/testt) not served by this instance. Please redo the lookup. Request is denied: namespace=test/testn","reqId":100250947237580162, "remote":"broker-0.brokers.pulsar-dev.svc.cluster.local/10.42.4.195:6651", "local":"/10.42.13.202:53428"}
2022-08-23T18:15:09,744+0000 [pulsar-io-7-6] WARN org.apache.pulsar.client.impl.ConnectionHandler - [persistent://test/testn/testt] [pulsar.repl.cluster1-->cluster2] Could not get connection to broker: {"errorMsg":"Namespace bundle for topic (persistent://test/testn/testt) not served by this instance. Please redo the lookup. Request is denied: namespace=test/testn","reqId":100250947237580162, "remote":"broker-0.brokers.pulsar-dev.svc.cluster.local/10.42.4.195:6651", "local":"/10.42.13.202:53428"} -- Will try again in 24.362 s

log local broker-0

2022-08-23T18:15:09,743+0000 [pulsar-io-7-2] INFO org.apache.pulsar.broker.service.ServerCnx - [/10.42.13.202:53428][persistent://test/testn/testt] Creating producer. producerId=78
2022-08-23T18:15:09,743+0000 [pulsar-io-7-2] WARN org.apache.pulsar.broker.service.BrokerService - Namespace bundle for topic (persistent://test/testn/testt) not served by this instance. Please redo the lookup. Request is denied: namespace=test/testn

Another thing to notice is that I'm using the same k8s namespace name on both environments.
Is this causing the problem or is the local-broker/remote-proxy that is not working as expected?