Active-active and no data loss. Is it possible?

[kitekite2020]

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[kitekite2020]

We would like to have pulsar clusters in two data centers. They are active and active. And we would like to replicate messages from one site to the other. i.e. the subscriber from one site will receive messages from publishers at two sites. And no message loss. If one site is down, the other site can continue to work.
We are thinking of geo replication. But since geo replication is asynchronous, we are worried about message loss. Every single message cannot be lost.
Then we are thinking to have synchronous replication, at bookkeeper level, cross two sites. First difficulty is there is no examples of how to do it without global zookeeper. If we use local clusters to manage each site, we need a global configuration store to manage two sites information which becomes a single point failure. If we use global zookeeper, one zookeeper for two sites, due to zookeeper cluster has to be 2*n+1 nodes, it means if the site with n+1 node fails, the other site also stops working.
Is my understanding right? Is there a possible solution with Pulsar for us? Thanks.

[codelipenghui]

We are thinking of geo replication. But since geo replication is asynchronous, we are worried about message loss. Every single message cannot be lost.

The geo-replication is asynchronous, but it does not result in message loss. The replication state is stored in a cursor(ledger backend), even the broker crash, the replication state not lost. So the new broker can resume the replicate task.

[kitekite2020]

Is the message replication guaranteed?
The replication state is stored in a cursor(ledger backend) -> Does this mean it is stored in bookie? What if this bookie crashes?

[codelipenghui]

Yes, it is stored in bookies(data is replicated in the certain bookies). So you can increase the data replicas to avoid data loss.

[kitekite2020]

Yes, it is stored in bookies(data is replicated in the certain bookies). So you can increase the data replicas to avoid data loss.

Thank you. So I understand from you that there is a mechanism to trace or manage which message is replicated and make sure all messages are replicated. That is great.
Is there a mechanism to trace which is message replicated and received by the target cluster/site?
I am also trying if possible to use grafana and prometheus to validate if there is message loss. Any advice here?

[codelipenghui]

Yes, the metrics related to the replication is described at https://pulsar.apache.org/docs/en/reference-metrics/#replication-metrics-1, and https://github.com/streamnative/apache-pulsar-grafana-dashboard already contains message replication related dashboards. You can checkout and try it.

[kitekite2020]

Yes, the metrics related to the replication is described at https://pulsar.apache.org/docs/en/reference-metrics/#replication-metrics-1, and https://github.com/streamnative/apache-pulsar-grafana-dashboard already contains message replication related dashboards. You can checkout and try it.

Thanks. Was trying this dashboard. Had some problems and will figure out why.

[kitekite2020]

Would like to clarify for geo-replication message delivery. Is it more like TCP, every message got acknowledged when it is replicated successfully to the other site, or more like UDP, fire and forget?