Architecture: streaming data consolidation

[vaibhav-rbs]

I am trying to come up with an architecture we have some streams/RPC clients running on python-Autobahn. We have a crossbar broker which allows a consumer application to take the RPC/stream and precess those values in the databases. Currently we have strategy of using


[
N client (publishing stream /registering/calling RPC) —> Xbar1 —> a consumer application which takes care of all these data points by N clients and stored them in the DB
M client (publishing stream /registering/calling RPC) —> Xbar2 —> a consumer application which takes care of all these data points by M clients  and stored them in the DB
]

This works good if we have one to one mapping, but this also means that we can’t independently scale the number of Xbar servers and consumer application, as if we did that we will have problems as follows

All client (publishing stream /registering/calling RPC) —> Xbar1 —>  a consumer application which takes care of all these data points   and stored them in the DB

Reached 80 % capacity, we have to spin up a new xbar instance for balancing the load.

All client (publishing stream /registering/calling RPC) —> Xbar1 —>  a consumer application which takes care of all these data points and stored them in the DB
											               Xbar2 —>  a consumer application which takes care of all these data points and stored them in the DB

The problem is we don’t want to have double entries in the DB as same message will be processed by Xbar1 and Xbar2 if we scale up based on the load
We need to have some type of service which consolidates the data points and process them once, I would like to listen some ideas and hear about potential solution.

[oberstet]

it's unclear to me what exactly you want to achieve, but anyways, maybe this helps: https://github.com/crossbario/crossbar-examples/tree/master/scaling-microservices