Using Edomata for real-time streaming use case

[cloudcompute]

Hi

Can we use this library for an architecture something like this:

Using this library, an application writes events to a database (using the library's Event souring feature, and not CQRS). Then a CDC connector (Debezium) pulls the changes made to the database and writes to Kafka.

Or, do we need to make any modifications to the library so that our app can handle hundreds of events in a second before it is able to write them in the database.. something like back pressure and caching.

Secondly, there is little confusion relating to the implementation of Edomata, the app that uses it can support only one at a time, either Event sourcing Or CQRS. Why is this so?

So to implement both Event Sourcing and CQRS in our entire project, the architecture would be like this:

App --> immutable Write database (Event store).. this is making use of the Event Sourcing part of Edomata). Using the DB's builtin ability to observe that new writes have been made (like MongoDB has Change stream feature) which forwards those changes to another application that supports Read-only view (this app is using the CQRS feature of Edomata)

Is this how things are working in Edomata?

Thank you.

[hnaderi]

Hi

Using this library, an application writes events to a database (using the library's Event souring feature, and not CQRS). Then a CDC connector (Debezium) pulls the changes made to the database and writes to Kafka.

Yes, you can; but if you want to use CDC only to read data, you can also read the journal and outbox using the backend itself. Here is the documentation.

Or, do we need to make any modifications to the library so that our app can handle hundreds of events in a second before it is able to write them in the database.. something like back pressure and caching.

Probably no. The default configuration of edomata backend has a builtin cache that is enabled by default and doesn't interact with database unless for writing or in case of cache misses, and I've been able to easily reach a couple of thousands of commands per second using a single machine in a few of my projects that needed it. That being said, anything performance related depends on your specific problem at hand and I can't give certain answers.

Also, in case you needed to have a custom storage strategy or implementation, you can simply implement the StorageDriver and use your own driver in the backend, which basically gives you all the control and is probably the most flexible way of doing it.
But avoid it if not necessary.

Secondly, there is little confusion relating to the implementation of Edomata, the app that uses it can support only one at a time, either Event sourcing Or CQRS. Why is this so?

No, that's not correct. Edomata is only a toolkit, and you can use it however you want and doesn't have any opinions on how to organize your overall application. You can have several Edomatons and Stomatons in a single application, in conjunction with other non-edomata modules (e.g. crud services etc...).
Each of these state machines are only a way to define a single aggregate in DDD terms.
Note that a single aggregate can't be both event-sourced and CQRS (this is a contradiction, not a limitation of this specific library), but you can have (and probably most of the time have) several different aggregates.

As a matter of fact, the project's example has done exactly this! There is an account aggregate that is ES and an order aggregate that is CQRS, both in the same application. They don't even use different connection pools, and the users can't distinguish the implementation details either.

App --> immutable Write database (Event store).. this is making use of the Event Sourcing part of Edomata). Using the DB's builtin ability to observe that new writes have been made (like MongoDB has Change stream feature) which forwards those changes to another application that supports Read-only view (this app is using the CQRS feature of Edomata)

You can use the backend to read the changes and events in real-time as mentioned above and there is no need to delve into to the database stuff.
For projections, you can simply read the journal and update your views.

I hope I answered all your questions, if there are any missing let me know, and also if there are new questions as well.

[cloudcompute]

Hi @hnaderi

Thank you for the detailed response. I got to understand things to a certain extent. I have some more queries though:

You can use the backend to read the changes and events in real-time as mentioned above and there is no need to delve into to the database stuff.

So this means an application using Edomata is capable of doing both things:
a. Writing to a database (event store)
b. Writing to Kafka (this code will pull the new events from the backend and posts them to Kafka)

We will read events from the backend only once. Am I right?

If yes, how can I make sure that both the steps are done in a single transaction. What if the app is able to write to the db and then crashes (not able to to write to Kafka).

Looks like I need to take care of myself because Edomata is not doing any CRUD stuff, right? Wondering how can I accomplish it.. I am very much new to Scala, will do the coding for the first time. In Spring, there is a Transactional annotation.

Also, in case you needed to have a custom storage strategy or implementation, you can simply implement the StorageDriver and use your own driver in the backend...

in conjunction with other non-edomata modules (e.g. crud services etc...)

Okay this means I'd use the Edomata backend to read the events and then use any DB library/driver to write those events in the db. I am looking forward to use Cassandra. What do I need to do, implement StorageDriver and use the Cassandra-Java driver?

How can I accomplish this? The objective is to separate the Read and Writes on the same database.

Microservices 1, 2 and 3: Writing Product aggregate, Pricing aggregate and User aggregate events to Event store (using the ES feature of Edomata) respectively.

Microservice 4.. Reading the Event store (all or few aggregates) and sending the result to the browser.

In Microservice 4, do I need to use the Edomata - CQRS thing? What exactly is Edomata - CQRS doing.. what role does Edomata-backend play in it?

If I need to do these two (both writes and reads) in a single transaction i.e. strong consistency (and not eventual consistency), I have to implement it of my own. Edomata won't place any restriction on it. Am I correct?

using gRPC

Hopefully I can use gRPC with Edomata. Which library would you recommend, I want to implement a gRPC enabled microservice.

Can I adjust the cache size.. increase/decrease it as per the requirements?

Thank you again
Raman

[hnaderi]

So this means an application using Edomata is capable of doing both things:
a. Writing to a database (event store)
b. Writing to Kafka (this code will pull the new events from the backend and posts them to Kafka)

Yes, you can read both the events and notifications once they are commited.

We will read events from the backend only once. Am I right?

It depends on how you process the data after reading.
Each event in the journal has a version, seq nr. and an Id you can use to implement idempotency.
Also, for notifications, the default behavior is to only read the notifications that are not marked as read yet. (like a kafka manual ack consumer)

If yes, how can I make sure that both the steps are done in a single transaction. What if the app is able to write to the db and then crashes (not able to to write to Kafka).

Having a transaction doesn't solve this problem, as you are dealing with two different databases (e.g. postgres and kafka). If one fails, the other might succeed and basically you'll enter the realm of 2 phase commit! which is not the right path to take anyway.

If you want to store data in two different databases reliably, you need to implement an idempotent writer pattern. This way, you can achieve at least once delivery of messages and changes.

Looks like I need to take care of myself because Edomata is not doing any CRUD stuff, right? Wondering how can I accomplish it..

No, edomata is a toolkit for creating event driven applications using state machines. If you want to have CRUD like entities as well, it's beyond the scope of this library.
You can use the database libraries like Skunk, Doobie or other alternatives.

I am very much new to Scala, will do the coding for the first time. In Spring, there is a Transactional annotation.

If you want to continue the FP way of scala, you probably need to gain more familiarity with the ecosystem and the language.
Things are done very differently in scala compared to java and it might be really confusing if you want to think using the java mindset.
For instance, the java way of doing transactions is annotating the handlers (as you mentioned), but in scala, most of the libraries use the composition pattern, so that all of the actions are composed together, and the result is run in a transaction.
If you want to get familiar with these stuff, one of the best ways is to join the typelevel discord or other scala communities like zio and akka.

Okay this means I'd use the Edomata backend to read the events and then use any DB library/driver to write those events in the db. I am looking forward to use Cassandra. What do I need to do, implement StorageDriver and use the Cassandra-Java driver?

If all you need is to read data and store in cassandra, you don't need to implement a StorageDriver, you just need to read the data and handle it however you want.
If you want to store data directly into the cassandra, you need to implement a StorageDriver, and take into consideration the semantics of cassandra. e.g. if you want consistency you need to choose the right write mod and ...

Microservices 1, 2 and 3: Writing Product aggregate, Pricing aggregate and User aggregate events to Event store (using the ES feature of Edomata) respectively.

Microservice 4.. Reading the Event store (all or few aggregates) and sending the result to the browser.

In Microservice 4, do I need to use the Edomata - CQRS thing? What exactly is Edomata - CQRS doing.. what role does Edomata-backend play in it?

The topology you mentioned has a few bad practices:

1. Aggregates should encapsulate their state, in an event sourced application, event are the state.
2. Reading internal events from a different service couple all the services together.
3. Event store is not meant to be used as a message bus, you might need to use a proper message bus and/or integration events (notifications in edomata)

What exactly is Edomata - CQRS doing.. what role does Edomata-backend play in it?

Stomaton (the state machine behind edomata CQRS), is a state machine that doesn't emit events and is not event sourced, but is able to publish notifications. It's storage can be a CRUDish table, a NoSQL database or anything capable of storing the state (like memento design pattern).

If I need to do these two (both writes and reads) in a single transaction i.e. strong consistency (and not eventual consistency), I have to implement it of my own. Edomata won't place any restriction on it. Am I correct?

You can use the CQRS approach, as it let you control the transaction in its handler.
However, IMHO if all you want is direct transactions and CRUD, this library is not the right choice.

Hopefully I can use gRPC with Edomata. Which library would you recommend, I want to implement a gRPC enabled microservice.

Yes, there are no limitations. As I told earlier, this library doesn't impose any limitations or opinions on users.
If you want to go the principled functional way of typelevel ecosystem, I suggest fs2-grpc

[cloudcompute]

@hnaderi

Things are more clear than before. After learning some more FP and going through your code, I will try to connect the remaining dots.

Yeah, I am new to.. functional programming, Scala, and event sourcing. The main reason for switching to Scala is only one: it has great functional support than Java and Kotlin has. And I want to do my next project using FP, OO just sucks.

basically you'll enter the realm of 2 phase commit! which is not the right path to take anyway.

I full agree with this fact. We need to implement the Saga orchestration pattern for this.. using a library like Temporal. IBut it will definitely slow down the application.

The CDC pattern can solve this problem.. Write to db first and use CDC that reads the db changes from its transactional log and write them to Kafka.

Thank you once again :) Will let you know if I decide to use this library.