WIP: Example for how the validationFn could work. #45
Conversation
I wanted to give people an example in code of my idea for validation functions as an additional and more powerful custom validation & access control mechanism than Access Controllers are currently.
|
I like the idea of having more customizable indexing in stores via validation functions. The problem of doing it on the index/store level is that the oplog (which is the ultimate source of truth for any db) can’t access that function to validate them upon processing the ops. This means that one could enter a valid oplog entry but it’s not a valid entry from the perspective of the validation function. Second part of this is that any validation should also be done before writing to the oplog in order to prevent user accidentally adding invalid entries to the oplog. Based on some work I did some time ago creating new ACs, I believe it’d be possible to do this kind of validation in an AC. However, iirc, the ACs don’t currently have access to the store that is using them, so they can’t query the current state of the database as part of the validation (eg. “If this key already has value, don’t allow the op that sets a new value to it” as per your example). It’d be worth experimenting more on the AC level and change them (or their interface) accordingly. What do you think @CSDUMMI? |
|
That is just the problem I want to address. Nor is it possible to recreate the important parts of the state of a store in the AC. (i.e. having another db in the AC with all the keys that have already been used contained within it) Because Thus the AC interface as it currently stands does not, to my knowledge at least, This has the unfortunate consequence, that complex validation is only possible once the oplog has already been created. The perfect place for Validation (Access Rules under consideration of the State of the Store) would be somewhere in the middle, after the oplog has been assembled and before the oplog becomes that ultimate source of truth in the database. |
|
What kind of validation would be possible if ACs had access to their store? Is there a way, for example, to create logic that “delays” returning invalid (false) until it can be sure something was invalid earlier and that all ops have been processed? |
|
I think that's another case entirely, to invalidate an entry based on future entries. And that I don't think is very intuitive for any developer. If an AC had access to their store, they could perform just such validation |
Do you mean that this (access to the store) would solve your particular use case? If so, it sounds like a PR that passes the store to the ACs would get us forward quite a bit and open new possibilities to explore even if it doesn’t allow all possible validation cases atm. |
|
I'm doubtful though, that validation (AC Rules using Store State) should even be able to mutate the oplog. Since I don't know what consequences this could have on replication. Consider the following scenario:
Now if A and B sync their oplogs, they will not have the same oplog. The question thus is: Should two peers syncing their |
@tabcat I think has been opposed to this idea @haadcode when I proposed it in the Matrix channel. Though I failed to understand @tabcat in this point. |
|
a possible solution for your case could be to make an access controller that checks entries for what key they want to edit and requiring them to use a namespace if they are not an admin. for example the namespace could be the writers id, if they want to edit the key |
|
That's not achieving the same behavior as my example tries to achieve. Namespaces don't help me here. |
|
@CSDUMMI would it be possible for you to make such functionality by implementing a custom Store/Index (as opposed to trying to right now generalize it into KVStore)? |
|
As side note, eventual consistency doesn't lend itself well for the use case described above, which is "making sure there's only one unique value for a key in database". This is a use case where strongly consistent db models fit better, especially if "anyone can write to the database". |
As an expansion to this: Because of eventual consistency, the result of the AC should be generally independent from the contents of the Store. This mostly arises of the fact that you don't know the order that entries will get replicated to your node. Given this, I think that giving access to the Store from the AC will result to various unexpected behaviours. |
|
That is a problem which is why I think, we should work Even if that means that some operations will remain |
|
Unless there is another path forward, where most of the customizations are possible in the most efficient manner. But efficiency is an expected payoff for customization. |
I wanted to give people an example in code of my idea for validation functions as an additional and more powerful
custom validation & access control mechanism than Access Controllers are currently.
The validationFn is given an entry in the oplog and the current state of the store.
And only if validationFn returns a true-ish value, does the entry get to be applied
to the store's index.