Move MemoryOverflowModel and SailSourceModel to rdf4j-sail-base

[kenwenzel]

Problem description

Both, NativeStore and LmdbStore, use the classes MemoryOverflowModel and SailSourceModel with own implementations.

Preferred solution

Move MemoryOverflowModel and SailSourceModel to rdf4j-sail-base.

Are you interested in contributing a solution yourself?

None

Alternatives you've considered

No response

Anything else?

No response

[hmottestad]

We should make sure that we don't break backwards compatibility.

[abrokenjester]

I personally think we should get rid of SailSourceModel entirely, as it has quite poor performance, and replace with a MapDB-backed model implementation.

[kenwenzel]

I am not sure if MapDB will help here as we need multiple triple indexes and also some kind of value store. In the end we will be forced to implement something like NativeStore or LmdbStore.

[hmottestad]

I'm not sure if we need all that much safety wise with the MemoryOverflowModel. We could overflow to a n-quads file. The ShaclSail wouldn't enjoy the performance hit though, since it would have to query the data in the current transaction which would mean reading from the n-quads file.

[kenwenzel]

But this would require a linear scan to match different triple patterns like (S, _, ), (, P, ) or (, _, O).
As overflow is only triggered for large transactions it should be expected that the n-quads file is also rather large.
We could also experiment with Parquet files that support faster scanning by skipping row groups.

[hmottestad]

True, which is why it would be bad for the ShaclSail. In general I would assume that users who load in large files don't typically run queries before committing. We could try something like the DynamicModel, where we use a very simple and fast overflow until the user runs queries at which point we would migrate the data to a more advanced data structure.

[kenwenzel]

@hmottestad I think we could improve the current situation:
#4557

Maybe you can help with the logic for switching to disk-based storage.