Fix set semantics bug

[maartyman]

Fixes #6

[maartyman]

I opened this PR for conversation on what to do.

I'm starting to doubt that this is actually possible with the rdfjs Store interface. In some way, we need to await the import of each quad into the store to continue to match a new quad to the store. I made a new test that amplifies this problem.

[maartyman]

One solution could be to have a temporary "import store" where we can syncrounously add quads to. When checking if this quad already exists (with the match function) we check both the temporary "import store" and the actual store. If it doesn't exist yet we can add it to both stores. At the end of the stream we can delete the temporary "import store".

[maartyman]

nvm, this would break if there are multiple import streams. If, for example, two simultaneous imports are done with identical quads you would get duplicate results.

[rubensworks]

nvm, this would break if there are multiple import streams. If, for example, two simultaneous imports are done with identical quads you would get duplicate results.

You could work around this, by halting concurrent imports until the other import is done.
But this would complicate things of course, so I'd only do this if we're certain there's no easier way :-)

[rubensworks]

Actually, what you suggest would make the logic quite clean:

1. Wait until other concurrent imports are done
2. Determine unique triples synchronously
3. Import unique triples to store and listeners.

[maartyman]

This does mean that a non ending stream could block imports into the store.

[rubensworks]

This does mean that a non ending stream could block imports into the store.

True, that would be problematic indeed...
So a more fine-grained quad-per-quad approach may be needed.

[maartyman]

I was implementing this, but I realize now that there is no way of knowing when we should free the synchronous store.

1. Add quad in 'synchronous store'
2. Import unique quads to 'actual store' and listeners
3. After import is done delete 'synchronous store'

When do we delete the 'synchronous store'? We don't know when the import to the 'actual store' has finished. The only way I know is continuously match the quads to the 'actual store' until the match function returns an element, but this is really inefficient.

[rubensworks]

This may be getting too complicated for what it's worth.

Maybe we should concede and just assume a synchronous quad-containment checker in the rdf store.
This would come at the cost of interoperability, but perhaps this is fine given the complexity.

As long as it's works with n3 and rdf-stores, it's fine by me :-)

[maartyman]

So I switch from a rdfjs interface to a n3 Store interface?

[maartyman]

Or I switch to a rdfjs dataset interface?

[maartyman]

See branch: https://github.com/comunica/rdf-streaming-store.js/tree/fix-set-semantics-bug-with-other-interface

[rubensworks]

@maartyman This approach won't work with rdf-stores unfortunately, as that lib doesn't implement DatasetCore directly on the store, but through a separate asDataset() method.

What we could do instead, is make use of the sync countQuads method that both N3 and rdf-stores implement. We could just create a custom local interface with a copy of this countQuads method signature, so that we can pass both N3 stores and rdf-stores.