Ability to fill seqNr gaps with currentEventsByPersistenceId query for sources that expose such

[leviramsey]

Short description

Two possible rationales:

• resilience in the face of possible journal unreliability (e.g. querying from eventually consistent sources e.g. Cassandra). The targeted backtracking would be expected to have a performance impact, but it may often be preferable to be slow than failed. Given sufficient observability, this would not intrinsically be masking a problem (in any case, resilience arguably includes resilience against bugs). It would admit the possibility of missing a suffix of the events from a given persistence ID, but those would never be rejected anyway.

• many projections may want to start projecting from some recency window (e.g. "only shopping carts since some previous release") to avoid having to process an arbitrarily large backlog. Starting a projection in the middle should reject seqNrs for entities that started before that point-in-time and it's a reasonable expectation that there would be entities which persisted events on both sides of that starting point: this functionality would allow the projection to have "full history of entities which were active after the projection's starting point" which seems like something desirable in at least some cases from an application perspective.

Details

This could be accomplished with a trait on event sourced SourceProvider which would be added if the readjournal exposes CurrentEventsByPersistenceIdTypedQuery. Projection implementations could then (similar to the CanTriggerReplay mechanism) respond to seqNr rejections by using that interface to request a source of events (ultimately fulfilled by the read journal). The projection would fail if this source completed before fully satisfying the query (this indicates an actual absence from the journal, since the query is almost surely the same query used to rehydrate the entity in the write-side, which assumes strong consistency).