[Segment Replication] Sequence number based recoveries

[dreamer-89]

Coming from #6761 exercise, there are few tests failures (listed below) due to sequence number based recovery been not attempted. Seq no based recovery is not attempted as it needs replica shard to identify starting sequence number during recovery and replay (index) local translog operations on underlying engine. Since, NRTReplicationEngine does not support indexing operations, thus UNASSIGNED_SEQ_NO is always used as the starting seq no. On soure (primary) this results in sequence number based recovery to be not attempted. Created this issue to validate that this behavior does not have any undesired implications outside of test failures.

Sequence number based recovery is perf improvement as it recover data locally by replaying from local translog operations and skips phase 1 of recovery which copies over data (segment) files from primary, including files with different checksum from replica which adds to recovery time. With segment replication, replica sync segment files from primary and thus should not have files with different checksum. Thus, with segment replication avoiding seq no based recovery shouldn't have performance issues. With document replication, the replica and primary running on InternalEngine have different checksums data files on latest commit point which is not the case with segment replication.

Test failures

1. PrimaryAllocationIT.testPrimaryReplicaResyncFailed [BUG] [Segment Replication] Resync failures results in removal of in-sync allocation id  #7163
2. CloseIndexIT.testNoopPeerRecoveriesWhenIndexClosed [BUG] [Segment Replication] NO-OP recovery not attempted #7161
3. ReplicaShardAllocatorIT.testFullClusterRestartPerformNoopRecovery [BUG] [Segment Replication] NO-OP recovery not attempted #7161