[Remote Store] Translogs get re-downloaded on peer/existing/remote store recoveries

[Bukhtawar]

Describe the bug

As a part of performing recoveries we re-download translogs when all we need is to re-open them as a part of opening ore resetting the engine

java.lang.Thread.State: WAITING (parking)
        at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
        - parking to wait for  <0x00000017820c2028> (a java.util.concurrent.FutureTask)
        at java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:211)
        at java.util.concurrent.FutureTask.awaitDone([email protected]/FutureTask.java:447)
        at java.util.concurrent.FutureTask.get([email protected]/FutureTask.java:190)
        at org.opensearch.encryption.frame.CryptoInputStream.read(CryptoInputStream.java:193)
        at java.io.InputStream.transferTo([email protected]/InputStream.java:782)
        at java.nio.file.Files.copy([email protected]/Files.java:3171)
        at org.opensearch.index.translog.transfer.TranslogTransferManager.downloadToFS(TranslogTransferManager.java:263)
        at org.opensearch.index.translog.transfer.TranslogTransferManager.downloadTranslog(TranslogTransferManager.java:246)
        at org.opensearch.index.translog.RemoteFsTranslog.downloadOnce(RemoteFsTranslog.java:222)
        at org.opensearch.index.translog.RemoteFsTranslog.download(RemoteFsTranslog.java:189)
        at org.opensearch.index.translog.RemoteFsTranslog.<init>(RemoteFsTranslog.java:112)
        at org.opensearch.index.translog.RemoteBlobStoreInternalTranslogFactory.newTranslog(RemoteBlobStoreInternalTranslogFactory.java:67)
        at org.opensearch.index.translog.InternalTranslogManager.openTranslog(InternalTranslogManager.java:373)
        at org.opensearch.index.translog.InternalTranslogManager.<init>(InternalTranslogManager.java:73)
        at org.opensearch.index.engine.InternalEngine.<init>(InternalEngine.java:295)
        at org.opensearch.index.engine.InternalEngine.<init>(InternalEngine.java:224)
        at org.opensearch.index.engine.NRTReplicationEngineFactory.newReadWriteEngine(NRTReplicationEngineFactory.java:23)
        at org.opensearch.index.shard.IndexShard.resetEngineToGlobalCheckpoint(IndexShard.java:4866)
        - locked <0x000000114c790160> (a java.lang.Object)
        at org.opensearch.index.shard.IndexShard.lambda$resetToWriteableEngine$12(IndexShard.java:2006)
        at org.opensearch.index.shard.IndexShard$$Lambda$9640/0x00000001024f5c50.run(Unknown Source)
        at org.opensearch.index.shard.IndexShardOperationPermits.blockOperations(IndexShardOperationPermits.java:123)
        at org.opensearch.index.shard.IndexShard.resetToWriteableEngine(IndexShard.java:2006)
        at org.opensearch.indices.replication.SegmentReplicationTargetService$4.onReplicationDone(SegmentReplicationTargetService.java:669)
        at org.opensearch.indices.replication.SegmentReplicationTargetService$SegmentReplicationListener.onDone(SegmentReplicationTargetService.java:525)
        at org.opensearch.indices.replication.common.ReplicationTarget.markAsDone(ReplicationTarget.java:149)
        at org.opensearch.indices.replication.common.ReplicationCollection.markAsDone(ReplicationCollection.java:221)
        at org.opensearch.indices.replication.SegmentReplicationTargetService$3.onResponse(SegmentReplicationTargetService.java:577)
        at org.opensearch.indices.replication.SegmentReplicationTargetService$3.onResponse(SegmentReplicationTargetService.java:573)
        at org.opensearch.indices.replication.SegmentReplicationTarget.lambda$startReplication$3(SegmentReplicationTarget.java:187)
        at org.opensearch.indices.replication.SegmentReplicationTarget$$Lambda$7299/0x0000000101e0ae78.accept(Unknown Source)
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82)
        at org.opensearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:126)
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
        at org.opensearch.common.util.concurrent.OpenSearchExecutors$DirectExecutorService.execute(OpenSearchExecutors.java:412)
        at org.opensearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:120)
        at org.opensearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:82)
        at org.opensearch.action.StepListener.whenComplete(StepListener.java:95)
        at org.opensearch.indices.replication.SegmentReplicationTarget.startReplication(SegmentReplicationTarget.java:185)
        at org.opensearch.indices.replication.SegmentReplicationTargetService.start(SegmentReplicationTargetService.java:573)
        at org.opensearch.indices.replication.SegmentReplicationTargetService$ReplicationRunner.doRun(SegmentReplicationTargetService.java:559)
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:917)
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
        at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1136)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:635)
        at java.lang.Thread.run([email protected]/Thread.java:833)

    at org.opensearch.index.translog.transfer.BlobStoreTransferService.downloadBlob(BlobStoreTransferService.java:164)
    at org.opensearch.index.translog.transfer.TranslogTransferManager.downloadToFS(TranslogTransferManager.java:260)
    at org.opensearch.index.translog.transfer.TranslogTransferManager.downloadTranslog(TranslogTransferManager.java:246)
    at org.opensearch.index.translog.RemoteFsTranslog.downloadOnce(RemoteFsTranslog.java:222)
    at org.opensearch.index.translog.RemoteFsTranslog.download(RemoteFsTranslog.java:189)
    at org.opensearch.index.translog.RemoteFsTranslog.<init>(RemoteFsTranslog.java:112)
    at org.opensearch.index.translog.RemoteBlobStoreInternalTranslogFactory.newTranslog(RemoteBlobStoreInternalTranslogFactory.java:67)
    at org.opensearch.index.translog.InternalTranslogManager.openTranslog(InternalTranslogManager.java:373)
    at org.opensearch.index.translog.InternalTranslogManager.<init>(InternalTranslogManager.java:73)
    at org.opensearch.index.translog.WriteOnlyTranslogManager.<init>(WriteOnlyTranslogManager.java:43)
    at org.opensearch.index.engine.NRTReplicationEngine.<init>(NRTReplicationEngine.java:128)
    at org.opensearch.index.engine.NRTReplicationEngineFactory.newReadWriteEngine(NRTReplicationEngineFactory.java:21)
    at org.opensearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:2539)
    - locked <0x0000001137140b38> (a java.lang.Object)
    at org.opensearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:2481)
    at org.opensearch.index.shard.IndexShard.openEngineAndSkipTranslogRecovery(IndexShard.java:2463)
    at org.opensearch.indices.recovery.RecoveryTarget.lambda$prepareForTranslogOperations$1(RecoveryTarget.java:256)
    at org.opensearch.indices.recovery.RecoveryTarget$$Lambda$7402/0x0000000101f7ab20.get(Unknown Source)
    at org.opensearch.core.action.ActionListener.completeWith(ActionListener.java:344)
    at org.opensearch.indices.recovery.RecoveryTarget.prepareForTranslogOperations(RecoveryTarget.java:253)
    at org.opensearch.indices.recovery.PeerRecoveryTargetService$PrepareForTranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:402)
    at org.opensearch.indices.recovery.PeerRecoveryTargetService$PrepareForTranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:391)

Related component

Storage:Remote

To Reproduce

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior

Decouple translog open and download methods so that opening translogs just requires downloading the latest metadata and cached translog files are re-used.

Additional Details

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

[peternied]

[Triage - attendees 1 2 3 4 5 6 7 8]
@Bukhtawar Thanks for creating this issue; however, it isn't being accepted due to being unclear how to reproduce this issue or how to interpret the callstacks that were included. Please feel free to re-open after addressing the reason