[Repository] Perform benchmarking for multistream downloads

[kotwanikunal]

Is your feature request related to a problem? Please describe.

• Multi-stream download support was added as a part of Add async blob read and download support using multiple streams #9592
• The focus of this task is to run performance benchmarking for the newly added APIs to guage improvements

Describe the solution you'd like
Run benchmarks using OpenSearch/opensearch-benchmark for the newly added APIs

Describe alternatives you've considered

• N/A

Additional context

• [Repository] Support multi-stream downloads within Repository #8596

[Rishikesh1159]

I have added few logs on the recovery path to get some benchmarks for multistream downloads.

Test Summary :

• Dataset : Stack Overflow (34.1gb)
• Metrics : Recovery time (Ingested cluster with stackoverflow dataset and then added a replica to the index. The new replica would download all the data from remote store. The time taken here to download all data from remote store is recovery/restore time for replica shard)
• Results:

With Multistream Download:

From logs time taken to download is : 4.599 min

[so][0] starting time for download segments : 11618566048144637
[2023-09-30T00:05:12,414][INFO ][o.o.i.s.IndexShard    ] [node-3] [so][0] End time for download segments : 11618841991960896
[2023-09-30T00:05:12,420][INFO ][o.o.i.s.IndexShard    ] [node-3] [so][0] time taken to download segments : 275943816259

From recovery api:

$ curl localhost:9200/_cat/recovery?v
index shard time type    stage source_host  source_node target_host  target_node repository snapshot files files_recovered files_percent files_total bytes bytes_recovered bytes_percent bytes_total translog_ops translog_ops_recovered translog_ops_percent
so  0   143ms empty_store done n/a      n/a     node-ip node-1   n/a    n/a   0   0        0.0%     0      0   0        0.0%     0      0      0           100.0%
so  0   4.7m peer    done node-ip node-1   node-ip node-3   n/a    n/a   1   1        100.0%    169     3776 3776      100.0%    36711602919 0      0           100.0%

Without MultiStream Download (default case):

From logs time taken to download is : 14.867 min.

starting time for download segments : 11674473747749612
[2023-09-30T15:47:16,197][INFO ][o.o.i.s.IndexShard    ] [node-1] [so][0] End time for download segments : 11675365792826759
[2023-09-30T15:47:16,198][INFO ][o.o.i.s.IndexShard    ] [node-1] [so][0] time taken to download segments : 892045077147

From recovery api :

$ curl localhost:9200/_cat/recovery?v
index shard time type    stage source_host source_node target_host  target_node repository snapshot files files_recovered files_percent files_total bytes bytes_recovered bytes_percent bytes_total translog_ops translog_ops_recovered translog_ops_percent
so  0   181ms empty_store done n/a     n/a     node-ip  node-2   n/a    n/a   0   0        0.0%     0      0   0        0.0%     0      0      0           100.0%
so  0   14.8m peer    done node-ip node-2   node-ip node-1   n/a    n/a   1   1        100.0%    154     3356 3356      100.0%    36662256654 0      0           100.0%

From the results above we can see that: MultiStream Download is almost 3x+ faster than Non-MultiStream (Default case)