[Backport 2.x] Validate checksum of each segment file post download from remote store

[opensearch-trigger-bot]

Backport 6c02261 from #10119.

[github-actions]

Compatibility status:

Checks if related components are compatible with change a1533b5

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git]

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testCancelPrimaryAllocation

• URL: https://build.ci.opensearch.org/job/gradle-check/27741/
• CommitID: a1533b5
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Merging #10590 (a1533b5) into 2.x (52018ec) will increase coverage by 0.03%.
The diff coverage is 76.00%.

@@             Coverage Diff              @@
##                2.x   #10590      +/-   ##
============================================
+ Coverage     70.84%   70.87%   +0.03%     
- Complexity    58489    58536      +47     
============================================
  Files          4831     4831              
  Lines        276413   276436      +23     
  Branches      40581    40583       +2     
============================================
+ Hits         195822   195929     +107     
+ Misses        63872    63829      -43     
+ Partials      16719    16678      -41     

Files	Coverage Δ
...earch/index/store/RemoteSegmentStoreDirectory.java	88.88% <100.00%> (+0.03%)	⬆️
...rc/main/java/org/opensearch/index/store/Store.java	80.30% <75.00%> (-1.24%)	⬇️

... and 478 files with indirect coverage changes