Fix flaky test NRTReplicationEngineTests.testPreserveLatestCommit

[mch2]

Description

Fix flaky test NRTReplicationEngineTests.testPreserveLatestCommit.
This test occasionally failed when the primary's engine performed a flush and merge behind the scenes. The test would assert against a specific refCount on committed files with a value of 2 - one for being part of the replica's latest commit and one for being active on the reader. With the unexpected merge, the files will not be part of the latest infos on the reader but still be part of the replica's latest commit point, leaving the files referenced once. I've changed this to instead assert referenced vs unreferenced instead of the specific refCount so that the test can handle the random merges. Further cleaned this up so assertions include a refCount check and to validate the files exist in the directory.

I've run this ~5k times now without failure, was able to repro before this every 100-200 runs.

Related Issues

Resolves #9356

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[opensearch-trigger-bot]

Compatibility status:

Checks if related components are compatible with change b58bec4

Incompatible components

Incompatible components: [https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/security-analytics.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.search.SearchTimeoutIT.testSimpleTimeout {p0={"search.concurrent_segment_search.enabled":"true"}}

• URL: https://build.ci.opensearch.org/job/gradle-check/22854/
• CommitID: 6b4b17f
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Merging #9378 (6b4b17f) into main (4114009) will decrease coverage by 0.02%.
Report is 6 commits behind head on main.
The diff coverage is n/a.

@@             Coverage Diff              @@
##               main    #9378      +/-   ##
============================================
- Coverage     71.19%   71.17%   -0.02%     
+ Complexity    57455    57451       -4     
============================================
  Files          4777     4777              
  Lines        270712   270712              
  Branches      39566    39566              
============================================
- Hits         192729   192690      -39     
- Misses        61782    61783       +1     
- Partials      16201    16239      +38     

see 493 files with indirect coverage changes

[Poojita-Raj]

Gradle Check (Jenkins) Run Completed with:

* **RESULT:** UNSTABLE ❕

* **TEST FAILURES:**

      1 org.opensearch.search.SearchTimeoutIT.testSimpleTimeout {p0={"search.concurrent_segment_search.enabled":"true"}}

* **URL:** https://build.ci.opensearch.org/job/gradle-check/22854/

* **CommitID:** [6b4b17f](https://github.com/opensearch-project/OpenSearch/commit/6b4b17f75fd47780ccf1e7d769845de6792ef400)
  Please review all [flaky tests](https://github.com/opensearch-project/OpenSearch/blob/main/DEVELOPER_GUIDE.md#flaky-tests) that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Mentioned under cancellation failures here: #7357