[Remote Store] Avoid repeated delete calls for the stale segment files

[sachinpkale]

Description

• Currently, we trigger cleanup of stale segment files from remote segment store on each flush.
• This clean-up is based on metadata file which references to the segment file
• If multiple stale segment metadata files refer to the same segment file, we call remoteDataDirectory.deleteFile multiple times.
• Functionally, this is a no-op as RemoteDirectory internally calls blobContainer.deleteBlobsIgnoringIfNotExists.
• But this can still make a call to the remote store which is un-necessary.
• In this PR, we keep track of deleted files and avoid calling delete on the same deleted file.

Related Issues

• [Remote Store] Avoid repeated delete calls for the stale segment files #11530

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Compatibility status:

Checks if related components are compatible with change d6b88ec

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/sql.git]

[github-actions]

❕ Gradle check result for 47602dc: UNSTABLE

• TEST FAILURES:

      2 org.opensearch.index.shard.RemoteStoreRefreshListenerTests.classMethod
      1 org.opensearch.index.shard.RemoteStoreRefreshListenerTests.testRefreshAfterCommit

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Attention: 2 lines in your changes are missing coverage. Please review.

Comparison is base (87b3011) 71.31% compared to head (d6b88ec) 71.33%.
Report is 2 commits behind head on main.

Files	Patch %	Lines
...earch/index/store/RemoteSegmentStoreDirectory.java	87.50%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #11532      +/-   ##
============================================
+ Coverage     71.31%   71.33%   +0.02%     
- Complexity    59157    59176      +19     
============================================
  Files          4904     4904              
  Lines        278149   278152       +3     
  Branches      40419    40419              
============================================
+ Hits         198360   198422      +62     
- Misses        63254    63276      +22     
+ Partials      16535    16454      -81     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

❕ Gradle check result for d6b88ec: UNSTABLE

• TEST FAILURES:

      1 org.opensearch.remotestore.RemoteIndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.