Add encryption support for repository

[vikasvb90]

Description

This PR adds an encrypted layer on top of a repository which allows encryption to remain transparent to features using a repository.

Related Issues

Meta Issue : #7229

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/22594/
• CommitID: 3077261
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[opensearch-trigger-bot]

Compatibility status:

> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/performance-analyzer.git]
Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

BUILD SUCCESSFUL in 33m 21s

[github-actions]

Compatibility status:

Checks if related components are compatible with change d3b938b

Incompatible components

Incompatible components: [https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/neural-search.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/24339/
• CommitID: d3b938b

[github-actions]

Compatibility status:

Checks if related components are compatible with change bfb49b9

Incompatible components

Incompatible components: [https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/neural-search.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.client.PitIT.testDeleteAllAndListAllPits

• URL: https://build.ci.opensearch.org/job/gradle-check/24354/
• CommitID: bfb49b9
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Merging #9289 (bfb49b9) into main (e98ded6) will decrease coverage by 0.01%.
Report is 3 commits behind head on main.
The diff coverage is 46.00%.

@@             Coverage Diff              @@
##               main    #9289      +/-   ##
============================================
- Coverage     71.07%   71.06%   -0.01%     
- Complexity    57730    57776      +46     
============================================
  Files          4806     4814       +8     
  Lines        272238   272660     +422     
  Branches      39729    39783      +54     
============================================
+ Hits         193480   193779     +299     
- Misses        62531    62556      +25     
- Partials      16227    16325      +98     

Files Changed	Coverage Δ
...rg/opensearch/repositories/s3/S3BlobContainer.java	76.07% <0.00%> (-2.59%)	⬇️
.../repositories/put/PutRepositoryRequestBuilder.java	50.00% <0.00%> (-5.56%)	⬇️
...main/java/org/opensearch/common/StreamContext.java	62.50% <0.00%> (-37.50%)	⬇️
...ommon/blobstore/AsyncMultiStreamBlobContainer.java	0.00% <ø> (ø)
...bstore/AsyncMultiStreamEncryptedBlobContainer.java	0.00% <0.00%> (ø)
...earch/common/blobstore/EncryptedBlobContainer.java	0.00% <0.00%> (ø)
...search/common/blobstore/EncryptedBlobMetadata.java	0.00% <0.00%> (ø)
...pensearch/common/blobstore/EncryptedBlobStore.java	0.00% <0.00%> (ø)
...ch/common/blobstore/stream/write/WriteContext.java	64.28% <0.00%> (-35.72%)	⬇️
...ch/repositories/blobstore/BlobStoreRepository.java	60.82% <0.00%> (+0.68%)	⬆️
... and 13 more

... and 495 files with indirect coverage changes

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-9289-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 63ce8324b5a4a2d1afa3f76c1dd758f55f4cd0e8
# Push it to GitHub
git push --set-upstream origin backport/backport-9289-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-9289-to-2.x.

[peternied]

This change is pulling in bouncy castle libraries which are causing havoc with the Security plugins use of SecurityManager. Can we revert the change to server/build.gradle by removing api project(":libs:opensearch-encryption-sdk"), or is there an alternative way we can prevent the downstream impact?

[vikasvb90]

@peternied We had a discussion here and have been having discussions earlier around the design of encryption and how we want to place it. I believe by removing you are suggesting making it a plugin since removing is not really an option. We decided in the last discussion that security needs to be a first class member of the core and hence, encryption is to be built as a lib. Regarding bouncy castle library, can't we try excluding this from plugin? Not sure if there's an alternative to avoid impacting downstreams because of dependencies pulled by lib.

[reta]

@vikasvb90 the BC (Bouncycastle) provider is added to core but it is not loaded anywhere (programmatically at least, since modifying the JVM security properties is out of scope), are there any plans to have it added to the list of security providers? (Security.addProvider)?