use multipart upload to parallelize index metadata uplaod #6
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
soosinha
force-pushed
the
remote_state_publish
branch
2 times, most recently
from
155e369 to
3773049
Compare
…thing specified by user (opensearch-project#9264) --------- Signed-off-by: bansvaru <[email protected]> Signed-off-by: Varun Bansal <[email protected]>
…rch-project#8951) Signed-off-by: Sachin Kale <[email protected]>
) * Add support to use trace propagated from client Signed-off-by: Gagan Juneja <[email protected]> * Add support to use trace propagated from client Signed-off-by: Gagan Juneja <[email protected]> * Refactor code Signed-off-by: Gagan Juneja <[email protected]> * Add support to use trace propagated from client Signed-off-by: Gagan Juneja <[email protected]> * Add support to use trace propagated from client Signed-off-by: Gagan Juneja <[email protected]> * Refactor code Signed-off-by: Gagan Juneja <[email protected]> * Refactor code Signed-off-by: Gagan Juneja <[email protected]> * Merged CHANGELOG Signed-off-by: Gagan Juneja <[email protected]> * Empty-Commit Signed-off-by: Gagan Juneja <[email protected]> * Empty-Commit Signed-off-by: Gagan Juneja <[email protected]> * Empty-Commit Signed-off-by: Gagan Juneja <[email protected]> * Empty-Commit Signed-off-by: Gagan Juneja <[email protected]> * Empty-Commit Signed-off-by: Gagan Juneja <[email protected]> * Empty-Commit Signed-off-by: Gagan Juneja <[email protected]> --------- Signed-off-by: Gagan Juneja <[email protected]> Signed-off-by: Gagan Juneja <[email protected]> Co-authored-by: Gagan Juneja <[email protected]>
* Update BWC testing documentation Signed-off-by: Stephen Crawford <[email protected]> * Fix code block Signed-off-by: Stephen Crawford <[email protected]> * Fix code block Signed-off-by: Stephen Crawford <[email protected]> * Add placeholder Signed-off-by: Stephen Crawford <[email protected]> * Fix typos Signed-off-by: Stephen Crawford <[email protected]> * Fix table of contents order Signed-off-by: Stephen Crawford <[email protected]> * Add colon Signed-off-by: Stephen Crawford <[email protected]> * trigger retry Signed-off-by: Stephen Crawford <[email protected]> --------- Signed-off-by: Stephen Crawford <[email protected]>
linuxpi
force-pushed
the
remote-state-publish-multipart
branch
from
d049abe to
1dc6bc0
Compare
…pensearch-project#9562) * For sort request on timeseries field use non concurrent search path Signed-off-by: Sorabh Hamirwasia <[email protected]> * Address review feedback Signed-off-by: Sorabh Hamirwasia <[email protected]> --------- Signed-off-by: Sorabh Hamirwasia <[email protected]>
…ing engine reset. (opensearch-project#9495) * Fix bug where replica shows stale doc count during engine reset. This change fixes an issue where replica shards can temporarily return stale results while converting to a RO engine during an engine reset. This is possible because NRTReplicationEngine did not previously implement flush and the freshest data is only active on the reader. Fixed by implementing flush and also honoring acquireLatestCommit's flushFirst parameter. Signed-off-by: Marc Handalian <[email protected]> * Add changelog entry. Signed-off-by: Marc Handalian <[email protected]> * Add unit test for search during engine reset. Signed-off-by: Marc Handalian <[email protected]> * Remove useless test. Signed-off-by: Marc Handalian <[email protected]> --------- Signed-off-by: Marc Handalian <[email protected]>
…plication (opensearch-project#9593) * [Segment Replication] Add ClusterState utility to identify SEGMENT replication Signed-off-by: Suraj Singh <[email protected]> * Address review comment Signed-off-by: Suraj Singh <[email protected]> * Address review comments Signed-off-by: Suraj Singh <[email protected]> --------- Signed-off-by: Suraj Singh <[email protected]>
opensearch-project#9578) * Bump io.grpc:grpc-api from 1.57.1 to 1.57.2 in /plugins/repository-gcs Bumps [io.grpc:grpc-api](https://github.com/grpc/grpc-java) from 1.57.1 to 1.57.2. - [Release notes](https://github.com/grpc/grpc-java/releases) - [Commits](grpc/grpc-java@v1.57.1...v1.57.2) --- updated-dependencies: - dependency-name: io.grpc:grpc-api dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
* Add SearchExtBuilders to SearchResponse. [Issue opensearch-project#9328](opensearch-project#9328) Signed-off-by: Austin Lee <[email protected]> * Keep SearchResponse immutable, add a constructor to take a List of SearchExtBuilders. Signed-off-by: Austin Lee <[email protected]> * Fix spotlessJavaCheck findings. Signed-off-by: Austin Lee <[email protected]> * Move SearchExtBuilders into SearchResponseSections, fix indenting in SearchRequest. Signed-off-by: Austin Lee <[email protected]> * Updated changelog (mixed minor formatting issues), added version checks on serialization/deserialization, added a Builder for making copies of SearchResponse easier. Signed-off-by: Austin Lee <[email protected]> * Add GenericSearchExtBuilder as a catch-all for SearchExtBuilders not registered in xcontent registry. Signed-off-by: Austin Lee <[email protected]> * Simplify GenericSearchExtBuilder using a single Object member. Signed-off-by: Austin Lee <[email protected]> * Address additional review comments. Signed-off-by: Austin Lee <[email protected]> * Add Javadocs. Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]>
…t#9596) Signed-off-by: Sayali Gaikawad <[email protected]>
…oject#9597) Signed-off-by: Suraj Singh <[email protected]>
6 tasks
…tations (part 1) (opensearch-project#9520) * [BWC and API enforcement] Decorate the existing APIs with proper annotations (part 1) Signed-off-by: Andriy Redko <[email protected]> * Address code review comments Signed-off-by: Andriy Redko <[email protected]> --------- Signed-off-by: Andriy Redko <[email protected]>
…tion. (opensearch-project#9471) * Fix SegmentReplicationUsingRemoteStoreIT#testDropPrimaryDuringReplication. This test is failing because a concurrent flush can wipe out an old commit file while we are in the remote store refresh listener. The listener will fetch the latest infos from the reader which will reference a segments_n tht has been deleted by an incoming flush. To fix this, InternalEngine will preserve the latest commit until a new commit is loaded onto the readerManager. Signed-off-by: Marc Handalian <[email protected]> * update InternalEngine to preserve commit file until a new commit is refreshed on. Signed-off-by: Marc Handalian <[email protected]> * Update ReadOnlyEngine inside of resetEngineToGlobalCheckpoint to implement getSegmentInfosSnapshot. This ensures access to this function is not permitted on the ReadOnlyEngine and is delegated to the new IE once opened. Signed-off-by: Marc Handalian <[email protected]> * Update javadoc. Signed-off-by: Marc Handalian <[email protected]> * spotless. Signed-off-by: Marc Handalian <[email protected]> --------- Signed-off-by: Marc Handalian <[email protected]>
…hResponse (Issue # 9328) (opensearch-project#9641) Signed-off-by: Austin Lee <[email protected]>
…n before validating spans (opensearch-project#9561) * Allow MockTracingTelemetry to await for asynchronous tasks termination before validating spans Signed-off-by: Andriy Redko <[email protected]> * Address code review comments Signed-off-by: Andriy Redko <[email protected]> --------- Signed-off-by: Andriy Redko <[email protected]>
…pensearch-project#9426) Benchmarks show that ArrayList performs better than TreeSet. Added a comment on where to find the results and benchmark. Signed-off-by: Kiran Reddy <[email protected]>
…ensearch-project#9622) Signed-off-by: Jay Deng <[email protected]>
…ch-project#9536) * pick oldest OS version replica to promote as primary Signed-off-by: Poojita Raj <[email protected]> * add test Signed-off-by: Poojita Raj <[email protected]> * refactor Signed-off-by: Poojita Raj <[email protected]> * refactor to avoid coupling Signed-off-by: Poojita Raj <[email protected]> * add comments Signed-off-by: Poojita Raj <[email protected]> --------- Signed-off-by: Poojita Raj <[email protected]>
…ers (opensearch-project#9546) --------- Signed-off-by: Bhumika Saini <[email protected]>
…h-project#9507) * Decouple replication lag from replication timer logic used to fail stale replicas Signed-off-by: Ankit Kala <[email protected]> * Added changelog entry Signed-off-by: Ankit Kala <[email protected]> * Addressed comments Signed-off-by: Ankit Kala <[email protected]> * Addressed comments 2 Signed-off-by: Ankit Kala <[email protected]> * Addressed comments Signed-off-by: Ankit Kala <[email protected]> * Retry gradle Signed-off-by: Ankit Kala <[email protected]> * fix UT Signed-off-by: Ankit Kala <[email protected]> * Addressed comments Signed-off-by: Ankit Kala <[email protected]> * Retry Gradle Signed-off-by: Ankit Kala <[email protected]> --------- Signed-off-by: Ankit Kala <[email protected]>
opensearch-project#9160) * Upload all index metadata to remote store using BlobStoreRepository interface Signed-off-by: Sooraj Sinha <[email protected]>
…Key Provider Plugin to provide keys (opensearch-project#8466) Signed-off-by: Vikas Bansal <[email protected]> Signed-off-by: Gaurav Bafna <[email protected]>
…de for on demand (opensearch-project#9522) Signed-off-by: Dev Agarwal <[email protected]>
* Expose DelimitedTermFrequencyTokenFilter Relates: opensearch-project#9413 This commit exposes Lucene's delimited term frequency token filter to be able to provide term frequencies along with terms. Signed-off-by: Russ Cam <[email protected]> * fix format violations Signed-off-by: Russ Cam <[email protected]> * fix test and add to changelog Signed-off-by: Russ Cam <[email protected]> * Address PR feedback - Add unit tests for DelimitedTermFrequencyTokenFilterFactory - Remove IllegalArgumentException as caught exception - Add skip to yaml rest tests to skip for version < 2.10 Signed-off-by: Russ Cam <[email protected]> * formatting Signed-off-by: Russ Cam <[email protected]> * Rename filter Signed-off-by: Russ Cam <[email protected]> * update naming in REST tests Signed-off-by: Russ Cam <[email protected]> --------- Signed-off-by: Russ Cam <[email protected]>
…search-project#9592) Signed-off-by: Kunal Kotwani <[email protected]>
…ensearch-project#9584) Signed-off-by: Ashish Singh <[email protected]>
…project#9615) Signed-off-by: panguixin <[email protected]>
…t referenced by reader with matching checksum (opensearch-project#9630) * [Segment Replication] Allow segment replication with on disk files not referenced by reader with matching checksum Signed-off-by: Suraj Singh <[email protected]> * Address review comments Signed-off-by: Suraj Singh <[email protected]> * Address review comments Signed-off-by: Suraj Singh <[email protected]> * Spotless fix Signed-off-by: Suraj Singh <[email protected]> --------- Signed-off-by: Suraj Singh <[email protected]>
…ation (opensearch-project#9617) * Add SpanBuilder support Signed-off-by: Gagan Juneja <[email protected]> * Refactor code Signed-off-by: Gagan Juneja <[email protected]> * Redefine telemetry context restoration Signed-off-by: Gagan Juneja <[email protected]> * Update changelog Signed-off-by: Gagan Juneja <[email protected]> * Stores the SpanScope in ThreadLocal Signed-off-by: Gagan Juneja <[email protected]> * Revert the context name changes Signed-off-by: Gagan Juneja <[email protected]> * Change the span::endSpan and SpanScope::close behaviour Signed-off-by: Gagan Juneja <[email protected]> * Supressed warnings Signed-off-by: Gagan Juneja <[email protected]> * Add more test cases Signed-off-by: Gagan Juneja <[email protected]> * Address review comment Signed-off-by: Gagan Juneja <[email protected]> * Address review comment Signed-off-by: Gagan Juneja <[email protected]> * Fix java doc Signed-off-by: Gagan Juneja <[email protected]> * Address review comment Signed-off-by: Gagan Juneja <[email protected]> * Fix failing test Signed-off-by: Gagan Juneja <[email protected]> * Empty-Commit Signed-off-by: Gagan Juneja <[email protected]> * Empty-Commit Signed-off-by: Gagan Juneja <[email protected]> --------- Signed-off-by: Gagan Juneja <[email protected]> Signed-off-by: Gagan Juneja <[email protected]> Co-authored-by: Gagan Juneja <[email protected]>
…#9686) * Fix Segment Replication stats bytes behind metric. This metric currently gives an estimate of the bytes behind based on the difference in size of the segments referenced by the active readers between shards. This does not give a good indication of the amount of bytes that need to be fetched and is inaccurate after deletes and merges. Fixed by sending file metadata with each checkpoint and computing a diff between checkpoints when SegmentReplicationShardStats is built. Signed-off-by: Marc Handalian <[email protected]> * Skip SegRep bwc test until this is backported to 2.x. Signed-off-by: Marc Handalian <[email protected]> * Add changelog entry. Signed-off-by: Marc Handalian <[email protected]> --------- Signed-off-by: Marc Handalian <[email protected]>
Signed-off-by: Sooraj Sinha <[email protected]>
Signed-off-by: Sooraj Sinha <[email protected]>
Signed-off-by: Sooraj Sinha <[email protected]>
Signed-off-by: Sooraj Sinha <[email protected]>
Signed-off-by: bansvaru <[email protected]>
linuxpi
force-pushed
the
remote-state-publish-multipart
branch
from
1dc6bc0 to
823fab4
Compare
Signed-off-by: bansvaru <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
[Describe what this change achieves]
Related Issues
Resolves #[Issue number to be closed when this PR is merged]
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.