Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Tiramisu Splitting Up Work] Tiramisu Keystore #11710

Closed
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
73a4d1a
[Tiered Caching] Enabling serialization for IndicesRequestCache key o…
sgup432 Sep 29, 2023
db3d61d
Fixing javadoc issue
sgup432 Sep 29, 2023
29c5375
Tiered caching framework draft changes
sgup432 Oct 9, 2023
502938f
fixed double counting of on heap hit count
Oct 11, 2023
f57e2b0
[Tiered caching] Framework changes
sgup432 Oct 19, 2023
dfef981
Addressing comments
sgup432 Oct 26, 2023
dc6cf5b
Adding changelog
sgup432 Oct 26, 2023
9134543
Added javadoc for new files/packages
sgup432 Oct 26, 2023
50fc167
Added changelog
sgup432 Oct 26, 2023
be1fe01
Fixing javadoc warnings
sgup432 Oct 26, 2023
76c859a
Ehcache disk tier integration
sgup432 Oct 31, 2023
1d8d21f
Adds Serializer interface for use in ehcache disk tier and elsewhere;…
Nov 1, 2023
9d0cfea
addressed sagar's comments
Nov 13, 2023
498b5cb
Fixed broken test
Nov 13, 2023
a65c649
Fixed ehcache init issue in test cases
Nov 14, 2023
caa125e
cleanup, fixed commented line
Nov 14, 2023
609e175
Addressed Sagar's other comments
Nov 22, 2023
758fc3b
removed more dead code
Nov 22, 2023
f37daf3
Added notification for REMOVED case in ehcache event listener
Dec 1, 2023
ff1e90c
Fixed bug with how disk tier memory was tracked
Dec 21, 2023
e33f43f
Cleanup from separation cherry pick
Dec 28, 2023
1feb988
Merge branch 'tiramisu-framework-before-recent-changes' into tiramisu…
Dec 29, 2023
5f4ca9d
Cleanup commit part 1
Jan 2, 2024
0faf761
Removed stats updates which should be in stats PR
Jan 2, 2024
8831632
Merge branch 'tiramisu-framework-before-recent-changes' into tiramisu…
Jan 2, 2024
ad25390
cleanup commit
Jan 2, 2024
8f615fb
Removed serializer implementations which should be part of their own PR
Jan 2, 2024
9ef1254
Implements a memory-efficient roaring bitmap-based keystore for use i…
Sep 22, 2023
0085306
Addressed Sagar's comments besides new counter/arraylist setup for re…
Nov 2, 2023
cf8a806
Implemented/tested counter+removal list setup to allow more removals
Nov 6, 2023
35a22c2
Integrated keystore
Nov 6, 2023
f5a0eb8
Simplified RBM size estimator
Nov 11, 2023
835e6c2
Optimized modulo calculation with bitmask
Nov 22, 2023
3c37126
Changed memory size estimator to reflect optimized modulo
Nov 27, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -64,4 +64,5 @@ testfixtures_shared/
.ci/jobs/

# build files generated
doc-tools/missing-doclet/bin/
doc-tools/missing-doclet/bin/
server/src/main/java/org/opensearch/indices/KLSPerformanceTest.java
19 changes: 9 additions & 10 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,15 +82,14 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),

## [Unreleased 2.x]
### Added
- Add coordinator level stats for search latency ([#8386](https://github.com/opensearch-project/OpenSearch/issues/8386))
- Add metrics for thread_pool task wait time ([#9681](https://github.com/opensearch-project/OpenSearch/pull/9681))
- Async blob read support for S3 plugin ([#9694](https://github.com/opensearch-project/OpenSearch/pull/9694))
- [Telemetry-Otel] Added support for OtlpGrpcSpanExporter exporter ([#9666](https://github.com/opensearch-project/OpenSearch/pull/9666))
- Async blob read support for encrypted containers ([#10131](https://github.com/opensearch-project/OpenSearch/pull/10131))
- Add capability to restrict async durability mode for remote indexes ([#10189](https://github.com/opensearch-project/OpenSearch/pull/10189))
- Add Doc Status Counter for Indexing Engine ([#4562](https://github.com/opensearch-project/OpenSearch/issues/4562))
- Add unreferenced file cleanup count to merge stats ([#10204](https://github.com/opensearch-project/OpenSearch/pull/10204))
- [Remote Store] Add support to restrict creation & deletion if system repository and mutation of immutable settings of system repository ([#9839](https://github.com/opensearch-project/OpenSearch/pull/9839))
- Per request phase latency ([#10351](https://github.com/opensearch-project/OpenSearch/issues/10351))
- [Remote Store] Add repository stats for remote store([#10567](https://github.com/opensearch-project/OpenSearch/pull/10567))
- Add search query categorizer ([#10255](https://github.com/opensearch-project/OpenSearch/pull/10255))
- Introduce ConcurrentQueryProfiler to profile query using concurrent segment search path and support concurrency during rewrite and create weight ([10352](https://github.com/opensearch-project/OpenSearch/pull/10352))
- [Remote cluster state] Make index and global metadata upload timeout dynamic cluster settings ([#10814](https://github.com/opensearch-project/OpenSearch/pull/10814))
- Added cluster setting cluster.restrict.index.replication_type to restrict setting of index setting replication type ([#10866](https://github.com/opensearch-project/OpenSearch/pull/10866))
- Add cluster state stats ([#10670](https://github.com/opensearch-project/OpenSearch/pull/10670))
- [Tiered caching] Framework changes ([#10753](https://github.com/opensearch-project/OpenSearch/pull/10753)

### Dependencies
- Bump `peter-evans/create-or-update-comment` from 2 to 3 ([#9575](https://github.com/opensearch-project/OpenSearch/pull/9575))
Expand Down Expand Up @@ -143,4 +142,4 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
### Security

[Unreleased 3.0]: https://github.com/opensearch-project/OpenSearch/compare/2.x...HEAD
[Unreleased 2.x]: https://github.com/opensearch-project/OpenSearch/compare/2.11...2.x
[Unreleased 2.x]: https://github.com/opensearch-project/OpenSearch/compare/2.11...2.x
5 changes: 4 additions & 1 deletion buildSrc/version.properties
Original file line number Diff line number Diff line change
Expand Up @@ -68,4 +68,7 @@ jzlib = 1.1.3
resteasy = 6.2.4.Final

# opentelemetry dependencies
opentelemetry = 1.30.1
opentelemetry = 1.31.0
opentelemetrysemconv = 1.21.0-alpha

ehcache = 3.10.8
1 change: 0 additions & 1 deletion plugins/discovery-ec2/licenses/slf4j-api-1.7.36.jar.sha1

This file was deleted.

21 changes: 0 additions & 21 deletions plugins/discovery-ec2/licenses/slf4j-api-LICENSE.txt

This file was deleted.

This file was deleted.

21 changes: 0 additions & 21 deletions plugins/identity-shiro/licenses/slf4j-api-LICENSE.txt

This file was deleted.

This file was deleted.

21 changes: 0 additions & 21 deletions plugins/ingest-attachment/licenses/slf4j-api-LICENSE.txt

This file was deleted.

Empty file.

This file was deleted.

21 changes: 0 additions & 21 deletions plugins/repository-azure/licenses/slf4j-api-LICENSE.txt

This file was deleted.

Empty file.

This file was deleted.

21 changes: 0 additions & 21 deletions plugins/repository-hdfs/licenses/slf4j-api-LICENSE.txt

This file was deleted.

Empty file.
1 change: 0 additions & 1 deletion plugins/repository-s3/licenses/slf4j-api-1.7.36.jar.sha1

This file was deleted.

21 changes: 0 additions & 21 deletions plugins/repository-s3/licenses/slf4j-api-LICENSE.txt

This file was deleted.

Empty file.
7 changes: 7 additions & 0 deletions server/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -158,6 +158,13 @@ dependencies {
api "com.google.protobuf:protobuf-java:${versions.protobuf}"
api "jakarta.annotation:jakarta.annotation-api:${versions.jakarta_annotation}"

api "org.ehcache:ehcache:${versions.ehcache}"
api "org.slf4j:slf4j-api:${versions.slf4j}"

// roaring bitmaps
api 'org.roaringbitmap:RoaringBitmap:0.9.49'
runtimeOnly 'org.roaringbitmap:shims:0.9.49'

testImplementation(project(":test:framework")) {
// tests use the locally compiled version of server
exclude group: 'org.opensearch', module: 'server'
Expand Down
1 change: 1 addition & 0 deletions server/licenses/RoaringBitmap-0.9.49.jar.sha1
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
b45b49c1ec5c5fc48580412d0ca635e1833110ea
Loading
Loading