Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

deprecated job-metadata-index #2339

Merged

Conversation

penghuo
Copy link
Collaborator

@penghuo penghuo commented Oct 21, 2023

Description

  1. deprecated job-metadata index. reuse .query_execution_request index. Using type:jobmeta to different with sessionDoc, and statementDoc.
  2. Store AsyncQueryId to JobMetadata. AsyncQueryId is used async_query id.
  3. Extract fetchResponse and cancel logic from SparkDispatcher.

Issues Resolved

n/a

Check List

  • New functionality includes testing.
    • All tests pass, including unit test, integration test and doctest
  • New functionality has been documented.
    • New functionality has javadoc added
    • New functionality has user manual doc added
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@codecov
Copy link

codecov bot commented Oct 21, 2023

Codecov Report

Merging #2339 (b0dac26) into main (7b4156e) will decrease coverage by 0.01%.
The diff coverage is 98.43%.

@@             Coverage Diff              @@
##               main    #2339      +/-   ##
============================================
- Coverage     96.49%   96.49%   -0.01%     
- Complexity     4884     4891       +7     
============================================
  Files           458      463       +5     
  Lines         13398    13365      -33     
  Branches        909      901       -8     
============================================
- Hits          12929    12896      -33     
  Misses          452      452              
  Partials         17       17              
Flag Coverage Δ
sql-engine 96.49% <98.43%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
...park/asyncquery/AsyncQueryExecutorServiceImpl.java 100.00% <100.00%> (ø)
...OpensearchAsyncQueryJobMetadataStorageService.java 100.00% <100.00%> (ø)
...search/sql/spark/dispatcher/AsyncQueryHandler.java 100.00% <100.00%> (ø)
...search/sql/spark/dispatcher/BatchQueryHandler.java 100.00% <100.00%> (ø)
.../sql/spark/dispatcher/InteractiveQueryHandler.java 100.00% <100.00%> (ø)
...rch/sql/spark/dispatcher/SparkQueryDispatcher.java 100.00% <100.00%> (ø)
...ql/spark/execution/session/InteractiveSession.java 100.00% <100.00%> (ø)
...nsearch/sql/spark/execution/session/SessionId.java 100.00% <ø> (ø)
...rch/sql/spark/execution/statement/StatementId.java 100.00% <100.00%> (ø)
...rch/sql/spark/execution/statestore/StateStore.java 79.81% <100.00%> (+3.18%) ⬆️
... and 3 more

@penghuo penghuo force-pushed the enhance/removeMetaJobIndices-v1 branch from 4604869 to 0d2b36f Compare October 21, 2023 19:54
Signed-off-by: Peng Huo <[email protected]>
dai-chen
dai-chen previously approved these changes Oct 23, 2023
vmmusings
vmmusings previously approved these changes Oct 23, 2023
Signed-off-by: Peng Huo <[email protected]>
@penghuo penghuo dismissed stale reviews from vmmusings and dai-chen via be2c70b October 23, 2023 20:20
Signed-off-by: Peng Huo <[email protected]>
@penghuo penghuo merged commit b30d3c9 into opensearch-project:main Oct 23, 2023
22 of 23 checks passed
opensearch-trigger-bot bot pushed a commit that referenced this pull request Oct 23, 2023
* deprecate job-metadata-index

Signed-off-by: Peng Huo <[email protected]>

* upgrade log4j2

Signed-off-by: Peng Huo <[email protected]>

* update codestyle

Signed-off-by: Peng Huo <[email protected]>

* upgrade log4j

Signed-off-by: Peng Huo <[email protected]>

---------

Signed-off-by: Peng Huo <[email protected]>
(cherry picked from commit b30d3c9)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
opensearch-trigger-bot bot pushed a commit that referenced this pull request Oct 23, 2023
* deprecate job-metadata-index

Signed-off-by: Peng Huo <[email protected]>

* upgrade log4j2

Signed-off-by: Peng Huo <[email protected]>

* update codestyle

Signed-off-by: Peng Huo <[email protected]>

* upgrade log4j

Signed-off-by: Peng Huo <[email protected]>

---------

Signed-off-by: Peng Huo <[email protected]>
(cherry picked from commit b30d3c9)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
penghuo added a commit that referenced this pull request Oct 23, 2023
* deprecated job-metadata-index (#2339)

* deprecate job-metadata-index

Signed-off-by: Peng Huo <[email protected]>

* upgrade log4j2

Signed-off-by: Peng Huo <[email protected]>

* update codestyle

Signed-off-by: Peng Huo <[email protected]>

* upgrade log4j

Signed-off-by: Peng Huo <[email protected]>

---------

Signed-off-by: Peng Huo <[email protected]>
(cherry picked from commit b30d3c9)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* revert log4j upgrade

Signed-off-by: Peng Huo <[email protected]>

---------

Signed-off-by: Peng Huo <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Peng Huo <[email protected]>
opensearch-trigger-bot bot pushed a commit that referenced this pull request Oct 23, 2023
* deprecated job-metadata-index (#2339)

* deprecate job-metadata-index

Signed-off-by: Peng Huo <[email protected]>

* upgrade log4j2

Signed-off-by: Peng Huo <[email protected]>

* update codestyle

Signed-off-by: Peng Huo <[email protected]>

* upgrade log4j

Signed-off-by: Peng Huo <[email protected]>

---------

Signed-off-by: Peng Huo <[email protected]>
(cherry picked from commit b30d3c9)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* revert log4j upgrade

Signed-off-by: Peng Huo <[email protected]>

---------

Signed-off-by: Peng Huo <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Peng Huo <[email protected]>
(cherry picked from commit f3fdead)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
penghuo added a commit that referenced this pull request Oct 23, 2023
* deprecated job-metadata-index (#2339)

* deprecate job-metadata-index



* upgrade log4j2



* update codestyle



* upgrade log4j



---------


(cherry picked from commit b30d3c9)


* revert log4j upgrade



---------





(cherry picked from commit f3fdead)

Signed-off-by: Peng Huo <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Peng Huo <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants