Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[opt](inverted index) add performance profiling for remote io access in inverted index #43542

Merged
merged 1 commit into from
Nov 17, 2024
Merged

[opt](inverted index) add performance profiling for remote io access in inverted index #43542

merged 1 commit into from
Nov 17, 2024

Conversation

zzzxl1993
Copy link
Contributor

@zzzxl1993 zzzxl1993 commented Nov 10, 2024 •
edited by qidaye
Loading

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

  1. Implement the accounting of remote I/O operations for the inverted index in profiling statistics.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

github-actions[bot]
github-actions bot reviewed Nov 10, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/olap/rowset/segment_v2/inverted_index_reader.cpp
Status FullTextIndexReader::query(OlapReaderStatistics* stats, RuntimeState* runtime_state,
const std::string& column_name, const void* query_value,
InvertedIndexQueryType query_type,
Status FullTextIndexReader::query(const io::IOContext* io_ctx, OlapReaderStatistics* stats,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function 'query' exceeds recommended size/complexity thresholds [readability-function-size]

Status FullTextIndexReader::query(const io::IOContext* io_ctx, OlapReaderStatistics* stats,
                            ^
Additional context

be/src/olap/rowset/segment_v2/inverted_index_reader.cpp:250: 81 lines including whitespace and comments (threshold 80)

Status FullTextIndexReader::query(const io::IOContext* io_ctx, OlapReaderStatistics* stats,
                            ^

be/src/olap/rowset/segment_v2/inverted_index_reader.cpp
return Status::OK();
}

Status StringTypeInvertedIndexReader::query(OlapReaderStatistics* stats,
Status StringTypeInvertedIndexReader::query(const io::IOContext* io_ctx,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function 'query' exceeds recommended size/complexity thresholds [readability-function-size]

Status StringTypeInvertedIndexReader::query(const io::IOContext* io_ctx,
                                      ^
Additional context

be/src/olap/rowset/segment_v2/inverted_index_reader.cpp:348: 117 lines including whitespace and comments (threshold 80)

Status StringTypeInvertedIndexReader::query(const io::IOContext* io_ctx,
                                      ^

@zzzxl1993 zzzxl1993 force-pushed the 202411052006 branch 2 times, most recently from 028c8fd to 6e99cbb Compare November 11, 2024 07:00
@zzzxl1993
Copy link
Contributor Author

run buildall

1 similar comment
@zzzxl1993
Copy link
Contributor Author

run buildall

@zzzxl1993 zzzxl1993 force-pushed the 202411052006 branch 2 times, most recently from 025f6e7 to c0b7127 Compare November 12, 2024 11:14
@zzzxl1993
Copy link
Contributor Author

run buildall

@zzzxl1993
Copy link
Contributor Author

run buildall

@zzzxl1993
Copy link
Contributor Author

run buildall

@zzzxl1993
Copy link
Contributor Author

run buildall

@zzzxl1993
Copy link
Contributor Author

run buildall

airborne12
airborne12 previously approved these changes Nov 13, 2024
View reviewed changes
Copy link
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 13, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Nov 13, 2024
@zzzxl1993
Copy link
Contributor Author

run buildall

@zzzxl1993
Copy link
Contributor Author

run buildall

github-actions[bot]
github-actions bot reviewed Nov 14, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/olap/rowset/segment_v2/inverted_index_file_reader.cpp Outdated
@@ -42,7 +41,8 @@ Status InvertedIndexFileReader::init(int32_t read_buffer_size, bool open_idx_fil
return Status::OK();
}

Status InvertedIndexFileReader::_init_from_v2(int32_t read_buffer_size) {
Status InvertedIndexFileReader::_init_from_v2(const io::IOContext* io_ctx,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function '_init_from_v2' exceeds recommended size/complexity thresholds [readability-function-size]

Status InvertedIndexFileReader::_init_from_v2(const io::IOContext* io_ctx,
                                ^
Additional context

be/src/olap/rowset/segment_v2/inverted_index_file_reader.cpp:43: 94 lines including whitespace and comments (threshold 80)

Status InvertedIndexFileReader::_init_from_v2(const io::IOContext* io_ctx,
                                ^

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.95% (9891/26065)
Line Coverage: 29.14% (82650/283622)
Region Coverage: 28.27% (42481/150261)
Branch Coverage: 24.85% (21547/86692)
Coverage Report: http://coverage.selectdb-in.cc/coverage/521ef057683eb938723c89068dd5896e0eab752f_521ef057683eb938723c89068dd5896e0eab752f/report/index.html

@zzzxl1993 zzzxl1993 closed this Nov 14, 2024
@zzzxl1993 zzzxl1993 reopened this Nov 14, 2024
@zzzxl1993
Copy link
Contributor Author

run buildall

1 similar comment
@zzzxl1993
Copy link
Contributor Author

run buildall

@zzzxl1993
Copy link
Contributor Author

run p0

github-actions[bot]
github-actions bot reviewed Nov 15, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/olap/rowset/segment_v2/inverted_index_reader.cpp
Status FullTextIndexReader::query(OlapReaderStatistics* stats, RuntimeState* runtime_state,
const std::string& column_name, const void* query_value,
InvertedIndexQueryType query_type,
Status FullTextIndexReader::query(const io::IOContext* io_ctx, OlapReaderStatistics* stats,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function 'query' exceeds recommended size/complexity thresholds [readability-function-size]

Status FullTextIndexReader::query(const io::IOContext* io_ctx, OlapReaderStatistics* stats,
                            ^
Additional context

be/src/olap/rowset/segment_v2/inverted_index_reader.cpp:248: 81 lines including whitespace and comments (threshold 80)

Status FullTextIndexReader::query(const io::IOContext* io_ctx, OlapReaderStatistics* stats,
                            ^

be/src/olap/rowset/segment_v2/inverted_index_reader.cpp
return Status::OK();
}

Status StringTypeInvertedIndexReader::query(OlapReaderStatistics* stats,
Status StringTypeInvertedIndexReader::query(const io::IOContext* io_ctx,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function 'query' exceeds recommended size/complexity thresholds [readability-function-size]

Status StringTypeInvertedIndexReader::query(const io::IOContext* io_ctx,
                                      ^
Additional context

be/src/olap/rowset/segment_v2/inverted_index_reader.cpp:346: 117 lines including whitespace and comments (threshold 80)

Status StringTypeInvertedIndexReader::query(const io::IOContext* io_ctx,
                                      ^

@zzzxl1993
Copy link
Contributor Author

run buildall

@zzzxl1993
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.97% (9899/26074)
Line Coverage: 29.17% (82747/283683)
Region Coverage: 28.30% (42486/150132)
Branch Coverage: 24.87% (21542/86606)
Coverage Report: http://coverage.selectdb-in.cc/coverage/636764ebfe6c4361fb5e19ba4bb7fb0fa90fd2cd_636764ebfe6c4361fb5e19ba4bb7fb0fa90fd2cd/report/index.html

csun5285
csun5285 approved these changes Nov 17, 2024
View reviewed changes
Copy link
Contributor

@csun5285 csun5285 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

qidaye
qidaye approved these changes Nov 17, 2024
View reviewed changes
Copy link
Contributor

@qidaye qidaye left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 17, 2024
@qidaye qidaye merged commit 74238ae into apache:master Nov 17, 2024
27 of 29 checks passed
github-actions bot pushed a commit that referenced this pull request Nov 17, 2024
@zzzxl1993
[opt](inverted index) add performance profiling for remote io access …
0d65a2b 
…in inverted index (#43542)

Problem Summary:

1. Implement the accounting of remote I/O operations for the inverted
index in profiling statistics.

### Release note

Add performance profiling for remote IO access in inverted index
@github-actions github-actions bot mentioned this pull request Nov 17, 2024
Merged
airborne12 pushed a commit that referenced this pull request Nov 18, 2024
@github-actions @zzzxl1993
branch-3.0: [opt](inverted index) add performance profiling for remot…
fdbd9ca 
…e io access in inverted index #43542 (#44093)

Cherry-picked from #43542

Co-authored-by: zzzxl <[email protected]>
@zzzxl1993 zzzxl1993 mentioned this pull request Dec 3, 2024
Merged
16 tasks
gavinchou pushed a commit that referenced this pull request Dec 13, 2024
@zzzxl1993
[opt](inverted index) Add NumInvertedIndexRemoteIOTotal statistics in…
559a2f9 
… profile (#44863)

Add NumInvertedIndexRemoteIOTotal to count the number of remote IO
operations in the inverted index
Related PR: #43542
zzzxl1993 added a commit to zzzxl1993/doris that referenced this pull request Dec 19, 2024
@zzzxl1993
[opt](inverted index) Add NumInvertedIndexRemoteIOTotal statistics in…
6f2f8e0 
… profile (apache#44863)

Add NumInvertedIndexRemoteIOTotal to count the number of remote IO
operations in the inverted index
Related PR: apache#43542
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

@csun5285 csun5285 csun5285 approved these changes

@qidaye qidaye qidaye approved these changes

@airborne12 airborne12 airborne12 left review comments

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.3-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@zzzxl1993 @doris-robot @airborne12 @qidaye @csun5285