[CH] Shuffle writer connects to CH pipeline #6723
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues? https://github.com/apache/incubator-gluten/issues Then could you also rename commit message and pull request title in the following format? See also: |
|
Run Gluten Clickhouse CI |
liuneng1994
force-pushed
the
add_exechange_sink
branch
from
86b3227 to
17fd827
Compare
|
Run Gluten Clickhouse CI |
liuneng1994
force-pushed
the
add_exechange_sink
branch
from
17fd827 to
78431ef
Compare
|
Run Gluten Clickhouse CI |
liuneng1994
force-pushed
the
add_exechange_sink
branch
from
78431ef to
bf78670
Compare
|
Run Gluten Clickhouse CI |
liuneng1994
force-pushed
the
add_exechange_sink
branch
from
bf78670 to
02c3893
Compare
|
Run Gluten Clickhouse CI |
1 similar comment
|
Run Gluten Clickhouse CI |
liuneng1994
force-pushed
the
add_exechange_sink
branch
from
91df9a5 to
3eb182b
Compare
|
Run Gluten Clickhouse CI |
liuneng1994
force-pushed
the
add_exechange_sink
branch
from
3eb182b to
7f50cbd
Compare
|
Run Gluten Clickhouse CI |
liuneng1994
force-pushed
the
add_exechange_sink
branch
from
7f50cbd to
48781a3
Compare
|
Run Gluten Clickhouse CI |
liuneng1994
force-pushed
the
add_exechange_sink
branch
from
48781a3 to
749fb10
Compare
|
Run Gluten Clickhouse CI |
liuneng1994
force-pushed
the
add_exechange_sink
branch
from
749fb10 to
ee4bf11
Compare
|
Run Gluten Clickhouse CI |
liuneng1994
force-pushed
the
add_exechange_sink
branch
from
ee4bf11 to
c7cdffe
Compare
|
Run Gluten Clickhouse CI |
liuneng1994
force-pushed
the
add_exechange_sink
branch
from
c7cdffe to
0e9c9c7
Compare
|
Run Gluten Clickhouse CI |
liuneng1994
force-pushed
the
add_exechange_sink
branch
from
0e9c9c7 to
18435c7
Compare
|
Run Gluten Clickhouse CI |
liuneng1994
force-pushed
the
add_exechange_sink
branch
from
88c5b47 to
0f7982b
Compare
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
liuneng1994
force-pushed
the
add_exechange_sink
branch
from
2217b22 to
aa46eea
Compare
|
Run Gluten Clickhouse CI |
liuneng1994
force-pushed
the
add_exechange_sink
branch
from
aa46eea to
86d92e2
Compare
|
Run Gluten Clickhouse CI |
lgbo-ustc
reviewed
Sep 10, 2024
| auto * current_executor = local_engine::LocalExecutor::getCurrentExecutor(); | ||
| chassert(current_executor); | ||
| local_engine::SplitterHolder * splitter = nullptr; | ||
| // handle fallback, whole stage fallback or partial fallback |
There was a problem hiding this comment.
In which cases， current_executor will be nullptr or not? It's better to add a comment here
liuneng1994
force-pushed
the
add_exechange_sink
branch
from
86d92e2 to
e00a93c
Compare
|
Run Gluten Clickhouse CI |
liuneng1994
force-pushed
the
add_exechange_sink
branch
from
e00a93c to
da12381
Compare
|
Run Gluten Clickhouse CI |
liuneng1994
force-pushed
the
add_exechange_sink
branch
from
da12381 to
86087bf
Compare
|
Run Gluten Clickhouse CI |
|
LGTM |
lgbo-ustc
approved these changes
Sep 11, 2024
baibaichen
added a commit
to Kyligence/gluten
that referenced
this pull request
Sep 18, 2024
Fix UT build due to apache#7193
baibaichen
added a commit
to Kyligence/gluten
that referenced
this pull request
Sep 18, 2024
Fix UT build due to apache#7193
baibaichen
added a commit
that referenced
this pull request
Sep 18, 2024
* [GLUTEN-1632][CH]Daily Update Clickhouse Version (20240918) * Fix build due to ENABLE_ROCKSDB=OFF caused by #7239 * Fix UT build due to #6723 * Fix UT build due to #7193 * Fix Build due to ClickHouse/ClickHouse#69298 --------- Co-authored-by: kyligence-git <[email protected]> Co-authored-by: Chang Chen <[email protected]>
sharkdtu
pushed a commit
to sharkdtu/gluten
that referenced
this pull request
Nov 11, 2024
What changes were proposed in this pull request? shuffle writer现在可以作为Processor接入ClickHouse pipeline当中。 image 在fallback模式下，会在jni中以循环的形式完成stage的执行，主要原因是，某些情况下的fallback会有spark的whole code gen， 其中code gen生成的代码会使用TaskContext，需要保证执行线程为task线程 image 移除了CachedShuffleWriter，新的SparkExchangeSink行为与原有shuffleWriter保持一致 同时还做了一下改动： 支持native的inputFileName,InputBlockStart, InputBlockLength shuffle Wall time统计，在Processor层面统计完整的shuffle耗时 LocalExecutor移出SerilizedPlanParser DefaultHashAggregateResultStep与DefaultHashAggregateResultTransform的output header不匹配问题 How was this patch tested? unit tests (If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
sharkdtu
pushed a commit
to sharkdtu/gluten
that referenced
this pull request
Nov 11, 2024
) * [GLUTEN-1632][CH]Daily Update Clickhouse Version (20240918) * Fix build due to ENABLE_ROCKSDB=OFF caused by apache#7239 * Fix UT build due to apache#6723 * Fix UT build due to apache#7193 * Fix Build due to ClickHouse/ClickHouse#69298 --------- Co-authored-by: kyligence-git <[email protected]> Co-authored-by: Chang Chen <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
shuffle writer现在可以作为Processor接入ClickHouse pipeline当中。
在fallback模式下，会在jni中以循环的形式完成stage的执行，主要原因是，某些情况下的fallback会有spark的whole code gen，
其中code gen生成的代码会使用TaskContext，需要保证执行线程为task线程
移除了CachedShuffleWriter，新的SparkExchangeSink行为与原有shuffleWriter保持一致
同时还做了一下改动：
How was this patch tested?
unit tests
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)