[GLUTEN-7953][VL] Fetch and dump all inputs for micro benchmark on middle stage begin #7998
Conversation
|
does this PR only works for middle stage? or dump the data comes from shuffle? |
marin-ma
changed the title
|
@Yohahaha Yes. It only works for middle stage. Updated PR titile. |
zhztheplayer
changed the title
@zhztheplayer Currently, the input files are being dumped during the middle stage of execution. This PR will fetch all the data and save it into file before the pipeline starts. The stage input will then be read from this dumped file. This will help to save a complete input file even if the task fails. In this way, we can have the full input data of the failed task, and reproduce the failure with microbenchmark. |
cpp/core/jni/JniCommon.cc
Outdated
| auto rb = gluten::arrowGetOrThrow(arrow::ImportRecordBatch(array.get(), schema.get())); | ||
| GLUTEN_THROW_NOT_OK(writer_->initWriter(*(rb->schema().get()))); | ||
| GLUTEN_THROW_NOT_OK(writer_->writeInBatches(rb)); | ||
| } while (env->CallBooleanMethod(jColumnarBatchItr_, serializedColumnarBatchIteratorHasNext_)); |
There was a problem hiding this comment.
checkEnv after CallBooleanMethod
There was a problem hiding this comment.
We get all the batch, so the batch Iterator cannot be reused, will it affect the following stage
There was a problem hiding this comment.
@jinchengchenghh In what cases would the batch iterator be reused? Could you elaborate more on this? Thanks!
| } | ||
| return reader_->next(); | ||
| } | ||
| } // namespace gluten |
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
Collect all input data and save it into a Parquet file. Then, read the data from the Parquet file to feed it into the pipeline.
Update
spark.gluten.sql.benchmark_task.partitionIdandspark.gluten.sql.benchmark_task.taskIdto accept a comma-separated string of multiple partition ids/task idsCurrently, the input files are being dumped during the middle stage of execution. This PR will fetch all the data and save it into file before the pipeline starts. The stage input will then be read from this dumped file. This will help to save a complete input file even if the task fails. In this way, we can have the full input data of the failed task, and reproduce the failure with microbenchmark.