feat: use istziio_client trait for benchmarking
#53
Conversation
xx01cyx
changed the title
istziio_client trait for benchmarking
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #53 +/- ##
==========================================
- Coverage 87.36% 87.16% -0.20%
==========================================
Files 18 18
Lines 2263 2259 -4
Branches 2263 2259 -4
==========================================
- Hits 1977 1969 -8
+ Misses 172 171 -1
- Partials 114 119 +5 ☔ View full report in Codecov by Sentry. |
| @@ -1,7 +1,8 @@ | |||
| use arrow_array::Float64Array; | |||
| use arrow::array::Float64Array; | |||
There was a problem hiding this comment.
I believe in arrow 50.0.0 we should use arrow::array instead of arrow_array
| use hyper::Uri; | ||
| use lazy_static::lazy_static; | ||
| use parquet::arrow::{ParquetRecordBatchStreamBuilder, ProjectionMask}; | ||
| use parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder; |
There was a problem hiding this comment.
We have to switch to arrow 50.0.0 since istziio_client uses 50.0.0. And we also need to use arrow::record_batch::RecordBatch since istziio_client uses this one. Some changes are made accordingly including this.
| .unwrap() | ||
| .with_batch_size(BATCH_SIZE); | ||
|
|
||
| let file = File::open(file_path)?; |
| @@ -1,7 +1,7 @@ | |||
| pub mod client; | |||
|
|
|||
| use anyhow::Result; | |||
| use arrow_array::RecordBatch; | |||
| use arrow::record_batch::RecordBatch; | |||
There was a problem hiding this comment.
I think what EE ppl agreed on is the arrow_array one?
https://discord.com/channels/1198701067274297394/1200903142603239454/1201179689654419536
There was a problem hiding this comment.
🤔️ sounds reasonable but istziio_client already uses arrow::record_batch::RecordBatch. Shall we ask them to modify the interface?
There was a problem hiding this comment.
Could you talk to them about this first? We can discuss other things later tmr.
|
Please change the batch size. Other than this, LGTM. |
Co-authored-by: Kunle <[email protected]>
No description provided.