Host compression #17656
Host compression #17656
Conversation
Co-authored-by: Bradley Dice <[email protected]>
…into comp-headers-cleanup
…high-lvl-comp-api
…high-lvl-comp-api
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
vuule
added
feature request
…high-lvl-comp-api
…o high-lvl-comp-api
…o high-lvl-comp-api
| @@ -342,7 +342,7 @@ class writer::impl { | |||
| // Writer options. | |||
| stripe_size_limits const _max_stripe_size; | |||
| size_type const _row_index_stride; | |||
| CompressionKind const _compression_kind; | |||
| compression_type const _compression; | |||
There was a problem hiding this comment.
keeping this as cudf's compression type leads to much fewer conversions
| void set_compression(compression_type comp) | ||
| { | ||
| _compression = comp; | ||
| if (comp == compression_type::AUTO) { _compression = compression_type::SNAPPY; } |
There was a problem hiding this comment.
For my education, Is it common sense that AUTO should be just SNAPPY?
There was a problem hiding this comment.
compression_type is common for all file formats, so AUTO may mean different compression types for different formats.
I guess we could remove AUTO and set a concrete compression type as the default for each format.
cpp/src/io/comp/comp.cpp
Outdated
| stream.synchronize(); | ||
|
|
||
| std::vector<std::future<size_t>> tasks; | ||
| auto const streams = cudf::detail::fork_streams(stream, h_comp_pool().get_thread_count()); |
There was a problem hiding this comment.
Nit: should we cap the stream pool size to 32 here to avoid the warning in get_streams? (
cudf/cpp/src/utilities/stream_pool.cpp
Line 131 in dc2a75c
There was a problem hiding this comment.
I forgot about that warning! Yup, should be capped at stream pool size.
There was a problem hiding this comment.
We can consider moving the STREAM_POOL_SIZE variable to the header stream_pool.hpp and then invoke fork_streams as
| auto const streams = cudf::detail::fork_streams(stream, h_comp_pool().get_thread_count()); | |
| auto const streams = cudf::detail::fork_streams(stream, std::min(STREAM_POOL_SIZE, h_comp_pool().get_thread_count())); |
There was a problem hiding this comment.
I'm trying to use cudf::detail::global_cuda_stream_pool().get_stream_pool_size(), that should work without changes to the pool
There was a problem hiding this comment.
made the change, thanks for the suggestion!
also added an env var, since this reminded me that the default thread count could be bad.
There was a problem hiding this comment.
Ooh, good idea about the thread count env var, I'll use it in the JSON PR #17708 as well!
Description
Add compression APIs to make the nvCOMP use transparent.
Remove direct dependency on nvCOMP in the ORC and Parquet writers.
Add multi-threaded host-side compression; currently off by default, can only be enabled via
LIBCUDF_USE_HOST_COMPRESSIONenvironment variable.Currently the host compression adds D2H + H2D transfers. Avoiding the H2D transfer requires large changes to the writers.
Also moved handling of the AUTO compression type to the options classes, which should own such defaults (translate AUTO to SNAPPY in this case).
Checklist