IntoIterator and IndexedParallelIterator for Zip?

[adamreichold]

I have a problem where I was unable to apply the methods provided by Zip and therefore decay that into a Parallellterator to be able to use arbitrary fold and reduce operations. Due to the large size of the temporary arrays, I would like to use Rayon's with_min_len to avoid the Zip being split too often (and thereby allocating too many temporary arrays). However, this method is only available on IndexedParallelIterator instances which Zip is not. This seems to be related to Zip also not implementing IntoIterator in contrast to other NdProducers.

Is there a fundamental issue which prevents providing those traits for Zip (After all, it does implement SplitAt and provides a size method.) or is this just not yet implemented?

[bluss]

I think it's a fundamental issue. The NdProducers that are more than one-dimensional don't implement IntoIterator or indexing, since they don't have an efficient correspondance with a one dimensional sequence.

[adamreichold]

But doesn't the for_each-based design imply sequence-like access already? More concretely, wouldn't an iterator implementation basically unroll the loop in

ndarray/src/zip/mod.rs

Line 348 in 566177b

unsafe fn inner<F, Acc>(&self, mut acc: Acc, ptr: P::Ptr, strides: P::Stride,

? Similarly, doesn't

ndarray/src/zip/mod.rs

Line 892 in 566177b

fn split_at(self, axis: Axis, index: usize) -> (Self, Self) {

already split a Zip at a given index if the axis is fixed to max_stride_axis is it is already done for the UnindexedProducer?

[bluss]

this post talks about the conceptual problem with .next()-based iterators https://medium.com/@veedrac/rust-is-slow-and-i-am-the-cure-32facc0fdcb At the moment I won't go into all the details

[adamreichold]

Please note that I am not advocated for making this the primary interface for Zip.

In my particular case I have also already crossed that bridge as I have to use into_par_iter and thereby already downgraded to sequence-like access, even if in parallel. Providing an IntoIterator implementation for Zip would only be a means to enable that to be an IndexedParallelIterator (which should be possible due to having size and split_at methods). Also the optimization of providing an implementation of Producer::fold_with in terms of Zip::fold_while should be unaffected by that as far as I can see.

Or put differently, why provide into_par_iter but not into_iter?

(If providing an IntoIterator implementation for Zip is considered a performance foot gun, the above does not actually require it AFAIU and Paralell<Zip<...>>: IntoInterator should suffice.)

Thinking back to my original goal of being able to use IndexedParallelIterator::with_min_len on the parallel iterator backing a Zip, having the option of controlling SplitPreference::can_split for that iterator would suffice as well, e.g.

Zip::from(...).and(...).into_par_iter_wtih_min_len(42)

but I would also say that this feels a bit ad-hoc.

[adamreichold]

When trying to actually implement this, i.e. introduce an internal ZipIter<P, D>: Iterator + ExactSizeIterator + DoubleEndedIterator and use that to implement Parallel<Zip<P, D>>: IndexedParallelIterator, I find that the main complication seems to be that to implement

fn split_at(self, i: usize) -> (Self, Self);

one has to determine the axis which when used to split along, would allow splitting at exactly i elements. Furthermore, this and having to implement DoubleEndedIterator imply that ZipIter has to track both index and size. Finally, the fold_with implementation needs to respect those two fields as well.

[adamreichold]

Due to the complexity of implementing the above, I opened #1081 instead which does solve the immediate problem of limiting the splitting performed for (unindexed) parallel iterators on ArrayView and Zip.