Field type of BooleanArray?

[v1gnesh]

I'm trying to setup derive for a bitstring/bitfield.
Here's a minimal example where I write the bits as Utf8 instead.
The commented parts are my attempts at trying to make it work, but I'm unable to figure this out.
I've tried to put this MutableBooleanArray into a MutableFixedSizedListArray as you can see.
This is because sometimes my struct fields are like below:

pub struct Main {
    a: u8,
    b: [Flags; 2],
}

I'm getting the sense that it has to be a primitive datatype (Boolean), and that it can't be a BooleanArray.
Please help! I'm not sure how to proceed.

Example

use modular_bitfield::prelude::*;
use binrw::BinRead;
use std::io::Cursor;
use arrow2::datatypes::DataType;
use arrow2::datatypes::DataType::*;
use arrow2::array::*;
use arrow2_convert::{ArrowField, ArrowSerialize, field::ArrowField, serialize::ArrowSerialize};
use arrow2::array::Array;
use arrow2_convert::serialize::TryIntoArrow;

#[bitfield]
#[derive(Debug, BinRead, Clone, Copy, PartialEq)]
#[br(map = |x: [u8; 1]| Self::from_bytes(x.map(|b| b.reverse_bits())))]
pub struct Flags {
    b0: B1,
    b1: B1,
    b2: B1,
    b3: B1,
    b4: B1,
    b5: B1,
    b6: B1,
    b7: B1,
}

#[derive(Debug, BinRead, ArrowField, ArrowSerialize, Clone, PartialEq)]
#[br(big)]
pub struct Outside {
    a: u16,
    b: Flags,
}

impl ArrowField for Flags {
    type Type = Self;

    #[inline]
    fn data_type() -> DataType {
        // datatypes::DataType::Extension(
        //     "BitString".to_string(),
        //     Box::new(datatypes::DataType::Utf8),
        //     None,
        // )
        // FixedSizeList(Box::new(<Self as ArrowField>::field("item")), 1)
        Utf8
    }
}

impl ArrowSerialize for Flags {
    type MutableArrayType = MutableUtf8Array<i32>;
    // type MutableArrayType = MutableFixedSizeListArray<MutableBooleanArray>;

    #[inline]
    fn new_array() -> Self::MutableArrayType {
        Self::MutableArrayType::new()
        // MutableFixedSizeListArray::new_from(MutableBooleanArray::new(), Boolean, 1)
    }

    #[inline]
    fn arrow_serialize(v: &Self, array: &mut Self::MutableArrayType) -> arrow2::error::Result<()> {
        let ye = format!("{:08b}", v.into_bytes()[0]);
        array.try_push(Some(ye))

        /*
        let ye = PrimitiveArray::<u8>::from_slice(
            [v.b0(), v.b1(), v.b2(), v.b3(), v.b4(), v.b5(), v.b6(), v.b7()]
        );
        println!("impl-ye: {:?}", ye);
        let be = arrow2::compute::cast::primitive_to_boolean(&ye, <Self as ArrowField>::data_type());
        println!("impl-be: {:?}", be);

        // array.try_extend(Some(Some(be)))?;
        // array.try_push(Some(be))?;
        // array.push(Some(v.b0() != 0));
        // array.push(Some(v.b1() != 0));
        // array.push(Some(v.b2() != 0));
        // array.push(Some(v.b3() != 0));
        // array.push(Some(v.b4() != 0));
        // array.push(Some(v.b5() != 0));
        // array.push(Some(v.b6() != 0));
        // array.push(Some(v.b7() != 0));
        // println!("impl-array: {:?}", mbr);
        // Ok(())
        */
    }
}

fn main() {
    let data = [0x01, 0xB0, 0x1E];
    let value = Outside::read(&mut Cursor::new(data)).unwrap();
    println!("\n{:?}", &value);

    let arrow_array: Box<dyn Array> = [value].try_into_arrow().unwrap();
    let struct_array= arrow_array.as_any().downcast_ref::<StructArray>().unwrap();
    assert_eq!(struct_array.len(), 1);
    println!("main: {:#?}", struct_array);
    println!("main.fields: {:#?}", struct_array.fields());
    println!("main.values: {:#?}", struct_array.values());
}

Output:

Outside { a: 432, b: Flags { b0: 0, b1: 0, b2: 0, b3: 1, b4: 1, b5: 1, b6: 1, b7: 0 } }
main: StructArray[{a: 432, b: 00011110}]
main.fields: [
    Field {
        name: "a",
        data_type: UInt16,
        is_nullable: false,
        metadata: {},
    },
    Field {
        name: "b",
        data_type: Utf8,
        is_nullable: false,
        metadata: {},
    },
]
main.values: [
    UInt16[432],
    Utf8Array[00011110],
]

[ncpenke]

If I understand your question correctly, it has two parts:

1. How bitstrings are represented in arrow?
2. How bitstrings are represent in in-memory structs for the purposes of this crate?

For 1, internally arrow represents an array of bools very compactly as you would expect (by packing bits into bytes). Via arrow2, the MutableBooleanArray, and BooleanArrow are the right structures to use for this.

For 2, a boolean array translates to a Vec<bool>, it does not translate to a Vec<u8>, where each u8 has 8 values. You could create a wrapper type that makes this mapping. This is exactly the sort of use-case that the GAT use-case is expected to solve, but it should be possible with the current design as well.

Is the Vec<bool> an acceptable workaround for your use-case for now? It will use more memory in the expanded struct form, but still be a compact representation within arrow itself. If not, probably best to wait for the GAT refactor, and I'll add an example for how this can be done then.

[v1gnesh]

Vec<bool> is acceptable for now, thank you very much.
Once my row-based data is converted to StructArrays and RecordBatches, I can then index into this Vec<bool> for checks relying on those 8 bits/flags.

I may have a few more custom types where I will need your help. Hope it's ok to run them by you as well (as and when I'm stuck)? :)

[v1gnesh]

I see what you mean now.
I'm supposed to use it like

pub struct MStruct {
    a: Vec<BitString>,
}

pub struct BitString {
    bit: Vec<bool>,
}

The Vec wrapper in MStruct is necessary to 'fool' the program to allowing storing in a BooleanArray.
If I use just a: BitString, it complains that the field at this index is of length 8, when other fields are of length 1.

This is a bit troublesome for me as when I use a Vec<T>, I have to litter things like #[deku(count = 'n')] for each such Vec.
And the program goes from being readable to not so readable.

When you have time @ncpenke, could you please show (with psuedo-code) how this will look with GATs, when it lands?
Additionally, I hope it will allow field types of [u8; 22], [u16; 55], etc.

Const generics:
Is it possible to add general support for const generics in ArrowField and ArrowSerialize?
I believe it's currently required to manually create their impls.
Example:

pub struct MStruct {
    a: ChildStruct<5>,
}

pub struct ChildStruct<const N: usize> {
    #[some-macro-that-uses-N]
    b: u8,
}

requires

impl<N: usize> ArrowField for ChildStruct<N> { ... }
impl<N: usize> ArrowSerialize for ChildStruct<N> { ... }

Using a field like

    a: ChildStruct<6>,

is extremely desirable as it makes the code look clean & to-the-point.