Sqlite Use Case

[evbo]

Hi,

Sqlite is the undisputed champion in terms of embedded databases, and so many applications (that might use rust-ndarray) rely on embedded databases, it may be useful to document how rust-ndarray could interoperate with large arrays stored in formats common to these sorts of data stores.

That said, there's 2 camps of thought with most embedded databases: store arrays in Sqlite as tables, each row being an element (if you plan to individually access elements), or store arrays as byte array blobs (if you plan to retrieve entire array, never dealing with individual elements).

I fall into the 2nd camp, and I was wondering if folks at rust-ndarray had any thoughts to add on how best to interact with datatypes such as byte array (blobs) to do simple arithmetic to start.

So is there a simpler way to say, subtract to blobs from Sqlite?:

// Vec<u8> is generally how one would store numerical arrays
// but the sqlite client ("rusqlite" in my case) may vary
let left_blob: Vec<u8> = data_from_sqlite_connection();
let right_blob: Vec<u8> = data_from_sqlite_connection();

let left_f32 = left_blob
        .chunks_exact(4)
        .map(|bytes| f32::from_le_bytes(bytes.try_into().unwrap()))
        .collect();

let right_f32 = left_blob
        .chunks_exact(4)
        .map(|bytes| f32::from_le_bytes(bytes.try_into().unwrap()))
        .collect();

let fastest_subtraction_in_the_west = Array::from(left_f32) - Array::from(right_f32);

So in summary, how does this community feel about?:

• implementing connectors/extensions for ndarray data sources?
• sheding light on performance improvements to my above transforms for operating on Sqlite blobs in particular?

[bluss]

do the byteorder translation in place if possible - so that you get a &[f32]. It might require some setup or it might also not be possible (if you can't control how the data is stored). From a slice you can quickly get an ArrayView/Mut.

It sounds interesting but implementing sqlite specific stuff in the main crate is not in the plan right now.

[evbo]

Yeah, my only concern is endianess, I don't want to assume byte order since I don't know what platform I'm on...

Glad there's no obvious improvements I've missed. Is the overhead of ndarray huge? Would I be better off doing arithmetic with raw Vecs myself?

I like the simplicty of the ndarray API, which is also why I've opted not to do the low level processing myself ;)

[bluss]

ideally the overhead should be low. No reason an 1D array should be more complicated than a Vec. If you use ArrayD or write loops using indexing in the inner loop, then that's high overhead.

[jturner314]

I'd consider using the .npy format (via my ndarray-npy crate) for the bytes in the Vec<u8> binary blob. This is NumPy's standard file format. It is basically a small plain-text header (describing the element type and endianness, array shape, and whether the data is in C or Fortran layout) followed by the binary data. (Technically, the header format doesn't require the endianness to be specified, but at least the NumPy and ndarray-npy implementations always specify the endianness.) This wouldn't be a good choice if you have a lot of very small arrays (due to the space for each header), but it would be convenient with larger arrays. Also, there's overhead when writing arrays which aren't in contiguous C/Fortran layout (due to the need to transform the layout), but I'd think that would be acceptable for most use cases. The ndarray-npy crate is limited to only some element types (mostly primitive numeric types), but if it covers your element types, it's pretty convenient.

I suppose, though, if you're only dealing with 1-D arrays, then the overhead of the .npy header may be larger than you'd like, since all that's really necessary for the 1-D case is the element type and endianness. The byteorder crate provides convenient endianness handling.

[evbo]

Representing with npy format is an interesting idea I wouldn't have guessed. Unfortunately the array size vary wildly, from just a couple elements to several hundred, maybe thousands, but not much more than that.

I'm glad doing the arithmetic with Vec<f32> types naively isn't a recommendation. On SO there are several posts about linear algebra math in vanilla Rust, it might be helpful dropping a comment in there: "just use rust-ndarray" ;)