Draft for the Dataframe interchange protocol

[AlenkaF]

This is the first draft of the Dataframe protocol implementation for Vaex.
It works for

• numerical and boolean dtypes
• categorical columns constructed with categorize()

It is erroring with Arrow Dictionary when trying to convert dtype arrow_type.to_pandas_dtype(), line 305 in _dtype_from_vaexdtype. Do you have any suggestions?

What is still on my todo list:

• implement expression.codes see ✨ Expression.codes gives the numerical codes for categories/dict encoded #1503
• materializing virtual columns
• chunk handling
• support of missing values

cc @maartenbreddels @JovanVeljanoski

[maartenbreddels]

Excellent work, I gave it a first pass, hope that helps.

[maartenbreddels]

I don't think is needed. Could you explain what you are trying to do here?

[AlenkaF]

I also think this will not be needed when I will be able to use expressions.codes.

But for now, if I take this example:

df = vaex.from_arrays(year=[2012, 2015, 2019], weekday=[0, 4, 6])
df = df.categorize('year', min_value=2012, max_value=2019)
df = df.categorize('weekday', labels=['Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat', 'Sun'])

and use .values function on year and weekday column I get two different types of output. In the first case it is a list of labels and in the other case I get the codes.

array([2012, 2015, 2019])
array([0, 4, 6])

To get the codes for the year column I calculate them as in line 402-404.

For now I can't seem to find another way.

[AlenkaF]

This will have to be revised. See first two items in data-apis/dataframe-api#46 (comment)

[AlenkaF]

A nested list may be hard to read. Could be changed to a list of dictionaries with column names as keys if that would be better.

[AlenkaF]

@maartenbreddels @JovanVeljanoski don't know if it will stay green =) nevertheless I think this is ready for a review.

[AlenkaF]

Is offset used in Vaex?

[maartenbreddels]

Nice work! my biggest comment/change was the change of the describe_null, which can be 3 options (no null, np.ma or arrow). I hope you agree with that.
Would love to see string support!

PS: Please keep the line endings as line feeds.

[maartenbreddels]

This could make a copy. We can use .values/.evaluate() but then we'd have to support arrow and numpy arrays, which should be fine, maybe the following works:

Suggested change

	buffer = _VaexBuffer(self._col.to_numpy())
	buffer = _VaexBuffer(np.asarray(self._col.evaluate(), dtype=bool))

[maartenbreddels]

Not sure why that happens, I'll see if a test fails because of that.

[AlenkaF]

Maybe I should restate: the values changed in a strange way when transferring data with buffer_to_ndarray in the case of a bool arrow array if .to_numpy was used.

But I will try to use previous suggestion (.values/.evaluate()) for arrow and numpy and I will delete the comment if it works.

[AlenkaF]

I get the same strange change of values for evaluate() also. It happens in the case of arrow arrays (int and bool) with missing values.

df = vaex.from_arrays(
    arrow_int_m = pa.array([0, 1, 2, None, 0], mask=np.array([0, 0, 0, 1, 1], dtype=bool)),
    arrow_float_m = pa.array([0.5, 1.5, 2.5, None, 0.5], mask=np.array([0, 0, 0, 1, 0], dtype=bool)),
    arrow_bool_m = pa.array([True, False, True, None, True], mask=np.array([0, 0, 1, 1, 0], dtype=bool))
)

col = df.__dataframe__().get_column_by_name('arrow_int_m')
b, d = col.get_buffers()["data"]
buffer_to_ndarray(b, d)

output:

array([                  0, 4607182418800017408, 4611686018427387904,
         -2251799813685248,   -2251799813685248], dtype=int64)

I can commit with the error so you can have a look?

[maartenbreddels]

Maybe I misunderstood, but I think we can simply do

Suggested change

	size = self.num_rows()
	i = self._df.evaluate_iterator(self.get_column(0)._col, chunk_size=size // n_chunks)
	iterator = []
	for i1, i2, chunk in i:
	iterator.append(_VaexColumn(self._df[i1:i2]))
	return iterator
	yield self

Similar to https://github.com/data-apis/dataframe-api/blob/27b8e1cb676bf10704d1dfc3dca0d0d806e2e802/protocol/pandas_implementation.py#L766

[AlenkaF]

I added this part thinking one could read a dataframe (that is not chunked) in chunks specifying the n_chunks (item 12 from protocol-design-requirements).

But there are some errors left, sorry about that. It should be:

    def get_chunks(self, n_chunks: Optional[int] = None) -> Iterable["_VaexDataFrame"]:
        """
        Return an iterator yielding the chunks.
        TODO: details on ``n_chunks``
        """
        if n_chunks == None:
            size = self.num_rows()
            n_chunks = self.num_chunks()
            i = self._df.evaluate_iterator(self.get_column(0)._col, chunk_size=size // n_chunks)
            iterator = []
            for i1, i2, chunk in i:
                iterator.append(_VaexDataFrame(self._df[i1:i2]))
            return iterator
        elif self.num_chunks() == 1:
            size = self.num_rows()
            i = self._df.evaluate_iterator(self.get_column(0)._col, chunk_size=size // n_chunks)
            iterator = []
            for i1, i2, chunk in i:
                iterator.append(_VaexDataFrame(self._df[i1:i2]))
            return iterator
        else:
            raise ValueError("Dataframe is already chunked.")

[maartenbreddels]

Awesome to see the strings coming in, getting close!
I pushed some a change in how to consume the buffers for strings, which is more efficient (see vaex/arrow/convert.py for some more code examples), and I think we should also use that method to produce the buffers (so that my new mem copy test does not fail).

[AlenkaF]

@maartenbreddels I added the method suggested to produce the buffers for string dtpe. Also added the mask handling in the convert_string_column(). I think this is ready for another round of review. Thanks!

[maartenbreddels]

@AlenkaF many many things for your work, this is actually already released in vaex-core 4.6.0a3

[rgommers]

This is really great to see - thanks a lot @AlenkaF and @maartenbreddels!