Any plans to support Apache arrow bindings?

[chitralverma]

No description provided.

[csringhofer]

You mean extending the cursor to return fetch results in Arrow format instead of the current row oriented way?

I don't know of any plans but it sounds like a good addition to Impyla.

[chitralverma]

You mean extending the cursor to return fetch results in Arrow format instead of the current row oriented way?

I don't know of any plans but it sounds like a good addition to Impyla.

Yes, it will be great if we could have as_pyarrow_table and as_pyarrow_dataset options available some where to return the results as a PyArrow Table (eagerly) or PyArrow Dataset (lazyily) which doing zero-copy.

[Khalid-Nowaf]

I would second this strongly. +1

I'm not a Python guy, but I'm using this since it is the only (client/driver) I know for Impala that is stable and feature-complete. Adding arrow data format support will allow us to wrap it in different languages/systems with minimal cost.

[csringhofer]

Adding a basic implementation similar to as_pandas (

impyla/impala/util.py

Line 46 in a3d80ef

def as_pandas(cursor, coerce_float=False):

) seems quite simple. I see two things that could make this more complicated:

1. Performance - calling fetchall() converts the results to row based format from HS2's columnar format. Converting this back to a columnar format like arrow would mean two unnecessary transposition of the result set. Avoiding this overhead is possible but needs more work.
2. Type conversions (e.g. timestamps, which are returned in HS2 as strings). This adds both complexity and potential performance issues.