Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SNOW-889843: Feature Request: support direct read from SF in Apache Arrow format #613

Open
davisford opened this issue Aug 9, 2023 · 4 comments
Open

SNOW-889843: Feature Request: support direct read from SF in Apache Arrow format #613

davisford opened this issue Aug 9, 2023 · 4 comments
Assignees
@sfc-gh-anugupta
Labels
backend changes needed Change must be implemented on the Snowflake service, and not in the client driver. enhancement The issue is a request for improvement or a new feature parity this feature is supported in other drivers status-blocked Progress cannot be made to this issue due to an outside blocking factor. status-triage_done Initial triage done, will be further handled by the driver team

Comments

@davisford
Copy link

What is the current behavior?

A while back the Snowflake team announced support in the Python and JDBC drivers to read direct from the database in Apache Arrow format: https://www.snowflake.com/blog/fetching-query-results-from-snowflake-just-got-a-lot-faster-with-apache-arrow/

It doesn't appear that the NodeJS driver supports this now.

What is the desired behavior?

I would love to see this ability supported in the NodeJS driver, if possible. The client we are sending the data to supports arrow and so this would be a huge performance boon by avoiding the serdes steps during data transfer.

How would this improve snowflake-connector-nodejs?

Apache Arrow seems to be the up and coming de-facto standard for data transfer format, so I'm sure many other users of the driver would find it useful to be able to pull data from Snowflake in the arrow format.

References, Other Background

What is your Snowflake account identifier, if any?

NABANCARD
@github-actions github-actions bot changed the title Feature Request: support direct read from SF in Apache Arrow format SNOW-889843: Feature Request: support direct read from SF in Apache Arrow format Aug 9, 2023
@sfc-gh-dszmolka sfc-gh-dszmolka self-assigned this Aug 10, 2023
@sfc-gh-dszmolka
Copy link
Collaborator

hello and thank you for raising this request with us - it is a very valid one. There are already plans in place to support Arrow format natively in snowflake-sdk, but at this moment I cannot give any estimation for when it could be actually available. I'll keep this Issue updated as new information becomes available - which might take a while so thank you in advance bearing with us !

Since you already seem to be a Snowflake customer, if you did not do so yet, I would suggest reaching out to your Sales representative and explaining how this feature is important to you - these inputs can help the team prioritize the requests.

@sfc-gh-dszmolka sfc-gh-dszmolka added enhancement The issue is a request for improvement or a new feature status-in_progress Issue is worked on by the driver team labels Aug 10, 2023
@davisford
Copy link
Author

Thanks @sfc-gh-dszmolka -- I did reach out to our sales rep and let him know. We'll keep an eye on this ticket.

@bradleygore
Copy link

Not wanting to spam this with +1 comments - is there a project board or roadmap where we can view items like this and where they are in the In Progress flow?

@sfc-gh-dszmolka sfc-gh-dszmolka removed the status-in_progress Issue is worked on by the driver team label Dec 28, 2023
@sfc-gh-dszmolka
Copy link
Collaborator

unfortunately there is no publicly visible timeline / roadmap exposed, but I now asked for an update from the product team to share if they have any insight or even a rough estimation on this. Will update this thread with any new information as it becomes available to me. I think I added the In Progress label a bit prematurely; removed it for now.

If (any of) you are a Snowflake customer already; you can also ask your account team / Sales rep. for progress update on the matter; the internal tracking id is in the subject of the request - SNOW-889843. You can also express the importance of the feature implementation to them as well

@sfc-gh-dszmolka sfc-gh-dszmolka added the status-triage_done Initial triage done, will be further handled by the driver team label Feb 11, 2024
@sfc-gh-dszmolka sfc-gh-dszmolka added the parity this feature is supported in other drivers label Nov 18, 2024
@sfc-gh-dszmolka sfc-gh-dszmolka added status-blocked Progress cannot be made to this issue due to an outside blocking factor. backend changes needed Change must be implemented on the Snowflake service, and not in the client driver. labels Dec 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sfc-gh-anugupta sfc-gh-anugupta

Labels
backend changes needed Change must be implemented on the Snowflake service, and not in the client driver. enhancement The issue is a request for improvement or a new feature parity this feature is supported in other drivers status-blocked Progress cannot be made to this issue due to an outside blocking factor. status-triage_done Initial triage done, will be further handled by the driver team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@davisford @bradleygore @sfc-gh-dszmolka @sfc-gh-anugupta