Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[query] Unify CloudCredentials and Simplify BatchClient #14684

Open
wants to merge 19 commits into
base: main
Choose a base branch
from

Conversation

ehigham
Copy link
Member

@ehigham ehigham commented Sep 13, 2024

This change combines cloud auth logic that was previously duplicated
between the various FS implementations and the BatchClient.

The main refactoring is to make the interface between the ServiceBackend more
high-level and leave json serialisation to the BatchClient. To do this, I've
added a bunch of case classes that resemble the python objects the batch service
expects (or a subset of the data). To simplify the interface, I've split batch
creation from job submission (update). For QoB, the python client creates the
batch before handing control to the query driver; batch creation is necessary
for testing only.

This change has low security impact as there are minor changes to the creation
and scoping of service account credentials. Note that for each FS, credentials
are scoped to the default storage oauth2 scopes for each service.

Copy link
Member Author

ehigham commented Sep 13, 2024

@ehigham ehigham marked this pull request as ready for review September 13, 2024 19:05
@ehigham ehigham force-pushed the ehigham/cloud-credentials branch 2 times, most recently from fc06f02 to a101b55 Compare October 8, 2024 20:30
Base automatically changed from ehigham/enclosing to main October 17, 2024 14:26
Copy link
Member Author

ehigham commented Oct 29, 2024

Thanks Chris!

Copy link
Collaborator

@chrisvittal chrisvittal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work. Thanks for cleaning this up.

@ehigham ehigham changed the title [query] Unify CloudCredentials and Modernise BatchService API [query] Unify CloudCredentials and Simplify BatchClient Oct 31, 2024
@ehigham ehigham force-pushed the ehigham/cloud-credentials branch 2 times, most recently from 01aedac to 6e36a7a Compare November 4, 2024 22:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants