-
Notifications
You must be signed in to change notification settings - Fork 247
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[query] Unify CloudCredentials and Simplify BatchClient #14684
Open
ehigham
wants to merge
19
commits into
main
Choose a base branch
from
ehigham/cloud-credentials
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This stack of pull requests is managed by Graphite. Learn more about stacking. |
ehigham
force-pushed
the
ehigham/cloud-credentials
branch
from
September 16, 2024 17:45
cd38e9b
to
0ac3faa
Compare
ehigham
force-pushed
the
ehigham/enclosing
branch
from
September 16, 2024 17:47
992de5f
to
a11a232
Compare
ehigham
force-pushed
the
ehigham/cloud-credentials
branch
from
September 16, 2024 17:47
0ac3faa
to
9c4151a
Compare
ehigham
force-pushed
the
ehigham/enclosing
branch
from
September 16, 2024 17:48
a11a232
to
098f6a4
Compare
ehigham
force-pushed
the
ehigham/cloud-credentials
branch
from
September 16, 2024 17:48
9c4151a
to
16b6635
Compare
This was referenced Sep 17, 2024
ehigham
force-pushed
the
ehigham/enclosing
branch
from
September 19, 2024 20:52
098f6a4
to
2b028ba
Compare
ehigham
force-pushed
the
ehigham/cloud-credentials
branch
from
September 19, 2024 20:52
9cff2fc
to
d14679d
Compare
This was referenced Sep 20, 2024
ehigham
force-pushed
the
ehigham/enclosing
branch
from
October 1, 2024 19:44
2b028ba
to
8c7b1c3
Compare
ehigham
force-pushed
the
ehigham/cloud-credentials
branch
from
October 1, 2024 19:44
d14679d
to
ed2e8e8
Compare
ehigham
force-pushed
the
ehigham/enclosing
branch
from
October 8, 2024 19:18
8c7b1c3
to
3c702c0
Compare
ehigham
force-pushed
the
ehigham/cloud-credentials
branch
2 times, most recently
from
October 8, 2024 20:30
fc06f02
to
a101b55
Compare
ehigham
force-pushed
the
ehigham/cloud-credentials
branch
from
October 16, 2024 20:02
a101b55
to
1682a0f
Compare
ehigham
force-pushed
the
ehigham/enclosing
branch
from
October 16, 2024 21:30
2f05ecd
to
83ff2cc
Compare
ehigham
force-pushed
the
ehigham/cloud-credentials
branch
from
October 16, 2024 21:30
1682a0f
to
6e63e69
Compare
1 task
1 task
Thanks Chris! |
chrisvittal
approved these changes
Oct 29, 2024
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great work. Thanks for cleaning this up.
ehigham
changed the title
[query] Unify CloudCredentials and Modernise BatchService API
[query] Unify CloudCredentials and Simplify BatchClient
Oct 31, 2024
ehigham
force-pushed
the
ehigham/cloud-credentials
branch
2 times, most recently
from
November 4, 2024 22:15
01aedac
to
6e36a7a
Compare
ehigham
force-pushed
the
ehigham/cloud-credentials
branch
from
November 19, 2024 17:08
6e36a7a
to
e121222
Compare
ehigham
commented
Nov 19, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change combines cloud auth logic that was previously duplicated
between the various
FS
implementations and theBatchClient
.The main refactoring is to make the interface between the
ServiceBackend
morehigh-level and leave json serialisation to the
BatchClient
. To do this, I'veadded a bunch of case classes that resemble the python objects the batch service
expects (or a subset of the data). To simplify the interface, I've split batch
creation from job submission (update). For QoB, the python client creates the
batch before handing control to the query driver; batch creation is necessary
for testing only.
This change has low security impact as there are minor changes to the creation
and scoping of service account credentials. Note that for each
FS
, credentialsare scoped to the default storage oauth2 scopes for each service.