Support Workload Identity authentication to Azure Storage for State and Logging Storage #46720
rowanmoul
started this conversation in
Platform Features or Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
It's really great that Azure Storage support was recently added for State and Logging Storage!
Unfotunately only support for shared access keys embedded into the connection string was added.
I would like to be able to use Workload Identity to authenticate with Microsoft Entra ID (this is the recommended way to authenticate). Supporting this should be really simple to add. The
BlobServiceClientBuilder
already supports providing aTokenCredential
. Since Airbyte is intended to be run in Kubernetes, you could use aWorkloadIdentityCredential
here, since that is the only real option in kubernetes, or you could use aChainedTokenCredential
to support a few additional auth options (eg,AzureCliCredential
for local testing). All of these are provided in thecom.azure.azure-identity
package.The only other changes needed would be in Helm:
azure.workload.identity/use: 'true'
label to the Airbyte service account as well as any pods that need to connect to storage.An annotation is also needed on the service account, but the current chart already supports this.
TokenCredential
rather than a shared access key parsed from the connection string when setting up theBlobServiceClient
.I would be happy to contribute this myself, but there are so many layers of helm charts to thread these values through, and multiple components that need to log to storage, that I am not really sure were I would start (not to mention, I have zero experience with Kotlin).
Beta Was this translation helpful? Give feedback.
All reactions