Almost nothing under "Deploy Airbyte" leads to a working installation using Helm on K8s.

[jesperbagge]

Page URL

No response

Description

Hello. I have for three days tried to deploy Airbyte 1.1.0 on K8s using the official documentation and have failed monumentally. Almost every aspect of the documentation to deploy Airbyte is outright wrong.

State and Logging Storage

For S3, the documentation details secret keys that should be named s3-access-key-id and s3-secret-access-key and that I should invoke their values using the key global.storage.storageSecretName. This breaks the Helm parser. According to an archived thread on Slack it should instead be global.storage.secretName. At least when using this key the parser doesn't break here. However the Helm deployment will instead fail due to that Secret.stringData.AWS_ACCESS_KEY_ID and Secret.stringData.AWS_SECRET_ACCESS_KEY has unknown objects of type "nil". It seems that the supplied secrets never gets picked up. Adding Secret.stringData.AWS_ACCESS_KEY_ID and Secret.stringData.AWS_SECRET_ACCESS_KEY to my K8s secrets does not help.

There is also a sidenote in the documentation that if I want to use another S3-compatible interface, then the endpoint-key should be supplied. The documentation does not mention where this key should be supplied. Perhaps it should be under global.storage.s3 but I can't test that since the deployment fails anyhow.

Secret Management

According to the documentation the four supported secret managers are AWS Secrets Manager, Google Secrets Manager, Azure Key Vault and Hashicorp Vault. However there are only configuration examples of the first three. I asked ChatGPT if it knew how to configure Hashicorp Vault as a secrets manager for Airbyte. It gave a suggestion that looked promising but the deployment failed due to Helm parsing errors.

External database

The documentation first outlines how to disable the internal Postgres db by setting postgresql.enabled: false. This will make the Helm parser fail, complaining about "nil" in Secret.stringData.DATABASE_PASSWORD and Secret.stringData.DATABASE_USER.

Skipping the setting postgresql.enabled: false will allow the deployment to continue but will make the airbyte-bootloader pod fail because it cannot connect to the external database. After a lot of trial and errors and printenv inside the failing pod I came to the conclusion that the only way to get at least something to work is to set the env DATABASE_URL to something like jdbc:postgresql://external-db.com:5432/airbyte. That was the only way to make the bootloader care about anything else than references to a K8s-internal service.

It still ultimately fails, though because the temporal pod fails to connect to a database. Probably another reference somewhere that doesn't get set properly.

All in all. The current state of the documentation and Helm chart is a mess.

[kagodarog]

@jesperbagge, you need to add these keys s3-secret-access-key and s3-access-key-id to the YOUR_NAMESPACE/airbyte-secrets secret if using S3 storage. This worked for me.

[jesperbagge]

@kagodarog i added those keys to the secret airbyte-config-secrets exactly like the documentation have in its example. Doesn’t do anything.

[kagodarog]

@jesperbagge, It's not that secret. At least with the latest helm version 1.1.0 there's a secret named airbyte-secrets that you need to update.

[remisalmon]

I am running into the same failures with the documentation of custom integrations (https://docs.airbyte.com/deploying-airbyte/#4-optional-for-customized-installations-only-customize-your-deployment) being wrong.

The secrets in airbyte-config-secrets do not get picked at all, either for external database or s3.

There is that new airbyte-secrets secret that is not documented, setting the global.database.user and global.database.password values directly in the helm chart seems to populate that secret instead, but the values are not propagated to DATABASE_USER and DATABASE_PASSWORDin the airbyte-bootloader pod which fails to boot.

[jesperbagge]

@jesperbagge, It's not that secret. At least with the latest helm version 1.1.0 there's a secret named airbyte-secrets that you need to update.

Thanks! Will try this at work tomorrow. This however further proves the mess that the docs / chart is in.

[jesperbagge]

@remisalmon setting DATABASE_USER, DATABASE_PASSWORD and DATABASE_URL is the only way I’ve managed to get the airbyte-bootloader not to fail. But Temporal isn’t picking up on those either so the deployment still fails. This is all very far from what’s in the documentation.

[sepsi77]

@jesperbagge I successfully deployed Airbyte to GKE and using Cloud SQL for database, cloud storage for logs and secret manager for secrets. Here are the relevant bits from my Terraform file. I'm not sure if it's significantly different for AWS. In addition to this, in the values.yaml I'm setting postgresql.enabled: false

Configure log and state storage

set {
name = "global.storage.type"
value = "GCS"
}

set {
name = "global.storage.storageSecretName"
value = kubernetes_secret.airbyte_config_secret.metadata[0].name
}

set {
name = "global.storage.bucket.log"
value = google_storage_bucket.airbyte_logs.name
}

set {
name = "global.storage.bucket.state"
value = google_storage_bucket.airbyte_logs.name
}

set {
name = "global.storage.bucket.workloadOutput"
value = google_storage_bucket.airbyte_logs.name
}
set {
name = "global.storage.gcs.projectId"
value = var.gcp_project_id
}
set {
name = "global.storage.gcs.credentialsPath"
value = "/secrets/gcs-log-creds/gcp.json"
}

Configure secret manager

set {
name = "global.secretsManager.type"
value = "googleSecretManager"
}

set {
name = "global.secretsManager.secretManagerSecretName"
value = kubernetes_secret.airbyte_config_secret.metadata[0].name
}

set {
name = "global.secretsManager.googleSecretManager.projectId"
value = var.gcp_project_id
}
set {
name = "global.secretsManager.googleSecretManager.credentialsSecretKey"
value = "gcp.json"
}

Configure external database

set {
name = "global.database.type"
value = "external"
}
set {
name = "global.database.host"
value = google_sql_database_instance.airbyte.private_ip_address
}
set {
name = "global.database.port"
value = "5432"
}
set {
name = "global.database.database"
value = google_sql_database.airbyte_db.name
}
set {
name = "global.database.secretName"
value = "airbyte-config-secrets"
}
set {
name = "global.database.userSecretKey"
value = "database-user"
}
set {
name = "global.database.passwordSecretKey"
value = "database-password"
}

[jesperbagge]

@sepsi77 Congratulations to a successful deployment. However, many of the yaml keys that you are setting in your Terraform file are exactly the same that is in the documentation and has also been proven not to work when deploying version 1.1.0 using Helm.
As an example, setting global.storage.storageSecretName will break the Helm parser and just specifying the external database by setting global.database.host will not get picked up by the airbyte-bootloader. But all these keys might have worked in the past.

[sepsi77]

@jesperbagge I've deployed the version 1.1.0.

[jesperbagge]

@sepsi77 well perhaps it is Terraform or GKE that does things differently than plain Helm and Kubernetes then, since the parser currently breaks with those config values.

[bgroff]

@jesperbagge can you post your values.yaml and secret.yaml file with the sensitive data stripped?

[bgroff]

Attached is a secrets.yaml file and values.yaml file that I have successfully used to deploy Airbyte 1.1.0 on AWS, using RDS, S3, and Secrets Manager:

Secrets.yaml

apiVersion: v1
kind: Secret
metadata:
  name: airbyte-config-secrets
type: Opaque
stringData:
  database-password: "airbyte"

  # AWS S3 Secrets
  s3-access-key-id: <MY_ID>
  s3-secret-access-key: <MY_SECRET_KEY>

  aws-secret-manager-access-key-id: <MY_ID>
  aws-secret-manager-secret-access-key: <MY_SECRET_KEY>

values.yaml

postgresql:
  enabled: false

global:
  database:
    type: external

    # -- Secret name where database credentials are stored
    secretName: "airbyte-config-secrets"

    # -- The database host
    host: "airbyte.aws-id.us-west-2.rds.amazonaws.com"

    # -- The database port
    port: 5432

    # -- The database name
    database: "airbyte"

    # -- The database user
    user: "airbyte"

    # -- The key within `secretName` where password is stored
    passwordSecretKey: "database-password"

  storage:
    type: "S3"
    secretName: "airbyte-config-secrets" # Name of your Kubernetes secret.
    bucket: ## S3 bucket names that you've created. We recommend storing the following all in one bucket.
      log: airbyte-bucket
      state: airbyte-bucket
      workloadOutput: airbyte-bucket
    s3:
      region: "us-west-2" ## e.g. us-east-1
      authenticationType: credentials ## Use "credentials" or "instanceProfile"

  secretsManager:
    type: awsSecretManager
    secretName: "airbyte-config-secrets" # Name of your Kubernetes secret.
    awsSecretManager:
      region: us-east-2
      authenticationType: credentials ## Use "credentials" or "instanceProfile"

[jesperbagge]

Yes, I get the same error. I suspect that the error I pasted in my last message is due to the state and log store.

[bgroff]

and if you do a kubectl get secret airbyte-config-secrets -n airbyte you see the secret and the correct keys are set? It looks like the secret is found but the keys are not matching or are not set.

[jesperbagge]

This is what I get when running kubectl -n airbyte get secret airbyte-config-secrets -o yaml:

apiVersion: v1
data:
  database-password: <base64 snip>
  s3-access-key-id: <base64 snip>
  s3-secret-access-key: <base64 snip>
kind: Secret
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Secret","metadata":{"annotations":{},"name":"airbyte-config-secrets","namespace":"airbyte"},"stringData":{"database-password":"<snip>","s3-access-key-id":"<snip>","s3-secret-access-key":"<snip>"},"type":"Opaque"}
  creationTimestamp: "2024-11-13T07:26:39Z"
  name: airbyte-config-secrets
  namespace: airbyte
  resourceVersion: "391371065"
  uid: 525e48fc-056c-4b70-96d4-3a2c3cb1546d
type: Opaque

[bgroff]

And you are using the latest chart version? Can you use the secret and values file I posted above and see if those work for you (I have used them to deploy 1.1.0 so they should work). There is nothing about your values file that stands out as incorrect, but I cannot reproduce the issue you are having.

[jesperbagge]

The last try I did with a deployment was actually with a clean copy from your files, substituted with my values ofc. I even did a try with external storage as the only customization, as I noted a few posts above this one. Still breaks with error validating data: [unknown object type "nil" in Secret.stringData.AWS_ACCESS_KEY_ID.

[auguste-elax]

hey @jesperbagge, I ran into an issue similar to yours while upgrading airbyte to 1.2.0 from 0.67 with helm where the temporal pods would end in a crashloop state. If you're using postgres as an external database which enforces ssl, turns out you have to change the env variables names from SQL_XXX to POSTGRES_XXX.

Hope this helps!

    - name: POSTGRES_TLS
      value: "true"
    - name: POSTGRES_TLS_DISABLE_HOST_VERIFICATION
      value: "true"
    - name: POSTGRES_TLS_ENABLED
      value: "true"
    - name: POSTGRES_HOST_VERIFICATION
      value: "false"

[jesperbagge]

Hi @auguste-elax ! Thank you for the info. However, I’m not upgrading and there’s nothing in the documentation that supports this solution.