Terraform thinks there are changes due to comparing encrypted and plain environment variables for the app platform resources

[snack-0verflow]

Bug Report

---

Describe the bug

The environment variable is a secret and will be encrypted, once applied the value the encrypted value will always be different to the encrypted value. Which will cause changes to occur with terraform plan.

I'm just curious has anyone got a suggest for this? (Avoid unchanged encrypted values to not cause changes, due to comparison to provided value and applied value from digitalocean as they encrypt it for you)

Affected Resource(s)

• digitalocean_app

Expected Behavior

No changes to environment vars, expect no changes to terraform plan

Actual Behavior

Digitalocean can encrypt the value which causes value to be different to the unencrypted value provided. This results in terraform plan always thinks there are changes to the environment variables (ones i'm attaching to the service within digitalocean app)

Steps to Reproduce

• Create a terraform setup with 1 digitalocean_app with secret environment variables
• execute terraform plan & terraform apply
• execute terraform plan again (it will think you have changes)

Terraform Configuration Files

resource "digitalocean_app" "service" {
  spec {
    name   = "App"
    region = var.app.hosting.region
    service {
      env {
        key   = "DATABASE_HOST"
        value = var.app.database.cluster.host
        type  = "SECRET"
        scope = "RUN_AND_BUILD_TIME"
      }
  }
}

Terraform version

Debug Output

Panic Output

Additional context

Important Factoids

References

[driedel1987]

We have the same behaviour on using Pulumi app deployments. Pulumi is using the terraform provider for doing these activities --> pulumi/pulumi-digitalocean#308.

So this seems that it is the same bug.

[Tobjoern]

Yes,
I'm also facing the same issue.
It's really quite annoying, since it always takes quite a lot of time to re-apply a bunch of changes.

Are there any updates on this? @danaelhe

[StanScates]

+1, apps take a long time to deploy, and if you have many of them in your workspace it can add an unsustainable amount of time to every apply.

[danaelhe]

@Tobjoern @StanScates Thank you for bubbling this back up. We've pushed up an example that offers a workaround in #1030

We are aware this workaround adds a few more manual steps, but it should mitigate the issue enough until we find a better solution. Please feel free to offer feedback in that PR.

[StanScates]

Thanks for publishing this workaround @danaelhe! It does appear to be working for me.

A few notes on my experience for others wishing to use this workaround:

• Since I'm using Terraform Cloud I needed to set the extension on the variables file to .auto.tfvars for TF Cloud to pick up the variables.
• The first time I ran the apply after implementing this workaround, the plan detected changes to the env vars which was concerning to me, but after running the apply, subsequent plans no longer detect them as changed. I verified that the values remain correct and unchanged on the DO side, despite them ostensibly having been changed by the initial workaround apply.

Thanks again 💯

[StanScates]

@danaelhe Hi Danael, unfortunately it appears that this workaround no longer works - it seems that a change was made to the underlying REST API recently that rejects already encrypted values instead of the previous behavior - we're now receiving this error when utilizing the workaround I described above:

Error: Error creating App: POST https://api.digitalocean.com/v2/apps: 400 (request "<uuid>") error validating app spec field "envs.0.value": secret env value must not be encrypted before app is created

This puts us in a similar situation as before with no workaround:

• Specifying a non-encrypted value for this env var results in TF detecting the env as changed on every run and destroying/recreating the app
• Specifying an encrypted value (after the app has been created successfully with the unencrypted value) results in the error above

We are still using the DO TF Provider v2.29.0 but we've also tried the latest v2.39.2 with the same results. Please let me know if I should open a new issue or if I am perhaps missing something in the documentation relevant to this workaround - doctl seems to distinguish between apps update and doctl apps create --upsert as behaviors that could impact this based on this community question however I wasn't able to find an equivalent feature in the TF provider.

Thank you for any advice you can offer us 🙏

[danaelhe]

@danaelhe Hi Danael, unfortunately it appears that this workaround no longer works - it seems that a change was made to the underlying REST API recently that rejects already encrypted values instead of the previous behavior - we're now receiving this error when utilizing the workaround I described above:

Error: Error creating App: POST https://api.digitalocean.com/v2/apps: 400 (request "<uuid>") error validating app spec field "envs.0.value": secret env value must not be encrypted before app is created

This puts us in a similar situation as before with no workaround:

• Specifying a non-encrypted value for this env var results in TF detecting the env as changed on every run and destroying/recreating the app
• Specifying an encrypted value (after the app has been created successfully with the unencrypted value) results in the error above

We are still using the DO TF Provider v2.29.0 but we've also tried the latest v2.39.2 with the same results. Please let me know if I should open a new issue or if I am perhaps missing something in the documentation relevant to this workaround - doctl seems to distinguish between apps update and doctl apps create --upsert as behaviors that could impact this based on this community question however I wasn't able to find an equivalent feature in the TF provider.

Thank you for any advice you can offer us 🙏

Hmmm. Doesn't look like there was a change done in the API side. Could you create a new issue so we can keep better track of it? We'll investigate on our end. Thanks in advance.

[danaelhe]

@danaelhe Hi Danael, unfortunately it appears that this workaround no longer works - it seems that a change was made to the underlying REST API recently that rejects already encrypted values instead of the previous behavior - we're now receiving this error when utilizing the workaround I described above:

Error: Error creating App: POST https://api.digitalocean.com/v2/apps: 400 (request "<uuid>") error validating app spec field "envs.0.value": secret env value must not be encrypted before app is created

This puts us in a similar situation as before with no workaround:

• Specifying a non-encrypted value for this env var results in TF detecting the env as changed on every run and destroying/recreating the app
• Specifying an encrypted value (after the app has been created successfully with the unencrypted value) results in the error above

We are still using the DO TF Provider v2.29.0 but we've also tried the latest v2.39.2 with the same results. Please let me know if I should open a new issue or if I am perhaps missing something in the documentation relevant to this workaround - doctl seems to distinguish between apps update and doctl apps create --upsert as behaviors that could impact this based on this community question however I wasn't able to find an equivalent feature in the TF provider.

Thank you for any advice you can offer us 🙏

Interesting, just tried the workaround it still works for me.

Just to be sure....are you still following these instructions noted here:

> // Work around: Manually copy the encrypted values returned by the API into a tfvars file:
// - On the first run, use `terraform apply -var "secret_token=thisisverysecureandwillbeencrypted"`
// - Then, retrieve the encrypted value using: `doctl apps get <app-uuid>  -o json` or from passing 
//   TF_ACC=debug in the terraform apply command in the first step. 
// - Next, put the encrypted value into app-platform-encrypted.tvars
// - Now running terraform apply -var-file="app-platform-encrypted.tvars" does not detect any changes.
// - If I want to replace that value, I can run terraform apply -var "secret_token=a-new-secret" and the change is detected.

[iwahbe]

Can we keep the issue open? While there is a work-around, it still requires treating digitalocean_app differently then other resources and requires users to understand the workaround.

[brainrake]

newest report of this issue: #1075