Engine marking non-secret values as secret

[iwahbe]

What happened?

As part of diagnosing pulumi/pulumi-digitalocean#308, I saw this diff:

Do you want to perform this update? details
  pulumi:pulumi:Stack: (same)
    [urn=urn:pulumi:dev::dev-yaml::pulumi:pulumi:Stack::dev-yaml-dev]
    ~ digitalocean:index/app:App: (update)
        [id=acfa96e1-7680-4dae-bd2d-3317ff15b24c]
        [urn=urn:pulumi:dev::dev-yaml::digitalocean:index/app:App::topic]
      ~ spec: {
          ~ services: [
              ~ [0]: {
                      ~ envs            : [
                          ~ [0]: {
                                  ~ key  : [secret] => "CA_CERT"
                                  ~ scope: [secret] => "RUN_AND_BUILD_TIME"
                                  ~ value: [secret] => [secret]
                                }
                        ]
                      - instanceCount   : [secret]
                      - instanceSizeSlug: [secret]
                    }
            ]
        }

Only value should be a secret. instanceCount and instanceSizeSlug should not be secrets.

Example

Run pulumi up with the providers.all ESC env on this program:

name: dev-yaml
runtime: yaml
resources:
  topic:
    type: digitalocean:App
    properties:
      spec:
        region: ams
        name: app
        services:
          - name: service-name
            git:
              repoCloneUrl: "https://github.com/digitalocean/sample-golang.git"
              branch: main
            envs:
              - key: CA_CERT
                scope: RUN_AND_BUILD_TIME
                type: GENERAL
                value: "my-secret-value"

After the program is created, run pulumi preview --diff to see the extra secrets.

Output of pulumi about

CLI
Version 3.134.2-dev.0
Go Version go1.23.1
Go Compiler gc

Plugins
KIND NAME VERSION
resource digitalocean unknown
language yaml unknown

Host
OS darwin
Version 14.6.1
Arch arm64

This project is written in yaml

Current Stack: pulumi/dev-yaml/dev

TYPE URN
pulumi:pulumi:Stack urn:pulumi:dev::dev-yaml::pulumi:pulumi:Stack::dev-yaml-dev
pulumi:providers:digitalocean urn:pulumi:dev::dev-yaml::pulumi:providers:digitalocean::default
digitalocean:index/app:App urn:pulumi:dev::dev-yaml::digitalocean:index/app:App::topic

Found no pending operations associated with dev

Backend
Name pulumi.com
URL https://app.pulumi.com/ian-pulumi-corp
User ian-pulumi-corp
Organizations ian-pulumi-corp, pulumi
Token type personal

No dependencies found

Pulumi locates its logs in /var/folders/fg/_1q36r4j6yx0rwz2fbhjd5y40000gn/T/ by default

Additional context

I'm running against digitalocean v4.33.0.

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[justinvp]

Just confirming I was able to reproduce this. The value: "my-secret-value" input is marked as a secret automatically because it's marked as a secret in the schema.

After the initial pulumi up, exporting the stack shows the spec input property does indeed have the nested value that's a secret (within spec, services, envs):

                   "spec": {
                        "__defaults": [],
                        "name": "app",
                        "region": "ams",
                        "services": [
                            {
                                "__defaults": [],
                                "envs": [
                                    {
                                        "__defaults": [],
                                        "key": "CA_CERT",
                                        "scope": "RUN_AND_BUILD_TIME",
                                        "type": "GENERAL",
                                        "value": {
                                            "4dabf18193072939515e22adb298388d": "1b47061264138c4ac30d75fd1eb44270",
                                            "ciphertext": "XXX"
                                        }
                                    }
                                ],
                                "git": {
                                    "__defaults": [],
                                    "branch": "main",
                                    "repoCloneUrl": "https://github.com/digitalocean/sample-golang.git"
                                },
                                "name": "service-name"
                            }
                        ]
                    }

It also shows the entire services array of the spec output property is marked as a secret.

                    "spec": {
                        "alerts": [],
                        "databases": [],
                        "domainNames": [],
                        "domains": [],
                        "egresses": [],
                        "envs": [],
                        "features": [
                            "buildpack-stack=ubuntu-22"
                        ],
                        "functions": [],
                        "ingress": {
                            "rules": [
                                {
                                    "component": {
                                        "name": "service-name",
                                        "preservePathPrefix": false,
                                        "rewrite": ""
                                    },
                                    "cors": null,
                                    "match": {
                                        "path": {
                                            "prefix": "/"
                                        }
                                    },
                                    "redirect": null
                                }
                            ]
                        },
                        "jobs": [],
                        "name": "app",
                        "region": "ams",
                        "services": {
                            "4dabf18193072939515e22adb298388d": "1b47061264138c4ac30d75fd1eb44270",
                            "ciphertext": "XXX"
                        },
                        "staticSites": [],
                        "workers": []
                    },

This is almost certainly due to:

pulumi/sdk/go/common/resource/plugin/provider_plugin.go

Lines 1370 to 1375 in aa9ca8a

	// If we could not pass secrets to the provider, retain the secret bit on any property with the same name. This
	// allows us to retain metadata about secrets in many cases, even for providers that do not understand secrets
	// natively.
	if !pcfg.acceptSecrets {
	annotateSecrets(outs, req.NewInputs)
	}

and:

pulumi/sdk/go/common/resource/plugin/provider_plugin.go

Lines 578 to 591 in aa9ca8a

	// annotateSecrets copies the "secretness" from the ins to the outs. If there are values with the same keys for the
	// outs and the ins, if they are both objects, they are transformed recursively. Otherwise, if the value in the ins
	// contains a secret, the entire out value is marked as a secret. This is very close to how we project secrets
	// in the programming model, with one small difference, which is how we treat the case where both are objects. In the
	// programming model, we would say the entire output object is a secret. Here, we actually recur in. We do this because
	// we don't want a single secret value in a rich structure to taint the entire object. Doing so would mean things like
	// the entire value in the deployment would be encrypted instead of a small chunk. It also means the entire property
	// would be displayed as `[secret]` in the CLI instead of a small part.
	//
	// NOTE: This means that for an array, if any value in the input version is a secret, the entire output array is
	// marked as a secret. This is actually a very nice result, because often arrays are treated like sets by providers
	// and the order may not be preserved across an operation. This means we do end up encrypting the entire array
	// but that's better than accidentally leaking a value which just moved to a different location.
	func annotateSecrets(outs, ins resource.PropertyMap) {

The note about arrays is telling:

NOTE: This means that for an array, if any value in the input version is a secret, the entire output array is marked as a secret.

Which is what is happening here. And is likely the reason for the strange diff.