opensearch domain warning "ForceNew: No changes for engine_version"

[ploxiln]

What happened?

For all AWS OpenSearch Domain resources managed by my pulumi projects/stacks, I now get this warning in Diagnostics:

aws:opensearch:Domain (rodeo-production--non-ro-pharmacies--os):
warning: customdiff/force_new.go:32: sdk.helper_schema: unable to require attribute replacement: [email protected] tf_attribute_path=engine_version error="ForceNew: No changes for engine_version"

There is no actual diff to apply for the opensearch instance, the EngineVersion in particular had no change. I can just ignore the warning and everything continues to work ok, but it does make the plan output messier, and it even corrupts the plan terminal output even more sometimes. Example below. But before that, some more notes:

• I tried updating to the latest versions [email protected] and [email protected] and saw the exact same warning "ForceNew: No changes for engine_version"
• I was able to avoid this by downgrading back to [email protected] and [email protected] so this is probably a regression since v6.0.4
• Adding pulumi.IgnoreChanges([]string{"engineVersion"}) doesn't help
• My googling suggests this must be some kind of bug in the provider, using the terraform sdk

Example of corrupted preview terminal output (happens occasionally):

Previewing update (healthbyro/...)

View in Browser (Ctrl+O): https://app.pulumi.com/healthbyro/...

     Type                            Name                                                    Plan        Info
     Type                            Name                                                    Plan        Info
     Type                            Name                                                    Plan        Info
     Type                            Name                                                    Plan        Info
     Type                            Name                                                    Plan        Info
     Type                            Name                                                    Plan        Info
     Type                            Name                                                    Plan        Info
     Type                            Name                                                    Plan        Info
     Type                            Name                                                    Plan        Info
     Type                            Name                                                    Plan        Info
     Type                            Name                                                    Plan        Info
     Type                            Name                                                    Plan        Info
     Type                            Name                                                    Plan        Info
     Type                            Name                                                    Plan        Info
     Type                            Name                                                    Plan        Info
     Type                            Name                                                    Plan        Info
     Type                            Name                                                    Plan        Info
     Type                                 Name                                                    Plan        Info
     Type                                 Name                                                    Plan        Info
     Type                                 Name                                                    Plan        Info
     Type                                 Name                                                    Plan        Info
     Type                                 Name                                                    Plan        Info
     Type                                 Name                                                    Plan        Info
     Type                                 Name                                                    Plan        Info
     Type                                 Name                                                    Plan        Info
     Type                                 Name                                                    Plan        Info
     Type                                 Name                                                    Plan        Info
     Type                                 Name                                                    Plan        Info
     Type                                 Name                                                    Plan        Info
... about 500 more lines of this

Example

	domainName := fmt.Sprintf("rodeo-%s-shared-os", input.EnvironmentName)

	_, err := opensearch.NewDomain(ctx, domainName, &opensearch.DomainArgs{
		DomainName:    pulumi.String(domainName),
		EngineVersion: pulumi.String("OpenSearch_1.3"),
		ClusterConfig: &opensearch.DomainClusterConfigArgs{
			InstanceCount: pulumi.Int(1),
			InstanceType:  pulumi.String("m5.xlarge.search"),
		},
		EbsOptions: &opensearch.DomainEbsOptionsArgs{
			EbsEnabled: pulumi.Bool(true),
			VolumeType: pulumi.String("gp2"),
			VolumeSize: pulumi.Int(25),
		},
		EncryptAtRest: &opensearch.DomainEncryptAtRestArgs{
			Enabled: pulumi.Bool(true),
		},
		NodeToNodeEncryption: &opensearch.DomainNodeToNodeEncryptionArgs{
			Enabled: pulumi.Bool(true),
		},
		AccessPolicies: elasticSearchDomainAccessPolicies(input.NodeGroupRoleARN),
		VpcOptions: &opensearch.DomainVpcOptionsArgs{
			// Use a single subnet to allow a single-instance domain.
			SubnetIds:        pulumi.StringArray{input.Subnets[0].ID()},
			SecurityGroupIds: pulumi.StringArray{input.SecurityGroup.ID()},
		},
		Tags: pulumi.StringMap{
			"Name":   pulumi.String(domainName),
			"Env":    pulumi.String(input.EnvironmentName),
			"System": pulumi.String("rodeo"),
		},
	})

Output of pulumi about

CLI          
Version      3.96.2
Go Version   go1.21.4
Go Compiler  gc

Plugins
NAME        VERSION
aws         6.14.0
aws-native  0.90.0
go          unknown
kubernetes  4.6.1
random      4.13.4
tls         4.10.0

Host     
OS       darwin
Version  14.3
Arch     arm64

This project is written in go: executable='/opt/homebrew/bin/go' version='go version go1.21.6 darwin/arm64'

Current Stack: healthbyro/rodeo-infra/beta

TYPE ... URN ... a lot! seriously

Found no pending operations associated with healthbyro/beta

Backend        
Name           pulumi.com
URL            https://app.pulumi.com/plo-ro
User           plo-ro
Organizations  plo-ro, healthbyro
Token type     personal

Dependencies:
NAME                                        VERSION
github.com/apparentlymart/go-cidr           1.1.0
github.com/healthbyro/rodeo                 1.93.1704389193
github.com/pulumi/pulumi-aws-native/sdk     0.90.0
github.com/pulumi/pulumi-aws/sdk/v6         6.14.0
github.com/pulumi/pulumi-kubernetes/sdk/v4  4.6.1
github.com/pulumi/pulumi-random/sdk/v4      4.13.4
github.com/pulumi/pulumi-tls/sdk/v4         4.10.0
github.com/pulumi/pulumi/sdk/v3             3.96.2
github.com/rs/zerolog                       1.31.0
gopkg.in/yaml.v2                            2.4.0

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[t0yv0]

Thank you for reporting this issue @ploxiln this is indeed a recent change and likely side-effect of pulumi/pulumi-terraform-bridge#1569 . Our team will have a look as soon as we have a chance.

[ploxiln]

That makes sense; thanks for the update!

[iwahbe]

I was able to reproduce the issue (thanks for the great repro). It looks like pulumi/pulumi-terraform-bridge#1569 set a different log level then TF uses. Running TF_LOG=WARN terraform apply on an equivalent program produces the same warning:

2024-02-14T13:09:00.719-0800 [WARN]  provider.terraform-provider-aws_v5.32.1_x5: unable to require attribute replacement: tf_resource_type=aws_opensearch_domain tf_rpc=PlanResourceChange tf_mux_provider=*schema.GRPCProviderServer tf_provider_addr=registry.terraform.io/hashicorp/aws tf_req_id=39fbbc5d-8f19-4edd-e84c-cd0aec9c45dd @caller=github.com/hashicorp/terraform-plugin-sdk/[email protected]/helper/customdiff/force_new.go:32 @module=sdk.helper_schema error="ForceNew: No changes for engine_version" tf_attribute_path=engine_version timestamp=2024-02-14T13:09:00.719-0800

We should be able to fix this by reducing the default TF log level in the bridge to ERROR (instead of WARN).

[iwahbe]

The warning is triggered by the underlying AWS provider, and can be safely ignored. pulumi/pulumi-terraform-bridge#1684 will resolve the issue after it releases.