Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Panic when trying to update AWS wafv2 WebAcl #2815

Closed
DanielSchiavini opened this issue Sep 16, 2023 · 11 comments
Closed

Panic when trying to update AWS wafv2 WebAcl #2815

DanielSchiavini opened this issue Sep 16, 2023 · 11 comments
Assignees
Labels
customer/feedback Feedback from customers impact/panic This bug represents a panic or unexpected crash kind/bug Some behavior is incorrect or out of spec p1 A bug severe enough to be the next item assigned to an engineer resolution/no-repro This issue wasn't able to be reproduced
Milestone

Comments

@DanielSchiavini
Copy link

DanielSchiavini commented Sep 16, 2023

What happened?

When trying to apply updates, the following error is received:

  pulumi:pulumi:Stack (infra-shared):
    error: update failed

    panic: interface conversion: interface {} is nil, not map[string]interface {}
    goroutine 105 [running]:
    github.com/hashicorp/go-cty/cty.Value.GetAttr({{{0xe773160?, 0xc00f5a1330?}}, {0x0?, 0x0?}}, {0xd41e5b7, 0x8})
        /home/runner/go/pkg/mod/github.com/hashicorp/[email protected]/cty/value_ops.go:711 +0x308
    github.com/hashicorp/terraform-provider-aws/internal/provider.tagsResourceInterceptor.run({0xc000e49ce0?, 0xd6fa988?, 0xd6fa980?}, {0xe771b48, 0xc00ea474a0}, {0xe77fb70, 0xc00ea3ec00}, {0xd3ad660?, 0xc002bf4820?}, 0x1, ...)
        /home/runner/work/pulumi-aws/pulumi-aws/upstream/internal/provider/intercept.go:250 +0x1474
    github.com/hashicorp/terraform-provider-aws/internal/provider.interceptedHandler[...].func1(0xba4df60?, {0xd3ad660?, 0xc002bf4820?})
        /home/runner/work/pulumi-aws/pulumi-aws/upstream/internal/provider/intercept.go:100 +0x16b
    github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).update(0xe771ad8?, {0xe771ad8?, 0xc0000b2078?}, 0xd?, {0xd3ad660?, 0xc002bf4820?})
        /home/runner/go/pkg/mod/github.com/pulumi/terraform-plugin-sdk/[email protected]/helper/schema/resource.go:767 +0x87
    github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).Apply(0xc001d74380, {0xe771ad8, 0xc0000b2078}, 0xc00d5da820, 0xc00df90100, {0xd3ad660, 0xc002bf4820})
        /home/runner/go/pkg/mod/github.com/pulumi/terraform-plugin-sdk/[email protected]/helper/schema/resource.go:879 +0x845
    github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfshim/sdk-v2.v2Provider.Apply({0xc000627ce0?, {0xc002b821e8?, 0xc0157874d0?, 0x268247?}}, {0xd46e2b8, 0x11}, {0xe773630?, 0xc009ccf950}, {0xe7808d8, 0xc00df90100})
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tfshim/sdk-v2/provider.go:100 +0x188
    github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.(*Provider).Update(0xc009cd8580, {0xe771b48?, 0xc0136a1f80?}, 0xc0136a4000)
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tfbridge/provider.go:943 +0x88f
    github.com/pulumi/pulumi-terraform-bridge/x/muxer.(*muxer).Update.func1({0xe7998d0?, 0xc009cd8580?})
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/x/[email protected]/muxer.go:356 +0x39
    github.com/pulumi/pulumi-terraform-bridge/x/muxer.resourceMethod[...](0xc00995ac30?, 0x40, 0xc0136c5780?)
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/x/[email protected]/muxer.go:303 +0xbd
    github.com/pulumi/pulumi-terraform-bridge/x/muxer.(*muxer).Update(0x0?, {0xe771b48?, 0xc0136a1f80?}, 0x40?)
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/x/[email protected]/muxer.go:355 +0x68
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Update_Handler.func1({0xe771b48, 0xc0136a1f80}, {0xd015f80?, 0xc0136a4000})
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/[email protected]/proto/go/provider_grpc.pb.go:609 +0x7b
    github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1({0xe771b48, 0xc0136a0030}, {0xd015f80, 0xc0136a4000}, 0xc0136ae080, 0xc004bc6150)
        /home/runner/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/go/otgrpc/server.go:57 +0x3e8
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Update_Handler({0xd235ba0?, 0xc00995ac30}, {0xe771b48, 0xc0136a0030}, 0xc0136a2000, 0xc000ab06a0)
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/[email protected]/proto/go/provider_grpc.pb.go:611 +0x138
    google.golang.org/grpc.(*Server).processUnaryRPC(0xc0005be780, {0xe781960, 0xc00985f6c0}, 0xc001950480, 0xc0032879e0, 0x18b44bd0, 0x0)
        /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1337 +0xdf3
    google.golang.org/grpc.(*Server).handleStream(0xc0005be780, {0xe781960, 0xc00985f6c0}, 0xc001950480, 0x0)
        /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1714 +0xa36
    google.golang.org/grpc.(*Server).serveStreams.func1.1()
        /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:959 +0x98
    created by google.golang.org/grpc.(*Server).serveStreams.func1
        /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:957 +0x18c

  aws:wafv2:WebAcl (firewall):
    error: error reading from server: read tcp 127.0.0.1:51995->127.0.0.1:51992: wsarecv: An existing connection was forcibly closed by the remote host.

Example

This is the code that raises the error:

import { wafv2 } from '@pulumi/aws';

new wafv2.WebAcl('firewall', {
  name: 'firewall',
  description: 'Blocks IPs if they do too many requests per minute.',
  defaultAction: { allow: {} },
  rules: [
    {
      name: "rate-limit",
      priority: 0,
      statement: {
        rateBasedStatement: {
          limit: 2000,
          aggregateKeyType: "IP"
        }
      },
      action: {
        block: {
          customResponse: {
            responseCode: 429,
            customResponseBodyKey: "rate-limit-response"
          }
        }
      },
      visibilityConfig: {
        sampledRequestsEnabled: true,
        cloudwatchMetricsEnabled: true,
        metricName: "rate-limit"
      }
    },
    {
      name: "account-creation-rate-limit",
      priority: 1,
      statement: {
        rateBasedStatement: {
          limit: 200,
          aggregateKeyType: "IP",
          scopeDownStatement: {
            andStatement: {
              statements: [
                {
                  byteMatchStatement: {
                    searchString: "/api/v1/account",
                    fieldToMatch: { uriPath: {} },
                    textTransformations: [
                      {
                        priority: 0,
                        type: "NONE"
                      }
                    ],
                    positionalConstraint: "CONTAINS"
                  }
                },
                {
                  notStatement: {
                    statements: [{
                      byteMatchStatement: {
                        searchString: "GET",
                        fieldToMatch: {
                          method: {}
                        },
                        textTransformations: [
                          {
                            priority: 0,
                            type: "NONE"
                          }
                        ],
                        positionalConstraint: "EXACTLY"
                      }
                    }]
                  }
                }
              ]
            }
          }
        }
      },
      action: {
        block: {
          customResponse: {
            responseCode: 429,
            customResponseBodyKey: "rate-limit-response"
          }
        }
      },
      visibilityConfig: {
        sampledRequestsEnabled: true,
        cloudwatchMetricsEnabled: true,
        metricName: "account-creation-rate-limit"
      }
    }
  ],
  visibilityConfig: { sampledRequestsEnabled: true, cloudwatchMetricsEnabled: true, metricName: 'firewall' },
  customResponseBodies: [{ key: `firewall-rate-limit-response`, content: 'Too many requests, please try again later.', contentType: 'TEXT_PLAIN' }],
  scope: 'REGIONAL',
});

Output of pulumi about


CLI          
Version      3.83.0
Go Version   go1.21.1
Go Compiler  gc

Plugins
NAME    VERSION
nodejs  unknown

Host
OS       Microsoft Windows 11 Pro
Version  10.0.22621 Build 22621
Arch     x86_64

This project is written in nodejs: executable='C:\Program Files\nodejs\node.exe' version='v18.13.0'

Current Stack: DanielSchiavini/infra/shared

TYPE                                                 URN
pulumi:pulumi:Stack                                  urn:pulumi:shared::infra::pulumi:pulumi:Stack::infra-shared
pulumi:providers:pulumi                              urn:pulumi:shared::infra::pulumi:providers:pulumi::default
pulumi:pulumi:StackReference                         urn:pulumi:shared::infra::pulumi:pulumi:StackReference::DanielSchiavini/services/prod
pulumi:providers:aws                                 urn:pulumi:shared::infra::pulumi:providers:aws::default_6_0_4
pulumi:pulumi:StackReference                         urn:pulumi:shared::infra::pulumi:pulumi:StackReference::DanielSchiavini/services/staging
aws:acm/certificate:Certificate                      urn:pulumi:shared::infra::aws:acm/certificate:Certificate::certificate
pulumi:providers:github                              urn:pulumi:shared::infra::pulumi:providers:github::default_5_17_0
aws:ec2/vpc:Vpc                                      urn:pulumi:shared::infra::aws:ec2/vpc:Vpc::network
github:index/repository:Repository                   urn:pulumi:shared::infra::github:index/repository:Repository::services-repository
aws:ec2/internetGateway:InternetGateway              urn:pulumi:shared::infra::aws:ec2/internetGateway:InternetGateway::gateway
aws:ec2/subnet:Subnet                                urn:pulumi:shared::infra::aws:ec2/subnet:Subnet::subnet-b
aws:ec2/subnet:Subnet                                urn:pulumi:shared::infra::aws:ec2/subnet:Subnet::subnet-c
aws:ec2/subnet:Subnet                                urn:pulumi:shared::infra::aws:ec2/subnet:Subnet::subnet-a
github:index/branch:Branch                           urn:pulumi:shared::infra::github:index/branch:Branch::develop-branch
aws:ec2/routeTable:RouteTable                        urn:pulumi:shared::infra::aws:ec2/routeTable:RouteTable::localRouteTable
aws:ec2/routeTableAssociation:RouteTableAssociation  urn:pulumi:shared::infra::aws:ec2/routeTableAssociation:RouteTableAssociation::route-table-association-1
aws:ec2/routeTableAssociation:RouteTableAssociation  urn:pulumi:shared::infra::aws:ec2/routeTableAssociation:RouteTableAssociation::route-table-association-3
aws:ec2/routeTableAssociation:RouteTableAssociation  urn:pulumi:shared::infra::aws:ec2/routeTableAssociation:RouteTableAssociation::route-table-association-2
pulumi:pulumi:StackReference                         urn:pulumi:shared::infra::pulumi:pulumi:StackReference::DanielSchiavini/housing/west1
github:index/branchDefault:BranchDefault             urn:pulumi:shared::infra::github:index/branchDefault:BranchDefault::github-default-branch
aws:alb/loadBalancer:LoadBalancer                    urn:pulumi:shared::infra::aws:alb/loadBalancer:LoadBalancer::load-balancer
aws:alb/listener:Listener                            urn:pulumi:shared::infra::aws:alb/listener:Listener::https-listener
aws:alb/listener:Listener                            urn:pulumi:shared::infra::aws:alb/listener:Listener::http-listener
aws:wafv2/webAcl:WebAcl                              urn:pulumi:shared::infra::aws:wafv2/webAcl:WebAcl::firewall
pulumi:providers:aws                                 urn:pulumi:shared::infra::pulumi:providers:aws::default_5_30_0
aws:wafv2/webAclAssociation:WebAclAssociation        urn:pulumi:shared::infra::aws:wafv2/webAclAssociation:WebAclAssociation::balancer-firewall-association
pulumi:providers:github                              urn:pulumi:shared::infra::pulumi:providers:github::default


Found no pending operations associated with shared

Backend
Name           pulumi.com
URL            https://app.pulumi.com/DanielSchiavini
User           DanielSchiavini
Organizations  DanielSchiavini

Additional context

When downloading the JSON from the AWS console, the andStatement seems to expect a single statement, but Pulumi requires a list of statements

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

@DanielSchiavini DanielSchiavini added kind/bug Some behavior is incorrect or out of spec needs-triage Needs attention from the triage team labels Sep 16, 2023
@DanielSchiavini
Copy link
Author

OK I have now managed to apply the update after running pulumi refresh. It seems the resource was changed directly in AWS and that caused the crash on the Pulumi provider.

@mikhailshilkov
Copy link
Member

@DanielSchiavini Do you know which attributes were changed in AWS? Or any other concrete steps that I could you to reproduce the issue locally?

@mikhailshilkov mikhailshilkov added awaiting-feedback Blocked on input from the author and removed needs-triage Needs attention from the triage team labels Sep 19, 2023
@DanielSchiavini
Copy link
Author

@mikhailshilkov the following rule had been added to AWS and then "backported" to pulumi:

 {
                  byteMatchStatement: {
                    searchString: "/api/v1/account",
                    fieldToMatch: { uriPath: {} },
                    textTransformations: [
                      {
                        priority: 0,
                        type: "NONE"
                      }
                    ],
                    positionalConstraint: "CONTAINS"
                  }
                },

@mikhailshilkov mikhailshilkov added impact/panic This bug represents a panic or unexpected crash p1 A bug severe enough to be the next item assigned to an engineer and removed awaiting-feedback Blocked on input from the author labels Sep 29, 2023
@mikhailshilkov mikhailshilkov added this to the 0.95 milestone Sep 29, 2023
@mikhailshilkov
Copy link
Member

@DanielSchiavini Thank you! One more question - which pulumi-aws version are you on? pulumi about somehow did not print that info.

@DanielSchiavini
Copy link
Author

DanielSchiavini commented Sep 29, 2023

These are my dependencies

  "dependencies": {
    "@pulumi/aws": "^6.2.1",
    "@pulumi/awsx": "^1.0.5",
    "@pulumi/github": "^5.19.0",
    "@pulumi/mongodbatlas": "^3.10.1",
    "@pulumi/pulumi": "^3.85.0",
    "@pulumi/random": "^4.14.0"
  }

@lukehoban
Copy link
Contributor

A few questions:

  1. Do we have a repro for this?
  2. Do we have a list of known impacted versions of pulumi-aws? We would need the exact version from package-lock.json, not just the range from package.json.
  3. Do we have a list of know unaffected versions?

@DanielSchiavini
Copy link
Author

  1. No, the problem is fixed by refresh. Feel free to close the issue if there is not enough info
  2. See the section of the lock file below
  3. I haven't tried with different versions
    "node_modules/@pulumi/aws": {
      "version": "6.0.4",
      "resolved": "https://registry.npmjs.org/@pulumi/aws/-/aws-6.0.4.tgz",
      "integrity": "sha512-g8t+LuKwEEGX7bKUcYpB8gr1xtrJm3PAt26Js5QztSWDiujy9ehmk4CeEgKLfMR9EGmav6jFSSvwX5IQXSPgog==",
      "hasInstallScript": true,
      "dependencies": {
        "@pulumi/pulumi": "^3.0.0",
        "builtin-modules": "3.0.0",
        "mime": "^2.0.0",
        "read-package-tree": "^5.2.1",
        "resolve": "^1.7.1"
      }
    },
    "node_modules/@pulumi/awsx": {
      "version": "1.0.5",
      "resolved": "https://registry.npmjs.org/@pulumi/awsx/-/awsx-1.0.5.tgz",
      "integrity": "sha512-iGkDzPalPhzRlfqCaWgwJkaA8EfgPwzWkcqdg0TgcnUefNwKGEXfEelORxMXKxZe5M0VV3U3ljrEq6P0SLCtWg==",
      "hasInstallScript": true,
      "dependencies": {
        "@pulumi/aws": "^5.35.0",
        "@pulumi/docker": "^3.6.1",
        "@pulumi/pulumi": "^3.0.0",
        "@types/aws-lambda": "^8.10.23",
        "mime": "^2.0.0"
      }
    },
    "node_modules/@pulumi/awsx/node_modules/@pulumi/aws": {
      "version": "5.42.0",
      "resolved": "https://registry.npmjs.org/@pulumi/aws/-/aws-5.42.0.tgz",
      "integrity": "sha512-1h7Q5DjwoWVGxhBMcNragx/Q1US1KT7g29Tk3RghTg/9N7rGUbzTQKEXSrGgRSjGA/aKTbU+gt5A9ZmhONLiLg==",
      "hasInstallScript": true,
      "dependencies": {
        "@pulumi/pulumi": "^3.0.0",
        "aws-sdk": "^2.0.0",
        "builtin-modules": "3.0.0",
        "mime": "^2.0.0",
        "read-package-tree": "^5.2.1",
        "resolve": "^1.7.1"
      }
    },

@t0yv0
Copy link
Member

t0yv0 commented Oct 2, 2023

Unfortunately I cannot reproduce this. Judging by the stack trace it is a panic in the tags interceptor due to an unexpected nil in GetRawPlan:

https://github.com/hashicorp/terraform-provider-aws/blob/522a58443aa503714444e5333641300a6af34621/internal/provider/intercept.go#L250

I'm not sure how refresh fixed this, because it seems it should be happening during planning an Update before hitting up the cloud. I've tried running a few variations of the program through an Update on the listed versions, no repro.

We did historically have issues with panics in GetRawPlan, fixed since v3.45.0 of the terraform bridge framework used in this provider but it seems the stack trace is referencing a newer v3.58.0 version.

new wafv2.WebAcl('firewall', {
  name: 'firewall',
  description: 'Blocks IPs if they do too many requests per minute.',
  defaultAction: { allow: {} },
  rules: [
    {
      name: "rate-limit",
      priority: 0,
        rateBasedStatement: {
          limit: 2000,
          aggregateKeyType: "IP"
        }
      },
      action: {
        block: {
          customResponse: {
            responseCode: 429,
            //customResponseBodyKey: "rate-limit-response"
          }
        }
      },
      visibilityConfig: {
        sampledRequestsEnabled: true,
        cloudwatchMetricsEnabled: true,
        metricName: "rate-limit"
      }
    },
    {
      name: "account-creation-rate-limit",
      priority: 1,
      statement: {
        rateBasedStatement: {
          limit: 200,
          aggregateKeyType: "IP",
          scopeDownStatement: {
            andStatement: {
              statements: [
                {
                  byteMatchStatement: {
                    searchString: "/api/v1/account",
                    fieldToMatch: { uriPath: {} },
                    textTransformations: [
                      {
                        priority: 0,
                        type: "NONE"
                      }
                    ],
                    positionalConstraint: "CONTAINS"
                  }
                },
                {
                  notStatement: {
                    statements: [{
                      byteMatchStatement: {
                        searchString: "GET",
                        fieldToMatch: {
                          method: {}
                        },
                        textTransformations: [
                          {
                            priority: 0,
                            type: "NONE"
                          }
                        ],
                        positionalConstraint: "EXACTLY"
                      }
                    }]
                  }
                }
              ]
            }
          }
        }
      },
      action: {
        block: {
          customResponse: {
            responseCode: 429,
              //customResponseBodyKey: "rate-limit-response"
          }
        }
      },
      visibilityConfig: {
        sampledRequestsEnabled: true,
        cloudwatchMetricsEnabled: true,
        metricName: "account-creation-rate-limit"
      }
    }
  ],
  visibilityConfig: { sampledRequestsEnabled: true, cloudwatchMetricsEnabled: true, metricName: 'firewall' },
    customResponseBodies: [{
        key: `firewall-rate-limit-response`,
        content: 'Too many requests, please try again later.',
        contentType: 'TEXT_PLAIN',
    }],
  scope: 'REGIONAL',
});

@mnlumi mnlumi added the customer/feedback Feedback from customers label Oct 3, 2023
@t0yv0
Copy link
Member

t0yv0 commented Oct 3, 2023

I've tried harder here testing a few upgrade scenarios along these lines and tracing the place where this panic is received, but still wasn't able to reproduce.

pulumi destroy --yes

npm i @pulumi/[email protected] # tested 5.29.0
pulumi up --yes --skip-preview

npm i @pulumi/[email protected]
pulumi up --yes --skip-preview

Verbose logs or a repro would be extremely valuable here.

@DanielSchiavini
Copy link
Author

Sorry, I also cannot reproduce the issue anymore. Thanks for all your effort.

@mikhailshilkov mikhailshilkov added the resolution/no-repro This issue wasn't able to be reproduced label Oct 6, 2023
@mikhailshilkov
Copy link
Member

Closing for now... please let us know if the issue still occurs.

@mikhailshilkov mikhailshilkov closed this as not planned Won't fix, can't repro, duplicate, stale Oct 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
customer/feedback Feedback from customers impact/panic This bug represents a panic or unexpected crash kind/bug Some behavior is incorrect or out of spec p1 A bug severe enough to be the next item assigned to an engineer resolution/no-repro This issue wasn't able to be reproduced
Projects
None yet
Development

No branches or pull requests

5 participants