Error initializing provisioner 'shell' on packer version 1.6.2 and 1.6.3 due to OOM

[ture-karlsson]

After upgrading packer from version 1.6.0 to 1.6.2, we started experiencing the following kind of errors in our builds:

error initializing provisioner 'shell': EOF

and

error initializing provisioner 'shell': timeout while waiting for plugin to start

We experience the same error after upgrading to 1.6.3, but when downgrading to 1.6.0, we can run our builds again without issues.

Are there any changes in versions later than 1.6.0 that can cause these issues?

[sylviamoss]

Hey there, thanks for reaching out! Could you provide us more information on what builder you're using and a simplified template and steps we can use to reproduce the issue?
Are you only experiencing issues with the shell provisioner? If so do you have any custom plugins in your plugin directory?

[ture-karlsson]

Hello!

We are using the google compute builder and running it in cloud build. We are only experiencing this with the shell provisioner and don't use any custom plugins.

Simplified template looks something like this:

{
  "builders": [
    {
      "type": "googlecompute",
      "name": "example-name",
      "project_id": "example-project",
      "image_name": "example-image",
      ...
    }
  ],
  "provisioners": [
    {
      "type": "shell",
      "script": "path/to/script.sh"
    }
  ]
}

I will try to come up with a reproducer.

[ture-karlsson]

Hello!

I have tried to reproduce the error without running our full build, but with no success. As I said, this is in GCP Cloud Build and we have 20+ steps running in parallel and the errors does not appear on the same step every time and does not appear when I run any of the steps individually, nor if I create a simple mock template like the one above and run it in 30 parallel steps.

Do you have any suggestions on how we can narrow this down?

[SwampDragons]

Hi, it sounds to me like you're hitting some kind of compute limit of the instance you're building on. Try splitting your Packer run across multiple cloud builds, or running on a more powerful instance. Here's some documentation about available instance types: https://cloud.google.com/compute/docs/machine-types#n1_machine_type

[ture-karlsson]

Yes, I think you are correct! Running on a bigger machine worked fine:

gcloud builds submit --config=config.yaml ---machine-type=n1-highcpu-32

Are there any changes between 1.6.0 and 1.6.2/1.6.3 that would make the builds require more resources?

[SwampDragons]

I'm definitely going to look into that based on this report.

[github-actions]

This issue has been synced to JIRA for planning.

JIRA ID: HPR-767