Terraform deploy of Ubuntu 22.04.3 LTS customised with cloud-init resets to DHCP after reboot.

[mrdanielmartins]

Community Guidelines

• I have read and agree to the HashiCorp Community Guidelines .
• Vote on this issue by adding a 👍 reaction to the original issue initial description to help the maintainers prioritize.
• Do not leave "+1" or other comments that do not add relevant information or questions.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Terraform

1.6.1

Terraform Provider

2.5.1

VMware vSphere

7.0.3o

Description

Excuse me, I've tried various bets across the internet, from blogs to VMware docs, but I'm still far from solving this issue I'm facing.

Overview

Ubuntu VM has cloud-init enabled, and Terraform assigns static IP. Upon reboot, the VM loses its static IP configuration.

What is strange is the static IP customisation seems to be configured on the 50-cloud-init.yaml, which, of course, does not persist after a reboot.

Please do pardon me if this is not the correct place to raise the issue I'm facing!

What I've tried as part of the VM template

• Remove subiquity-disable-cloudinit-networking.cfg
• Remove etc/cloud/cloud.cfg.d/99-installer.cfg
• Add disable_vmware_customization: false to /etc/cloud/cloud.cfg
• Add to datasource_list the values of OVF and VMware

Cloud-init version for the VM is cloud-init 23.3.1-0ubuntu1~22.04.1.

Am I going entirely about this incorrectly, and this is expected behaviour?

Affected Resources or Data Sources

resource vsphere_virtual_machine

Terraform Configuration

  clone {
    template_uuid = data.vsphere_virtual_machine.template.id

    customize {
      timeout = 15
      linux_options {
        host_name = each.value.host_name
        domain    = var.domain
        time_zone = var.time_zone
      }

      dynamic "network_interface" {
        for_each = { for index, network in each.value.interfaces : index => network }

        content {
          ipv4_address = network_interface.value.ipv4_address != "" ? network_interface.value.ipv4_address : null
          ipv4_netmask = network_interface.value.ipv4_netmask != "" ? network_interface.value.ipv4_netmask : null
        }
      }

      ipv4_gateway    = each.value.ipv4_gateway != "" ? each.value.ipv4_gateway : null # NOTE - TF vsphere_virtual_machine resources has no per-interface gateway setting.
      dns_server_list = length(var.dns_server_list) != 0 ? var.dns_server_list : null
      dns_suffix_list = length(var.dns_suffix_list) != 0 ? var.dns_suffix_list : null
    }
  }

Debug Output

I can provide a debug later.

Panic Output

No response

Expected Behavior

Static IP to persist after reboot.

Actual Behavior

Static IP resets to DHCP.

Steps to Reproduce

Terraform apply on a custom Packer VM Ubuntu image.

Environment Details

No response

Screenshots

No response

References

No response

[github-actions]

Hello, mrdanielmartins! 🖐

Thank you for submitting an issue for this provider. The issue will now enter into the issue lifecycle.

If you want to contribute to this project, please review the contributing guidelines and information on submitting pull requests.

[tenthirtyam]

Your configuration look correct.

It's the same that I apply to Ubuntu with Ansible during the Packer build in Packer Examples for VMware vSphere project.

rm -rf /etc/cloud/cloud.cfg.d/subiquity-disable-cloudinit-networking.cfg
rm -rf /etc/cloud/cloud.cfg.d/99-installer.cfg
rm -rf /etc/netplan/00-installer-config.yaml
echo "disable_vmware_customization: false" >> /etc/cloud/cloud.cfg
echo "datasource_list: [ VMware, OVF, None ]" > /etc/cloud/cloud.cfg.d/90_dpkg.cfg

https://github.com/vmware-samples/packer-examples-for-vsphere/blob/613709506e443b9b6285da197fcdabc68ed6bceb/ansible/roles/configure/tasks/ubuntu.yml#L17C1-L21C87

Once you do that, you will customize with cloud-init using extra_config - not VMware Tools.

resource "vsphere_virtual_machine" "vm" {
  name                    = var.vm_name
  folder                  = var.vsphere_folder
  num_cpus                = var.vm_cpus
  memory                  = var.vm_memory
  firmware                = var.vm_firmware
  efi_secure_boot_enabled = var.vm_efi_secure_boot_enabled
  guest_id                = data.vsphere_virtual_machine.template.guest_id
  datastore_id            = data.vsphere_datastore.datastore.id
  resource_pool_id        = data.vsphere_resource_pool.pool.id
  network_interface {
    network_id = data.vsphere_network.network.id
  }
  disk {
    label            = "disk0"
    size             = data.vsphere_virtual_machine.template.disks.0.size
    eagerly_scrub    = data.vsphere_virtual_machine.template.disks.0.eagerly_scrub
    thin_provisioned = data.vsphere_virtual_machine.template.disks.0.thin_provisioned
  }
  clone {
    template_uuid = data.vsphere_virtual_machine.template.id
  }
  lifecycle {
    ignore_changes = [
      clone[0].template_uuid,
    ]
  }
  extra_config = {
      "guestinfo.metadata"          = base64encode(file("${path.module}/metadata.yml"))
      "guestinfo.metadata.encoding" = "base64"
      "guestinfo.userdata"          = base64encode(file("${path.module}/userdata.yml"))
      "guestinfo.userdata.encoding" = "base64"
  }
}

Example: tenthirtyam/terrafom-examples-vmware

[mrdanielmartins]

Good morning @tenthirtyam,

That repo is amazing!

I'll give the extra_config a go referencing your Terraform example.

Sure the VMware bit makes sense. I was using this VMware KB 90331 which lists a solution using Perl and cloud-init together. However, I've just read through it again, and it specifies only that user-data (which I wasn't doing in the first place) is to be provided to cloud-init. I'll also give this a go now that I've got your illustration with the exta_config block.

Thank you

Daniel

[perrfect]

Your configuration look correct.

It's the same that I apply to Ubuntu with Ansible during the Packer build in Packer Examples for VMware vSphere project.

rm -rf /etc/cloud/cloud.cfg.d/subiquity-disable-cloudinit-networking.cfg
rm -rf /etc/cloud/cloud.cfg.d/99-installer.cfg
rm -rf /etc/netplan/00-installer-config.yaml
echo "disable_vmware_customization: false" >> /etc/cloud/cloud.cfg
echo "datasource_list: [ VMware, OVF, None ]" > /etc/cloud/cloud.cfg.d/90_dpkg.cfg

https://github.com/vmware-samples/packer-examples-for-vsphere/blob/613709506e443b9b6285da197fcdabc68ed6bceb/ansible/roles/configure/tasks/ubuntu.yml#L17C1-L21C87

Once you do that, you will customize with cloud-init using extra_config - not VMware Tools.

resource "vsphere_virtual_machine" "vm" {
  name                    = var.vm_name
  folder                  = var.vsphere_folder
  num_cpus                = var.vm_cpus
  memory                  = var.vm_memory
  firmware                = var.vm_firmware
  efi_secure_boot_enabled = var.vm_efi_secure_boot_enabled
  guest_id                = data.vsphere_virtual_machine.template.guest_id
  datastore_id            = data.vsphere_datastore.datastore.id
  resource_pool_id        = data.vsphere_resource_pool.pool.id
  network_interface {
    network_id = data.vsphere_network.network.id
  }
  disk {
    label            = "disk0"
    size             = data.vsphere_virtual_machine.template.disks.0.size
    eagerly_scrub    = data.vsphere_virtual_machine.template.disks.0.eagerly_scrub
    thin_provisioned = data.vsphere_virtual_machine.template.disks.0.thin_provisioned
  }
  clone {
    template_uuid = data.vsphere_virtual_machine.template.id
  }
  lifecycle {
    ignore_changes = [
      clone[0].template_uuid,
    ]
  }
  extra_config = {
      "guestinfo.metadata"          = base64encode(file("${path.module}/metadata.yml"))
      "guestinfo.metadata.encoding" = "base64"
      "guestinfo.userdata"          = base64encode(file("${path.module}/userdata.yml"))
      "guestinfo.userdata.encoding" = "base64"
  }
}

Example: tenthirtyam/terrafom-examples-vmware

Hello. I have the same issue, but your fix doesn't work in full.
I'm cloning an Ubuntu 22.04 template to a new virtual machine, but I don't have resolv on it, DNS doesn't work.

I run the next commands:

rm -rf /etc/cloud/cloud.cfg.d/subiquity-disable-cloudinit-networking.cfg
rm -rf /etc/netplan/00-installer-config.yaml
echo "disable_vmware_customization: false" >> /etc/cloud/cloud.cfg
echo "datasource_list: [ VMware, OVF, None ]" > /etc/cloud/cloud.cfg.d/90_dpkg.cfg

Don't run the command rm -rf /etc/cloud/cloud.cfg.d/99-installer.cfg because this part from the terraform clone works correctly.

      linux_options {
        host_name         = var.vm_template_clone_customize_host_name
        domain		   = var.vm_template_clone_customize_domain_name
      } 

Now only the problem with DNS. Could you please help me fix it?

My full the clone part:

  clone {
    template_uuid         = data.vsphere_virtual_machine.template.id
    customize {
      linux_options {
        host_name         = var.vm_template_clone_customize_host_name
        domain		  = var.vm_template_clone_customize_domain_name
      }
      dynamic "network_interface" {
        for_each = var.vm_vsphere_networks
        content {
          ipv4_address	  = network_interface.value.ipv4_address
          ipv4_netmask	  = network_interface.value.ipv4_netmask
          dns_server_list = network_interface.value.dns_server_list
        }
      }
    ipv4_gateway          = var.vm_template_network_interface_ip_gateway
    }
  }

[tenthirtyam]

You should be customizing with cloud-init using extra_config,
otherwise it's using guest tools for customization; hence the issue.

See https://github.com/tenthirtyam/terrafom-examples-vmware/tree/main/vsphere/vsphere-virtual-machine/clone-template-linux-cloud-init for an example using cloud-init.

If you don't want to use cloud-init you can simply remove it with apt and not make any of the changes above.

[perrfect]

You should be customizing with cloud-init using extra_config, otherwise it's using guest tools for customization; hence the issue.

Hmm...I'm using the terraform for Rocky Linux and Ubuntu. For Rocky works fine.
I will try using extra_config, but in this case I should change terraform code for Rocky too.

[perrfect]

@tenthirtyam And should I change my extra_config all time for each VM if I have static IP for each?

[tenthirtyam]

For now, yes. There is an open issue for extra_config not working with dynamic blocks.

[perrfect]

@tenthirtyam let me know when the problem is solved please

[tenthirtyam]

As a commenter, you will
be notified when the issue closes unless you unsubscribe from the issue.

[perrfect]

@tenthirtyam
Hello. I tried do the next configuration:

main.tf

resource "vsphere_virtual_machine" "vm" {
  name                    = var.vm_name
  resource_pool_id        = data.vsphere_host.host.resource_pool_id
  host_system_id          = data.vsphere_host.host.id
  datastore_id            = data.vsphere_datastore.datastore.id
  wait_for_guest_net_timeout = var.vm_wait_for_guest_net_timeout
  wait_for_guest_ip_timeout  = var.vm_wait_for_guest_ip_timeout
  num_cpus                = var.vm_cpu_numbers
  memory                  = var.vm_memory
  firmware		            = var.vm_firmware
  guest_id                = var.vm_guest_id
  network_interface {
    network_id = data.vsphere_network.network.id
  }
  disk {
    label                 = var.vm_disk_label
    size                  = var.vm_disk_size
  }
  clone {
    template_uuid         = data.vsphere_virtual_machine.template.id
  }
  extra_config = {
    "guestinfo.metadata"         = base64encode(templatefile("${path.module}/cloudinit/metadata.yml", {
      interface   = var.interface
      dhcp        = var.dhcp
      hostname    = var.vm_hostname_ubuntu
      ip_address  = var.ip_address
      netmask     = var.netmask
      nameservers = jsonencode(var.nameservers)
      gateway     = var.gateway
    }))
    "guestinfo.metadata.encoding" = "base64"
  }
}

matadata.yml

local-hostname: ${hostname}
instance-id: ${hostname}
network:
  version: 2
  ethernets:
    ${interface}:
      %{ if dhcp == "true" }dhcp4: true
      %{ else }addresses:
        - ${ip_address}/${netmask}
      gateway4: ${gateway}
      nameservers:
        addresses: ${nameservers}
      %{ endif }

It works fine and I can set up different network settings for each virtual machine.

module "test-03" {
    source = "../modules/vms"
    
    vsphere_datacenter_name       = var.vsphere_datacenter_name
    vsphere_host_name             = var.vsphere_host_name
    vsphere_datastore_name        = var.vsphere_datastore_name
    vm_name                       = "test-03"
    vm_wait_for_guest_net_timeout = "0"
    vm_wait_for_guest_ip_timeout  = "0"
    vm_cpu_numbers                = "2"
    vm_memory                     = "2048"
    vm_firmware                   = "efi"
    vm_guest_id                   = "ubuntu64Guest"
    vm_disk_label                 = "disk0"
    vm_disk_size                  = "20"
    vsphere_template_name         = "template_ubuntu"
    interface                     = "ens192"
    dhcp                          =  false
    vm_hostname_ubuntu            = "test-03"
    vm_vsphere_networks_ubuntu    = "test-03"
    ip_address                    = "172.16.1.100"
    netmask                       = "24"
    gateway                       = "172.16.1.1"
    nameservers                   = ["172.16.1.1", "1.1.1.1"]
}

But I have to add additional routes to VMs. For that I add to metadata.yml file the next lines:

      routes:
        %{ for route in routes_list ~}
        - to: ${route.to}
          via: ${route.via}
        %{ endfor }
      %{ endif }

To main.tf I add:
      add_routes  = var.add_routes
      routes_list = var.routes_list

  extra_config = {
    "guestinfo.metadata"         = base64encode(templatefile("${path.module}/cloudinit/metadata.yml", {
      interface   = var.interface
      dhcp        = var.dhcp
      add_routes  = var.add_routes
      routes_list = var.routes_list
      hostname    = var.vm_hostname_ubuntu
      ip_address  = var.ip_address
      netmask     = var.netmask
      nameservers = jsonencode(var.nameservers)
      gateway     = var.gateway
    }))
    "guestinfo.metadata.encoding" = "base64"
  }

And to file with VM:

    add_routes                    =  true
    routes_list = [
      {
        to = "172.16.3.0/24"
        via = "172.16.1.3"
      },
      {
        to = "172.16.4.0/24"
        via = "172.16.1.4"
      },
      {
        to = "172.16.5.0/24"
        via = "172.16.1.5"
      }
    ]

And in this case, when I'm adding routes they are not transferable to VM and not working.
Where can be the problem?

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.