How to update the Assisted Installer to latest version

[gojeaqui]

Bug description

When trying to deploy the cluster (Openshift version 4.13.10) with the deploy_cluster.yml playbook I'm getting the error (in log output)

I think this is because the ai_version: v2.12.1 is from one year ago and Openshift 4.13 was GA in may this year.

I tried to update to the latest in quay.io: v.26.1, but the playbooks keep using the same version, I don't know how to update the version now that the prerequisites have been deployed.

OpenShift version

other (provide in the description)

Assisted Installer version

other (provide in the description)

Relevant log output

TASK [create_cluster : Create cluster] *******************************************************************************************************************************************************
task path: /home/core/crucible/roles/create_cluster/tasks/main.yml:75
<192.168.207.9> ESTABLISH LOCAL CONNECTION FOR USER: core
<192.168.207.9> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /tmp `"&& mkdir "` echo /tmp/ansible-tmp-1699640367.79549-80093-17192045367958 `" && echo ansible-tmp-1699640367.79549-80093-17192045367958="` echo /tmp/ansible-tmp-1699640367.79549-80093-17192045367958 `" ) && sleep 0'
Using module file /usr/lib/python3.6/site-packages/ansible/modules/net_tools/basics/uri.py
<192.168.207.9> PUT /home/core/.ansible/tmp/ansible-local-78920z_gstkyd/tmpv728xz0m TO /tmp/ansible-tmp-1699640367.79549-80093-17192045367958/AnsiballZ_uri.py
<192.168.207.9> EXEC /bin/sh -c 'chmod u+x /tmp/ansible-tmp-1699640367.79549-80093-17192045367958/ /tmp/ansible-tmp-1699640367.79549-80093-17192045367958/AnsiballZ_uri.py && sleep 0'
<192.168.207.9> EXEC /bin/sh -c '/usr/libexec/platform-python /tmp/ansible-tmp-1699640367.79549-80093-17192045367958/AnsiballZ_uri.py && sleep 0'
<192.168.207.9> EXEC /bin/sh -c 'rm -f -r /tmp/ansible-tmp-1699640367.79549-80093-17192045367958/ > /dev/null 2>&1 && sleep 0'
fatal: [bastion]: FAILED! => changed=false
  connection: close
  content: |-
    {"code":"400","href":"","id":400,"kind":"Error","reason":"Openshift version 4.13 for CPU architecture x86_64 is not supported: The requested CPU architecture (x86_64) isn't specified in release images list"}
  content_length: '208'
  content_type: application/json
  date: Fri, 10 Nov 2023 18:19:28 GMT
  elapsed: 0
  invocation:
    module_args:
      attributes: null
      backup: null
      body: '{"name": "cloud2", "high_availability_mode": "Full", "openshift_version": "4.13", "base_dns_domain": "interbanking.com.ar", "cluster_network_cidr": "10.104.0.0/14", "cluster_network_host_prefix": 23, "service_network_cidr": "10.151.0.0/16", "pull_secret": "{\"auths\": {\"cloud.openshift.com\": {\"auth\": \"b3BlbnNoaWZ0LXJlbGVhc2UtZGV2K29jbV9hY2Nlc3NfNDAwYjU0NjJiYTRkNGYzMGFjMWIwMjQxNTA2MDk3ZTU6REFQSE9GN1lUMTNDU1JSOFJDRU5NVDBXWUgyWE05R0xNWlNROVFTTTNXTjEwQk43VkVXRjZJUkpFMFdJWklMQw==\", \"email\": \"[email protected]\"}, \"quay.io\": {\"auth\": \"b3BlbnNoaWZ0LXJlbGVhc2UtZGV2K29jbV9hY2Nlc3NfNDAwYjU0NjJiYTRkNGYzMGFjMWIwMjQxNTA2MDk3ZTU6REFQSE9GN1lUMTNDU1JSOFJDRU5NVDBXWUgyWE05R0xNWlNROVFTTTNXTjEwQk43VkVXRjZJUkpFMFdJWklMQw==\", \"email\": \"[email protected]\"}, \"registry.connect.redhat.com\": {\"auth\": \"fHVoYy1wb29sLTI3ZDFhODczLTIzZWEtNGYxMi1iOGQ4LWNjNmYyODQ2NjQyYjpleUpoYkdjaU9pSlNVelV4TWlKOS5leUp6ZFdJaU9pSTJaakEyWkdFd1lqSTRaalEwTnpaaE9UZ3dNbUZoTm1SaVptUmlaRE0wWVNKOS5vZ0pvOVE5M3g1TDV5SHV4UlNfYzVwUko0SlZaYktabXhLSkN0V0c4b2ZlQmhEM0ZsQjFjOXhFbzg1V0JMLUtJbFhjTzBWQ0Fac2pPSWR0UjUzOW1NRGNNSUg4elE2UnRNQ0p4Z2NQdU4xN3dNcVlZR0tzY3VwR09tQnlIaWx1MEFDaXhKbDdUZnFpZC1YMDZER3F4VWY5b2JOQ0ZHcUpSc0pnOHkyM2RQMDM1X2dFRnpGYmstWmZoTGRLUGhqaHg0d1pkVjVnTDl0RENRQ3NHSUlCZVJTTmRXaS1YTzcyVnYzQWZpTTEwSjY2TXlrVndPTE5IODc1ZWYySXRHSUo0SEpJZ2lxNGktLVRRenZOYkxURHlkb1FWTWxMMTlnMXRkeFRMMm5fMlZfMm9LWkhOczY1aHhpdjNxNFFqXzlhMjU2dmRFODk1YmZwZUgzM2hWQzRfM0ktTno3aHRLdUVDc0hkQk9wTkNZU09zX2tqZEVqeVk5bVBfLUFvTXhlcWVOY202WTdLZThEQ2djSTFOQUVtRGgxdUlmNkNCdGdTTEJoblAzeTlwTnl3MlRrV21PQUFGbTFVTlZETWVndUpIdVVQTkpyZElEc2dDRVgxZ29rMWlWX2tGZVVRV1N3ZF9hWE1zYWw5c3paaW9LTEZsLUZhdTFZX1gtNWpab21US2k4YkpjZENSa1Y5aXFoNzV1YUZvbW9PM3doLUhURmo1cERiTHZQb1k2Yzd6cDJxVV9OVmY1ZXpUeHpMdXoxZGlBcERSck9MWWxhVmpDT3RoSzkyXzhaMWItWFVCYUFKeU5heDg2SmdLdFUtbUp1NFZJb294cDNzRDdyM2xlRDB2M1djTlFHQ2RxdWZ6eDdsV0pGSlNSZktBYkk3dmlfandraERwZTNKekpDbw==\",
        \"email\": \"[email protected]\"}, \"registry.redhat.io\": {\"auth\": \"fHVoYy1wb29sLTI3ZDFhODczLTIzZWEtNGYxMi1iOGQ4LWNjNmYyODQ2NjQyYjpleUpoYkdjaU9pSlNVelV4TWlKOS5leUp6ZFdJaU9pSTJaakEyWkdFd1lqSTRaalEwTnpaaE9UZ3dNbUZoTm1SaVptUmlaRE0wWVNKOS5vZ0pvOVE5M3g1TDV5SHV4UlNfYzVwUko0SlZaYktabXhLSkN0V0c4b2ZlQmhEM0ZsQjFjOXhFbzg1V0JMLUtJbFhjTzBWQ0Fac2pPSWR0UjUzOW1NRGNNSUg4elE2UnRNQ0p4Z2NQdU4xN3dNcVlZR0tzY3VwR09tQnlIaWx1MEFDaXhKbDdUZnFpZC1YMDZER3F4VWY5b2JOQ0ZHcUpSc0pnOHkyM2RQMDM1X2dFRnpGYmstWmZoTGRLUGhqaHg0d1pkVjVnTDl0RENRQ3NHSUlCZVJTTmRXaS1YTzcyVnYzQWZpTTEwSjY2TXlrVndPTE5IODc1ZWYySXRHSUo0SEpJZ2lxNGktLVRRenZOYkxURHlkb1FWTWxMMTlnMXRkeFRMMm5fMlZfMm9LWkhOczY1aHhpdjNxNFFqXzlhMjU2dmRFODk1YmZwZUgzM2hWQzRfM0ktTno3aHRLdUVDc0hkQk9wTkNZU09zX2tqZEVqeVk5bVBfLUFvTXhlcWVOY202WTdLZThEQ2djSTFOQUVtRGgxdUlmNkNCdGdTTEJoblAzeTlwTnl3MlRrV21PQUFGbTFVTlZETWVndUpIdVVQTkpyZElEc2dDRVgxZ29rMWlWX2tGZVVRV1N3ZF9hWE1zYWw5c3paaW9LTEZsLUZhdTFZX1gtNWpab21US2k4YkpjZENSa1Y5aXFoNzV1YUZvbW9PM3doLUhURmo1cERiTHZQb1k2Yzd6cDJxVV9OVmY1ZXpUeHpMdXoxZGlBcERSck9MWWxhVmpDT3RoSzkyXzhaMWItWFVCYUFKeU5heDg2SmdLdFUtbUp1NFZJb294cDNzRDdyM2xlRDB2M1djTlFHQ2RxdWZ6eDdsV0pGSlNSZktBYkk3dmlfandraERwZTNKekpDbw==\",
        \"email\": \"[email protected]\"}}}", "ssh_public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBu9gTr+d3r4+8Q5GHs5qY+Rt9kyX4iqIQNqsNz97npt core@ibocpbastion", "vip_dhcp_allocation": false, "http_proxy": "http://10.205.28.20:3128", "https_proxy": "http://10.205.28.20:3128", "no_proxy": "", "additional_ntp_source": "192.168.251.14", "api_vip": "192.168.207.10", "ingress_vip": "192.168.207.11", "network_type": "OVNKubernetes"}'
      body_format: json
      client_cert: null
      client_key: null
      content: null
      creates: null
      delimiter: null
      dest: null
      directory_mode: null
      follow: false
      follow_redirects: safe
      force: false
      force_basic_auth: false
      group: null
      headers:
        Content-Type: application/json
      http_agent: ansible-httpget
      method: POST
      mode: null
      owner: null
      regexp: null
      remote_src: null
      removes: null
      return_content: true
      selevel: null
      serole: null
      setype: null
      seuser: null
      src: null
      status_code:
      - 201
      timeout: 30
      unix_socket: null
      unsafe_writes: false
      url: http://192.168.207.9:8090/api/assisted-install/v2/clusters
      url_password: VALUE_SPECIFIED_IN_NO_LOG_PARAMETER
      url_username: VALUE_SPECIFIED_IN_NO_LOG_PARAMETER
      use_proxy: true
      validate_certs: true
  json:
    code: '400'
    href: ''
    id: 400
    kind: Error
    reason: 'Openshift version 4.13 for CPU architecture x86_64 is not supported: The requested CPU architecture (x86_64) isn''t specified in release images list'
  msg: 'Status code was 400 and not [201]: HTTP Error 400: Bad Request'
  redirected: false
  status: 400
  url: http://192.168.207.9:8090/api/assisted-install/v2/clusters
  vary: Accept-Encoding

Inventory file

No response

Required statements

• I have removed all sensitive details from the attached logs and inventory files.
• I acknowledge that Red Hat does not provide commercial support for the content of this repository.
• I acknowledge that any assistance is offered purely on a best-effort basis, as resource permits.

[nocturnalastro]

If you want to update the assisted installer version you can set ai_version in the all section of your inventory.

This being said to updated the OCP versions you need these three variables.
os_images
release_images
supported_ocp_versions

The relation between Z version of os_image and release_image. This can be a useful reference.

[gojeaqui]

Hello, unfortunately that is not enough.

There are things that changed between versions and it is not just changing the versions.

For example the people of nmstate.io remove the word "slave" and are using the word "port" for the bonding so that needs to change in the inventory and also in the templates. (that change is easy though)

Another thing is that something is different with regards to how Assisted Installer works, because the new version does not have the ISO where the playbook expects it. In fact the README from (https://github.com/openshift/assisted-image-service) says: None of these APIs should be considered stable for end-users of assisted installer. Users should never construct URLs that match these APIs; instead users should obtain an ISO URL from an InfraEnv resource, as provided by assisted installer. These APIs represent a contract between assisted-image-service and assisted installer only.

It might be that they already have changed those URLs.

[nocturnalastro]

There is a fix for the nmstate issue, I haven't had chance to test it. #281

I tested v2.26.0 the other day and it worked for a fairly basic flow. We use the download_url returned by the api to fetch the iso and it doesn't look like the have removed that based on the swagger. Do you have the error you saw?

It might be worth using the agent based install method use_agent_based_installer: True which extracts the openshift installer and uses the agent subcommand.

[gojeaqui]

Do you have the error you saw?

No, I'm sorry. I should have kept it but it was lost.

The issue was in the task:

TASK [generate_discovery_iso : Download discovery ISO] 
  status_code: 403
  uid: 0
  url: http://192.168.207.9:8888/images/9a63de32-4892-472a-bd44-90d4c02d03e5?arch=x86_64&type=full-iso&version=4.12

(something like this)

And the problem was that in the URL there was nothing, no ISO.

When I went back the the current version of Assisted Installed the issue dissapeared.

[gojeaqui]

I found this in my logs, but it's not the result of the ansible task:

[core@xx crucible]$ curl -vk  http://192.168.xx.9:8888/byid/a54181f5-53b6-4411-bfaf-9e9b098f0eca
* Uses proxy env variable no_proxy == 'localhost,127.0.0.1,192.168.x.9,192.168.x.9'
*   Trying 192.168.207.9...
* TCP_NODELAY set
* Connected to 192.168.207.9 (192.168.207.9) port 8888 (#0)
> GET /byid/a54181f5-53b6-4411-bfaf-9e9b098f0eca HTTP/1.1
> Host: 192.168.207.9:8888
> User-Agent: curl/7.61.1
> Accept: */*
> 
< HTTP/1.1 404 Not Found
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Fri, 10 Nov 2023 22:24:36 GMT
< Content-Length: 19
< 404 page not found