Enable remote tasks to be run in cluster

[arewm]

By default, we should run builds matching the local architecture in-cluster to reduce the overhead of provisioning platforms. This will enable a fully matrixed build for all images using only the remote builds. This change will require the multi-platform controller to set the /ssh/host to localhost in order for the builds to run in-cluster.

In a change from the prior behavior of auto-appending the architecture to the tag, we will now append a sanitized version of the entire PLATFORM to the image tag upon request. This behavior is now needs to be explicitly requested which fixes a bug where users cannot specify the specific tag desired when using a remote build.

NOTE: There are not currently any local platforms configured. These need to be added before this PR have an effect in production.

[ifireball]

I found what seems to be one major issue where the task essentially becomes a noop in the local case.

Also since the buildah-remote task is now gaining local capabilities, it may be desirable to stop using the task-generator script and instead switch to maintaining the generated code directly. We can do this under the assumption that the remote-capable task will eventually replace the local-only task. Doing this will IMO simplify maintenance going forward.

[chmeliik]

Also since the buildah-remote task is now gaining local capabilities, it may be desirable to stop using the task-generator script and instead switch to maintaining the generated code directly.

+1, the task-generator is quickly becoming very difficult to understand (the diff here is kind of impossible to review)

Can we instead add the remote capabilities to regular buildah

• and make buildah-remote an alias for buildah (the same way as all the buildah-*gb variants, but with no patch.yaml)
• or deprecate buildah-remote altogether?

[arewm]

We certainly can consolidate the tasks. Changes like the one that I proposed in #1185 will become much simpler as well.

If we consolidate, however, we will have to ensure that we support builds when the multi-platform-controller is not enabled. I don't think that we will be able to consolidate before the controller supports local builds.

[chmeliik]

This change will require the multi-platform controller to set
the /ssh/host to localhost in order for the builds to run in-cluster.

Ah, so we can't really replace replace the regular buildah task with this one if we don't get rid of this requirement. The basic buildah use case shouldn't require multi-platform-controller

[chmeliik]

This change will require the multi-platform controller to set
the /ssh/host to localhost in order for the builds to run in-cluster.

Ah, so we can't really replace replace the regular buildah task with this one if we don't get rid of this requirement. The basic buildah use case shouldn't require multi-platform-controller

Another problem for unifying the tasks is that the user doesn't get to decide whether to build in-cluster or on a VM - the multi-platform-controller config does.

[chmeliik]

Another problem for unifying the tasks is that the user doesn't get to decide whether to build in-cluster or on a VM - the multi-platform-controller config does.

Now that I think about it, won't these changes together with https://github.com/redhat-appstudio/infra-deployments/pull/4329/files break the option to run x86 builds on a VM?

[chmeliik]

Looks reasonable, my main concern is if we're not removing the option to run x86 builds on a VM (#1216 (comment))

[arewm]

Another problem for unifying the tasks is that the user doesn't get to decide whether to build in-cluster or on a VM - the multi-platform-controller config does.

Now that I think about it, won't these changes together with https://github.com/redhat-appstudio/infra-deployments/pull/4329/files break the option to run x86 builds on a VM?

The user decides what platform they want to build by specifying the parameter. The configuration determines whether that should happen locally or on a VM if necessary. If it doesn't matter whether builds happen locally or remotely, then this implementation is left up to the system. One challenge for the host configs/MPC controller will be to identify the best way to migrate workloads from a VM-configured platform to a local-configured platform (or vice versa). I think that these challenges exist outside of the scope of the currently proposed changes to buildah-remote.

For now, there are only AMD nodes in clusters, but there might be ARM nodes on the cluster in the future. With the changes in redhat-appstudio/infra-deployments/pull/4329, users can still run x86 builds on a VM, they just need to select one of the configured non-local hosts (i.e. using any of the */amd64 platforms. There is a desire to move linux/amd64 to be run in-cluster but this will be a breaking change and we need to migrate any current users off of it onto the new linux-mlarge/amd64 platform first (this change will happen in the future if it happens at all).

Today, localhost can be easily used to indicate that you want to build in-cluster as I used in my tests. When multiple node platforms are available, we will have to figure out how to enable the local builds to be scheduled to the right nodes.

[chmeliik]

users can still run x86 builds on a VM, they just need to select one of the configured non-local hosts (i.e. using any of the */amd64 platforms

There is a desire to move linux/amd64 to be run in-cluster but this will be a breaking change and we need to migrate any current users off of it onto the new linux-mlarge/amd64 platform first

Ah right, it's amd64, not x86_64. local-platforms: "linux/x86_64, in that PR confused me. This will need good docs 😅 It's all opaque behavior of the system, not something easily understandable to a user. Also, how is linux-mlarge/amd64 a "platform"?

[ifireball]

@chmeliik

Another problem for unifying the tasks is that the user doesn't get to decide whether to build in-cluster or on a VM - the multi-platform-controller config does.

Now that I think about it, won't these changes together with https://github.com/redhat-appstudio/infra-deployments/pull/4329/files break the option to run x86 builds on a VM?

They still get to decide, by a platform name that is explicit about it such as linux-mlarge/amd64 as opposed to assuming linux/amd64 would end on a VM, which is something they shouldn't care about if they only want to build for a foreign platform as opposed to explicitly asking to be on a VM.