Add 'make airgap-package' target and associated scripts for bundling images for air gap

[squizzi]

This adds new scripts for creating a bundle for airgap customers. We can easily add a step to the release CI as well, but the resulting tarball of saved images is 4.2G so I decided to not include that for now since we'll need to figure out the best place to host the tarball.

At this time this script deploys a management cluster, waits for all the providers to come up and be ready then gets a list of all the images used and pulls/saves those.

It then iterates helm template ... output across each of the k0s extensions we deploy and pulls/saves images affiliated with the extensions. I've also updated helm-package to pull in these extensions during it's packaging process.

$ IMG=ghcr.io/mirantis/hmc/controller-ci:v0.0.1-287-gd9c38c3 make bundle-images
...
Done! Images bundled into /home/squizzi/go/src/github.com/Mirantis/hmc/bin/images/hmc-images-v0.0.2-14-gf02c79b.tgz

$ docker load < ./bin/images/hmc-images-v0.0.2-14-gf02c79b.tgz
...
49dc5e73a32a: Loading layer [==================================================>]  70.34MB/70.34MB
Loaded image: mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.30.4
c2453148319b: Loading layer [==================================================>]  85.39MB/85.39MB
Loaded image: quay.io/k0sproject/k0smotron:v1.0.4
4a2377d5d202: Loading layer [==================================================>]   1.69MB/1.69MB
d9486140fcec: Loading layer [==================================================>]  76.06MB/76.06MB
Loaded image: registry.k8s.io/provider-aws/cloud-controller-manager:v1.30.3
a7c367db8420: Loading layer [==================================================>]  61.15MB/61.15MB
Loaded image: registry.k8s.io/sig-storage/csi-snapshotter:v7.0.2
e4d8c7836c02: Loading layer [==================================================>]  73.34MB/73.34MB
...

Closes #423

[a13x5]

Several issues with make targets
The scripts should be revisited.

One general important note regarding bash syntax:

There is a convention that local scope variable names should be always lower snake case. Uppercase names are usually reserved only for env variables (in rare cases for global ones in scope of script)
Ditching this rule makes script harder to read and reason about. Please fix

[squizzi]

EXPORT_ALL_VARIABLES does not appear to play nicely with other Makefile targets and export does not appear to actually populate the variables so I unfortunately can't seem to make that work without just populating the vars prior to script execution. Not sure the best way to do this.

[a13x5]

EXPORT_ALL_VARIABLES does not appear to play nicely with other Makefile targets and export does not appear to actually populate the variables

export should work at least with GNU Make 4.4 (especially if .EXPORT_ALL_VARIABLES works) (see manual)

Anyway if it's became too complex I will not block this PR. Issues with Makefile will be fixed in scope of #589

A couple of small issues should be fixed in the script however (Note that NIT comment are optional to fix)

[squizzi]

Anyway if it's became too complex I will not block this PR. Issues with Makefile will be fixed in scope of #589

Ya, I'm not sure what's going on, I've configured the exports just as the docs suggest with:

export var = value

and they end up empty in the script invocation, debugged with set -x.

[squizzi]

I think I need to bring back the custom_tag logic, for example kube-vip when templated with our values produces the following error:

helm template --repo https://kube-vip.github.io/helm-charts kube-vip -f values.yaml
Error: execution error at (kube-vip/templates/daemonset.yaml:33:22): A valid config.address required!

It's because we're trying to remove the config.address entry prior to templatizing it. Sure we could work around this instance with something like --set config.address "12345" but do we potentially want to avoid custom logic like this? The custom_tag logic, although a little quirky didn't have to deal with these potential custom validations that chart maintainers may add since it templates the chart with default values.

[squizzi]

Found a bug in the airgap-push.sh script, it wasn't dealing with multiple tags of images well, that's been fixed.

[a13x5]

I tested the scripts on the airgapped ubuntu machine.
While testing I had multiple issues with airgapped k0s (some irrelevant to this PR).
I added several comments related to the issues I experienced.

Regarding the kube-vip issue. It's caused by the empty value we're providing via values, so it's not related to kube-vip chart itself. Thus I suggest adding a special case for it either adding --debug (so it will output some yaml) or by removing empty value from the values we're providing.
custom_tag logic also could be considered, but IMO it looks more flaky.

[squizzi]

Regarding the kube-vip issue. It's caused by the empty value we're providing via values, so it's not related to kube-vip chart itself. Thus I suggest adding a special case for it either adding --debug (so it will output some yaml) or by removing empty value from the values we're providing.

Outputting --debug if the helm template fails seems to do the trick for this instance.

[a13x5]

Issue with hmc-controller image
With --insecure-skip-tls-verify flag we're not covering http registries, but it's out of scope of this PR and should be covered in the docs.

[squizzi]

Looks like kube-vip still didn't want to render out, for now I've just added some logic to workaround this case, I kept the --debug logic in there though for other instances where that may be viable.

[squizzi]

I've squashed the changes down to prepare for merge/carrying of this PR since this is my last day at Mirantis and I'll be taking some time off for a bit.

[a13x5]

LGTM