Add validation for paths in config

[wjt]

Previously, if a filename referred to in the configuration does not exist, that would only be discovered when the hook which uses that key runs, which may be minutes or hours into the build.

Add a way to specify in the schema that certain keys must be set to a single path to an extant file, or a list of paths to extant files.

Add key_type = path(s) annotations to the schema for all (I think) of the file-like keys used by hooks in this repo.

https://phabricator.endlessm.com/T35517

[wjt]

I should be clear that this is only very lightly tested. I aimed to spend no more time on it than the image builder spent on a doomed image build overnight :)

[wjt]

So lightly tested that it fails its tests!

[wjt]

Oh I guess I can't use match unless we bump our minimum Python version to 3.10 (i.e. Endless OS 6 is required).

@dbnicholson I'm curious what you think about this approach, before I mess with going back to if/elif etc.

Of course this wouldn't catch the error at PR time – I think the way to do that would be to have a list of known combinations of fields and have a GitHub Actions job which validates the config for all those. (And a similar job in the private config repo which is where the error occurred.) My old idea there is to again abuse the schema. Add the following to the config:

[build]
configtuple = ${product}-${arch}-${platform}-${personality}

and then add to the schema:

[build]
configtuple_values = eos-amd64-amd64-base
                     eos-amd64-amd64-en
                     # ... etc

[dbnicholson]

Oh I guess I can't use match unless we bump our minimum Python version to 3.10 (i.e. Endless OS 6 is required).

@dbnicholson I'm curious what you think about this approach, before I mess with going back to if/elif etc.

Sorry, I meant to reply to this long ago. I think it's a nice idea, and I would switch away from match since the intention of this builder is that you can run it on basically any host that has python3 and ostree. Limiting that to only Python >= 3.10 would be a little limiting, although less so as time marches on. You wouldn't be able to run it on Debian bullseye (oldstable) or Ubuntu focal (20.04LTS).

Of course this wouldn't catch the error at PR time – I think the way to do that would be to have a list of known combinations of fields and have a GitHub Actions job which validates the config for all those. (And a similar job in the private config repo which is where the error occurred.) My old idea there is to again abuse the schema. Add the following to the config:

[build]
configtuple = ${product}-${arch}-${platform}-${personality}

and then add to the schema:

[build]
configtuple_values = eos-amd64-amd64-base
                     eos-amd64-amd64-en
                     # ... etc

Generally I like this idea, but we'd only be able to manage the allowed values in our private config repo. It would be artificially limiting to 3rd parties to constrain those fields in this repo. Oh, it does look like I added the ability to read the schema from the local directory in 3e03f8a, so that's good. Unfortunately, it doesn't merge values, so any schema value we put in this repo would have to be duplicated.

It might be nice to be able to use wildcards to you could have an allowed value like eoscustom-amd64-amd64-*, but that's probably a whole can of worms. For our purposes, a strict list of allowed values in our private repo would be enough.

[wjt]

I think wildcards would be useful even in the private repo. We have often made an identical copy of an existing personality with a project-specific suffix just to give it a distinctive name for our activation/metrics data. With the changes on #169 we'd no longer need to change anything in the repo to build an identical-in-all-but-name derived image. It would be sad to have to update the schema each time...

I'll see if I can find time this week to revive this one.

[wjt]

Revised this to use more traditional Python syntax and add a test. I didn't add any of the config tuple validation stuff described above.

[dbnicholson]

👍