Build system & cargo manifest

[o-smirnov]

Continuing the discussion from #637.

@SpheMakh I pushed some initial revisions to configuratt, see 2021dc5. Quoting the commit message:

• load image and cab info from yaml files at startup using OmegaConf
• add stimela.config module, a local config file (~/.config/stimela.conf), and "stimela config" subcommand
• refactor individual sub-commands into stimela.commands.*
• use standard ArgumentParser facilities for sub-command parsers
• refactor contaioner backends into stimela.backends.*
• reimplement "stimela images" to interrogate docker directly for available images
• reimplement "stimela build", using the new OmegaConf system to get build info
• implement "stimela push"

Please take a look before too long so that our thinking stays in sync. Note that this pretty much implements #692, albeit only for docker for now. In particular, stimela images, build and push now works with the new config.

[o-smirnov]

I realized that if you ship multiple images like so:

name: shadems

images:   # image versions (tags) to be built
  0.5.0:  
    dockerfile: Dockerfile       
  0.4.0:  # key is version/tag
    dockerfile: Dockerfile.v040     

...then you need an obvious mechanism to select a default image, when the recipe does not specify a version number. The obvious suggestion is to select the highest version. (Comparing semvers properly: https://stackoverflow.com/questions/11887762/how-do-i-compare-version-numbers-in-python/21065570)

Then, in development I often find myself building "test" images with pre-release or branch versions of packages, we don't want those ever selected by default. I have added a production option (default Trrue). So:

images:   # image versions (tags) to be built
  0.6.test:  
    dockerfile: Dockerfile.v06test
    production: false       
  0.5.0:  
    dockerfile: Dockerfile       
  0.4.0:  # key is version/tag
    dockerfile: Dockerfile.v040     

The default image version should then be the highest-numbered production version. NB: need to modify stimela build to add a "production" label to the image.

[SpheMakh]

Why load all the configs here, and not on the fly as they are needed?

https://github.com/ratt-ru/Stimela/blob/b7eb63632339e3c35fa2fe5ed4d7cc77ed721670/stimela/config.py#L117-L123

[o-smirnov]

Well, it's the simplest thing to do, and it ensures any schema compliance errors are caught up front (as opposed to post-release, when remote user X has tried to use obscure cab Y.) So why not? If we find it becomes costly once every cab has a config, then let's think about on-demand loading...

[o-smirnov]

If we find it becomes costly once every cab has a config, then let's think about on-demand loading...

Hmmm, with the current (2.0.6) release of OmegaConf, it was taking ~10s to load the two full cab definitions that you checked in. However, when I installed it from master, things got a lot faster, and load times dropped to <1s. I think omry/omegaconf#477 was the fix. So install from master for now, at least until the next release.

If gets too expensive once we load all the cabs, we can think about caching this stuff -- it is pretty static, after all.

[omry]

You can also depend on the pre-release packages, latest is omegaconf==2.1.0dev20.

[o-smirnov]

Thanks, @omry. We're moving slower than you, so I'm pretty sure 2.1 will be out long before we're ready to release.