Command Generation Recipes

[dinatale2]

Within the various script adapters of maestrowf, there are command flags which are used to generate commands. Currently, in order to support various command line commands such as srun or lsf, a new script adapter would need to be written.

To simplify the process, why not introduce script adapter recipes? Using YAML, we can specify the command line arguments a new command expects. For instance, a slurm adapter would be:

Adapter:
    name: slurm
    cmd: srun
    arguments:
        depends : --dependency
        ntasks : -n
        nodes: -N

Then in the study spec, a user can specify that they would like to use the slurm adapter and use the keys provided in arguments to specify the value for ntasks, nodes, or depends.

This could even be expanded later to other parts of maestro (i.e. script header generation).

[FrankD412]

I like this idea. There's one thing about it though -- we'll need to specify the format that the arguments take. --dependency for example may support only a single dependency or multiple, and the adapter needs to know how to format it.

[FrankD412]

Another thought -- we also need a way to specify if an argument is required. That can be accomplished in the following way:

Adapter:
    name: slurm
    cmd: srun
    arguments:
        required:
            ntasks : -n
        optional:
            depends : --dependency
            nodes: -N

Further, we could clean it up by moving the adapter name to the key.

slurm
    cmd: srun
    arguments:
        required:
            ntasks : -n
        optional:
            depends : --dependency
            nodes: -N

We may be able to do something like this for headers also.

[FrankD412]

@dinatale2 -- I wanted to revisit this because I've been thinking about it. These recipes could be a good way to provide different launcher commands since they essentially provide mappings.

What we will need is a way to annotate the format the arguments take. For example, dependency takes a list of (I think) comma separated job identifiers. The single item parameters are simple in a sense since we could default to Python standard notation (%s, %d, etc.).