Merge pull request #125 from xylar/add-shared-configs

Add support for shared config options

deploy/default.cfg

@@ -24,7 +24,7 @@ geometric_features = 1.2.0
 jigsaw = 0.9.14
 jigsawpy = 0.3.3
 mache = 1.16.0
-mpas_tools = 0.23.0
+mpas_tools = 0.25.0
 otps = 2021.10
 parallelio = 2.6.0
 

docs/developers_guide/api.md

@@ -120,6 +120,9 @@ seaice/api
 
    Component
    Component.add_task
+   Component.add_step
+   Component.remove_step
+   Component.add_config
 ```
 
 #### Task
@@ -133,6 +136,8 @@ seaice/api
    Task
    Task.configure
    Task.add_step
+   Task.remove_step
+   Task.set_shared_config
 ```
 
 #### Step
@@ -152,6 +157,8 @@ seaice/api
    Step.add_input_file
    Step.add_output_file
    Step.add_dependency
+   Step.validate_baselines
+   Step.set_shared_config
 ```
 
 
@@ -199,6 +206,7 @@ seaice/api
    :toctree: generated/
 
    PolarisConfigParser
+   PolarisConfigParser.setup
 ```
 
 ### io

docs/developers_guide/framework/config.md

@@ -9,21 +9,36 @@ Here, we include some specific details relevant to using the
 
 Here, we provide the {py:class}`polaris.config.PolarisConfigParser` that has
 almost the same functionality but also ensures that certain relative paths are
-converted automatically to absolute paths.
+converted automatically to absolute paths.  `PolarisConfigParser` also has
+attributes for a `filepath` where the config file will be written out and a
+list of `symlinks` that will point to `filepath`.  It also has a
+{py:meth}`polaris.config.PolarisConfigParser.setup()` method that can be
+overridden to add config options (e.g. algorithmically from other config
+options) as part of setting up polaris tasks and steps.  These features are
+included to accommodate sharing config options across shared steps and/or
+multiple tasks.
 
 The {py:meth}`mpas_tools.config.MpasConfigParser.add_from_package()` method can
 be used to add the contents of a config file within a package to the config
-options. Examples of this can be found in many tasks as well as
-{py:func}`polaris.setup.setup_task()`. Here is a typical example from
-{py:func}`polaris.ocean.tasks.baroclinic_channel.decomp.Decomp.configure()`:
+options. Examples of this can be found in many tasks as well as in the
+`polaris.setup` module. Here is a typical example from
+{py:class}`polaris.ocean.tasks.inertial_gravity_wave.InertialGravityWave`:
 
 ```python
-def configure(self):
-    """
-    Add the config file common to baroclinic channel tests
-    """
-    self.config.add_from_package('polaris.ocean.tasks.baroclinic_channel',
-                                 'baroclinic_channel.cfg')
+from polaris import Task
+
+
+class InertialGravityWave(Task):
+    def __init__(self, component):
+        name = 'inertial_gravity_wave'
+        subdir = f'planar/{name}'
+        super().__init__(component=component, name=name, subdir=subdir)
+
+        ...
+
+        self.config.add_from_package(
+            'polaris.ocean.tasks.inertial_gravity_wave',
+            'inertial_gravity_wave.cfg')
 ```
 
 The first and second arguments are the name of a package containing the config
@@ -38,9 +53,14 @@ example from {py:func}`polaris.setup.setup_task()`, there may not be a config
 file for the particular machine we're on, and that's fine:
 
 ```python
-if machine is not None:
-    config.add_from_package('mache.machines', f'{machine}.cfg',
-                            exception=False)
+from polaris.config import PolarisConfigParser
+
+
+def _get_basic_config(config_file, machine, component_path, component):
+    config = PolarisConfigParser()
+    if machine is not None:
+        config.add_from_package('mache.machines', f'{machine}.cfg',
+                                exception=False)
 ```
 If there isn't a config file for this machine, nothing will happen.
 
@@ -59,6 +79,107 @@ be used to get python dictionaries, lists and tuples as well as a small set
 of functions (`range()`, {py:meth}`numpy.linspace()`,
 {py:meth}`numpy.arange()`, and {py:meth}`numpy.array()`)
 
+## Shared config files
+
+Often, it makes sense for many tasks and steps to share the same config 
+options.  The default behavior is for a task and its "owned" steps to share
+a config file in the task's work directory called `{task.name}.cfg` and 
+symlinks with that same name in each step's work directory.  The default for
+a shared step is to have its own `{step.name}.cfg` in its work directory.
+
+Developers can create shared config parsers that define the location of the
+shared config file and add them to tasks and steps using
+{py:meth}`polaris.Task.set_shared_config()` and 
+{py:meth}`polaris.Step.set_shared_config()`.  The location of the shared
+config file should be intuitive to users but local symlinks will also make
+it easy to modify the shared config options from within any of the tasks and
+steps that use them.
+
+As an example, the baroclinic channel tasks share a single 
+`baroclinic_channel.cfg` config file for each resolution that resides in the
+resolution's work directory:
+
+```python
+from polaris.config import PolarisConfigParser
+from polaris.ocean.resolution import resolution_to_subdir
+from polaris.ocean.tasks.baroclinic_channel.default import Default
+from polaris.ocean.tasks.baroclinic_channel.init import Init
+from polaris.ocean.tasks.baroclinic_channel.rpe import Rpe
+
+
+def add_baroclinic_channel_tasks(component):
+    for resolution in [10., 4., 1.]:
+        resdir = resolution_to_subdir(resolution)
+        resdir = f'planar/baroclinic_channel/{resdir}'
+
+        config_filename = 'baroclinic_channel.cfg'
+        config = PolarisConfigParser(filepath=f'{resdir}/{config_filename}')
+        config.add_from_package('polaris.ocean.tasks.baroclinic_channel',
+                                'baroclinic_channel.cfg')
+
+        init = Init(component=component, resolution=resolution, indir=resdir)
+        init.set_shared_config(config, link=config_filename)
+
+        default = Default(component=component, resolution=resolution,
+                          indir=resdir, init=init)
+        default.set_shared_config(config, link=config_filename)
+        component.add_task(default)
+
+        ...
+
+        component.add_task(Rpe(component=component, resolution=resolution,
+                               indir=resdir, init=init, config=config))
+```
+
+For most tasks and steps, it is convenient to call `set_shared_config()`
+after constructing the step or task and before adding it to the component.
+In the example of the `Rpe` task here, we need the shared config in the
+constructor so it has to be passed in.  We call `self.set_shared_config()`
+in the constructor, and then use config options to determine the steps to be
+added as follows:
+
+```python
+from polaris import Task
+from polaris.ocean.tasks.baroclinic_channel.forward import Forward
+from polaris.ocean.tasks.baroclinic_channel.rpe.analysis import Analysis
+
+
+class Rpe(Task):
+    def __init__(self, component, resolution, indir, init, config):
+        super().__init__(component=component, name='rpe', indir=indir)
+        self.resolution = resolution
+
+        # this needs to be added before we can use the config options it
+        # brings in to set up the steps
+        self.set_shared_config(config, link='baroclinic_channel.cfg')
+        self.add_step(init, symlink='init')
+        self._add_rpe_and_analysis_steps()
+
+    def _add_rpe_and_analysis_steps(self):
+        """ Add the steps in the test case either at init or set-up """
+        config = self.config
+        component = self.component
+        resolution = self.resolution
+
+        nus = config.getlist('baroclinic_channel_rpe', 'viscosities',
+                             dtype=float)
+        for nu in nus:
+            name = f'nu_{nu:g}'
+            step = Forward(
+                component=component, name=name, indir=self.subdir,
+                ntasks=None, min_tasks=None, openmp_threads=1,
+                resolution=resolution, nu=nu)
+
+            step.add_yaml_file(
+                'polaris.ocean.tasks.baroclinic_channel.rpe',
+                'forward.yaml')
+            self.add_step(step)
+
+        self.add_step(
+            Analysis(component=component, resolution=resolution, nus=nus,
+                     indir=self.subdir))
+```
+
 ## Comments in config files
 
 One of the main advantages of {py:class}`mpas_tools.config.MpasConfigParser`

docs/developers_guide/framework/remapping.md

@@ -27,8 +27,8 @@ from polaris import Task
 from polaris.remap import MappingFileStep
 
 class MyTestCase(Task):
-    def __int__(self):
-        step = MappingFileStep(task=self, name='make_map', ntasks=64, 
+    def __int__(self, component):
+        step = MappingFileStep(component=component, name='make_map', ntasks=64, 
                                min_tasks=1, method='bilinear')
         # indicate the the mesh from another step is an input to this step
         # note: the target is relative to the step, not the task.
@@ -62,8 +62,8 @@ from polaris.remap import MappingFileStep
 
 
 class VizMap(MappingFileStep):
-    def __init__(self, task, name, subdir, mesh_name):
-        super().__init__(task=task, name=name, subdir=subdir,
+    def __init__(self, component, name, subdir, mesh_name):
+        super().__init__(component=component, name=name, subdir=subdir,
                          ntasks=128, min_tasks=1)
         self.mesh_name = mesh_name
         self.add_input_file(filename='mesh.nc', target='../mesh/mesh.nc')

docs/developers_guide/ocean/framework.md

@@ -194,16 +194,19 @@ barotropic time step in a similar way.  The `run_duration` and
 
 Each convergence test can override these defaults with its own defaults by 
 defining them in its own config file.  Convergence tests should bring in this
-config file in their `configure()` methods, then add its own config options
-after that to make sure they take precedence, e.g.:
+config file in their constructor or by adding them to a shared `config`.  The
+options from the shared infrastructure should be added first, then those from 
+its own config file to make sure they take precedence, e.g.:
 
 ```python
-from polaris import Task
-class CosineBell(Task):
-    def configure(self):
-        super().configure()
-        config = self.config
-        config.add_from_package('polaris.mesh', 'mesh.cfg')
+from polaris.config import PolarisConfigParser
+
+
+def add_cosine_bell_tasks(component):
+    for icosahedral, prefix in [(True, 'icos'), (False, 'qu')]:
+
+        filepath = f'spherical/{prefix}/cosine_bell/cosine_bell.cfg'
+        config = PolarisConfigParser(filepath=filepath)
         config.add_from_package('polaris.ocean.convergence.spherical',
                                 'spherical.cfg')
         config.add_from_package('polaris.ocean.tasks.cosine_bell',

docs/developers_guide/organization/steps.md

@@ -261,7 +261,7 @@ number of CPUs per task, the number of OpenMP threads, and (currently as
 placeholders) the amount of memory the step is allowed to use.
 
 Then, the step can add {ref}`dev-step-inputs-outputs` as well as
-{ref}`dev-step-namelists-and-streams`, as described below.
+{ref}`dev-model-yaml-namelists-and-streams`, as described below.
 
 As with the task's {ref}`dev-task-init`, it is important that the
 step's constructor doesn't perform any time-consuming calculations, download
@@ -282,6 +282,10 @@ Each of these functions just caches information about the the inputs, outputs,
 namelists, streams or YAML files to be read later if the task in question gets
 set up, so each takes a negligible amount of time.
 
+If this is a shared step with its own config options, it is also okay to call 
+{py:meth}`mpas_tools.config.MpasConfigParser.add_from_package()` from the
+constructor.
+
 The following is from
 {py:class}`polaris.ocean.tasks.baroclinic_channel.forward.Forward()`:
 
@@ -357,10 +361,6 @@ class Forward(OceanModelStep):
                          indir=indir, ntasks=ntasks, min_tasks=min_tasks,
                          openmp_threads=openmp_threads)
 
-        if nu is not None:
-            # update the viscosity to the requested value
-            self.add_model_config_options(options=dict(config_mom_del2=nu))
-
         # make sure output is double precision
         self.add_yaml_file('polaris.ocean.config', 'output.yaml')
 
@@ -372,6 +372,11 @@ class Forward(OceanModelStep):
         self.add_yaml_file('polaris.ocean.tasks.baroclinic_channel',
                            'forward.yaml')
 
+        if nu is not None:
+            # update the viscosity to the requested value *after* loading
+            # forward.yaml
+            self.add_model_config_options(options=dict(config_mom_del2=nu))
+
         self.add_output_file(
             filename='output.nc',
             validate_vars=['temperature', 'salinity', 'layerThickness',
@@ -1027,14 +1032,12 @@ on that output file as an input use {py:meth}`polaris.Step.add_input_file()`.
 Under these circumstances, it is useful to be able to specify that a step
 is a dependency of another (dependent) step.  This is accomplished by passing 
 the  dependency to the step's {py:meth}`polaris.Step.add_dependency()` method 
-either  during the creation of the dependent step, within the `configure()` 
-method of  the parent task, or in the `setup()` method of the dependent
-step.  The  dependency does not need to belong to the same task as the
-dependent step.  But the dependent step will fail to run if the dependency
-has not run.  Also all dependencies must be set up along with dependent steps
-(even if they are not run by default, because they are added to the task
-with `run_by_default=False`).  This is because a user could modify which steps
-they wish to run and all dependencies should be available if they do so.
+either during the creation of the dependent step, within the `configure()` 
+method of the parent task, or in the `setup()` method of the dependent
+step.  The dependency does not need to belong to the task, it can be a shared
+step, but it should be a step in any tasks that also use the dependent step.
+This is because the dependent step will fail to run if the dependency has not 
+run.
 
 When a step is added as a dependency, after it runs, its state will be stored
 in a pickle file (see {ref}`dev-setup`) that contains any modifications to its