(1) Scenario Requirements Specification

We use cucumber specifications to describe example end-to-end black-box modelling scenarios and how the model is expected to behave throughout this scenario. Before we outline the specification process, we need to define some terminology.

Terminology

We use the term end-to-end modelling scenario to refer to a specific execution of a computational model from some initial starting to state to some final stopping condition. The term black-box means that we don't have access to the inner-workings of the model and only have access to user-facing information about the model, such as logs and other outputs.

Process

In CauseCumber, we use executable cucumber specifications as way to capture requirements and generate or collect appropriate execution data for causal inference. For a specific end-to-end modelling scenario a user should define a feature file which describes different variations of the scenario the expected change in outcomes.

Each feature file has the following components:

• Background: a list of default input parameter settings that all scenarios will use unless overwritten. E.g.

  Background:
    Given a simulation with parameters
      | parameter    | value  | type |
      | quar_period  | 14     | int  |
      | n_days       | 84     | int  |
      | pop_type     | hybrid | str  |
      | pop_size     | 50000  | int  |
      | pop_infected | 100    | int  |
      | location     | UK     | str  |
    And the following variables are recorded weekly
      | variable          | type     |
      | cum_infections    | int      |
      | cum_symptomatic   | int      |
      | cum_severe        | int      |
      | cum_critical      | int      |
      | cum_deaths        | int      |

• A baseline Scenario: a simple scenario which captures the vanilla version of the scenario which may be used for comparison in future scenarios. E.g.

  Scenario: Baseline
    Given a simulation run with no vaccine available
    When the simulation is finished
    Then the weekly cumulative infections should be reported

• A number of Scenarios and Scenario Outlines: for each variation of the baseline scenario, a scenario or scenario outline is defined that describes a different version of the baseline scenario. Specifically, each variation should modify the model in some way (an intervention) that is expected to cause a change to some output. E.g.

  Scenario Outline: Single vaccine
    All vaccines should reduce the cumulative number of infections relative to the
    baseline scenario where no vaccine is available.

    Given the <vaccine_name> vaccine is available
    When the simulation is finished
    Then the cumulative number of infections should be less than the baseline
    Examples:
      | vaccine_name |
      | pfizer       |
      | moderna      |
      | az           |