-
Notifications
You must be signed in to change notification settings - Fork 1
Home
CauseCumber is a causality-driven approach to software testing for comptuational models that is centred round the idea that software testing can be viewed as a problem of causal inference. Causal inference is a statistical methodology designed to answer causal questions by estimating causal effects; the change in some outcome that is caused by changing a particular variable, known as a treatment. Unlike traditional statistical techniques, causal inference has the ability to make causal rather than merely associational claims about the effect of some change to a system. Therefore, CauseCumber translates testing to a causal inference problem in which the goal is to answer causal questions about particular modelling scenarios in place of traditional test cases.
CauseCumber has four key stages:
-
Scenario Requirements Specification: for a given modelling scenario, specify the expected behaviour under different input configurations in a cucumber feature file. Each requirement should consist of a causal question about the modelling scenario (e.g. when I set parameter X to x, does output Y take on value y at time t?) and an expected answer (e.g. yes/no).
-
Causal Graph Generation: given a modelling scenario specified in a cucumber feature file, generate a causal graph that depicts the key cause-effect relationships that exist between variables in the scenario. This will allow us to apply powerful graphical causal inference techniques to test causality in the modelling scenario.
-
Causal Inference: given a causal graph and a particular requirement (causal question and expected answer), use causal inference techniques to estimate the causal effect and provide an answer to the question.
-
Test Oracle: given the outcome of the causal inference step and the requirement in question, automatically check the actual outcome (causal effect estimate) is suitably similar to the expected answer. Depending on the causal question, the actual and expected answer need not be identical. A user can specify a fault tolerance e.g. +/- 10% of the actual outcome.