Remove unused input from previous experiments in suites

[akunft]

Currently we only remove the output from previous experiments when executing a suite.

I experienced some problems lately in suites with lots of different input data while having a fixed system and cluster setup. This resulted in out of disk space errors because the input of previous experiments used all the disk space.

I would suggest we introduce a flag for suites which, when enabled, removes input from previous experiments (if not needed by the current experiment).

I would be up to integrate this feature but I'm not sure if we should make it the default or enable it by flag.

[aalexandrov]

Currently we only remove the output from previous experiments when executing a suite.

I think that this is just because we were lazy to do it better from the onset.

I would be up to integrate this feature but I'm not sure if we should make it the default or enable it by flag.

👍 IMHO the implementation should be based on some sort of Graph analysis which determines whether an input is going to be needed later in the suite or not.

[aalexandrov]

Sorry it is not clear to me whether we talk about ExperimentOutput or DataSet beans?

[akunft]

I mean the DataSet beans from the experiment inputs.

👍 for the graph analysis, instead of only checking the previous experiment. Then we can also make it by default.

Do you think we should extend that also to the output data, to achieve suite with dependencies on previous output?

[aalexandrov]

Here is an idea for the implementation:

• construct the initial dependency graph
• keep track of the initial set of input and output nodes
• after each experiment
  • remove the experiment node from the graph
  • query each input and output for dependent nodes. If the result is empty, delete the data.

I suggest to introduce flags --preserve-input and --preserve-output that disable the logic for input and output data.

[akunft]

Actually I already have a prototype for the inputs only, but I could extend it for the output sets as well. Its not based on the graph though.

akunft@437f917

There are actually two problems regarding the configuration:

1. We would have to set the config for all decentants for the exp beforehand. We need this to resolve the paths of the DataSet
2. As DataSet for the same input path are currently not represented as the same object, we have to overload the equals and hashcode which requires the config to be resolved beforehand.

In the prototype, i did the implementation on the unresolved paths, but IMO we have to do it on the resolved paths.

I just ref this here for future discussions.