Caching issue when deploying unit tests

[sitapriyamoorthi]

1. Is the default state of call caching turned on (both read from cache and write to cache) when we run unit tests?

• Background: Cromwell enables call caching by default in its configuration. This means that when workflows are executed, their results can be:

- Written to the cache: Outputs of successful tasks are stored.

- Read from the cache: If a task with identical inputs and runtime conditions is executed again, Cromwell retrieves the outputs from the cache instead of re-executing the task.

2. If call caching is enabled by default, does that mean re-running tests will read outputs from the cached calls?

Potential Problem: In a typical unit testing scenario, workflows and tasks should be executed afresh every time to verify their correctness. However, if call caching is enabled and outputs are read from the cache during a re-run:

• It could falsely indicate that tasks are working correctly, even if they are not.

• Test coverage would be compromised since the actual execution of the workflow or task is bypassed.

Implications: This behavior defeats the purpose of unit testing, as it relies on cached results instead of performing actual executions.

3. For unit tests that explicitly test call caching, could there be issues if the environment enforces caching irrespective of the runtime cache settings?

Potential Problem: Unit tests designed to validate call caching behavior (e.g., testing read_from_cache and write_to_cache runtime attributes) could yield misleading results if:

• The testing environment forces caching on/off, overriding the runtime settings specified in the WDL.

• This would prevent testing scenarios where caching is explicitly enabled or disabled for specific tasks.

Example: If we set read_from_cache: false in the WDL runtime block for a task, but the environment ignores this setting and uses cached outputs anyway, the test would fail to evaluate the desired behavior.