How to deal with long running processes?

[sckott]

(pulling info from a slack thread convo with @dtenenba on 2024-12-20 so it doesn't get lost)

We have some "long" running processes that we'll have to deal with for tests. Long here may be 30 seconds or multiple hours, both of which need specific handling - a request to get metadata about job X after 2 seconds may error or return a different set of data than the same request after job X has completed.

Some options:

• use test fixtures - record a real http response and then use that cached http response insetad of making a real request to the cromwell server - this will be very fast
• use mocks - similar to fixtures, but instead of the whole http response saved to a file that we read (the fixture) we just specify the returned json in a variable
• forget about tests that depend on longer running operations? seems not ideal
• on a cron schedule submit all the wdls/jsons that have longer running operations - say once a day - when we run tests we just use the results already in the test user DB

---

If things can take a long time and we're okay with that (we don't use the above approaches of fixtures, mocks, etc.), then we need a different testing setup than is typical (where you do X and Y and then run the unit test right away). How would that work?

Some background info:

• Self-hosted runners can have jobs that take up to 5 days
• You can trigger actions by calling the github api with a token. If you have a workflow that has on: workflow_dispatch: then you can do e.g.,

curl -X POST \
  -H "Accept: application/vnd.github+json" \
  -H "Authorization: Bearer YOUR_GITHUB_TOKEN" \
  https://api.github.com/repos/OWNER/REPO/actions/workflows/WORKFLOW_ID/dispatches \
  -d '{"ref":"main"}'

It's not clear yet what the solution is, but likely involves some system where one workflow (.yml file) submits workflows, then another runs tests that depend on those workflows (presumably all in final state on cromwell) - how the 2nd system knows when to run is up for debate; possibly some script polling for all workflows being done