Run the tests separately

[thofma]

In AA, we test four packages (https://github.com/Nemocas/AbstractAlgebra.jl/blob/master/OscarCI.toml). Would it be possible to have the four tests in separate jobs? The problem we have at the moment is that the Oscar tests run first, but they might (and often do) fail for unrelated reasons. Then all the tests of the other packages are not run.

I am not sure we want to have separate jobs or handle it inside the julia command here: https://github.com/oscar-system/OscarDevTools.jl/blob/master/src/OscarCI.jl#L283

Thoughts?

Edit: Sorry, I just checked. The Oscar tests run last. But my problem is that I have to scroll through everything to spot which tests passed, which did not, and to find out which did not even run. I hope you get my point.

[benlorenz]

Not totally sure how to handle this:

• Expanding this into separate jobs (via an additional axis) will increase the already rather large job-list even further and make it difficult to find the correct build. (also this will probably get rather complicated in the combination with the includes ...)
• Unfortunately I don't think there is a way to dynamically create additional steps (e.g. per package). But we could pre-define steps for all package-tests that might run (e.g. all main oscar packages) and just skip those that are not requested in the current job. (And at the end run any remaining package-tests in one extra step)
• Maybe we can try running the tests with TestReports.jl and use something like https://github.com/marketplace/actions/junit-report-action or https://github.com/marketplace/actions/publish-unit-test-results to summarize the results.

[thofma]

Thanks for your feedback. I agree that doing it on the level of jobs is not the best idea. I will probably play around with TestReports.jl.