[LUR-27] CI: Reject large perf regressions

[huitseeker]

#996 is a textbook case where the bench CI pipeline (which may well be buggy) reported a characterised regression. This is not a case of the CI failing to run or finding no suitable referential.

The regression factor is > 2 for all tests.

When that happens we should at least file an issue (if not fail the merge tests) and not count on folks reading the bench comments.

_LUR-27

[huitseeker]

from @samuelburnham :

It appears that the bc command is not found in https://github.com/lurk-lab/lurk-rs/actions/runs/7452711363/job/20276565702, which I think is preinstalled on GitHub runners (which I tested) but not our self-hosted machine

[porcuquine]

Although I know sorting through these mechanics is tedious, having this in place will be a big step forward. Given how performance-sensitive we are, and how many interacting systems can lead to regressions, it's important that we have an automated way of preventing us from accidentally veering down paths that will be expensive or difficult to recover from later.

[huitseeker]

The bc fix is there, but it's not tested, so I'm gonna insist this remains open.

Further, #1029 shows we have trouble locating the regression, which may have been in a dependency, because no job outside of the merge-test one runs through a differential benchmark.
The job we have under workflow-dispatch runs a single bench:
https://github.com/lurk-lab/lurk-rs/blob/main/.github/workflows/gpu-bench-workflow-dispatch.yml

It would simplify forensics to change that.

[samuelburnham]

Agreed, sorry I didn't realize "Should fix X" actually closes the issue