Fuzz for panics

[petervdonovan]

This is a proposal for an testing enhancement that might help us eliminate panics.

I should preface this by saying that this issue is not urgent and I am not asking any one specific person to worry about it right now. The purpose of this issue is to document an idea in case we later decide to pursue it.

Probably lingo should not panic.

Here is an example of lingo panicking:

lfc: fatal error: /home/peter/xronos/tracing-plugin-reproducible-example/lf/laggy: No such file or directory.
thread 'main' panicked at lingo/src/backends/mod.rs:160:21:
build step failed because of laggy with main reactor /home/peter/xronos/tracing-plugin-reproducible-example/lf/laggy!
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

If the branch that creates liblingo gets merged, the library part of lingo should be fuzzed by injecting mock file system access functions that sometimes but not always misbehave/fail. This is representative of a real file system, because real file systems will be concurrently accessed by other processes on the system, making them inherently unpredictable. This is easy to fuzz because panics are easy to detect without any false positives or false negatives. This issue would have been addressed when we hit all the error paths. It is possible to just grab a mutex and record which error paths have been hit as a way to track coverage without needing any fancy build system tricks.

There are other properties that would be easy to fuzz for, like not going for more than ten seconds without producing any output on stdout or stderr.

[tanneberger]

right now the panic is explicit see: https://github.com/lf-lang/lingo/blob/main/src/backends/mod.rs#L155 if you set --keep-going then it wont. But probably smart to just propagate the error to the main function and return an error code.