Using roswell to run tests.

[glennj]

In #758 (comment), the topic of roswell was introduced.

Let's discuss it here.

[glennj]

So far, it looks very promising as a "one true" way to run tests.

Using homebrew on this Debian install, and brew install roswell is all that's needed.

The first time it's invoked, it will immediately install SBCL and Quicklisp:

$ brew install roswell
...

$ ros help
Installing sbcl-bin...
No SBCL version specified. Downloading sbcl-bin_uri.tsv to see the available versions...
[##########################################################################]100%
Installing sbcl-bin/2.3.9...
Downloading https://github.com/roswell/sbcl_bin/releases/download/2.3.9/sbcl-2.3.9-x86-64-linux-binary.tar.bz2
[##########################################################################]100%
Extracting sbcl-bin-2.3.9-x86-64-linux.tar.bz2 to /home/glennj/.roswell/src/sbcl-2.3.9-x86-64-linux/
Building sbcl-bin/2.3.9... Done.
'patchelf' not detected.no patching for sbcl
Install Script for sbcl-bin...
Installing Quicklisp... Done 21832
Making core for Roswell...
Common Lisp environment setup Utility.

Usage:

   ros [options] Command [arguments...]
or
   ros [options] [[--] script-path arguments...]

commands:
   run       Run repl
   install   Install a given implementation or a system for roswell environment
   update    Update installed systems.
   build     Make executable from script.
   use       Change default implementation.
   init      Creates a new ros script, optionally based on a template.
   fmt       Indent lisp source.
   list      List Information
   template  Manage templates
   delete    Delete installed implementations
   config    Get and set options
   version   Show the roswell version information

Use "ros help [command]" for more information about a command.

Additional help topics:

   options

Use "ros help [topic]" for more information about the topic.

$ ros list installed
Installed implementations:

Installed versions of sbcl-bin:
sbcl-bin/2.3.9

This is great: the only installation instruction needs to be "go install roswell". The first time ros run is invoked, it will seamlessly install sbcl and quicklisp, which seamlessly installs fiveam

[glennj]

Running tests installs dependencies seamlessly thanks to quicklisp:

$ cd acronym
$ ros run -l acronym-test.lisp -e '(acronym-test:run-tests)' -q
To load "fiveam":
  Load 3 ASDF systems:
    alexandria asdf trivial-backtrace
  Install 2 Quicklisp releases:
    asdf-flv fiveam
Downloading http://beta.quicklisp.org/archive/asdf-flv/2023-10-21/asdf-flv-version-2.2.tgz
##########################################################################
Downloading http://beta.quicklisp.org/archive/fiveam/2023-10-21/fiveam-20231021-git.tgz
##########################################################################
; Loading "fiveam"
[package alexandria]..............................
..................................................
..................................................
[package alexandria-2]............................
[package net.didierverna.asdf-flv]................
[package trivial-backtrace].......................
[package it.bese.fiveam]..........................
.................

Running test suite ACRONYM-SUITE
 Running test EMPTY-GIVES-EMPTY .
 Running test PNG-TEST .
 Running test ROR-TEST .
 Running test FIFO-TEST .
 Running test PHP-TEST .
 Running test CMOS-TEST .
 Did 6 checks.
    Pass: 6 (100%)
    Skip: 0 ( 0%)
    Fail: 0 ( 0%)

$ echo $?
0

[glennj]

But, adding an incorrect test

(test wrong-test (is (equal "WRONG" (acronym:acronym "Readily ignited gasoline-hydrocarbon turbine"))))

ros does not exit with a non-zero exit status:

$ ros run -l acronym-test.lisp -e '(acronym-test:run-tests)' -q
To load "fiveam":
  Load 1 ASDF system:
    fiveam
; Loading "fiveam"


Running test suite ACRONYM-SUITE
 Running test EMPTY-GIVES-EMPTY .
 Running test PNG-TEST .
 Running test WRONG-TEST f
 Running test ROR-TEST .
 Running test FIFO-TEST .
 Running test PHP-TEST .
 Running test CMOS-TEST .
 Did 7 checks.
    Pass: 6 (85%)
    Skip: 0 ( 0%)
    Fail: 1 (14%)

 Failure Details:
 --------------------------------
 WRONG-TEST in ACRONYM-SUITE []:

(ACRONYM:ACRONYM "Readily ignited gasoline-hydrocarbon turbine")

 evaluated to

"RIGHT"

 which is not

EQUAL

 to

"WRONG"


 --------------------------------

$ echo $?
0

[glennj]

In playing with sbcl, ccl, clisp, each interpreter has a unique way to set the exit status

interp	exit status
sbcl	(exit :code 42)
ccl	(quit 42)
clisp	(exit 42)

[glennj]

This little shell script would work:

#!/bin/sh
exercise=${PWD##*/}
tests="${exercise}-test.lisp"

if ! [ -f "${tests}" ]; then
    echo "Test file not found: $tests" >&2
    exit 2
fi

ros run -l "${tests}" \
        -e "(${exercise}-test:run-tests)" \
        -q 2>&1 \
| (
    status=0;
    while IFS= read -r line; do
        case "$line" in
            '    Fail: '[1-9]*) status=1 ;;
        esac
        echo "$line"
    done
    exit $status
)

Running it:

$ sh run-tests.sh
...
Running test suite ACRONYM-SUITE
...
 Did 7 checks.
    Pass: 6 (85%)
    Skip: 0 ( 0%)
    Fail: 1 (14%)

 Failure Details:
...

$ echo $?
1

[glennj]

That would mean:

• bundling that run-tests script with every exercise
• coming up with a powershell equivalent as well
• making roswell mandatory for the track

I'm not entirely happy with the shell script: it's a bit fragile in that if fiveam changes the format of the "Fail: nnn" line, then the exit status won't be set correctly.

[verdammelt]

You might want to look at the UIOP package (which is a dependency of ASDF and lots of other things). Already used in the repo. It is contains a facade layer to provide consistent interface to lots of implementation things (such as exit codes).

If RUN-TESTS evaluates to the test results then checking for failed/errored tests could be done.

[glennj]

Ah, that's it!

$ ros run -l acronym-test.lisp -e '(uiop/image:quit (if (acronym-test:run-tests) 0 42))'
To load "fiveam":
  Load 1 ASDF system:
    fiveam
; Loading "fiveam"


Running test suite ACRONYM-SUITE
 Running test EMPTY-GIVES-EMPTY .
 Running test PNG-TEST .
 Running test WRONG-TEST f
 Running test ROR-TEST .
 Running test FIFO-TEST .
 Running test PHP-TEST .
 Running test CMOS-TEST .
 Did 7 checks.
    Pass: 6 (85%)
    Skip: 0 ( 0%)
    Fail: 1 (14%)

 Failure Details:
 --------------------------------
 WRONG-TEST in ACRONYM-SUITE []:

(ACRONYM:ACRONYM "Readily ignited gasoline-hydrocarbon turbine")

 evaluated to

"RIGHT"

 which is not

EQUAL

 to

"WRONG"


 --------------------------------

then

$ echo $?
42

[glennj]

If we still want to add a script to launch the tests:

#!/bin/sh
exercise=${PWD##*/}
tests="${exercise}-test.lisp"

if ! [ -f "${tests}" ]; then
    echo "Test file not found: $tests" >&2
    exit 2
fi

# exit status: 0 if all tests pass, 1 otherwise
ros run --load "${tests}" \
        --eval "(uiop/image:quit (if (${exercise}-test:run-tests) 0 1))"

Or, this might be clearer:

ros run --load "${tests}" \
        --eval "
            (let* ((test-status (${exercise}-test:run-tests))
                   (exit-status (if test-status 0 1)))
              (uiop/image:quit exit-status))
        "

[verdammelt]

Looks good. I think UIOP:QUIT is sufficient - likely an alias for UIOP/IMAGE:QUIT - but using the former will keep us insulated from changes to the internal arrangement of packages.

I think the final lines of the shell script you have above are good (rather than the second which uses let*. Not that there is a problem with let* but I just don't think it is necessary here.

Can you put together a PR with instructions on installing ros as well as using it for running tests?