Testing FitLins regressions on example data

So you've found that FitLins tests are failing with results like:

        # check niftis against reference
        ref_niis = sorted(reference_dir.glob('**/*.nii.gz'))
        for ref_nii in ref_niis:
            out_nii = Path(output_dir) / ref_nii.relative_to(reference_dir)
            ref_data = nib.load(ref_nii).get_fdata(dtype=np.float64)
            out_data = nib.load(out_nii).get_fdata(dtype=np.float64)
            difs = get_nan_diff(ref_nii, out_nii, ref_data, out_data, max_abs=1e-06, max_rel=1e-04)
>           assert len(difs) == 0
E           AssertionError: assert 1 == 0
E            +  where 1 = len(['Relative difference (max of 0.062146558687224976) greater than 0.0001 for /tmp/.cache/data/fitlins-tests/outputs/nistats_smooth/node-subject/sub-01/sub-01_contrast-taskVsBaseline_stat-p_statmap.nii.gz and /tmp/ds003/derivatives/node-subject/sub-01/sub-01_contrast-taskVsBaseline_stat-p_statmap.nii.gz.'])

How do we debug this locally?

First we fetch the https://gin.g-node.org/markiewicz/fitlins-tests repository:

$ datalad install -r https://gin.g-node.org/markiewicz/fitlins-tests
$ cd fitlins-tests

And we retrieve the files needed to run FitLins as well as the outputs for comparison:

$ datalad get inputs/ds000003-fmriprep/sub-0{1,2,3}/func/*_space-MNI152NLin2009cAsym_desc-*.nii.gz \
              inputs/ds000003-fmriprep/sub-0{1,2,3}/func/*_desc-confounds_*.tsv \
              inputs/ds000003-fmriprep/dataset_description.json \
              inputs/ds000003-fmriprep/sub-*/*/*.json \
              outputs

Then we set up a testing environment, if we don't already have one. Using conda:

$ conda create -n fitlins-test python=3.9
$ conda activate fitlins-test

We're pretty likely to want FitLins and PyBIDS installed in editable mode.

$ pip install -e /projects/fitlins
$ pip install -e /projects/pybids

Now we're ready to test. Let's start on a fresh branch and move the old outputs:

$ git checkout -b regression
$ git mv outputs outputs.old
$ git commit -m 'Rename old outputs for comparison'

And finally we'll datalad run our script (with -vv for extra verbosity):

$ datalad run code/nistats_smooth.sh -vv

From here we can compare the results between the outputs directory and outputs.old.