Napari-workflows tests fail with stacview 0.9.0

[tcompa]

The same tests do run when using poetry, which suggests this is a problem of some breaking dependency update.

---

Ref https://github.com/fractal-analytics-platform/fractal-tasks-core/actions/runs/10628700134/job/29464301618

FAILED tests/tasks/test_workflows_napari_workflows.py::test_napari_workflow - ValueError: could not broadcast input array from shape (540,640) into shape (691199,)
FAILED tests/tasks/test_workflows_napari_workflows.py::test_napari_worfklow_label_input_only - ValueError: could not broadcast input array from shape (540,640) into shape (691199,)
FAILED tests/tasks/test_workflows_napari_workflows.py::test_relabeling[wf_relab_1-labeling_only.yaml-input_specs0-output_specs0-False] - ValueError: could not broadcast input array from shape (2160,2560) into shape (11059199,)
FAILED tests/tasks/test_workflows_napari_workflows.py::test_relabeling[wf_relab_2-measurement_only.yaml-input_specs1-output_specs1-True] - ValueError: could not broadcast input array from shape (2160,2560) into shape (11059199,)
FAILED tests/tasks/test_workflows_napari_workflows.py::test_relabeling[wf_relab_3-labeling_and_measurement.yaml-input_specs2-output_specs2-False] - ValueError: could not broadcast input array from shape (2160,2560) into shape (11059199,)
FAILED tests/tasks/test_workflows_napari_workflows.py::test_fail_if_no_relabeling - ValueError: could not broadcast input array from shape (2160,2560) into shape (11059199,)
FAILED tests/tasks/test_workflows_napari_workflows.py::test_expected_dimensions[2-2-True-True] - ValueError: could not broadcast input array from shape (269,320) into shape (86399,)
FAILED tests/tasks/test_workflows_napari_workflows.py::test_expected_dimensions[2-2-False-True] - ValueError: could not broadcast input array from shape (269,320) into shape (86399,)
FAILED tests/tasks/test_workflows_napari_workflows.py::test_expected_dimensions[3-2-False-True] - ValueError: could not broadcast input array from shape (0,270,320) into shape (86399,)
FAILED tests/tasks/test_workflows_napari_workflows.py::test_expected_dimensions[3-3-False-True] - ValueError: could not broadcast input array from shape (270,320) into shape (172799,)
FAILED tests/tasks/test_workflows_napari_workflows.py::test_napari_workflow_CYX - ValueError: could not broadcast input array from shape (539,640) into shape (345599,)
= 11 failed, 151 passed, 2 skipped, 1 xpassed, 278 warnings in 163.84s (0:02:43) =

2024-08-30T07:36:34.9811190Z ___________________________ test_napari_workflow_CYX ___________________________
2024-08-30T07:36:34.9811200Z 
2024-08-30T07:36:34.9811740Z tmp_path = PosixPath('/private/var/folders/m4/5dz5h26x329cqq4fx333f8gm0000gn/T/pytest-of-runner/pytest-0/test_napari_workflow_CYX0')
2024-08-30T07:36:34.9812100Z testdata_path = PosixPath('/Users/runner/work/fractal-tasks-core/fractal-tasks-core/tests/data')
2024-08-30T07:36:34.9813190Z zenodo_zarr = ['/Users/runner/work/fractal-tasks-core/fractal-tasks-core/tests/data/10_5281_zenodo_13305156/plate.zarr', '/Users/runner/work/fractal-tasks-core/fractal-tasks-core/tests/data/10_5281_zenodo_13305156/plate_mip.zarr']
2024-08-30T07:36:34.9813200Z 
2024-08-30T07:36:34.9813280Z     def test_napari_workflow_CYX(
2024-08-30T07:36:34.9813360Z         tmp_path: Path,
2024-08-30T07:36:34.9813480Z         testdata_path: Path,
2024-08-30T07:36:34.9813600Z         zenodo_zarr: list[str],
2024-08-30T07:36:34.9813680Z     ):
2024-08-30T07:36:34.9813740Z     
2024-08-30T07:36:34.9813800Z         # Init
2024-08-30T07:36:34.9813890Z         zarr_dir = tmp_path / "tmp_out/"
2024-08-30T07:36:34.9813970Z         zarr_urls = prepare_2D_zarr(
2024-08-30T07:36:34.9814040Z             str(zarr_dir),
2024-08-30T07:36:34.9814110Z             zenodo_zarr,
2024-08-30T07:36:34.9814180Z             remove_labels=True,
2024-08-30T07:36:34.9814240Z             make_CYX=True,
2024-08-30T07:36:34.9814300Z         )
2024-08-30T07:36:34.9814370Z         debug(zarr_dir)
2024-08-30T07:36:34.9814440Z         debug(zarr_urls)
2024-08-30T07:36:34.9814490Z     
2024-08-30T07:36:34.9814640Z         # First napari-workflows task (labeling)
2024-08-30T07:36:34.9814840Z         workflow_file = str(testdata_path / "napari_workflows/wf_1.yaml")
2024-08-30T07:36:34.9814990Z         input_specs: dict[str, NapariWorkflowsInput] = {
2024-08-30T07:36:34.9815260Z             "input": {"type": "image", "channel": {"wavelength_id": "A01_C01"}},  # type: ignore # noqa
2024-08-30T07:36:34.9815320Z         }
2024-08-30T07:36:34.9815460Z         output_specs: dict[str, NapariWorkflowsOutput] = {
2024-08-30T07:36:34.9815790Z             "Result of Expand labels (scikit-image, nsbatwm)": {  # type: ignore # noqa
2024-08-30T07:36:34.9815900Z                 "type": "label",
2024-08-30T07:36:34.9816020Z                 "label_name": "label_DAPI",
2024-08-30T07:36:34.9816080Z             },
2024-08-30T07:36:34.9816170Z         }
2024-08-30T07:36:34.9816280Z         for zarr_url in zarr_urls:
2024-08-30T07:36:34.9816370Z >           napari_workflows_wrapper(
2024-08-30T07:36:34.9816450Z                 zarr_url=zarr_url,
2024-08-30T07:36:34.9816550Z                 input_specs=input_specs,
2024-08-30T07:36:34.9816670Z                 output_specs=output_specs,
2024-08-30T07:36:34.9816790Z                 workflow_file=workflow_file,
2024-08-30T07:36:34.9816900Z                 input_ROI_table="FOV_ROI_table",
2024-08-30T07:36:34.9816970Z                 expected_dimensions=2,
2024-08-30T07:36:34.9817050Z                 level=2,
2024-08-30T07:36:34.9817140Z             )
2024-08-30T07:36:34.9817150Z 
2024-08-30T07:36:34.9817320Z tests/tasks/test_workflows_napari_workflows.py:540: 
2024-08-30T07:36:34.9817520Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
2024-08-30T07:36:34.9818070Z /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pydantic/validate_call_decorator.py:60: in wrapper_function
2024-08-30T07:36:34.9818200Z     return validate_call_wrapper(*args, **kwargs)
2024-08-30T07:36:34.9818670Z /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pydantic/_internal/_validate_call.py:96: in __call__
2024-08-30T07:36:34.9818920Z     res = self.__pydantic_validator__.validate_python(pydantic_core.ArgsKwargs(args, kwargs))
2024-08-30T07:36:34.9819150Z fractal_tasks_core/tasks/napari_workflows_wrapper.py:549: in napari_workflows_wrapper
2024-08-30T07:36:34.9819230Z     unique_labels = np.unique(mask)
2024-08-30T07:36:34.9819670Z /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/numpy/lib/arraysetops.py:274: in unique
2024-08-30T07:36:34.9819870Z     ret = _unique1d(ar, return_index, return_inverse, return_counts,
2024-08-30T07:36:34.9820050Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
2024-08-30T07:36:34.9820060Z 
2024-08-30T07:36:34.9820530Z ar = <[TypeError('%d format: a real number is required, not numpy.ndarray') raised in repr()] StackViewNDArray object at 0x15ea0d850>
2024-08-30T07:36:34.9820900Z return_index = False, return_inverse = False, return_counts = False
2024-08-30T07:36:34.9820900Z 
2024-08-30T07:36:34.9821050Z     def _unique1d(ar, return_index=False, return_inverse=False,
2024-08-30T07:36:34.9821170Z                   return_counts=False, *, equal_nan=True):
2024-08-30T07:36:34.9821230Z         """
2024-08-30T07:36:34.9821370Z         Find the unique elements of an array, ignoring shape.
2024-08-30T07:36:34.9821420Z         """
2024-08-30T07:36:34.9821510Z         ar = np.asanyarray(ar).flatten()
2024-08-30T07:36:34.9821570Z     
2024-08-30T07:36:34.9821690Z         optional_indices = return_index or return_inverse
2024-08-30T07:36:34.9821750Z     
2024-08-30T07:36:34.9821820Z         if optional_indices:
2024-08-30T07:36:34.9822050Z             perm = ar.argsort(kind='mergesort' if return_index else 'quicksort')
2024-08-30T07:36:34.9822130Z             aux = ar[perm]
2024-08-30T07:36:34.9822190Z         else:
2024-08-30T07:36:34.9822250Z             ar.sort()
2024-08-30T07:36:34.9822310Z             aux = ar
2024-08-30T07:36:34.9822430Z         mask = np.empty(aux.shape, dtype=np.bool_)
2024-08-30T07:36:34.9822490Z         mask[:1] = True
2024-08-30T07:36:34.9822670Z         if (equal_nan and aux.shape[0] > 0 and aux.dtype.kind in "cfmM" and
2024-08-30T07:36:34.9822770Z                 np.isnan(aux[-1])):
2024-08-30T07:36:34.9822970Z             if aux.dtype.kind == "c":  # for complex all NaNs are considered equivalent
2024-08-30T07:36:34.9823200Z                 aux_firstnan = np.searchsorted(np.isnan(aux), True, side='left')
2024-08-30T07:36:34.9823270Z             else:
2024-08-30T07:36:34.9823490Z                 aux_firstnan = np.searchsorted(aux, aux[-1], side='left')
2024-08-30T07:36:34.9823570Z             if aux_firstnan > 0:
2024-08-30T07:36:34.9823650Z                 mask[1:aux_firstnan] = (
2024-08-30T07:36:34.9823820Z                     aux[1:aux_firstnan] != aux[:aux_firstnan - 1])
2024-08-30T07:36:34.9823910Z             mask[aux_firstnan] = True
2024-08-30T07:36:34.9823990Z             mask[aux_firstnan + 1:] = False
2024-08-30T07:36:34.9824060Z         else:
2024-08-30T07:36:34.9824170Z >           mask[1:] = aux[1:] != aux[:-1]
2024-08-30T07:36:34.9824440Z E           ValueError: could not broadcast input array from shape (539,640) into shape (345599,)
2024-08-30T07:36:34.9824450Z 
2024-08-30T07:36:34.9824910Z /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/numpy/lib/arraysetops.py:352: ValueError

[tcompa]

I can reproduce it locally, in a python3.11 environment with this pip freeze

alabaster==1.0.0
anndata==0.10.9
annotated-types==0.7.0
anyio==4.4.0
app-model==0.2.8
appdirs==1.4.4
argon2-cffi==23.1.0
argon2-cffi-bindings==21.2.0
array_api_compat==1.8
arrow==1.3.0
asciitree==0.3.3
astropy==6.1.2
astropy-iers-data==0.2024.8.27.10.28.29
asttokens==2.4.1
async-lru==2.0.4
attrs==24.2.0
autopep8==2.3.1
babel==2.16.0
beautifulsoup4==4.12.3
bleach==6.1.0
build==1.2.1
cachey==0.2.1
cellpose==2.2.3
certifi==2024.8.30
cffi==1.17.0
charset-normalizer==3.3.2
click==8.1.7
cloudpickle==3.0.0
cmake==3.30.2
comm==0.2.2
contourpy==1.3.0
cycler==0.12.1
dask==2024.8.1
debugpy==1.8.5
decorator==5.1.1
defusedxml==0.7.1
devtools==0.12.2
docstring-parser==0.15
docutils==0.21.2
executing==2.0.1
fasteners==0.19
fastjsonschema==2.20.0
fastremap==1.15.0
filelock==3.13.4
flexcache==0.3
flexparser==0.3.1
fonttools==4.53.1
fqdn==1.5.1
-e git+ssh://[email protected]/fractal-analytics-platform/fractal-tasks-core.git@8e776f0a98e7dec51963220e02a547737b2e890b#egg=fractal_tasks_core
freetype-py==2.5.1
fsspec==2024.6.1
h11==0.14.0
h5py==3.11.0
HeapDict==1.0.1
hsluv==5.0.4
httpcore==1.0.5
httpx==0.27.2
idna==3.8
image_registration==0.2.9
imagecodecs==2024.6.1
imageio==2.35.1
imageio-ffmpeg==0.4.9
imagesize==1.4.1
importlib_metadata==8.4.0
in-n-out==0.2.1
iniconfig==2.0.0
ipycanvas==0.13.2
ipyevents==2.0.2
ipykernel==6.29.5
ipympl==0.9.4
ipython==8.26.0
ipython-genutils==0.2.0
ipywidgets==8.1.5
isoduration==20.11.0
jedi==0.19.1
Jinja2==3.1.4
json5==0.9.25
jsonpointer==3.0.0
jsonschema==4.23.0
jsonschema-specifications==2023.12.1
jupyter==1.1.1
jupyter-console==6.6.3
jupyter-events==0.10.0
jupyter-lsp==2.2.5
jupyter_client==8.6.2
jupyter_core==5.7.2
jupyter_server==2.14.2
jupyter_server_terminals==0.5.3
jupyterlab==4.2.5
jupyterlab_pygments==0.3.0
jupyterlab_server==2.27.3
jupyterlab_widgets==3.0.13
jupytext==1.16.4
kiwisolver==1.4.5
lazy_loader==0.4
lit==18.1.8
llvmlite==0.43.0
locket==1.0.0
loguru==0.7.2
lxml==4.9.4
magicgui==0.9.1
markdown-it-py==3.0.0
MarkupSafe==2.1.5
matplotlib==3.9.2
matplotlib-inline==0.1.7
mdit-py-plugins==0.4.1
mdurl==0.1.2
mistune==3.0.2
mpmath==1.3.0
napari==0.5.2
napari-assistant==0.4.7
napari-console==0.0.9
napari-plugin-engine==0.2.0
napari-segment-blobs-and-things-with-membranes==0.3.8
napari-skimage-regionprops==0.8.2
napari-svg==0.2.0
napari-time-slicer==0.5.0
napari-tools-menu==0.1.19
napari-workflows==0.2.11
natsort==8.4.0
nbclient==0.10.0
nbconvert==7.16.4
nbformat==5.10.4
nest-asyncio==1.6.0
networkx==3.3
notebook==7.2.2
notebook_shim==0.2.4
npe2==0.7.7
numba==0.60.0
numcodecs==0.13.0
numpy==1.26.4
numpydoc==1.8.0
nvidia-cublas-cu11==11.10.3.66
nvidia-cuda-cupti-cu11==11.7.101
nvidia-cuda-nvrtc-cu11==11.7.99
nvidia-cuda-runtime-cu11==11.7.99
nvidia-cudnn-cu11==8.5.0.96
nvidia-cufft-cu11==10.9.0.58
nvidia-curand-cu11==10.2.10.91
nvidia-cusolver-cu11==11.4.0.1
nvidia-cusparse-cu11==11.7.4.91
nvidia-nccl-cu11==2.14.3
nvidia-nvtx-cu11==11.7.91
opencv-python-headless==4.10.0.84
overrides==7.7.0
packaging==24.1
pandas==1.5.3
pandocfilters==1.5.1
parso==0.8.4
partd==1.4.2
pexpect==4.9.0
pillow==10.4.0
Pint==0.24.3
platformdirs==4.2.2
pluggy==1.5.0
pooch==1.8.2
prometheus_client==0.20.0
prompt_toolkit==3.0.47
psutil==6.0.0
psygnal==0.11.1
ptyprocess==0.7.0
pure_eval==0.2.3
pycodestyle==2.12.1
pyconify==0.1.6
pycparser==2.22
pydantic==2.8.2
pydantic-compat==0.1.2
pydantic_core==2.20.1
pyerfa==2.0.1.4
Pygments==2.18.0
PyOpenGL==3.1.7
pyparsing==3.1.4
pyperclip==1.9.0
pyproject_hooks==1.1.0
pytest==8.3.2
python-dateutil==2.9.0.post0
python-json-logger==2.0.7
pytz==2024.1
PyYAML==6.0.2
pyzmq==26.2.0
qtconsole==5.6.0
QtPy==2.4.1
referencing==0.35.1
requests==2.32.3
rfc3339-validator==0.1.4
rfc3986-validator==0.1.1
rich==13.8.0
roifile==2024.5.24
rpds-py==0.20.0
scikit-image==0.24.0
scipy==1.14.1
Send2Trash==1.8.3
shellingham==1.5.4
six==1.16.0
sniffio==1.3.1
snowballstemmer==2.2.0
soupsieve==2.6
Sphinx==8.0.2
sphinxcontrib-applehelp==2.0.0
sphinxcontrib-devhelp==2.0.0
sphinxcontrib-htmlhelp==2.1.0
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==2.0.0
sphinxcontrib-serializinghtml==2.0.0
stack-data==0.6.3
stackview==0.9.0
superqt==0.6.7
sympy==1.13.2
tabulate==0.9.0
terminado==0.18.1
tifffile==2024.8.28
tinycss2==1.3.0
tomli_w==1.0.0
toolz==0.12.1
torch==2.0.0
tornado==6.4.1
tqdm==4.66.5
traitlets==5.14.3
triton==2.0.0
typer==0.12.5
types-python-dateutil==2.9.0.20240821
typing_extensions==4.12.2
uri-template==1.3.0
urllib3==2.2.2
vispy==0.14.3
wcwidth==0.2.13
webcolors==24.8.0
webencodings==0.5.1
websocket-client==1.8.0
wget==3.2
widgetsnbextension==4.0.13
wrapt==1.16.0
zarr==2.18.2
zipp==3.20.1

[tcompa]

Here is an incomplete diff of the working/failing environmnets:

anndata 0.10.8 -> 0.10.9
tiffile 2024.8.10 -> 2024.8.10
stackview 0.8.2 -> 0.9.0
zipp 3.20.0 -> 3.20.1
astropy_iers_data
contourpy
pyparsing
certifi
jupyter
qtconsole
rich
typer
freetype
...

I confirm that the issue comes from stackview going from 0.8.2 to 0.9.0 (released on August 29th):

• stackview 0.8.2: napari-workflow tests pass
• stackview 0.9.0: napari-workflow tests fail - as described in the current issue

Here is the full diff between the two versions: haesleinhuepf/stackview@0.8.2...0.9.0

[tcompa]

I verified that haesleinhuepf/stackview#68 is the specific change that breaks the tests.
Starting from a stacview=0.9.0 environment and manually reverting this change makes the tests pass.

[tcompa]

Issue opened at haesleinhuepf/napari-workflows#47. Meanwhile I will simply pin stackview to 0.8.2.