Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add read_nwb_method for local paths in both hdf5 and zarr #1994

Open
wants to merge 10 commits into
base: dev
Choose a base branch
from
11 changes: 6 additions & 5 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,13 @@
## PyNWB 3.0.0 (Upcoming)

### Enhancements and minor changes
- Added `pynwb.read_nwb` convenience method to simplify reading an NWBFile written with any backend @h-mayorquin [#1994](https://github.com/NeurodataWithoutBorders/pynwb/pull/1994)
- Added support for NWB schema 2.8.0. @rly [#2001](https://github.com/NeurodataWithoutBorders/pynwb/pull/2001)
- Removed `SpatialSeries.bounds` field that was not functional. This will be fixed in a future release. @rly [#1907](https://github.com/NeurodataWithoutBorders/pynwb/pull/1907), [#1996](https://github.com/NeurodataWithoutBorders/pynwb/pull/1996)
- Added support for `NWBFile.was_generated_by` field. @stephprince [#1924](https://github.com/NeurodataWithoutBorders/pynwb/pull/1924)
- Added support for `model_number`, `model_name`, and `serial_number` fields to `Device`. @stephprince [#1997](https://github.com/NeurodataWithoutBorders/pynwb/pull/1997)
- Deprecated `EventWaveform` neurodata type. @rly [#1940](https://github.com/NeurodataWithoutBorders/pynwb/pull/1940)
- Deprecated `ImageMaskSeries` neurodata type. @rly [#1941](https://github.com/NeurodataWithoutBorders/pynwb/pull/1941)
- Removed `SpatialSeries.bounds` field that was not functional. This will be fixed in a future release. @rly [#1907](https://github.com/NeurodataWithoutBorders/pynwb/pull/1907), [#1996](https://github.com/NeurodataWithoutBorders/pynwb/pull/1996)
- Added support for `NWBFile.was_generated_by` field. @stephprince [#1924](https://github.com/NeurodataWithoutBorders/pynwb/pull/1924)
- Added support for `model_number`, `model_name`, and `serial_number` fields to `Device`. @stephprince [#1997](https://github.com/NeurodataWithoutBorders/pynwb/pull/1997)
- Deprecated `EventWaveform` neurodata type. @rly [#1940](https://github.com/NeurodataWithoutBorders/pynwb/pull/1940)
- Deprecated `ImageMaskSeries` neurodata type. @rly [#1941](https://github.com/NeurodataWithoutBorders/pynwb/pull/1941)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- Removed `SpatialSeries.bounds` field that was not functional. This will be fixed in a future release. @rly [#1907](https://github.com/NeurodataWithoutBorders/pynwb/pull/1907), [#1996](https://github.com/NeurodataWithoutBorders/pynwb/pull/1996)
- Added support for `NWBFile.was_generated_by` field. @stephprince [#1924](https://github.com/NeurodataWithoutBorders/pynwb/pull/1924)
- Added support for `model_number`, `model_name`, and `serial_number` fields to `Device`. @stephprince [#1997](https://github.com/NeurodataWithoutBorders/pynwb/pull/1997)
- Deprecated `EventWaveform` neurodata type. @rly [#1940](https://github.com/NeurodataWithoutBorders/pynwb/pull/1940)
- Deprecated `ImageMaskSeries` neurodata type. @rly [#1941](https://github.com/NeurodataWithoutBorders/pynwb/pull/1941)
- Removed `SpatialSeries.bounds` field that was not functional. This will be fixed in a future release. @rly [#1907](https://github.com/NeurodataWithoutBorders/pynwb/pull/1907), [#1996](https://github.com/NeurodataWithoutBorders/pynwb/pull/1996)
- Added support for `NWBFile.was_generated_by` field. @stephprince [#1924](https://github.com/NeurodataWithoutBorders/pynwb/pull/1924)
- Added support for `model_number`, `model_name`, and `serial_number` fields to `Device`. @stephprince [#1997](https://github.com/NeurodataWithoutBorders/pynwb/pull/1997)
- Deprecated `EventWaveform` neurodata type. @rly [#1940](https://github.com/NeurodataWithoutBorders/pynwb/pull/1940)
- Deprecated `ImageMaskSeries` neurodata type. @rly [#1941](https://github.com/NeurodataWithoutBorders/pynwb/pull/1941)

These updates were indented to indicate schema 2.8.0 related changes


## PyNWB 2.8.3 (November 19, 2024)

Expand Down
14 changes: 8 additions & 6 deletions docs/gallery/general/plot_read_basics.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,6 @@
import matplotlib.pyplot as plt
import numpy as np

from pynwb import NWBHDF5IO

####################
# We will access NWB data on the `DANDI Archive <https://gui.dandiarchive.org/>`_,
Expand Down Expand Up @@ -103,14 +102,17 @@
# read the data into a :py:class:`~pynwb.file.NWBFile` object.

filepath = "sub-P11HMH_ses-20061101_ecephys+image.nwb"
# Open the file in read mode "r",
io = NWBHDF5IO(filepath, mode="r")
nwbfile = io.read()
from pynwb import read_nwb

nwbfile = read_nwb(filepath)
Comment on lines +105 to +107
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think once the simplified read_nwb with streaming is supported we may want to go through the other tutorials that use io.read() in read only mode and update those as well.

nwbfile

#######################################
# :py:class:`~pynwb.NWBHDF5IO` can also be used as a context manager:
# For more advanced use cases, the :py:class:~pynwb.NWBHDF5IO class provides additional functionality.
# Below, we demonstrate how :py:class:~pynwb.NWBHDF5IO can be used as a context manager
# to read data from an NWB file in a more controlled manner:

from pynwb import NWBHDF5IO
with NWBHDF5IO(filepath, mode="r") as io2:
nwbfile2 = io2.read()

Expand Down Expand Up @@ -291,4 +293,4 @@
# -----------------------
# It is good practice, especially on Windows, to close any files that you have opened.

io.close()
nwbfile.get_read_io().close()
3 changes: 3 additions & 0 deletions requirements-opt.txt
Original file line number Diff line number Diff line change
Expand Up @@ -6,3 +6,6 @@ oaklib==0.5.32; python_version >= "3.9"
fsspec==2024.10.0
requests==2.32.3
aiohttp==3.10.11

# For read_nwb tests
hdmf-zarr
65 changes: 65 additions & 0 deletions src/pynwb/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -535,6 +535,71 @@

return nwbfile

@docval({'name': 'path', 'type': (str, Path), 'doc': 'the path to the nwbfile'},
is_method=False)
def read_nwb(**kwargs):
"""Read an NWB file from a local path.

High-level interface for reading NWB files. Automatically handles both HDF5
and Zarr formats. For advanced use cases (parallel I/O, custom namespaces),
use NWBHDF5IO or NWBZarrIO.

Parameters
----------
path : str or pathlib.Path
Path to the NWB file. Can be either a local filesystem path to an HDF5 (.nwb)
or Zarr (.zarr) file

Returns
-------
pynwb.NWBFile
The loaded NWB file object.

See Also
--------
pynwb.NWBHDF5IO : Core I/O class for HDF5 files with advanced options.
hdmf_zarr.nwb.NWBZarrIO : Core I/O class for Zarr files with advanced options.

Notes
-----
This function uses the following defaults:
* Always opens in read-only mode
* Automatically loads namespaces
* Detects file format based on extension
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* Detects file format based on extension
* Detects file format based on whether supported backend IO classes can read the file

I think in the current implementation other extensions will work since can_read and IO.read_nwb methods just check whether the IO can successfully open the file?


Advanced features requiring direct use of IO classes include:
* Streaming data from s3
* Custom namespace extensions
* Parallel I/O with MPI
* Custom build managers
* Write or append modes
* Pre-opened HDF5 file objects or Zarr stores
* Remote file access configuration

Examples
--------
Read a local NWB file:

>>> from pynwb import read_nwb
>>> nwbfile = read_nwb("path/to/file.nwb")


"""

path = popargs('path', kwargs)
backend_is_hdf5 = NWBHDF5IO.can_read(path=path)

Check warning on line 590 in src/pynwb/__init__.py

View check run for this annotation

Codecov / codecov/patch

src/pynwb/__init__.py#L589-L590

Added lines #L589 - L590 were not covered by tests
if backend_is_hdf5:
return NWBHDF5IO.read_nwb(path=path)

Check warning on line 592 in src/pynwb/__init__.py

View check run for this annotation

Codecov / codecov/patch

src/pynwb/__init__.py#L592

Added line #L592 was not covered by tests
else:
from hdmf_zarr import NWBZarrIO
backend_is_zarr = NWBZarrIO.can_read(path=path)

Check warning on line 595 in src/pynwb/__init__.py

View check run for this annotation

Codecov / codecov/patch

src/pynwb/__init__.py#L594-L595

Added lines #L594 - L595 were not covered by tests
if backend_is_zarr:
return NWBZarrIO.read_nwb(path=path)

Check warning on line 597 in src/pynwb/__init__.py

View check run for this annotation

Codecov / codecov/patch

src/pynwb/__init__.py#L597

Added line #L597 was not covered by tests
else:
raise ValueError(f"Unsupported backend for file: {path}")

Check warning on line 599 in src/pynwb/__init__.py

View check run for this annotation

Codecov / codecov/patch

src/pynwb/__init__.py#L599

Added line #L599 was not covered by tests
Comment on lines +590 to +595
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
from hdmf_zarr import NWBZarrIO
backend_is_zarr = NWBZarrIO.can_read(path=path)
if backend_is_zarr:
return NWBZarrIO.read_nwb(path=path)
else:
raise ValueError(f"Unsupported backend for file: {path}")
try:
from hdmf_zarr import NWBZarrIO
backend_is_zarr = NWBZarrIO.can_read(path=path)
if backend_is_zarr:
return NWBZarrIO.read_nwb(path=path)
else:
raise ValueError(f"Unsupported backend for file: {path}")
except ImportError:
raise ValueError(f"Unsupported backend for file: '{path}'. If you are trying to read a Zarr file, make sure you have hdmf-zarr installed.")

I believe read_nwb will currently return an ImportError if the user provides an invalid path but does not have hdmf_zarr installed. Can you support this case?




from . import io as __io # noqa: F401,E402
from .core import NWBContainer, NWBData # noqa: F401,E402
from .base import TimeSeries, ProcessingModule # noqa: F401,E402
Expand Down
2 changes: 1 addition & 1 deletion tests/integration/hdf5/test_io.py
Original file line number Diff line number Diff line change
Expand Up @@ -613,7 +613,7 @@ def test_read_nwb_method_file(self):
io.write(self.nwbfile)

import h5py

file = h5py.File(self.path, 'r')

read_nwbfile = NWBHDF5IO.read_nwb(file=file)
Expand Down
48 changes: 48 additions & 0 deletions tests/integration/io/test_read.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
from pathlib import Path
import tempfile

from pynwb import read_nwb
from pynwb.testing.mock.file import mock_NWBFile
from pynwb.testing import TestCase

import unittest
try:
from hdmf_zarr import NWBZarrIO # noqa f401
HAVE_NWBZarrIO = True
except ImportError:
HAVE_NWBZarrIO = False


class TestReadNWBMethod(TestCase):
"""
Test that H5DataIO functions correctly on round trip with the HDF5IO backend
"""
def setUp(self):
self.nwbfile = mock_NWBFile()


def test_read_nwb_hdf5(self):
from pynwb import NWBHDF5IO

with tempfile.TemporaryDirectory() as temp_dir:
path = Path(temp_dir) / "test.nwb"
with NWBHDF5IO(path, 'w') as io:
io.write(self.nwbfile)

read_nwbfile = read_nwb(path=path)
self.assertContainerEqual(read_nwbfile, self.nwbfile)
read_nwbfile.get_read_io().close()

@unittest.skipIf(not HAVE_NWBZarrIO, "NWBZarrIO library not available")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add tests for

  1. the user providing a path that does not exist
  2. if the user does not have hdmf_zarr installed and provides an unsupported file path

def test_read_zarr(self):
# from pynwb import NWBZarrIO
from hdmf_zarr import NWBZarrIO

with tempfile.TemporaryDirectory() as temp_dir:
path = Path(temp_dir) / "test.nwb"
with NWBZarrIO(path, 'w') as io:
io.write(self.nwbfile)

read_nwbfile = read_nwb(path=path)
self.assertContainerEqual(read_nwbfile, self.nwbfile)
read_nwbfile.get_read_io().close()
Loading