Allow Amending NWB Files

[lawrence-mbf]

fix #109 Allows for opening a preexisting NWB file and efficiently writing it back out without rewriting all the datasets.

[bendichter]

hey is this ready to test?

[lawrence-mbf]

Yes

[bendichter]

running ecephys and then nwbExport(nwb, 'ecephys_tutorial.nwb') yeilds this error:

Error using hdf5lib2
The HDF5 library encountered an unknown error.

Error in H5D.write (line 100)
H5ML.hdf5lib2('H5Dwrite', varargin{:});

Error in io.writeDataset (line 12)
H5D.write(did, tid, sid, sid, 'H5P_DEFAULT', data);

Error in types.core.NWBFile/export (line 521)
                io.writeDataset(fid, [fullpath
                '/file_create_date'],
                class(obj.file_create_date),
                obj.file_create_date, true);

Error in nwbfile/export (line 58)
                refs = [email protected](obj,
                output_file_id, '/', {});

Error in nwbExport (line 35)
    export(nwb(i), fn);

[lawrence-mbf]

So this looks like an error due to increasing the size of a dataset. This is due to file_create_date increasing in size every time the nwb file is exported. Unfortunately, the only way to allow dynamically resizing a dataset that's already written is to guarantee that it was saved in chunked mode (which we don't use at the moment). Considering chunking mode is optional, do you happen to know if this constraint also exists in pynwb if the dataset layout is not chunked?

[bendichter]

Yeah good point it looks like pynwb stores file_create_date as contiguous

[oruebel]

The chunked vs. contiguous store for file_create_date is really an HDF5 issue, i.e., if HDF5 can't expand it in Matlab than I'd expect PyNWB to see the same problem.

[lawrence-mbf]

It looks like this particular feature doesn't quite exist in pynwb yet: NeurodataWithoutBorders/pynwb#990.

Is there another way to append data to a previously written dataset?

[lawrence-mbf]

I've added a solution. For the special property file_create_date and only file_create_date, we force chunking. This will allow for the regular modification behavior. For pynwb-created files, we print a warning but ignore writing the dataset which does match pynwb behavior for file_create_date if the user doesn't modify it themselves.

[bendichter]

That sounds like a good solution to me

[bendichter]

@ln-vidrio could you write a test for this?

[lawrence-mbf]

@bendichter sure, I'll take a look at it.

[bendichter]

@ln-vidrio Thanks for working on this! It looks like there are a bunch of merge conflicts now. Would you mind sorting through that?

[bendichter]

ecephys:

Undefined function or variable 'fid'.

Error in NwbFile/export (line 54)
                obj.embedSpecifications(fid);

Error in nwbExport (line 35)
    ex
port(nwb(i), filename);

Error in ecephys (line 278)
nwbExport(nwb, 'ecephys_tutorial.nwb')

[lawrence-mbf]

yeah it's not done yet. I'll let you know when I get the tests running.

[lawrence-mbf]

@bendichter This should be okay now I think. Let me know if there are any issues. I've also added a test type for testing amend with any of the other system tests.

[bendichter]

@ln-vidrio

I ran into a bug with links:

>> ecephys

...

Error using hdf5lib2
The HDF5 library encountered an error and produced the following
stack trace information:

    H5L_link_cb           name already exists
    H5G_traverse_real     traversal operator failed
    H5G_traverse          internal path traversal failed
    H5L_create_real       can't insert link
    H5L_create_ud         unable to register new name for object
    H5Lcreate_external    unable to create link

Error in H5L.create_external (line 34)
H5ML.hdf5lib2('H5Lcreate_external', varargin{:});

Error in types.untyped.ExternalLink/export (line 71)
            H5L.create_external(obj.filename, obj.path, fid,
            fullpath, plist, plist);

Error in types.core.NWBFile/export (line 690)
            refs = obj.general_subject.export(fid, [fullpath
            '/general/subject'], refs);

Error in NwbFile/export (line 55)
                refs = [email protected](obj, output_file_id,
                '/', {});

Error in nwbExport (line 35)
    export(nwb(i), filename);

Error in ecephys (line 350)
nwbExport(nwb3, 'link_test.nwb')

[lawrence-mbf]

works for me now.

[bendichter]

Works for me as well. I think this is ready.