Nellie

Automated organelle segmentation, tracking, and hierarchical feature extraction in 2D/3D live-cell microscopy

arXiv

Preprint Link | Cite

Abstract: The analysis of dynamic organelles remains a formidable challenge, though key to understanding biological processes. We introduce Nellie, an automated and unbiased pipeline for segmentation, tracking, and feature extraction of diverse intracellular structures. Nellie adapts to image metadata, eliminating user input. Nellie’s preprocessing pipeline enhances structural contrast on multiple intracellular scales allowing for robust hierarchical segmentation of sub-organellar regions. Internal motion capture markers are generated and tracked via a radius-adaptive pattern matching scheme, and used as guides for sub-voxel flow interpolation. Nellie extracts a plethora of features at multiple hierarchical levels for deep and customizable analysis. Nellie features a Napari-based GUI that allows for code-free operation and visualization, while its modular open-source codebase invites customization by experienced users.

Nellie's pipeline and Napari plugin are both very much in early stages, therefore I highly encourage any and all feedback.

Example output intermediates

full_mito_lower_res.mp4

Installation (~ 1 minute)

Notes:

• It is recommended (but usually not required) to create a new environment for Nellie to avoid conflicts with other packages.
• May take several minutes to install.
• Choose one of the following methods, and only one!
• If you do not already have Python 3.9 or higher installed, download it via the python website.

0_download_python.mov

Option 1. Via Napari plugin manager:

If not already installed, install Napari: https://napari.org/stable/tutorials/fundamentals/installation

1_install_nellie_napari.mov

1. Open Napari
2. Go to Plugins > Install/Uninstall Plugins...
3. Search for Nellie and click Install
4. Make sure Nellie is updated to the latest version.
5. Restart Napari.

Option 2. Via PIP:

1_install_nellie_terminal.mov

python3 -m pip install nellie

Option 2a for NVIDIA GPU acceleration, optional (Windows, Linux):

To use GPU acceleration via NVIDIA GPUs, you also need to install cupy:

pip install cupy-cudaXXx

• replace cupy-cudaXXx with the appropriate version for your CUDA version.
  • i.e. cupy-cuda11x for CUDA 11.x or cupy-cuda12x for CUDA 12.x
• if you don't have CUDA installed, go here.
• Mac Metal GPU-acceleration coming... eventually.

Usage

The sample dataset shown below is in the repo if you want to play around without, and can be downloaded here.

General data preparation

• It is strongly recommended to have your data in a parsable format, such as .ome.tif, .nd2, or other raw data files from microscopes.
  • Importing into ImageJ/FIJI and saving via BioFormats with the proper image dimensions should do the trick.
  • If the metadata cannot be parsed, you will have to manually enter it.
• It is also recommended to crop your image as much as possible to reduce processing time and memory usage. But really, unless you have massive lightsheet data, it should be pretty fast (seconds to minutes on a typical modern desktop computer).

3D + Timeseries dataset

2_use_nellie_tzyx.mov

3D (no Timeseries) dataset

2_use_nellie_zyx.mov

2D + Timeseries dataset

2_use_nellie_tyx.mov

Running Nellie's processing pipeline

1. Start Napari (open a Terminal and type napari)

   napari

2. Go to Plugins > Nellie (nellie) then to the File select tab.
3. Click Select File of Select Folder to select your image(s).
   • If the metadata boxes do not fill in automatically and turn red, this means Nellie did not detect that metadata portion from your image, and you must manually enter it or reformat your image and try again.
     • The metadata slot will appear green if it is in the correct format.
   • *Note, if you are batch processing, the metadata must be the same for all images if any of them are in an incorrect format (this will be fixed eventually). If they are different, but all pass validation, then it will process fine.
   • You can preview your image via the Open preview button once the metadata is filled in to ensure it looks correct.
   • From this tab, you can also choose what time points and channel you want to analyze, if your file contains more than one slice in those dimensions.
4. Click the Process tab.
5. You can run the full pipeline with Run Nellie, or run individual steps below.
   • Steps can only be run once its previous step has been run.
   • Likewise, visualizations in the Visualization tab can only be opened once its respective step has been run.
6. All intermediate files and output csvs will be saved to [image_directory]/nellie_output/, which can be accessed via the Open output directory button.
   • A separate .csv is created for each level of the organellar hierarchy.
7. Once features have been exported, Nellie will automatically detect this, and allow analysis via the Analyze tab.
   • Analysis at this point is optional, but can be helpful for visualizing, and selectively exporting data.

Using Nellie's visualization plugin

1. Follow the previous processing steps, you only need to do this once per file as long as you don't move or delete the files.
2. Open the Visualization tab
3. Select a visualization from the list.
   1. Raw: Visualize the raw data for the processed channel.
   2. Preprocessed: Visualize the contrast-enhanced data.
   3. Segmentation: Visualize the organelle and branch instance segmentation masks.
   4. Mocap Markers: Visualize the mocap markers used for waypoints.
   5. Reassigned Labels: Visualize the organelle and branch instance segmentation masks where voxels are reassigned based on the first timepoint.
4. To visualize tracks, open and select one of the segmentation layers.
5. To visualize all tracks of all organelles/branches, click the Visualize all frame labels' tracks button.
6. To visualize all tracks of a specific organelle/branch:
   1. Click on the layer, and use the eyedropper tool at the top to select an organelle/branch to track.
   2. Click the Visualize selected label's tracks.

Using Nellie's analysis plugin

1. Follow the previous processing steps, you only need to do this once per file as long as you don't move or delete the files.
2. Open the Analyze tab, select the hierarchy level you want to visualize from the dropdown.
3. Select the level-specific feature you want to visualize from the new dropdown.
4. A histogram of all the data will be displayed.
   • This histogram can be directly exported via the Save graph button. A .png will be saved to [image_directory]/nellie_output/graphs/ with the current datetime.
   • The values of the histogram can be exported via the Export graph data button. A .csv will be saved to [image_directory]/nellie_output/graphs/ with the current datetime.
   • The histogram's x-axis can be viewed in log10 scale via the Log scale checkbox.
   • By default, the histogram shows lines at the mean +/- 1 standard deviation. This can instead be switched to median and quartiles via the Median view checkbox.
5. Press the Overlay mask button to colormap the organelle mask based on your selected feature.
   • Once overlaid, toggle the Timepoint data checkbox to allow you to select a specific timepoint to visualize via the slider.

Other features

• Nellie's plugin offers an Easy screenshot feature:
  • Press the button under Easy screenshot or hit Ctrl/Cmd-Shift-E after clicking your image.
  • The .png will be saved to [image_directory]/nellie_output/screenshots/ with the current datetime.

Feedback / Getting Help

A few options are available for providing feedback or getting help with Nellie:

Github Issues | email | X | wherever else you can find me!

To avoid any unnecessary back-and-forth, please include any/all (if possible) of the following information in your bug report:

• What kind of computer do you have, and what are its specs?
• Send me screenshots of what is not working.
• Send me any error logs in your terminal.
• Send me the file you ran (if possible).
• Any other information that might be helpful

Other Info

For a 16bit dataset, the output:input ratio is ~15x. There is an option in the GUI to automatically delete intermediates after processing, keeping only the CSV files containing the extracted features.

Requirements

Nellie has been tested on the following configurations:

• Mac, Linux, and Windows operating systems
• Python >= 3.9

License

Nellie © 2024 by Austin E. Y. T. Lefebvre is licensed under CC BY 4.0

Reference

If you used Nelly or found this work useful in your own research, please cite our arXiv preprint:

Lefebvre, A. E. Y. T., Sturm, G., et. al. Nellie: Automated organelle segmentation, tracking, and hierarchical feature extraction in 2D/3D live-cell microscopy, arXiv, 2024, https://arxiv.org/abs/2403.13214

@misc{lefebvre2024nellie,
      title={Nellie: Automated organelle segmentation, tracking, and hierarchical feature extraction in 2D/3D live-cell microscopy}, 
      author={Austin E. Y. T. Lefebvre and Gabriel Sturm and Ting-Yu Lin and Emily Stoops and Magdalena Preciado Lopez and Benjamin Kaufmann-Malaga and Kayley Hake},
      year={2024},
      eprint={2403.13214},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

More fun examples

Microtubule growing ends:

microtubules_full.mp4

Endoplasmic reticulum:

full_er.mp4

Peroxisomes:

peroxisomes_full.mp4

Code contents:

Full documentation can be found within the code, and compiled by Sphinx in the file docs/_build/html/index.html

Nellie pipeline

All the Nellie pipeline code is found within the nellie folder

• File and metadata loading, and file preparation is found at nellie/im_info/verifier.py
• Preprocessing is found at nellie/segmentation/filtering.py
• Segmentation of organelles is found at nellie/segmentation/labelling.py
• Skeletonization and segmentation of branches is found at nellie/segmentation/networking.py
• Mocap marker detection is found at nellie/segmentation/mocap_marking.py
• Mocap marker tracking is found at nellie/tracking/hu_tracking.py
• Voxel reassignment via flow interpolation is found at nellie/tracking/voxel_reassignment.py
• Hierarchical feature extraction is found at nellie/feature_extraction/hierarchical.py

Nellie Napari plugin

All the Napari plugin code is found with the nellie_napari folder

• The home tab is found at nellie_napari/nellie_home.py
• The file selection tab is found at nellie_napari/nellie_fileselect.py
• The processing tab is found at nellie_napari/nellie_processor.py
• The visualization tab is found at nellie_napari/nellie_visualizer.py
• The analysis tab is found at nellie_napari/nellie_analysis.py
• The settings tab is found at nellie_napari/nellie_settings.py