Merge pull request #95 from AICONSlab/jo-dependencies_fix_ace_docs

Release 2.2.6
AICONSlab · Dec 22, 2023 · 97f95f7 · 97f95f7
2 parents af562a2 + 94b0c46
commit 97f95f7
Show file tree

Hide file tree

Showing 13 changed files with 253 additions and 31 deletions.
diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -62,7 +62,7 @@ jobs:
             # Change permissions for code directory for any Singularity user
             sed -i '/#STOPUNCOMMENT#/a # Change permissions for code dir\nchmod -R 777 /code' miracl.def && \
             # Log in to cloud.sylabs.io
-            echo $SYLABS_TOKEN_APR_04_2023 > tokenfile && \
+            echo $SYLABS_TOKEN_DEC_22_2023 > tokenfile && \
             singularity remote login -u aiconslab --tokenfile tokenfile && \
             # Setting trap in case image cannot be deleted
             set -e

diff --git a/README.md b/README.md
@@ -6,25 +6,32 @@
 
 ___
 
-MIRACL (Multi-modal Image Registration And Connectivity anaLysis)
-is a general-purpose, open-source pipeline for automated:
+**MIRACL** (**M**ulti-modal **I**mage **R**egistration **A**nd **C**onnectivity ana**L**ysis) is a general-purpose, open-source pipeline for automated:
 
-1) Registration of mice clarity data to the Allen reference atlas
-2) Segmentation & feature extraction of mice clarity data in 3D (Sparse & nuclear staining)
-3) Registration of mice multimodal imaging data (MRI & CT, in-vivo & ex-vivo) to Allen reference atlas
-4) Tract or label specific connectivity analysis based on the Allen connectivity atlas
-5) Comparison of diffusion tensort imaging (DTI)/tractography, virus tracing using CLARITY &
-   Allen connectivity atlas
-6) Statistical analysis of CLARITY & Imaging data
-7) Atlas generation & Label manipulation
+1. Registration of cleared and imaging data (ex. LSFM and MRI) to atlases (ex. Allen Reference Atlas)
+2. 3D Segmentation and feature extraction of cleared data
+3. Tract-specific or network-level connectivity analysis
+4. Statistical analysis of cleared and imaging data
+5. Comparison of dMRI/tractography, virus tracing, and connectivity atlases
+6. Atlas generation and Label manipulation
 
-Copyright (c) 2022 Maged Goubran, [email protected]
+___
 
-All Rights Reserved
+\*\***NEW WORKFLOW/FEATURE RELEASE**\*\*
+
+We will be releasing our AI-based Cartography of Ensembles (ACE) workflow, an end-to-end, automated pipeline that integrates cutting-edge deep learning 
+segmentation models and advanced statistical methods to enable unbiased and generalizable brain-wide mapping of 3D alterations in neuronal activity, 
+morphology, or connectivity at the sub-regional and laminar levels beyond atlas-defined regions.
+
+ACE will be available as part of MIRACL in **January 2024**.
 
 ___
 
-We provide containers for using the software (Docker and Singularity) as well as
-local install instructions. For more details, see our [docs](https://miracl.readthedocs.io). 
-Note that the base image for the docker container can be found in [docker](docker) and
-the container `mgoubran/miracl` is built on top of that.
+We recommend using MIRACL with the Docker or Singularity containers we provide but it can also be installed locally.  For more details, see our 
+[docs](https://miracl.readthedocs.io).
+
+___
+
+Copyright (c) 2023 Maged Goubran
+
+All Rights Reserved
diff --git a/docs/conf.py b/docs/conf.py
@@ -10,8 +10,8 @@
 copyright = '2023, Maged Goubran @ AICONS Lab'
 author = 'Maged Goubran'
 # Version and release set to the same since no separation is needed
-version = '2.2.5'
-release = '2.2.5'
+version = '2.2.6'
+release = '2.2.6'
 
 # -- General configuration ---------------------------------------------------
 # https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration

diff --git a/docs/directives/miracl_overview_directive.rst b/docs/directives/miracl_overview_directive.rst
@@ -2,14 +2,10 @@
 **C**\ onnectivity ana\ **L**\ ysis) is a general-purpose, open-source pipeline 
 for automated:
 
-#. Registration of mice CLARITY data to the Allen Reference Atlas
-#. Segmentation and feature extraction of mice CLARITY data in 3D (Sparse and 
-   nuclear staining)
-#. Registration of mice multimodal imaging data (MRI & CT, in-vivo & ex-vivo) 
-   to Allen Reference Atlas
-#. Tract or label specific connectivity analysis based on the Allen 
-   Connectivity Atlas
-#. Comparison of diffusion tensort imaging (DTI)/tractography, virus tracing 
-   using CLARITY and Allen Connectivity Atlas
-#. Statistical analysis of CLARITY and imaging data
+#. Registration of cleared and imaging data (ex. LSFM and MRI) to atlases
+   (ex. Allen Reference Atlas)
+#. 3D Segmentation and feature extraction of cleared data
+#. Tract-specific or network-level connectivity analysis
+#. Statistical analysis of cleared and imaging data
+#. Comparison of dMRI/tractography, virus tracing, and connectivity atlases
 #. Atlas generation and Label manipulation
diff --git a/docs/index.rst b/docs/index.rst
@@ -33,6 +33,15 @@ Welcome to MIRACL's documentation!
 
 .. include:: ./directives/miracl_overview_directive.rst
 
+\*\*NEW WORKFLOW/FEATURE RELEASE\*\*
+"""""""""""""""""""""""""""""""""""""""""
+
+We will be releasing our AI-based Cartography of Ensembles (ACE) workflow, an end-to-end, automated pipeline that integrates cutting-edge deep learning segmentation models and advanced statistical methods to enable unbiased and generalizable brain-wide mapping of 3D alterations in neuronal activity, morphology, or connectivity at the sub-regional and laminar levels beyond atlas-defined regions.
+
+:program:`ACE` will be available as part of MIRACL in January 2024.
+
+------
+
 We recommend using MIRACL with the Docker or Singularity containers we provide 
 but it can also be installed locally. See our 
 :doc:`installation instructions <./installation/installation>` for more information.

diff --git a/docs/tutorials/getting_started.rst b/docs/tutorials/getting_started.rst
@@ -1,7 +1,7 @@
 Getting started
 ###############
 
-MIRACL can be run in the command-line or by using its GUI.
+:program:`MIRACL` can be run in the command-line or by using its GUI.
 
 Command-line
 ============

diff --git a/docs/tutorials/index.rst b/docs/tutorials/index.rst
@@ -20,5 +20,6 @@ for a tutorial on :program:`MIRACL's` general usage.
    workflows/index
    conversion/index
    registration/index
+   segmentation/index
    utilities/index
    slurm/index
diff --git a/docs/tutorials/segmentation/ace_seg/ace_seg.rst b/docs/tutorials/segmentation/ace_seg/ace_seg.rst
@@ -0,0 +1,79 @@
+ACE Segmentation Function
+#########################
+
+Cutting edge vision transformer and CNN-based DL architectures trained on very 
+large LSFM datasets to map cFos brain-wide.
+
+CLI
+===
+
+To look at the arguments that need to be provided to the function, invoke the 
+help menu using:
+
+.. code-block::
+
+   $ miracl seg ace -h
+
+The following menu will be printed to the terminal:
+
+.. code-block::
+
+   usage: miracl ace [-h] -sai SA_INPUT_FOLDER -sao SA_OUTPUT_FOLDER -sam
+                     {unet,unetr,ensemble} [-sas height width depth]
+                     [-sar X-res Y-res Z-res] [-saw SA_NR_WORKERS]
+                     [-sac SA_CACHE_RATE] [-sasw SA_SW_BATCH_SIZE] [-samc] [-sav]
+                     [-sau]
+   
+   AI-based Cartography of Ensembles (ACE) segmentation method
+   
+   optional arguments:
+     -h, --help            show this help message and exit
+     -sai SA_INPUT_FOLDER, --sa_input_folder SA_INPUT_FOLDER
+                           path to raw tif/tiff data folder
+     -sao SA_OUTPUT_FOLDER, --sa_output_folder SA_OUTPUT_FOLDER
+                           path to output file folder
+     -sam {unet,unetr,ensemble}, --sa_model_type {unet,unetr,ensemble}
+                           model architecture
+     -sas height width depth, --sa_image_size height width depth
+                           image size (type: int; default: fetched from image
+                           header)
+     -sar X-res Y-res Z-res, --sa_resolution X-res Y-res Z-res
+                           voxel size (type: _validate_vox_res)
+     -saw SA_NR_WORKERS, --sa_nr_workers SA_NR_WORKERS
+                           number of cpu cores deployed to pre-process image
+                           patches in parallel (type: int; default: 4)
+     -sac SA_CACHE_RATE, --sa_cache_rate SA_CACHE_RATE
+                           percentage of raw data that is loaded into cpu during
+                           segmentation (type: float; default: 0.0)
+     -sasw SA_SW_BATCH_SIZE, --sa_sw_batch_size SA_SW_BATCH_SIZE
+                           number of image patches being processed by the model
+                           in parallel on gpu (type: int; default: 4)
+     -samc, --sa_monte_dropout
+                           use Monte Carlo dropout (default: False)
+     -sav, --sa_visualize_results
+                           visualizing model output after predictions (default:
+                           False)
+     -sau, --sa_uncertainty_map
+                           enable map (default: False)
+
+.. table::
+
+   ===============================  =====================  =========  =======================================================================================  =============================
+   Flag                             Parameter              Type       Description                                                                              Default
+   ===============================  =====================  =========  =======================================================================================  =============================
+   \-sai, \-\-sa_input_folder       SA_INPUT_FOLDER        ``str``    path to raw tif/tiff data folder                                                         ``None`` (required)
+   \-sao, \-\-sa_output_folder      SA_OUTPUT_FOLDER       ``str``    path to output file folder                                                               ``None`` (required)
+   \-sam, \-\-sa_model_type         {unet,unetr,ensemble}  ``str``    model architecture                                                                       ``None`` (required)
+   \-sas, \-\-sa_image_size         height width depth     ``int``    image size; provided as three arguments                                                  ``fetched from image header``
+   \-sar, \-\-sa_resolution         X-res Y-res Z-res      ``int``    voxel resolution; provided as three arguments                                            ``None`` (required)
+   \-saw, \-\-sa_nr_workers         SA_NR_WORKERS          ``int``    number of cpu cores deployed to pre-process image patches in parallel                    ``4``
+   \-sac, \-\-sa_cache_rate         SA_CACHE_RATE          ``float``  percentage of raw data that is loaded into cpu during segmentation                       ``0.0``
+   \-sasw,\ \--sa_sw_batch_size     SA_SW_BATCH_SIZE       ``int``    number of image patches being processed by the model in parallel on gpu                  ``4``
+   \-samc,\ \--sa_monte_dropout     True/False             ``bool``   use Monte Carlo dropout                                                                  ``False``
+   \-sav, \-\-sa_visualize_results  True/False             ``bool``   visualizing model output after predictions                                               ``False``
+   \-sau, \-\-sa_uncertainty_map    True/False             ``bool``   enable map                                                                               ``False``
+   ===============================  =====================  =========  =======================================================================================  =============================
+
+.. note::
+
+   The ``-sa`` in the flag part stands for ``segmentation ACE``.
diff --git a/docs/tutorials/segmentation/index.rst b/docs/tutorials/segmentation/index.rst
@@ -0,0 +1,12 @@
+Segmentation
+############
+
+Functions to segment tiff image files.
+
+.. include:: ../../directives/missing_fns_tutorials_directive.rst
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Table of contents:
+
+   ace_seg/ace_seg
diff --git a/docs/tutorials/workflows/ace_flow/ace_flow.rst b/docs/tutorials/workflows/ace_flow/ace_flow.rst
@@ -0,0 +1,116 @@
+ACE Workflow
+############
+
+**A**\ I-based **C**\ artography of **E**\ nsembles (**ACE**) pipeline highlights:
+
+1. Cutting-edge vision transformer and CNN-based DL architectures trained on 
+   very large LSFM datasets to map brain-wide neuronal activity
+2. Optimized cluster-wise statistical analysis with a threshold-free 
+   enhancement approach to chart subpopulation-specific effects at the laminar 
+   and local level, without restricting the analysis to atlas-defined regions.
+3. Modules for providing DL model uncertainty estimates and fine-tuning
+4. Interface with MIRACL registration to create study-specific atlases
+5. Ability to account for covariates at the cluster level and map the 
+   connectivity between clusters of activations.
+
+.. note::
+
+   The :program:`ACE Segmentation Function` and the :program:`ACE 
+   Workflow` are currently in the final stages of testing and will 
+   be available for download in version ``2.3.1`` which will be 
+   released in **early January 2024**.
+
+Main outputs
+============
+
+.. code-block::
+
+   clar_allen_reg
+   conv_final
+   reg_final
+   seg_final
+   vox_final
+   warp_final
+   heatmap_final
+   cluster_final
+
+Executes:
+
+.. code-block::
+
+   seg/ace_interface.py
+   conv/miracl_conv_convertTIFFtoNII.py
+   reg/miracl_reg_clar-allen.sh
+   seg/miracl_seg_voxelize_parallel.py
+   reg/miracl_reg_warp_clar_data_to_allen.sh
+   stats/miracl_stats_heatmap_group.py
+   stats/miracl_workflow_ace_stats.py
+   stats/miracl_summarize_clusters_corr.py
+
+.. note::
+
+   :program:`ACE's` workflow is currently only available on the command line. 
+   We are currently working on the GUI integration which will be integrated
+   in **early 2024**.
+
+CLI
+===
+
+To get more information about the workflow and its required arguments 
+use the following command on the cli:
+
+.. code-block::
+
+   $ miracl flow ace -h
+
+The following information will be printed to the terminal:
+
+.. code-block::
+
+   usage: miracl flow ace -sai input_folder -sao output_folder -sam model_type 
+          -pcsw wild_type_dir -pcsd disease_group_dir -pcso output_directory
+
+     1) Segments images with ACE
+     2) Convert raw tif/tiff files to nifti for registration
+     3) Registers CLARITY data (down-sampled images) to Allen Reference mouse brain atlas
+     4) Voxelizes segmentation results into density maps with Allen atlas resolution
+     5) Warps downsampled CLARITY data/channels from native space to Allen atlas
+
+   required arguments:
+     -sai SA_INPUT_FOLDER, --sa_input_folder SA_INPUT_FOLDER
+                           path to raw tif/tiff data folder
+     -sao SA_OUTPUT_FOLDER, --sa_output_folder SA_OUTPUT_FOLDER
+                           path to output file folder
+     -sam {unet,unetr,ensemble}, --sa_model_type {unet,unetr,ensemble}
+                           model architecture
+     -pcsw PCS_WILD_TYPE, --pcs_wild_type PCS_WILD_TYPE
+                           wild type group directory (should contain warped
+                           voxelized nifti files)
+     -pcsd PCS_DISEASE, --pcs_disease PCS_DISEASE
+                           disease group directory (should contain warped
+                           voxelized nifti files)
+
+.. note::
+
+   There are a number of optional arguments that can be provided to the
+   respective function invoked by the workflow. These arguments have been 
+   ommitted here for readability but will be printed when actually using 
+   :program:`MIRACL`.
+
+.. table::
+
+   ===========================  =====================  =======  ================================
+   Flag                         Parameter              Type     Description                     
+   ===========================  =====================  =======  ================================
+   \-sai, \-\-sa_input_folder   SA_INPUT_FOLDER        ``str``  path to raw tif/tiff data folder
+   \-sao, \-\-sa_output_folder  SA_OUTPUT_FOLDER       ``str``  path to output file folder      
+   \-sam, \-\-sa_model_type     {unet,unetr,ensemble}  ``str``  model architecture              
+   \-pcsw, \-\-pcs_wild_type    PCS_WILD_TYPE          ``str``  wild type group directory
+   \-pcsd, \-\-pcs_disease      PCS_DISEASE            ``str``  disease group directory
+   ===========================  =====================  =======  ================================
+
+Example usage:
+
+.. code-block::
+
+   $ miracl flow ace -sai /data/cells -sao /data/output -sam unet -pcsw /data/wild -pcsd /data/disease -pcso 
diff --git a/docs/tutorials/workflows/index.rst b/docs/tutorials/workflows/index.rst
@@ -31,6 +31,7 @@ Or structure tensor analysis:
    :maxdepth: 2
    :caption: Table of contents:
 
+   ACE **new feature** <ace_flow/ace_flow>
    CLARITY-Allen registration <clarity_registration/clar_reg>
    STA <sta/sta>
    CLARITY segmentation <clarity_segmentation/clar_seg>
diff --git a/miracl/version.txt b/miracl/version.txt
@@ -1 +1 @@
-2.2.5
+2.2.6
diff --git a/setup.py b/setup.py
@@ -54,6 +54,7 @@
         'dipy==1.6.0',
         'seaborn==0.12.2',
         'pyqt5==5.15.8',
+        'SimpleITK==2.2.1',
         'nipype==1.8.5'
         ],
     entry_points={'console_scripts': ['miracl=miracl.cli:main'],