Merge pull request #51 from MELDProject/dev_docker

Final fixes

README.md

@@ -4,9 +4,6 @@
 
 **Full documentation: [here](https://meld-graph.readthedocs.io/en/latest/index.html)**
 
-## ! The code is still undergoing development and review. 
-## ! Please DO NOT INSTALL yet
-
 Graph based FCD lesion segmentation for the [MELD project](https://meldproject.github.io/).
 
 This package is a pipeline to segment FCD-lesions from MRI scans. 
@@ -45,9 +42,9 @@ The MELD surface-based graph FCD detection algorithm is intended for research pu
 
 ### Installations available 
 You can install and use the MELD FCD prediction pipeline with :
-- [**docker container**](https://meld-graph.readthedocs.io/en/latest/install_docker.html) (STILL IN PROGRESS) recommended for easy installation of the pipeline as all the prerequisite packages are already embeded into the container. Note: Dockers are not working on High Performance Computing (HCP) systems.
+- [**docker container**](https://meld-graph.readthedocs.io/en/latest/install_docker.html) (STILL IN PROGRESS) recommended for easy installation of the pipeline as all the prerequisite packages are already embeded into the container. Note: Dockers are not working on High Performance Computing (HCP) systems ; Not tested on Windows 
 - [**native installation**](https://meld-graph.readthedocs.io/en/latest/install_native.html) recommended for Mac and users that want to modify the code and/or use the code to train/test their own classifier. 
-- **singularity container (COMING SOON)** enables to run a container on High Performance Computing (HCP) systems. 
+- [**singularity container (Not tested yet)**](https://meld-graph.readthedocs.io/en/latest/install_singularity.html) enables to run a container on High Performance Computing (HCP) systems. 
 
 ### Running the pipeline 
 Once installed you will be able to use the MELD FCD prediction pipeline on your data following the steps:

docs/harmonisation.md

@@ -9,9 +9,10 @@ Harmonisation of your patient data is not mandatory but recommended, to remove a
 
 The harmonisation parameters are computed using [Distributed Combat](https://doi.org/10.1016/j.neuroimage.2021.118822).
 To get these parameters you will need a cohort of subjects acquired from the same scanner and under the same protocol (sequence, parameters, ...).
-Subjects can be controls and/or patients, but we advise to use ***at least 20 subjects*** to enable an accurate harmonisation. 
+Subjects can be controls and/or patients, but we advise to use ***at least 20 subjects*** to enable an accurate harmonisation (see (paper)[]). 
 Try to ensure the data are high quality (i.e no blurring, no artefacts, no cavities in the brain).
-Demographic information (e.g age and sex) will be required for this process.
+Demographic information (e.g age and sex) will be required for this process. 
+WARNING: zero variance in the demographics information (e.g. having the same age for all subjects) will lead to Combat failures or errors. 
 
 Once you have done the process once, you can follow the [general guidelines to predict on a new patient](https://meld-graph.readthedocs.io/en/latest/run_prediction_pipeline.html) 
 
@@ -31,15 +32,15 @@ Once you have done the process once, you can follow the [general guidelines to p
 Open a terminal and `cd` to where you extracted the release zip.
 
 ```bash
-docker compose run meld_graph python scripts/new_patient_pipeline/new_pt_pipeline.py -harmo_code <harmo_code> -ids <subjects_list> -demos <demographic_file> --harmo_only
+DOCKER_USER="$(id -u):$(id -g)" docker compose run meld_graph python scripts/new_patient_pipeline/new_pt_pipeline.py -harmo_code <harmo_code> -ids <subjects_list> -demos <demographic_file> --harmo_only
 ```
 :::
 :::{tab-item} Native
 :sync: native
 Open a terminal and `cd` to the meld graph folder.
 
 ```bash
-./meldgraph.sh new_pt_pipeline -harmo_code <harmo_code> -ids <subjects_list> -demos <demographic_file> --harmo_only
+./meldgraph.sh new_pt_pipeline.py -harmo_code <harmo_code> -ids <subjects_list> -demos <demographic_file> --harmo_only
 ```
 :::
 ::::

docs/install_docker.md

@@ -1,9 +1,11 @@
 # Docker container
 
+**WARNING: Installation and use not yet tested on Windows. Please do let us know if you are succeeding / failing to use the docker container on Windows**
+
 The Docker container has all the prerequisites embedded on it which makes it easier to install and compatible with most of the OS systems. 
 
 Notes: 
-- Currently only tested on **Linux and Windows** (HPC Singularity coming soon, Mac M chip computers have to do a [install_native](https://meld-graph.readthedocs.io/en/latest/install_native.html)
+- Currently only tested on **Linux** (Not yet tested on Windows, HPC users should use the [Singularity version (not tested yet)](https://meld-graph.readthedocs.io/en/latest/install_singularity.html), Mac M chip computers have to do a [install_native](https://meld-graph.readthedocs.io/en/latest/install_native.html))
 - You will need **~12GB of space** to install the container
 - The docker image contains Miniconda 3, Freesurfer V7.2, Fastsurfer V1.1.2 and torch 1.10.0+cu111. The whole image is 11.4 GB.  
 
@@ -49,7 +51,8 @@ In order to run the docker, you'll need to configure a couple of files
 
 1. Download `meld_graph.zip` from the [latest github release](https://github.com/MELDProject/meld_graph/releases/latest) and extract it.
 2. Copy the freesurfer `license.txt` into the extracted folder
-3. Create the meld_data folder, if it doesn't exist already, and edit the compose.yml `volumes` line before the `:` to point to it. For example, if you wanted the folder to be on a mounted drive in Linux it might be:
+3. Create the meld_data folder, if it doesn't exist already. This folder is where you would like to store MRI data to run the classifier
+4. Edit the compose.yml `volumes` line before the `:` to point to it. For example, if you wanted the folder to be on a mounted drive in Linux it might be:
 ```
     volumes:
       - /mnt/datadrive/meld-data:/data
@@ -64,11 +67,7 @@ On windows, if you're using absolute paths, use forward slashes and quotes:
       - "C:/Users/John/Desktop/meld-data:/data"
 ```
 :::
-4. In order to use docker as a non-root user, the compose.yml file pass the current user ID to the docker. For that we recommand to add a DOCKER_USER variable in your bashrc file by running:
-```
-echo 'export DOCKER_USER="$(id -u):$(id -g)"' >> ~/.bashrc
-```
-Alternatively, you can export this variable everytime you are using the docker 
+
 
 ## Set up paths and download model
 Before being able to use the classifier on your data, data paths need to be set up and the pretrained model needs to be downloaded. 
@@ -77,9 +76,23 @@ Before being able to use the classifier on your data, data paths need to be set
 
 2. Run this command to download the docker image and the training data
 
+::::{tab-set}
+
+:::{tab-item} Linux
+:sync: linux
+```bash
+DOCKER_USER="$(id -u):$(id -g)" docker compose run meld_graph python scripts/new_patient_pipeline/prepare_classifier.py
+```
+:::
+
+:::{tab-item} Windows
+:sync: windows
 ```bash
 docker compose run meld_graph python scripts/new_patient_pipeline/prepare_classifier.py
 ```
+:::
+
+::::
 
 :::{note}
 Append `--skip-download-data` to the python call to skip downloading the test data.
@@ -89,9 +102,24 @@ Append `--skip-download-data` to the python call to skip downloading the test da
 ## Verify installation
 To verify that you have installed all packages, set up paths correctly, and downloaded all data, this verification script will run the pipeline to predict the lesion classifier on a new patient. It takes approximately 15 minutes to run.
 
+::::{tab-set}
+
+:::{tab-item} Linux
+:sync: linux
+```bash
+DOCKER_USER="$(id -u):$(id -g)" docker compose run meld_graph pytest
+```
+:::
+
+:::{tab-item} Windows
+:sync: windows
 ```bash
 docker compose run meld_graph pytest
 ```
+:::
+
+::::
+
 
 ### Errors
 If you run into errors at this stage and need help, you can re-run by changing the last line of the command by the command below to save the terminal outputs in a txt file. Please send `pytest_errors.log` to us so we can work with you to solve any problems. [How best to reach us.](#contact)
@@ -101,7 +129,7 @@ If you run into errors at this stage and need help, you can re-run by changing t
 :::{tab-item} Linux
 :sync: linux
 ```bash
-docker compose run meld_graph pytest -s | tee pytest_errors.log
+DOCKER_USER="$(id -u):$(id -g)" docker compose run meld_graph pytest -s | tee pytest_errors.log
 ```
 :::
 

docs/install_native.md

@@ -111,11 +111,11 @@ pip install -e .
 ## Set up paths and download model
 Before being able to use the classifier on your data, some paths need to be set up and the pretrained model needs to be downloaded. For this, run:
 ```bash
-./meldgraph.sh prepare_classifier
+./meldgraph.sh prepare_classifier.py
 ```
 
 This script will ask you if you want to change the path to the data folder, answer **'y'** for yes. \
-Then, it will ask for the the location of your **MELD data folder** and download the pretrained model and test data to a folder inside your MELD data folder. Please provide the path to where you would like to store MRI data to run the classifier on.
+Then, it will ask for the the location of your **MELD data folder**, where you would like to store MRI data to run the classifier. Create the **MELD data folder**, if it doesn't exist, and provide the path. It will download the pretrained model and test data to a folder inside your MELD data folder
 
 
 Note: You can also skip the downloading of the test data. For this, append the option `--skip-download-data` to the call.

docs/install_singularity.md

@@ -1,5 +1,7 @@
 # Singularity container
 
+**WARNING: Installation and use not yet tested. Please do let us know if you are succeeding / failing to use the singularity container on HPC**
+
 The Singularity container has been created to be used on HPC supporting Linux as they do not work with Docker container. If you are not working on a HPC, we recommend to install the docker version of container. 
 
 Notes: 
@@ -33,9 +35,9 @@ singularity build meld_graph.sif docker://meldproject/meld_graph:latest
 Before being able to use the classifier on your data, data paths need to be set up and the pretrained model needs to be downloaded. 
 
 1. Make sure you have 2GB available for the meld data.
-
+2. Create the **meld_data** folder, if it doesn't exist already. This folder is where where you would like to store MRI data to run the classifier. 
 2. Run this command to set the paths needed:
--  <path_to_meld_data_folder> : folder where the data containing the MELD Graph model should be saved and where the output of the pipeline will be saved
+-  <path_to_meld_data_folder> : Add the path to meld_data folder
 - <path_to_FS_license>: path where the license.txt has been saved
 ```bash
 export SINGULARITY_BINDPATH=/<path_to_meld_data_folder>:/data,<path_to_FS_license>/license.txt:/license.txt:ro

docs/prepare_data.md

@@ -65,6 +65,9 @@ You can copy the *demographics_file.csv* that you can find in your <meld_data_fo
 - Sex: 1 if male, 0 if female
 - Scanner: the scanner strenght associated with the MRI data ('3T' for 3 Tesla or '15T' for 1.5 Tesla)
 
-Note: please ensure to not change the name of the columns and to have completed the files with the appropriate values, otherwise the pipeline will fail. 
-
+Note: 
+- please ensure the column names are unchanged and completed with the appropriate values, otherwise the pipeline will fail.
 
+WARNING: for harmonisation 
+- please make sure you add the appropriate age and sex of the patients. Adding dummy information can lead to suboptimal harmonisation. 
+- please ensure that there is non-zero variance in the age of your subjects. Similar age for all subjects will lead to harmonisation failure. If your patients have the same age, please add randomly +- 0.01 to all age to introduce variance.   

docs/run_prediction_pipeline.md

@@ -32,7 +32,7 @@ If you wish to use the harmonisation feature of the MELD pipeline, you will need
 Open a terminal and `cd` to where you extracted the release zip.
 
 ```bash
-docker compose run meld_graph python scripts/new_patient_pipeline/new_pt_pipeline.py -id <subject_id> 
+DOCKER_USER="$(id -u):$(id -g)" docker compose run meld_graph python scripts/new_patient_pipeline/new_pt_pipeline.py -id <subject_id> 
 ```
 :::
 
@@ -49,7 +49,7 @@ source $FREESURFER_HOME/SetUpFreeSurfer.sh
 
 Then run the command
 ```bash
-./meldgraph.sh new_pt_pipeline -id <subject_id> 
+./meldgraph.sh new_pt_pipeline.py -id <subject_id> 
 ```
 :::
 ::::
@@ -86,14 +86,14 @@ To run the whole prediction pipeline on 1 subject using fastsurfer:
 :::{tab-item} Docker
 :sync: docker
 ```bash
-docker compose run meld_graph python scripts/new_patient_pipeline/new_pt_pipeline.py -id sub-001 --fastsurfer
+DOCKER_USER="$(id -u):$(id -g)" docker compose run meld_graph python scripts/new_patient_pipeline/new_pt_pipeline.py -id sub-001 --fastsurfer
 ```
 :::
 
 :::{tab-item} Native
 :sync: native
 ```bash
-./meldgraph.sh new_pt_pipeline -id sub-001 --fastsurfer
+./meldgraph.sh new_pt_pipeline.py -id sub-001 --fastsurfer
 ```
 :::
 ::::
@@ -103,14 +103,14 @@ To run the whole prediction pipeline on 1 subject using harmonisation code H1:
 :::{tab-item} Docker
 :sync: docker
 ```bash
-docker compose run meld_graph python scripts/new_patient_pipeline/new_pt_pipeline.pyy -id sub-001 -harmo_code H1
+DOCKER_USER="$(id -u):$(id -g)" docker compose run meld_graph python scripts/new_patient_pipeline/new_pt_pipeline.py -id sub-001 -harmo_code H1
 ```
 :::
 
 :::{tab-item} Native
 :sync: native
 ```bash
-./meldgraph.sh new_pt_pipeline -id sub-001 -harmo_code H1
+./meldgraph.sh new_pt_pipeline.py -id sub-001 -harmo_code H1
 ```
 :::
 ::::
@@ -120,14 +120,14 @@ To run the whole prediction pipeline on multiples subjects with parallelisation:
 :::{tab-item} Docker
 :sync: docker
 ```bash
-docker compose run meld_graph python scripts/new_patient_pipeline/new_pt_pipeline.py -ids list_subjects.txt --parallelise
+DOCKER_USER="$(id -u):$(id -g)" docker compose run meld_graph python scripts/new_patient_pipeline/new_pt_pipeline.py -ids list_subjects.txt --parallelise
 ```
 :::
 
 :::{tab-item} Native
 :sync: native
 ```bash
-./meldgraph.sh new_pt_pipeline -ids list_subjects.txt --parallelise
+./meldgraph.sh new_pt_pipeline.py -ids list_subjects.txt --parallelise
 ```
 :::
 ::::
@@ -147,13 +147,29 @@ This script:
     * Moves the features to the template surface
     * Write feature in hdf5
 
+Example to use it on one patient without harmonisation:
+::::{tab-set}
+:::{tab-item} Docker
+:sync: docker
+```bash
+DOCKER_USER="$(id -u):$(id -g)" docker compose run meld_graph python scripts/new_patient_pipeline/run_script_segmentation.py -id sub-001
+```
+:::
+
+:::{tab-item} Native
+:sync: native
+```bash
+./meldgraph.sh run_script_preprocessing.py -id run_script_segmentation-001
+```
+:::
+::::
 
 To know more about the script and how to use it on its own:
 ::::{tab-set}
 :::{tab-item} Docker
 :sync: docker
 ```bash
-docker compose run meld_graph python scripts/new_patient_pipeline/run_script_segmentation.py -h
+DOCKER_USER="$(id -u):$(id -g)" docker compose run meld_graph python scripts/new_patient_pipeline/run_script_segmentation.py -h
 ```
 :::
 
@@ -179,12 +195,29 @@ This script :
   - Features need to have been extracted using script 1. 
   - (optional): this script can also be called to harmonise your data for new harmonisation code but will need to pass a file containing demographics information.
 
+Example to use it on one patient without harmonisation:
+::::{tab-set}
+:::{tab-item} Docker
+:sync: docker
+```bash
+DOCKER_USER="$(id -u):$(id -g)" docker compose run meld_graph python scripts/new_patient_pipeline/run_script_preprocessing.py -id sub-001
+```
+:::
+
+:::{tab-item} Native
+:sync: native
+```bash
+./meldgraph.sh run_script_preprocessing.py -id sub-001
+```
+:::
+::::
+
 To know more about the script and how to use it on its own:
 ::::{tab-set}
 :::{tab-item} Docker
 :sync: docker
 ```bash
-docker compose run meld_graph python scripts/new_patient_pipeline/run_script_preprocessing.py -h
+DOCKER_USER="$(id -u):$(id -g)" docker compose run meld_graph python scripts/new_patient_pipeline/run_script_preprocessing.py -h
 ```
 :::
 
@@ -206,12 +239,29 @@ This script :
 Notes: 
 - Features need to have been processed using script 2 and Freesurfer outputs need to be available for each subject
 
+Example to use it on one patient without harmonisation:
+::::{tab-set}
+:::{tab-item} Docker
+:sync: docker
+```bash
+DOCKER_USER="$(id -u):$(id -g)" docker compose run meld_graph python scripts/new_patient_pipeline/run_script_prediction.py -id sub-001
+```
+:::
+
+:::{tab-item} Native
+:sync: native
+```bash
+./meldgraph.sh run_script_prediction.py -id sub-001
+```
+:::
+::::
+
 To know more about the script and how to use it on its own:
 ::::{tab-set}
 :::{tab-item} Docker
 :sync: docker
 ```bash
-docker compose run meld_graph python scripts/new_patient_pipeline/run_script_prediction.py -h
+DOCKER_USER="$(id -u):$(id -g)" docker compose run meld_graph python scripts/new_patient_pipeline/run_script_prediction.py -h
 ```
 :::