Merge branch 'develop' of github.com:nrcan/geo-deep-learning

.gitignore

@@ -1,2 +1,3 @@
 *__pycache__**
-*.idea**
+*.idea**
+*.vscode**

.travis.yml

@@ -16,6 +16,7 @@ install:
   - source activate ci_env
   - conda install fiona
   - conda install mlflow -c conda-forge
+  - pip install git+https://github.com/qubvel/segmentation_models.pytorch
 before_script:
   - unzip ./data/massachusetts_buildings.zip -d ./data
   - unzip ./data/classification_data.zip -d ./data

Metrics.md

@@ -0,0 +1,75 @@
+# Metrics in GDL
+
+## Pixel based Metrics
+
+#### Intersection over Union (IoU)
+
+The IOU metric is a pixel based metric which measures overlap using the number of pixels common between groundtruth and predictions divided by the total pixels across both.
+
+```
+IoU=  (groundtruth ∩ prediction) /
+      (groundtruth u predictions)
+```
+
+#### Dice Similarity Coefficient  (dice)
+
+The dice metric scores model performance by measuring overlap between groundthruth and predictions divided by sum of pixels of both groundtruth and predictions. 
+
+```
+dice= 2 * (groundtruth ∩ prediction) /
+          (groundtruth + prediction) 
+
+```
+_Note:_ IoU and Dice metrics weigh factors differently, however both metrics are positively correlated. This means if model A is better than B then this is captured similarly in both metrics.
+
+#### Precision and Recall
+
+By ploting a confusion matrix which indicates ground-truth and predicted classes with number of pixels classified in each class, [precision](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html#sklearn.metrics.precision_score) and [recall](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.recall_score.html#sklearn.metrics.recall_score) is easily computed.
+
+```
+precision= true positives /
+           true positives + false positives
+```
+
+```
+recall= true positives /
+        true positives + false negatives
+```
+
+classes = A and B evaluating for class A 
+where,
+
+- True Positives(TP) = pixels correctly classified as class A 
+- False Positives(FP) = pixels incorrectly classified as class A
+- False Negatives(FN) = class A pixels incorrectly classified as class B
+- True Negatives(TN) = pixels correctly classified as class B
+
+#### Matthews Correlation Coefficient (MCC)
+
+The MCC metric takes all four confusion matrix categories (TP, FP, FN and TN) into account which in turn provides a more reliable score. This [article](https://bmcgenomics.biomedcentral.com/articles/10.1186/s12864-019-6413-7) compares MCC to other widely used metrics. This metric is insusceptible to the imbalanced dataset factor. Also [Scikit-Learn](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.matthews_corrcoef.html) defines this metric is simpler terms. 
+
+```
+MCC= TP * TN - FP * FN /
+     [(TP + FP) * (FN + TN) * (FP + TN) * (TP + FN)]^(1/2)
+```
+
+#### Accuracy
+
+Accuracy is simply defined here as the ratio of correctly classified pixels to total number of pixels between ground-truth and predicted.
+_Note:_ This metric is very susceptible to imbalanced datasets and may give an overly optimistic score when that is not the case. We can write down the accuracy formular using the confusion matrix categories. 
+
+```
+acc= TP + TN /
+     TP + TN + FP + FN 
+```
+## Comparing common pixel based metrics
+| Accuracy   | IoU  | Dice Coefficient  |
+|---|---|---|
+| Counts the number of correctly classified pixels   | Counts pixels in both label and pred   | Similar to IoU, has its own strengths   |
+| Not suitable for Imbalanced datasets    | Counts pixels in either label and pred  | Measures average performance to IoU’s measure of worst case performance.   |
+| High accuracy may not translate to quality predictions   | Statistically correlates the counts (ratio)   |   |
+|    | Accounts for imbalanced data by penalizing FP and FN   |   |
+
+## Shape based Metrics
+
+** New shape based metrics would be added soon ** 

README.md

@@ -45,25 +45,24 @@ After installing the required computing environment (see next section), one need
 - nvidia GPU highly recommended
 - The system can be used on your workstation or cluster and on [AWS](https://aws.amazon.com/).
 
-## Installation on your workstation
+## Installation on your workstation using miniconda
 1. Using conda, you can set and activate your python environment with the following commands:  
     With GPU (defaults to CUDA 10.0 if `cudatoolkit=X.0` is not specified):
     ```shell
-    conda create -p YOUR_PATH python=3.6 pytorch torchvision -c pytorch
-    source activate YOUR_ENV
-    conda install opencv -c conda-forge
-    conda install ruamel_yaml h5py fiona rasterio scikit-image scikit-learn tqdm -c conda-forge
-    conda install nvidia-ml-py3 -c fastai
-    conda install mlflow
+    conda create -n gpu_ENV python=3.6 -c pytorch pytorch torchvision 
+    conda activate gpu_ENV
+    conda install -c conda-forge ruamel_yaml h5py fiona rasterio geopandas scikit-image scikit-learn tqdm 
+    conda install -c fastai nvidia-ml-py3
+    conda install mlflow 
     ```
     CPU only:
     ```shell
-    conda create -p YOUR_PATH python=3.6 pytorch-cpu torchvision -c pytorch
-    source activate YOUR_ENV
-    conda install opencv -c conda-forge
-    conda install ruamel_yaml h5py fiona rasterio scikit-image scikit-learn tqdm -c conda-forge
-    conda install mlflow
-   ```
+    conda create -n cpu_ENV python=3.6 -c pytorch pytorch-cpu torchvision-cpu 
+    conda activate cpu_ENV
+    conda install -c conda-forge opencv
+    conda install -c conda-forge ruamel_yaml h5py fiona rasterio geopandas scikit-image scikit-learn tqdm
+    conda install mlflow 
+    ```
     > For Windows OS: 
     > - Install rasterio, fiona and gdal first, before installing the rest. We've experienced some [installation issues](https://github.com/conda-forge/gdal-feedstock/issues/213), with those libraries. 
     > - Mlflow should be installed using pip rather than conda, as mentionned [here](https://github.com/mlflow/mlflow/issues/1951)  
@@ -138,14 +137,30 @@ Structure as created by geo-deep-learning
 **See: [train_segmentation.py / Outputs](training_outputs)
 
 ## Models available
+
+Models: Train from Scratch
 - [Unet](https://arxiv.org/abs/1505.04597)
-- [Deeplabv3 (backbone: resnet101, optional: pretrained on coco dataset)](https://arxiv.org/abs/1706.05587)
-- Experimental: Deeplabv3 (default: pretrained on coco dataset) adapted for RGB-NIR(4 Bands) supported
 - Unet small (less deep version of Unet)
 - Checkpointed Unet (same as Unet small, but uses less GPU memory and recomputes data during the backward pass)
 - [Ternausnet](https://arxiv.org/abs/1801.05746)
+
+Models: Pre-trained (torch vision)
+- [Deeplabv3 (backbone: resnet101, optional: pretrained on coco dataset)](https://arxiv.org/abs/1706.05587)
+- Experimental: Deeplabv3 (default: pretrained on coco dataset) adapted for RGB-NIR(4 Bands) supported
 - [FCN (backbone: resnet101, optional: pretrained on coco dataset)](https://people.eecs.berkeley.edu/~jonlong/long_shelhamer_fcn.pdf)
 
+Models: Segmentation Models Pytorch Library
+
+The following highly configurable models are offered from this easy to use [library](https://github.com/qubvel/segmentation_models.pytorch).
+
+- unet_pretrained
+- [pan_pretrained](https://arxiv.org/abs/1805.10180)
+- [fpn_pretrained](http://presentations.cocodataset.org/COCO17-Stuff-FAIR.pdf)
+- [pspnet_pretrained](https://arxiv.org/abs/1612.01105)
+- [deeplabv3+_pretrained](https://arxiv.org/pdf/1802.02611.pdf)
+
+Models from this library support any number of image bands and offers modular encoder architectures. Check the official [github repo](https://github.com/qubvel/segmentation_models.pytorch) for more details.  
+
 ## `csv` preparation
 The `csv` specifies the input images and the reference vector data that will be use during the training.
 Each row in the `csv` file must contain 5 comma-separated items:
@@ -256,7 +271,7 @@ global:
   debug_mode: True           # Activates various debug features (ex.: details about intermediate outputs, detailled progress bars, etc.). Default: False
 
 sample:
-  overlap: 20                # % of overlap between 2 samples.
+  overlap: 20                # % of overlap between 2 samples Note: high overlap > 25 creates very similar samples between train and val sets.   
   min_annotated_percent: 10  # Min % of non background pixels in stored samples.
 
 training:
@@ -376,12 +391,12 @@ global:
 
 
 inference:
-  img_dir_or_csv_file: /path/to/list.csv        # Directory containing all images to infer on OR CSV file with list of images
+  img_dir_or_csv_file: /path/to/list.csv        # CSV file containing directory of images with or without gpkg labels(used in benchmarking) 
   working_folder: /path/to/output_images        # Folder where all resulting images will be written (DEPRECATED, leave blank)
   state_dict_path: /path/to/checkpoint.pth.tar  # Path to model weights for inference
   chunk_size: 512                               # (int) Size (height and width) of each prediction patch. Default: 512
-  overlap: 10                                   # (int) Percentage of overlap between 2 chunks. Default: 10
-  heatmaps: False                               # if True, heatmaps for each class will be saved along with inference .tif
+  smooth_prediction: True                       # Smoothening Predictions with 2D interpolation
+  overlap: 2                                    # overlap between tiles for smoothing. Must be an even number that divides chunk_size without remainder.
 ```
 ### Process
 - The process will load trained weights to the chosen model and perform a per-pixel inference task on all the images contained in the working_folder
@@ -460,6 +475,67 @@ The `vis_batch_range` parameter plays a central role in visualization. First num
 - if in inference, `vis()` will print all unique values in each heatmap array. If there are only a few values, it gives a hint on usefulness of heatmap.
 - if in inference, `vis()` will check number of predicted classes in output array. If only one predicted class, a warning is sent.
 
+### Training and Inference for Segmentation on RGB-NIR images
+For the majorities of the YAML file will be the same as before for RGB images, this section will present the modifications to do to be able the use a model that use RGBN images. For more informations on the implementation, see the article [Transfer Learning from RGB to Multi-band Imagery](https://www.azavea.com/blog/2019/08/30/transfer-learning-from-rgb-to-multi-band-imagery/) frome [Azavea](https://www.azavea.com/).
+
+Here some specifications on this fonctionnality:
+- At the moment this fonctionnality is only available for the [Deeplabv3 (backbone: resnet101)](https://arxiv.org/abs/1706.05587)
+- You may need to reduce the size of the `batch_size` to fit everything in the memory.
+
+To launch the training and the inference program, the commands are the same as normal, only the YAML need to change:
+```bash
+python train_segmentation.py path/to/config/file/config.yaml
+python inference.py path/to/config/file/config.yaml
+```
+
+Details on parameters used by this module:
+```yaml
+
+# Global parameters
+global:
+  samples_size: 256
+  num_classes: 4 
+  data_path: /home/cauthier/data/ # TODO: put it in the git ignor
+  number_of_bands: 4
+  # Model must be in the follow list:
+  # unet, unetsmall, checkpointed_unet, ternausnet,
+  # fcn_resnet101, deeplabv3_resnet101
+  model_name: deeplabv3_resnet101
+  bucket_name:   # name of the S3 bucket where data is stored. Leave blank if using local files
+  task: segmentation  # Task to perform. Either segmentation or classification
+  num_gpus: 2
+  BGR_to_RGB: True
+  scale_data: [0,1]
+  aux_vector_file:
+  aux_vector_attrib:
+  aux_vector_ids:
+  aux_vector_dist_maps:
+  aux_vector_dist_log:
+  aux_vector_scale:
+  debug_mode: True
+  coordconv_convert: False
+  coordvonc_scale:
+  
+  # Module to include the NIR
+  modalities: RGBN
+  concatenate_depth: 'layer4'
+```
+
+The rest of the YAML file will be the same as present bellow.
+
+The major changes are the `modalities`, `number_of_bands` and `concatenate_depth` parameters. If the model select is not **DeeplavV3** but the `nuber_of_band = 4` and the `modalities = RGBN`, the model will train with the basic architecture but the input will be an image with 4 dimmensions. 
+
+Since we have the concatenation point for the NIR band only for the **DeeplabV3**, the `concatenate_depth` parameter option are:
+- conv1
+- maxpool
+- layer2
+- layer3
+- layer4
+
+**Illustration of the principale will fellow soon**
+
+
+
 # Classification Task
 The classification task allows images to be recognized as a whole rather than identifying the class of each pixel individually as is done in semantic segmentation.
 

conf/config_template.yaml

@@ -8,14 +8,14 @@
 #   7) Visualization parameters
 
 # Global parameters
-
 global:
   samples_size: 512
   num_classes: 5
   data_path: path/to/data
-  mlflow_uri: path/to/mlflow_tracking_uri  # Default to ./mlruns
   number_of_bands: 4
-  model_name: unet     # One of unet, unetsmall, checkpointed_unet, ternausnet, fcn_resnet101, deeplabv3_resnet50,deeplabv3_resnet101
+  # Model available: unet, unetsmall, checkpointed_unet, ternausnet, fcn_resnet101, deeplabv3_resnet50, deeplabv3_resnet101
+  model_name: unet
+  mlflow_uri: path/to/mlflow_tracking_uri  # Default to ./mlruns
   bucket_name:   # name of the S3 bucket where data is stored. Leave blank if using local files
   task: segmentation  # Task to perform. Either segmentation or classification
   num_gpus: 2
@@ -33,7 +33,6 @@ global:
 
 
 # Data analysis parameters; used in data_analysis.py ------------------
-
 data_analysis:
   create_csv: False
   optimal_parameters_search : False
@@ -44,22 +43,19 @@ data_analysis:
     # values (int) represent class minimum threshold targeted in samples
 
 # Sample parameters; used in images_to_samples.py -------------------
-
 sample:
   prep_csv_file: /path/to/csv/file.csv
   val_percent: 5 # Percentage of validation samples created from train set (0 - 100)
-  overlap: 25
+  overlap: 25 # (int) Percentage of overlap between 2 chunks.
   sampling_method: # class_proportion or min_annotated_percent
     'min_annotated_percent': 0  # Min % of non background pixels in samples. Default: 0
     'class_proportion': {'1':0, '2':0, '3':0, '4':0} # See below:
     # keys (numerical values in 'string' format) represent class id
     # values (int) represent class minimum threshold targeted in samples
-
   mask_reference: False
 
 
 # Training parameters; used in train_segmentation.py ----------------------
-
 training:
   state_dict_path: path/to/pretrained/file/checkpoint.pth.tar    # optional
   num_trn_samples: 4960
@@ -68,31 +64,52 @@ training:
   batch_size: 32
   num_epochs: 100
   target_size: 128
-  loss_fn: Lovasz # One of CrossEntropy, Lovasz, Focal, OhemCrossEntropy (*Lovasz for segmentation tasks only)
-  optimizer: adabound # One of adam, sgd or adabound
+  # Loss function available: CrossEntropy, Lovasz, Focal, OhemCrossEntropy 
+  #                          (*Lovasz for segmentation tasks only)
+  loss_fn: Lovasz
+  # Optimizer available: adam. sgd, adabound
+  optimizer: adabound
   learning_rate: 0.0001
   weight_decay: 0
   step_size: 4
   gamma: 0.9
-  dropout: False    # (bool) Use dropout or not
+  dropout: False   # (bool) Use dropout or not
   dropout_prob:    # (float) Set dropout probability, e.g. 0.5
   class_weights: [1.0, 2.0]
-  batch_metrics:    # (int) Metrics computed every (int) batches. If left blank, will not perform metrics. If (int)=1, metrics computed on all batches.
-  ignore_index: 255 # Specifies a target value that is ignored and does not contribute to the input gradient. Default: None
-  normalization:  # Normalization parameters for finetuning (Ex. mean: [0.485, 0.456, 0.406], std: std: [0.229, 0.224, 0.225])
+  # Batch metrics (int): The metrics computed every (int) batches.
+  # If left blank, will not perform metrics.
+  # If (int)=1, metrics computed on all batches.
+  batch_metrics:
+  # Ignore index (default -> None): the target value that is ignored and
+  #                                  does not contribute to the input gradient.
+  ignore_index: 255 
+  # Normalization: parameters for finetuning.
+  # for esample
+  #    -> mean: [0.485, 0.456, 0.406]
+  #    -> std: std: [0.229, 0.224, 0.225])
+  normalization:
     mean:
     std:
-
+
+  # For each augmentation parameters, if not specified,
+  # the part of the augmentation will not be performed.
   augmentation:
-      rotate_limit: 45         # Specifies the upper and lower limits for data rotation. If not specified, no rotation will be performed.
-      rotate_prob: 0.5         # Specifies the probability for data rotation. If not specified, no rotation will be performed.
-      hflip_prob: 0.5          # Specifies the probability for data horizontal flip. If not specified, no horizontal flip will be performed.
-      random_radiom_trim_range: [0.1, 2.0] # Specifies the range in which a random percentile value will be chosen to trim values. This value applies to both left and right sides of the raster's histogram. If not specified, no enhancement will be performed.
-      brightness_contrast_range: # Not yet implemented
-      noise: # Not yet implemented
+    # Rotate limit: the upper and lower limits for data rotation.
+    rotate_limit: 45
+    # Rotate probability: the probability for data rotation.
+    rotate_prob: 0.5
+    # Horizontal flip: the probability for data horizontal flip.
+    hflip_prob: 0.5
+    # Range of the random percentile:
+    #   the range in which a random percentile value will
+    #   be chosen to trim values. This value applies to
+    #   both left and right sides of the raster's histogram.
+    random_radiom_trim_range: [0.1, 2.0]
+    brightness_contrast_range: # Not yet implemented
+    noise: # Not yet implemented
 
-# Inference parameters; used in inference.py --------
 
+# Inference parameters; used in inference.py --------
 inference:
   img_dir_or_csv_file: /path/to/csv/containing/images/list.csv
   state_dict_path: /path/to/model/weights/for/inference/checkpoint.pth.tar
@@ -102,7 +119,11 @@ inference:
 
 
 # Visualization parameters
+  smooth_prediction: True
+  overlap: 2 # overlap between tiles for smoothing. Must be an even number that divides chunk_size without remainder.
 
+
+# Visualization parameters
 visualization:
   vis_batch_range: [0,200,10] #start, finish, increment
   vis_at_checkpoint: True
@@ -114,4 +135,5 @@ visualization:
   vis_at_train: True
   grid: True
   heatmaps: True
-  colormap_file: ./data/colormap.csv
+  colormap_file: ./data/colormap.csv
+