Merge branch 'main' into regression_updates

.github/CONTRIBUTING.md

@@ -6,6 +6,7 @@ We welcome all forms of contributions, including but not limited to the followin
 - Incorporate downstream datasets
 - Add new decoder heads
 - Fix typo or bugs
+- Add new decoders
 
 ### Workflow
 
@@ -16,6 +17,37 @@ We welcome all forms of contributions, including but not limited to the followin
 
 Note: For significant modifications or any bugs spotting, please consider opening an issue for discussion beforehand.
 
+## Code structure 
+
+### engines
+In engines, basic modules in the training pipeline are defined including data_preprocessor, trainer and evaluator.
+1. data_preprocessor selects the bands needed by an encoder and pads unavailable bands with zeros, and different augmentations.
+2. trainer supports mixed precision/distributed training and print training stats and metrics in real time.
+3. evaluator can be called independently and evaluate a model also in distributed way and compute per class metrics.
+
+### datasets
+1. The implementations are simplified and standardized.
+2. Dataset metas are read from configs, including newly added classes (name), ignore_index, and so on.
+3. To add (register) a new dataset implementation, use the decorator ```@DATASET_REGISTRY.register()```.
+
+### foundation_models
+1. Support multi-stage output that may be needed by segmentors, specified by output layers in encoder config.
+2. All the encoder should work properly.
+3. To add (register) a new encoder implementation, use the decorator ```@ENCODER_REGISTRY.register()```.
+
+### segmentors
+1. The UperNet implementation is based on [mmsegmentation](https://github.com/open-mmlab/mmsegmentation/tree/main)
+2. To add (register) a new encoder implementation, use the decorator ```@SEGMENTOR_REGISTRY.register()```.
+3. So far, we have UPerNet for unitemporal semantic segmentation, UPerNetCD for change detection and MTUPerNet for multitemporal semantic segmentation
+4. for multi-temporal, L-TAE and linear projection are supported
+
+### augmentations
+1. All the available augmentations are in ```data_preproessor.py```
+2. To add (register) a new augmentation implementation, use the decorator ```@AUGMENTER_REGISTRY.register()```.
+
+All the parameters can also be set in the run config file.
+
+## Adding new features
 
 ### Adding a new geospatial foundation model
 1. Inside the `foundation_models` folder:

README.md

@@ -1,42 +1,51 @@
 [![Tests](https://github.com/yurujaja/geofm-bench/actions/workflows/python-test.yml/badge.svg)](https://github.com/yurujaja/geofm-bench/actions/workflows/python-test.yml)
 
-## Introduction
-(TBD)
-
-### engines
-In engines, basic modules in the training pipeline are defined including data_preprocessor, trainer and evaluator.
-1. data_preprocessor replaced the previous adaptation.py, i.e., selects the bands needed by an encoder and pads unavailable bands with zeros, and different augmentations.
-2. trainer now support mixed precision/distributed training and print training stats and metrics in real time.
-3. evaluator can be called independently and evaluate a model also in distributed way and compute per class metrics.
-4. see run.py for how to assemble these modules and concatenate them
-
-### datasets
-1. The implementations are simplified and standardized (I try my best).
-2. Dataset metas are read from configs, including newly added classes (name), ignore_index, and so on.
-3.Mados, sen1floods, hlsburnscars, xView2, biomasster are supported by this branch currently.
-4. To add (register) a new dataset implementation, use the decorator @DATASET_REGISTRY.register().
-
-### foundation_models
-1. Remove all irrelevant modules and functions used in pre-training. Only keep the essential modules in encoders for extracting features.
-2. Support multi-stage output that may be needed by segmentors, specified by output layers in encoder config.
-3. All the encoder should work properly.
-4. To add (register) a new encoder implementation, use the decorator @ENCODER_REGISTRY.register().
-
-### segmentors
-1. Now the UperNet implementation is based on mmsegmentation, which is more likely correct: https://github.com/open-mmlab/mmsegmentation/tree/main
-2. We can copypaste more segmentors later.
-3. To add (register) a new encoder implementation, use the decorator @SEGMENTOR_REGISTRY.register().
-4. So far, we have UPerNet for unitemporal semantic segmentation, UPerNetCD for change detection and MTUPerNet for multitemporal semantic segmentation
-5. for multi-temporal, L-TAE and linear projection are supported
-
-All of these parameters can also be set in the run config file.
-
-To use more gpus or nodes, set `--nnodes` and `--nproc_per_node` correspondingly, see:
-https://pytorch.org/docs/stable/elastic/run.html
-
-To use mixed precision training, specify either `--fp16` for float16 and or `--bf16` for bfloat16
-
-For fine-tuning instead of linear probing, specify `--finetune`.
+# TITLE 
+
+## 📚 Introduction
+
+While geospatial foundation models (GFMs) have proliferated rapidly, their evaluations remain inconsistent and narrow. Existing works often utilize suboptimal downstream datasets (e.g., EuroSAT) and tasks (e.g., land cover classification), which constrain comparability and real-world usability. Additionally, a lack of diversity in evaluation protocols, including image resolution and sensor types, further complicates the extensive assessments of GFM performance. To bridge this gap, we propose a standardized evaluation protocol that incorporates a wide-ranging selection of datasets, tasks, resolutions, and sensor types, establishing a robust and widely applicable benchmark for GFMs.
+
+In this repo, you can find the code to benchmark GFMs. For the moment we included several GFMs that present different approach. We look forward to adding new models and datasets.
+
+For the moment, we support the following **models**:
+
+|             | Paper | GitHub | Keywords |
+|:-----------:|:-----:|:------:|:--------:|
+|  SSL4EOS12  |       |        |          |
+|  Scale-MAE  |       |        |          |
+|  SatlasNet  |       |        |          |
+|     GFM     |       |        |          |
+| SpectralGPT |       |        |          |
+|     DOFA    |       |        |          |
+|    CROMA    |       |        |          |
+|   Prithvi   |       |        |          |
+|  RemoteCLIP |       |        |          |
+
+And the following **datasets**:
+
+|                     | Paper | Download | Domain | Task | Sensors | Location |
+|:-------------------:|:-----:|:--------:|:------:|:----:|---------|----------|
+|    HLS Burn Scars   |       |          |        |      |         |          |
+|        MADOS        |       |          |        |      |         |          |
+|        PASTIS       |       |          |        |      |         |          |
+|     Sen1Floods11    |       |          |        |      |         |          |
+|        xView2       |       |          |        |      |         |          |
+| Five Billion Pixels |       |          |        |      |         |          |
+|   DynamicEarthNet   |       |          |        |      |         |          |
+|   CropTypeMapping   |       |          |        |      |         |          |
+|      SpaceNet7      |       |          |        |      |         |          |
+|    AI4SmallFarms    |       |          |        |      |         |          |
+|     BioMassters     |       |          |        |      |         |          |
+
+The repository supports the following **tasks** using GFMs:
+ - [single temporal semantic segmentation](#single-temporal-semantic-segmentation)
+ - [multi-temporal semantic segmentation](#multi-temporal-semantic-segmentation)
+ - [change detection](#change-detection)
+ - [single temporal regression](#single-temporal-regression)
+ - [multi-temporal regression](#multi-temporal-regression)
+
+It is possible also to train some [supervised baselines](#-fully-supervised-training), based on UNet.
 
 ## 🛠️ Setup
 Clone the repository:
@@ -63,16 +72,25 @@ mamba activate geofm-bench8
 ```
 
 ## 🏋️ Training
+
 There are 5 basic component types in our config system:
-- `config`: Information of training settings such as batch size, epochs, use wandb. `limited_label` is to indicate the percentage of dataset used for training, for example, `-1` means the full training dataset is used while `0.5` means 50% used. 
+
+- `config`: Information of training settings such as batch size, epochs, use wandb. `limited_label` is to indicate the percentage of dataset used for training, for example, `-1` means the full training dataset is used while `0.5` means 50% used. #strategy used
 - `encoder_config`: GFM encoder related parameters. `output_layers` is used for which layers are used for Upernet decoder. 
 - `dataset_config`: Information of downstream datasets such as image size, band_statistics, etc. 
 - `segmentor_config`: Downstream task decoder fine-tuning related parameters, including the head type, loss, optimizer, scheduler, etc.
 - `augmentation_config`: Both preprocessing and augmentations steps required for the dataset, such as bands adaptation, normalization, resize/crop.
 
-We provide several examples of command lines to initilize different training tasks on single gpu.
+We provide several examples of command lines to initilize different training tasks on single GPU.
+
+Please note:
+ - Command line's parameters have the priority on the parameters in the config files. So, if you want to change e.g. the `batch size`, without changing the `config`, you can just add `--batch size n` to the command line
+ - To use more gpus or nodes, set `--nnodes` and `--nproc_per_node` correspondingly, see:
+https://pytorch.org/docs/stable/elastic/run.html
+ - To use mixed precision training, specify either `--fp16` for float16 and or `--bf16` for bfloat16
+
 ### 💻 Decoder Finetuning
-**Single Temporal Semantic Segmentation** 
+#### Single Temporal Semantic Segmentation
 
 Take MADOS dataset, Prithvi Encoder and Upernet Decoder as example:
 ```
@@ -85,7 +103,7 @@ torchrun --nnodes=1 --nproc_per_node=1 run.py  \
 --num_workers 4 --eval_interval 1  --use_wandb
 ```
 
-**Multi Temporal Semantic Segmentation**
+#### Multi-Temporal Semantic Segmentation
 
 Multi-temporal model `configs/segmentors/upernet_mt.yaml` should be used. In addition, in the dataset config, indicate the number of time frames, e.g., `multi_temporal: 6`
 ```
@@ -98,40 +116,94 @@ torchrun --nnodes=1 --nproc_per_node=1 run.py  \
 --num_workers 4 --eval_interval 1 --use_wandb
 ```
 
-**Multi Temporal Change Detection** 
+#### Change Detection
+```
+torchrun ...
+```
+#### Single Temporal Regression
 ```
 torchrun ...
 ```
 
-**Multi Temporal Regression** 
+#### Multi-Temporal Regression
 ```
 torchrun ...
 ```
 
 ### 💻 Fully Supervised Training
-**Single Temporal Change Detection** 
+#### Single Temporal Semantic Segmentation
 ```
 torchrun ...
 ```
+In general
+
+
 ## 🏃 Evaluation 
 Indicate the `eval_dir` where the checkpoints and configurations are stored.
+
 ```
 torchrun --nnodes=1 --nproc_per_node=1 run.py --batch_size 1 --eval_dir work-dir/the-folder-where-your-exp-is-saved
 ```
 
-
 ## ✏️ Contributing
-We appreciate all contributions to improve xxx. Please refer to [Contributing Guidelines](.github/CONTRIBUTING.md)
+We appreciate all contributions. Please refer to [Contributing Guidelines](.github/CONTRIBUTING.md)
+
+## ⚠️ Warnings
 
+Some features are under construction:
+ - the automatic download is working for all the datasets and models' weights but, respectively, **Five Billion Pixels**, **BioMassters**, and **GFM**.
 
 
+## 🧮 Some first results
 
-## Some numbers
+A pre-print is coming soon... Stay tuned!
 
-| Encoder | Dataset      | Epochs | mIoU   |
-|---------|--------------|--------|--------|
-| Prithvi | MADOS        | 80     | 53.455 |
-| Prithvi | HLSBurnScars | 80     | 86.208 |
-| Prithvi | Sen1Floods11 | 80     | 87.217 |
+| Encoder | Dataset       | Epochs | mIoU   |
+|---------|---------------|--------|--------|
+| Prithvi | MADOS         | 80     | 53.455 |
+| Prithvi | HLSBurnScars  | 80     | 86.208 |
+| Prithvi | Sen1Floods11  | 80     | 87.217 |
+| Prithvi | AI4SmallFarms | 80     | 33.796 |
+
+Please note: #add different conditions 
 
 ## 💡 Acknowledgements
+
+##  ©️ License
+
+MIT License
+
+Copyright (c) Microsoft Corporation.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE
+
+## 📝 Citing
+
+If you use this software in your work, please cite:
+
+```
+@misc{pangaea,
+  author = {},
+  title = {Pangaea},
+  year = {2024},
+  publisher = {GitHub},
+  journal = {GitHub repository},
+  howpublished = {\url{https://github.com/yurujaja/geofm-bench}},
+}
+```

configs/augmentations/pastis_seg.yaml

@@ -0,0 +1,11 @@
+train:
+  SegPreprocessor: ~
+  NormalizeMeanStd: ~
+  ResizeToEncoder: ~
+  # RandomFlip:
+  #   ud_probability: 0.3
+  #   lr_probability: 0.3
+test:
+  SegPreprocessor: ~
+  NormalizeMeanStd: ~
+  ResizeToEncoder: ~

configs/augmentations/regression_default.yaml

@@ -1,13 +1,11 @@
 train:
   RegPreprocessor: ~
-  NormalizeMinMax: ~
-#   NormalizeMeanStd: ~
+  NormalizeMeanStd: ~
   RandomCropToEncoder: ~
-#   RandomFlip:
-#     ud_probability: 0.3
-#     lr_probability: 0.3
+  RandomFlip:
+    ud_probability: 0.3
+    lr_probability: 0.3
 test:
   RegPreprocessor: ~
-  NormalizeMinMax: ~
-#   NormalizeMeanStd: ~
-  Tile: ~
+  NormalizeMeanStd: ~
+  Tile: ~

configs/datasets/dynamicen.yaml

@@ -4,7 +4,7 @@ download_url: None
 auto_download: False
 
 img_size: 1024
-multi_temporal: False
+multi_temporal: 6
 multi_modal: False
 
 # classes