Update CLI and others in reamde (#100)

* update gfm link in README * changed criterion for spacenet7 to dice loss * Update CTM whole inference mode * Update README.md * Update upernet.py * Update DATASET_GUIDE.md * Add files via upload * Update README.md --------- Co-authored-by: KerekesDavid <[email protected]> Co-authored-by: SebastianHafner <[email protected]> Co-authored-by: Valerio Marsocci <[email protected]>
VMarsocci · Dec 10, 2024 · e1d12ed · e1d12ed
1 parent 1380d35
commit e1d12ed
Show file tree

Hide file tree

Showing 4 changed files with 50 additions and 34 deletions.
diff --git a/.github/boxplot.png b/.github/boxplot.png
diff --git a/DATASET_GUIDE.md b/DATASET_GUIDE.md
@@ -37,10 +37,9 @@ This document provides a detailed overview of the datasets used in this reposito
    task=segmentation
   ```
 
-### PASTIS
+### PASTIS-R
 
 - The code supports automatic downloading of the dataset into `./data` folder.
-- SPOT-6 images are available for single-temporal semantic segmentation, otherwise the basic experimental setup for this dataset is a multi-temporal multi-modal semantic segmentation task.
 - Images are 128x128 patches, so a resize is needed to match input_size requirements of the encoders.
 - For models that don't support multi-temporal data, each time frame is processed separately for feature extraction and then mapped into a single representation. This setup requires the configuration file `configs/decoder/seg_upernet_mt_ltae.yaml`. Additionally, in the dataset configuration, specify the number of time frames, for example, `multi_temporal: 6`. Below is a CLI example for running the experiment using the RemoteCLIP pretrained encoder and multi-temporal UPerNet with L-TAE processing of temporal information:
 
@@ -52,6 +51,7 @@ This document provides a detailed overview of the datasets used in this reposito
    decoder=seg_upernet_mt_ltae \
    preprocessing=seg_resize \
    criterion=cross_entropy \
+   task.evaluator.inference_mode=whole \  
    task=segmentation
   ```
 
@@ -142,7 +142,8 @@ This document provides a detailed overview of the datasets used in this reposito
    decoder=seg_upernet_mt_ltae \
    preprocessing=seg_resize \
    criterion=cross_entropy \
-   task=segmentation
+   task=segmentation \
+   task.evaluator.inference_mode=whole 
   ```
 
 ### SpaceNet 7
@@ -159,7 +160,7 @@ This document provides a detailed overview of the datasets used in this reposito
    encoder=remoteclip \
    decoder=seg_upernet\
    preprocessing=seg_default \
-   criterion=cross_entropy \
+   criterion=dice \
    task=segmentation
   ```
 - Here is an example to run change detection:
@@ -170,7 +171,7 @@ This document provides a detailed overview of the datasets used in this reposito
    encoder=remoteclip \
    decoder=seg_siamupernet_conc\
    preprocessing=seg_default \
-   criterion=cross_entropy \
+   criterion=dice \
    task=change_detection
   ```
 

diff --git a/README.md b/README.md
@@ -2,6 +2,8 @@
 
 # PANGAEA: A Global and Inclusive Benchmark for Geospatial Foundation Models
 
+🔥 The [pre-print](https://arxiv.org/abs/2412.04204) is out!
+
 ## 📚 Introduction
 
 While geospatial foundation models (GFMs) have proliferated rapidly, their evaluations remain inconsistent and narrow. Existing works often utilize suboptimal downstream datasets (e.g., EuroSAT) and tasks (e.g., land cover classification), which constrain comparability and real-world usability. Additionally, a lack of diversity in evaluation protocols, including image resolution and sensor types, further complicates the extensive assessments of GFM performance. 
@@ -35,12 +37,12 @@ And the following **datasets**:
 |:-------------------:|:--------:|:------:|:----:|:-------:|:--------:|
 | [HLS Burn Scars](https://huggingface.co/datasets/ibm-nasa-geospatial/hls_burn_scars) | [link](https://huggingface.co/datasets/ibm-nasa-geospatial/hls_burn_scars) | Wildfire | Semantic Segmentation | HLS (Harmonized Landsat Sentinel-2) | USA |
 |        [MADOS](https://www.sciencedirect.com/science/article/pii/S0924271624000625)        |  [link](https://marine-pollution.github.io/index.html)        |  Marine      |  Semantic Segmentation    |    S2   | Global   |
-|        [PASTIS-HD](https://arxiv.org/abs/2404.08351)       |    [link](https://huggingface.co/datasets/IGNF/PASTIS-HD)       |   Agriculture     |  Semantic Segmentation    |    S1, S2, SPOT-6  | France   |
+|        [PASTIS-R](https://arxiv.org/abs/2404.08351)       |    [link](https://huggingface.co/datasets/IGNF/PASTIS-HD)       |   Agriculture     |  Semantic Segmentation    |    S1, S2, SPOT-6  | France   |
 |     [Sen1Floods11](http://openaccess.thecvf.com/content_CVPRW_2020/html/w11/Bonafilia_Sen1Floods11_A_Georeferenced_Dataset_to_Train_and_Test_Deep_Learning_CVPRW_2020_paper.html)    | [link](https://github.com/cloudtostreet/Sen1Floods11) |  Flood |Semantic Segmentation  | S1, S2 | Global |
 |        [xView2](https://openaccess.thecvf.com/content_CVPRW_2019/html/cv4gc/Gupta_Creating_xBD_A_Dataset_for_Assessing_Building_Damage_from_Satellite_CVPRW_2019_paper.html)       | [link](https://xview2.org/dataset) | HADR | Change Detection | Maxar | Global   |
 | [Five Billion Pixels](https://www.sciencedirect.com/science/article/pii/S0924271622003264) |  [original version](https://x-ytong.github.io/project/Five-Billion-Pixels.html) <br> (custom version coming soon)        |  (Urban) Land Cover     |  Semantic Segmentation    |    Gaofen-2     | China    |
 |   [DynamicEarthNet](https://arxiv.org/pdf/2203.12560)   |   [link](https://mediatum.ub.tum.de/1650201)        |    (Urban) Land Cover    |   Semantic Segmentation   |   PlanetFusion      | Global   |
-|   [CropTypeMapping](https://openaccess.thecvf.com/content_CVPRW_2019/papers/cv4gc/Rustowicz_Semantic_Segmentation_of_Crop_Type_in_Africa_A_Novel_Dataset_CVPRW_2019_paper.pdf) |   [link](https://sustainlab-group.github.io/sustainbench/docs/datasets/sdg2/crop_type_mapping_ghana-ss.html#download) | Agriculture |Semantic Segmentation |S1, S2, Planet|South Sudan|
+|   [CropTypeMapping-South Sudan](https://openaccess.thecvf.com/content_CVPRW_2019/papers/cv4gc/Rustowicz_Semantic_Segmentation_of_Crop_Type_in_Africa_A_Novel_Dataset_CVPRW_2019_paper.pdf) |   [link](https://sustainlab-group.github.io/sustainbench/docs/datasets/sdg2/crop_type_mapping_ghana-ss.html#download) | Agriculture |Semantic Segmentation |S1, S2, Planet|South Sudan|
 |      [SpaceNet 7](https://openaccess.thecvf.com/content/CVPR2021/papers/Van_Etten_The_Multi-Temporal_Urban_Development_SpaceNet_Dataset_CVPR_2021_paper.pdf)      |    [link](https://spacenet.ai/sn7-challenge/)      |    Urban    |   Change detection/ <br> Semantic Segmentation   |     Planet    | Global   |
 |    [AI4SmallFarms](https://ieeexplore.ieee.org/document/10278130)  | [link](https://doi.org/10.17026/dans-xy6-ngg6)  |  Agriculture     |  Semantic segmentation  |   S2   | Cambodia/Vietnam |
 |     [BioMassters](https://papers.nips.cc/paper_files/paper/2023/file/40daf2a00278c4bea1b26cd4c8a654f8-Paper-Datasets_and_Benchmarks.pdf)     |   [link](https://huggingface.co/datasets/nascetti-a/BioMassters)       | Forest       | Regression   |  S1, S2 | Finland   |
@@ -52,7 +54,7 @@ The repository supports the following **tasks** using geospatial (foundation) mo
  - [Single Temporal Regression](#single-temporal-regression)
  - [Multi-Temporal Regression](#multi-temporal-regression)
 
-It is also possible to train some [supervised baselines](#-fully-supervised-baseline), based on UNet.
+It is also possible to train some [supervised baselines](#-fully-supervised-baseline), based on UNet and ViT.
 
 ## 🗺️ Datasets details
 Please refer to [**Dataset Guide**](DATASET_GUIDE.md) to understand the processing requirements and commands specific to each dataset.
@@ -62,7 +64,7 @@ If you want to fast-prototype your model, maybe you want to run fast experiments
 ## 🛠️ Setup
 Clone the repository:
 ```
-git clone https://github.com/yurujaja/pangaea-bench.git
+git clone https://github.com/VMarsocci/pangaea-bench.git
 cd pangaea-bench
 ```
 
@@ -175,7 +177,7 @@ torchrun --nnodes=1 --nproc_per_node=1 pangaea/run.py \
    --config-name=train \
    dataset=croptypemapping \
    encoder=satlasnet_mi \
-   decoder=seg_upernet_mt_ltae decoder.multi_temporal_strategy=null \
+   decoder=seg_upernet_mt_ltae \
    preprocessing=seg_resize \
    criterion=cross_entropy \
    task=segmentation
@@ -258,8 +260,8 @@ torchrun --nnodes=1 --nproc_per_node=1 pangaea/run.py \
 
 ### 💻 Fully Supervised Baseline 
 
-The repo supports also training fully supervised baselines (e.g. UNet). To run these, follow the same command line rules as for other models. Keep in mind that setting finetune=True is necessary since this fully supervised approach trains the model from scratch. 
-An example for single temporal semantic segmentation is provided (Sen1Floods11 dataset):
+The repo supports also training fully supervised baselines (i.e. UNet and ViT). To run these, follow the same command line rules as for other models. Keep in mind that setting finetune=True is necessary since this fully supervised approach trains the model from scratch. 
+An example for single temporal semantic segmentation with UNet is provided (Sen1Floods11 dataset):
 ```
 torchrun --nnodes=1 --nproc_per_node=1 pangaea/run.py \
    --config-name=train \
@@ -271,7 +273,21 @@ torchrun --nnodes=1 --nproc_per_node=1 pangaea/run.py \
    task=segmentation \
    finetune=True
 ```
-For the moment, there is no multi-temporal baseline supported.
+There is no multi-temporal UNet supported.
+
+An example for multi-temporal semantic segmentation with ViT is provided (CropTypeMapping-SS dataset):
+```
+torchrun --nnodes=1 --nproc_per_node=1 pangaea/run.py \
+   --config-name=train \
+   dataset=croptypemapping \
+   encoder=vit_scratch \
+   decoder=seg_upernet_mt_ltae \
+   preprocessing=seg_default \
+   criterion=cross_entropy \
+   task=segmentation \
+   task.evaluator.inference_mode=whole \
+   finetune=true
+```
 
 ## 🔧 Customization
 
@@ -296,36 +312,33 @@ torchrun pangaea/run.py --config-name=test ckpt_dir=path_to_ckpt_dir
 ## ✏️ Contributing
 We appreciate all contributions. Please refer to [Contributing Guidelines](.github/CONTRIBUTING.md).
 
-## ⚠️ Warnings
-
-Some features are under construction:
- - the automatic download is working for all the datasets and models' weights but, respectively, **Five Billion Pixels**, **BioMassters**, and **GFM**.
+## ⚠️ TO DO
 
+ - host all weights/datasets/subsets on HF (the automatic download is working for all the datasets and models' weights but, respectively, **Five Billion Pixels**, **BioMassters**, and **GFM**. The GFM pretrained model can be downloaded from [OneDrive](https://onedrive.live.com/?authkey=%21AIx%5FHoX7JG4Ai18&id=93B3D3BDA9EFE744%21100937&cid=93B3D3BDA9EFE744).)
+ - add hyperparameters search ([Optuna](https://optuna.org/))
+ - support automatic running of all the experiments
+ - create an Arena to fast benchmark all the GFMs
 
-## 🧮 Some first results
+## 🧮 Some results
 
-A pre-print is coming soon... Stay tuned!
+<img src=".github/boxplot.png" alt="results" width="60%">
 
-| Encoder | Dataset       | Epochs | mIoU   |
-|---------|---------------|--------|--------|
-| Prithvi | MADOS         | 80     | 53.455 |
-| Prithvi | HLSBurnScars  | 80     | 86.208 |
-| Prithvi | Sen1Floods11  | 80     | 87.217 |
-| Prithvi | AI4SmallFarms | 80     | 33.796 |
+Check the paper for all the insights!
 
-NOTE: if you want to benchmark the results of your model, for a fair comparison do not change the hparams in the configs! When the pre-print will be out, we will publish also a set of "benchmark-configs".
+NOTE: if you want to benchmark the results of your model, for a fair comparison do not change the hparams in the configs! Soon we will publish also a set of "benchmark-configs", to support automatic running.
 
 ## 📝 Citation
 
-If you use this software in your work, please cite:
+If you find this work useful, please cite:
 
 ```
-@misc{pangaea,
-  author = {Pangaea Team},
-  title = {Pangaea},
-  year = {2024},
-  publisher = {GitHub},
-  journal = {GitHub repository},
-  howpublished = {\url{https://github.com/yurujaja/pangaea-bench}},
+@misc{marsocci2024pangaeaglobalinclusivebenchmark,
+      title={PANGAEA: A Global and Inclusive Benchmark for Geospatial Foundation Models}, 
+      author={Valerio Marsocci and Yuru Jia and Georges Le Bellier and David Kerekes and Liang Zeng and Sebastian Hafner and Sebastian Gerard and Eric Brune and Ritu Yadav and Ali Shibli and Heng Fang and Yifang Ban and Maarten Vergauwen and Nicolas Audebert and Andrea Nascetti},
+      year={2024},
+      eprint={2412.04204},
+      archivePrefix={arXiv},
+      primaryClass={cs.CV},
+      url={https://arxiv.org/abs/2412.04204}, 
 }
 ```
diff --git a/pangaea/decoders/upernet.py b/pangaea/decoders/upernet.py
@@ -1,3 +1,5 @@
+# Adapted from https://github.com/open-mmlab/mmsegmentation/blob/main/mmseg/models/decode_heads/uper_head.py
+
 import torch
 import torch.nn as nn
 import torch.nn.functional as F