From 07bf2e48a5d1434a0b3e803b0e5b42d34a668012 Mon Sep 17 00:00:00 2001 From: KerekesDavid Date: Fri, 27 Sep 2024 12:11:23 +0200 Subject: [PATCH] Install package in a way that doesn't mess with conda --- README.md | 136 +++++++++++++++++++++++++++++------------------------- 1 file changed, 74 insertions(+), 62 deletions(-) diff --git a/README.md b/README.md index 78ca5293..3b937ae0 100644 --- a/README.md +++ b/README.md @@ -4,63 +4,64 @@ ## 📚 Introduction -While geospatial foundation models (GFMs) have proliferated rapidly, their evaluations remain inconsistent and narrow. Existing works often utilize suboptimal downstream datasets (e.g., EuroSAT) and tasks (e.g., land cover classification), which constrain comparability and real-world usability. Additionally, a lack of diversity in evaluation protocols, including image resolution and sensor types, further complicates the extensive assessments of GFM performance. +While geospatial foundation models (GFMs) have proliferated rapidly, their evaluations remain inconsistent and narrow. Existing works often utilize suboptimal downstream datasets (e.g., EuroSAT) and tasks (e.g., land cover classification), which constrain comparability and real-world usability. Additionally, a lack of diversity in evaluation protocols, including image resolution and sensor types, further complicates the extensive assessments of GFM performance. To bridge this gap, we propose a standardized evaluation protocol that incorporates a wide-ranging selection of datasets, tasks, resolutions, and sensor types, establishing a robust and widely applicable benchmark for GFMs. - PANGAEA: a diverse benchmark for geospatial foundation models - In this repo, you can find the code to benchmark GFMs. For the moment we included several GFMs that present different approach. We look forward to adding new models and datasets. For the moment, we support the following **models**: -| | Paper | GitHub | Keywords | -|:-----------:|:-----:|:------:|:--------:| -| [SSL4EOS12](https://arxiv.org/abs/2211.07044) | SSL4EO-S12: A Large-Scale Multi-Modal, Multi-Temporal
Dataset for Self-Supervised Learning in Earth Observation | [link](https://github.com/zhu-xlab/SSL4EO-S12) | DINO, MAE, DATA2VEC, MOCO| -| [Scale-MAE](https://arxiv.org/pdf/2212.14532) | Scale-MAE: A Scale-Aware Masked Autoencoder for Multiscale Geospatial Representation Learning | [link](https://github.com/bair-climate-initiative/scale-mae) | Masked Autoencoders, Multiscale| -| [SatlasNet](https://arxiv.org/pdf/2211.15660) | SatlasPretrain: A Large-Scale Dataset for Remote Sensing Image Understanding | [link](https://github.com/allenai/satlas/tree/main) | Supervised, Multi-temporal | -| [GFM](https://arxiv.org/pdf/2302.04476) | Towards Geospatial Foundation Models via Continual Pretraining | [link](https://github.com/mmendiet/GFM) | Swin, Continual Pre-training | -| [SpectralGPT](https://arxiv.org/abs/2311.07113) | SpectralGPT: Spectral Remote Sensing Foundation Model | [link](https://github.com/danfenghong/IEEE_TPAMI_SpectralGPT) | MAE, Multi-spectral | -| [DOFA](https://arxiv.org/pdf/2403.15356) | Neural Plasticity-Inspired Multimodal Foundation Model for Earth Observation | [link](https://github.com/zhu-xlab/DOFA) | MAE, Dynamic bands | -| [CROMA](https://arxiv.org/pdf/2311.00566) | CROMA: Remote Sensing Representations with Contrastive Radar-Optical Masked Autoencoders | [link](https://github.com/antofuller/CROMA) | Contrastive Learning, MAE | -| [Prithvi](https://arxiv.org/pdf/2310.18660) | Foundation Models for Generalist Geospatial Artificial Intelligence | [link](https://github.com/NASA-IMPACT/hls-foundation-os) | MAE, Multi-temporal | -| [RemoteCLIP](https://arxiv.org/pdf/2306.11029) | RemoteCLIP: A Vision Language Foundation Model for Remote Sensing | [link](https://github.com/ChenDelong1999/RemoteCLIP) | Contrastive Learning | - +| | Paper | GitHub | Keywords | +| :---------------------------------------------: | :------------------------------------------------------------------------------------------------------------------: | :-----------------------------------------------------------: | :-----------------------------: | +| [SSL4EOS12](https://arxiv.org/abs/2211.07044) | SSL4EO-S12: A Large-Scale Multi-Modal, Multi-Temporal
Dataset for Self-Supervised Learning in Earth Observation | [link](https://github.com/zhu-xlab/SSL4EO-S12) | DINO, MAE, DATA2VEC, MOCO | +| [Scale-MAE](https://arxiv.org/pdf/2212.14532) | Scale-MAE: A Scale-Aware Masked Autoencoder for Multiscale Geospatial Representation Learning | [link](https://github.com/bair-climate-initiative/scale-mae) | Masked Autoencoders, Multiscale | +| [SatlasNet](https://arxiv.org/pdf/2211.15660) | SatlasPretrain: A Large-Scale Dataset for Remote Sensing Image Understanding | [link](https://github.com/allenai/satlas/tree/main) | Supervised, Multi-temporal | +| [GFM](https://arxiv.org/pdf/2302.04476) | Towards Geospatial Foundation Models via Continual Pretraining | [link](https://github.com/mmendiet/GFM) | Swin, Continual Pre-training | +| [SpectralGPT](https://arxiv.org/abs/2311.07113) | SpectralGPT: Spectral Remote Sensing Foundation Model | [link](https://github.com/danfenghong/IEEE_TPAMI_SpectralGPT) | MAE, Multi-spectral | +| [DOFA](https://arxiv.org/pdf/2403.15356) | Neural Plasticity-Inspired Multimodal Foundation Model for Earth Observation | [link](https://github.com/zhu-xlab/DOFA) | MAE, Dynamic bands | +| [CROMA](https://arxiv.org/pdf/2311.00566) | CROMA: Remote Sensing Representations with Contrastive Radar-Optical Masked Autoencoders | [link](https://github.com/antofuller/CROMA) | Contrastive Learning, MAE | +| [Prithvi](https://arxiv.org/pdf/2310.18660) | Foundation Models for Generalist Geospatial Artificial Intelligence | [link](https://github.com/NASA-IMPACT/hls-foundation-os) | MAE, Multi-temporal | +| [RemoteCLIP](https://arxiv.org/pdf/2306.11029) | RemoteCLIP: A Vision Language Foundation Model for Remote Sensing | [link](https://github.com/ChenDelong1999/RemoteCLIP) | Contrastive Learning | And the following **datasets**: -| | Download | Domain | Task | Sensors | Location | -|:-------------------:|:--------:|:------:|:----:|:-------:|:--------:| -| [HLS Burn Scars](https://huggingface.co/datasets/ibm-nasa-geospatial/hls_burn_scars) | [link](https://huggingface.co/datasets/ibm-nasa-geospatial/hls_burn_scars) | Wildfire | Semantic Segmentation | HLS (Harmonized Landsat Sentinel-2) | USA | -| [MADOS](https://www.sciencedirect.com/science/article/pii/S0924271624000625) | [link](https://marine-pollution.github.io/index.html) | Marine | Semantic Segmentation | S2 | Global | -| [PASTIS-HD](https://arxiv.org/abs/2404.08351) | [link](https://huggingface.co/datasets/IGNF/PASTIS-HD) | Agriculture | Semantic Segmentation | S1, S2, SPOT-6 | France | -| [Sen1Floods11](http://openaccess.thecvf.com/content_CVPRW_2020/html/w11/Bonafilia_Sen1Floods11_A_Georeferenced_Dataset_to_Train_and_Test_Deep_Learning_CVPRW_2020_paper.html) | [link](https://github.com/cloudtostreet/Sen1Floods11) | Flood |Semantic Segmentation | S1, S2 | Global | -| [xView2](https://openaccess.thecvf.com/content_CVPRW_2019/html/cv4gc/Gupta_Creating_xBD_A_Dataset_for_Assessing_Building_Damage_from_Satellite_CVPRW_2019_paper.html) | [link](https://xview2.org/dataset) | HADR | Change Detection | Maxar | Global | -| [Five Billion Pixels](https://www.sciencedirect.com/science/article/pii/S0924271622003264) | [original version](https://x-ytong.github.io/project/Five-Billion-Pixels.html)
(custom version coming soon) | (Urban) Land Cover | Semantic Segmentation | Gaofen-2 | China | -| [DynamicEarthNet](https://arxiv.org/pdf/2203.12560) | [link](https://mediatum.ub.tum.de/1650201) | (Urban) Land Cover | Semantic Segmentation | PlanetFusion | Global | -| [CropTypeMapping](https://openaccess.thecvf.com/content_CVPRW_2019/papers/cv4gc/Rustowicz_Semantic_Segmentation_of_Crop_Type_in_Africa_A_Novel_Dataset_CVPRW_2019_paper.pdf) | [link](https://sustainlab-group.github.io/sustainbench/docs/datasets/sdg2/crop_type_mapping_ghana-ss.html#download) | Agriculture |Semantic Segmentation |S1, S2, Planet|South Sudan| -| [SpaceNet 7](https://openaccess.thecvf.com/content/CVPR2021/papers/Van_Etten_The_Multi-Temporal_Urban_Development_SpaceNet_Dataset_CVPR_2021_paper.pdf) | [link](https://spacenet.ai/sn7-challenge/) | Urban | Change detection/
Semantic Segmentation | Planet | Global | -| [AI4SmallFarms](https://ieeexplore.ieee.org/document/10278130) | [link](https://doi.org/10.17026/dans-xy6-ngg6) | Agriculture | Semantic segmentation | S2 | Cambodia/Vietnam | -| [BioMassters](https://papers.nips.cc/paper_files/paper/2023/file/40daf2a00278c4bea1b26cd4c8a654f8-Paper-Datasets_and_Benchmarks.pdf) | [link](https://huggingface.co/datasets/nascetti-a/BioMassters) | Forest | Regression | S1, S2 | Finland | +| | Download | Domain | Task | Sensors | Location | +| :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------------------------------------: | :----------------: | :------------------------------------------: | :---------------------------------: | :--------------: | +| [HLS Burn Scars](https://huggingface.co/datasets/ibm-nasa-geospatial/hls_burn_scars) | [link](https://huggingface.co/datasets/ibm-nasa-geospatial/hls_burn_scars) | Wildfire | Semantic Segmentation | HLS (Harmonized Landsat Sentinel-2) | USA | +| [MADOS](https://www.sciencedirect.com/science/article/pii/S0924271624000625) | [link](https://marine-pollution.github.io/index.html) | Marine | Semantic Segmentation | S2 | Global | +| [PASTIS-HD](https://arxiv.org/abs/2404.08351) | [link](https://huggingface.co/datasets/IGNF/PASTIS-HD) | Agriculture | Semantic Segmentation | S1, S2, SPOT-6 | France | +| [Sen1Floods11](http://openaccess.thecvf.com/content_CVPRW_2020/html/w11/Bonafilia_Sen1Floods11_A_Georeferenced_Dataset_to_Train_and_Test_Deep_Learning_CVPRW_2020_paper.html) | [link](https://github.com/cloudtostreet/Sen1Floods11) | Flood | Semantic Segmentation | S1, S2 | Global | +| [xView2](https://openaccess.thecvf.com/content_CVPRW_2019/html/cv4gc/Gupta_Creating_xBD_A_Dataset_for_Assessing_Building_Damage_from_Satellite_CVPRW_2019_paper.html) | [link](https://xview2.org/dataset) | HADR | Change Detection | Maxar | Global | +| [Five Billion Pixels](https://www.sciencedirect.com/science/article/pii/S0924271622003264) | [original version](https://x-ytong.github.io/project/Five-Billion-Pixels.html)
(custom version coming soon) | (Urban) Land Cover | Semantic Segmentation | Gaofen-2 | China | +| [DynamicEarthNet](https://arxiv.org/pdf/2203.12560) | [link](https://mediatum.ub.tum.de/1650201) | (Urban) Land Cover | Semantic Segmentation | PlanetFusion | Global | +| [CropTypeMapping](https://openaccess.thecvf.com/content_CVPRW_2019/papers/cv4gc/Rustowicz_Semantic_Segmentation_of_Crop_Type_in_Africa_A_Novel_Dataset_CVPRW_2019_paper.pdf) | [link](https://sustainlab-group.github.io/sustainbench/docs/datasets/sdg2/crop_type_mapping_ghana-ss.html#download) | Agriculture | Semantic Segmentation | S1, S2, Planet | South Sudan | +| [SpaceNet 7](https://openaccess.thecvf.com/content/CVPR2021/papers/Van_Etten_The_Multi-Temporal_Urban_Development_SpaceNet_Dataset_CVPR_2021_paper.pdf) | [link](https://spacenet.ai/sn7-challenge/) | Urban | Change detection/
Semantic Segmentation | Planet | Global | +| [AI4SmallFarms](https://ieeexplore.ieee.org/document/10278130) | [link](https://doi.org/10.17026/dans-xy6-ngg6) | Agriculture | Semantic segmentation | S2 | Cambodia/Vietnam | +| [BioMassters](https://papers.nips.cc/paper_files/paper/2023/file/40daf2a00278c4bea1b26cd4c8a654f8-Paper-Datasets_and_Benchmarks.pdf) | [link](https://huggingface.co/datasets/nascetti-a/BioMassters) | Forest | Regression | S1, S2 | Finland | The repository supports the following **tasks** using geospatial (foundation) models: - - [Single Temporal Semantic Segmentation](#single-temporal-semantic-segmentation) - - [Multi-Temporal Semantic Segmentation](#multi-temporal-semantic-segmentation) - - [Change Detection](#change-detection) - - [Single Temporal Regression](#single-temporal-regression) - - [Multi-Temporal Regression](#multi-temporal-regression) + +- [Single Temporal Semantic Segmentation](#single-temporal-semantic-segmentation) +- [Multi-Temporal Semantic Segmentation](#multi-temporal-semantic-segmentation) +- [Change Detection](#change-detection) +- [Single Temporal Regression](#single-temporal-regression) +- [Multi-Temporal Regression](#multi-temporal-regression) It is also possible to train some [supervised baselines](#-fully-supervised-baseline), based on UNet. ## 🗺️ Datasets details + Please refer to [**Dataset Guide**](DATASET_GUIDE.md) to understand the processing requirements and commands specific to each dataset. If you want to fast-prototype your model, maybe you want to run fast experiments on smaller datasets. We suggest starting with MADOS, HLSBurnScars, SpaceNet7 and Sen1Floods11 and AI4SmallFarms. They offer good diversity in satellites and domains. In the future, we will release stratified subsets for each dataset to facilitate fast prototyping across all datasets. ## 🛠️ Setup + Clone the repository: + ``` git clone https://github.com/yurujaja/pangaea-bench.git cd pangaea-bench @@ -71,60 +72,67 @@ cd pangaea-bench We provide several ways to install the dependencies. 1. **Using either Conda or Mamba**: - ``` - conda env create -f environment.yaml - conda activate pangaea-bench - ``` - Optional: install [Mamba](https://github.com/conda-forge/miniforge/releases/) for faster resolution times - ``` - wget https://github.com/conda-forge/miniforge/releases/download/24.3.0-0/Mambaforge-24.3.0-0-Linux-x86_64.sh - sh ./Mambaforge-24.3.0-0-Linux-x86_64.sh + ``` + conda env create -f environment.yaml + conda activate pangaea-bench + ``` + + Optional: install [Mamba](https://github.com/conda-forge/miniforge/releases/) for faster resolution times + + ``` + wget https://github.com/conda-forge/miniforge/releases/download/24.3.0-0/Mambaforge-24.3.0-0-Linux-x86_64.sh + sh ./Mambaforge-24.3.0-0-Linux-x86_64.sh - mamba env create -f environment.yaml - mamba activate pangaea-bench - ``` + mamba env create -f environment.yaml + mamba activate pangaea-bench + ``` 2. **Using pip**, create a Python native virtual environment and install dependencies into it: + ``` export PANGAEA_PATH=/path/to/venv/pangaea-bench # change this python3 -m venv ${PANGAEA_PATH} source ${PANGAEA_PATH}/bin/activate - + pip install -r requirements.txt ``` - **Then install the code repository as a package** + + **then install the code repository as a development package** + ``` - pip install -e . + pip install --no-build-isolation --no-deps -e . ``` - ## 🏋️ Training To run experiments, please refer to `configs/train.yaml`. In it, in addition to some basic info about training (e.g. `finetune` for fine-tuning also the encoder, `limited_label` to train the model on a subset of labels, `num_workers`, `batch_size` and so on), there are 5 different basic configs: + - `dataset`: Information of downstream datasets such as image size, band_statistics, classes etc. - `decoder`: Downstream task decoder fine-tuning related parameters, like the type of architecture (e.g. UPerNet), which multi-temporal strategy to use, and other related hparams (e.g. nr of channels) -- `encoder`: GFM encoder related parameters. `output_layers` is used for which layers are used for Upernet decoder. +- `encoder`: GFM encoder related parameters. `output_layers` is used for which layers are used for Upernet decoder. - `preprocessing`: Both preprocessing and augmentations steps required for the dataset, such as bands adaptation, normalization, resize/crop. - `task`: Information about the trainer and evaluator. Most of the parameters are overwritten in run. Trainer and evaluator can be used for segmentation (`SegTrainer`) or regression (`RegTrainer`). Different parameter like precision training (`precision`) can be set in it. - Other 3 configs are used to set other training parameters: + - `criterion`: in which you can choose the loss for the training. Consider that if you want to add a custom loss, you should add to `pangaea/utils/losses.py`. Currently, we support `cross_entropy`, `weigthed_cross_entropy`, `dice` and `mae` loss functions. -- `lr_scheduler`: in which you can choose the scheduler. Consider that if you want to add a custom one, you should add to `pangaea/utils/schedulers.py`. +- `lr_scheduler`: in which you can choose the scheduler. Consider that if you want to add a custom one, you should add to `pangaea/utils/schedulers.py`. - `optimizer`: in which you can choose the optimizer. Consider that if you want to add a custom one, you should add to `pangaea/utils/optimizers.py`. - We provide several examples of command lines to initialize different training tasks on single GPU. Please note: - - The repo adopts [hydra](https://github.com/facebookresearch/hydra), so you can easily log your experiments and overwrite parameters from the command line. More examples are provided later. - - To use more gpus or nodes, set `--nnodes` and `--nproc_per_node` correspondingly. Please refer to the [torchrun doc](https://pytorch.org/docs/stable/elastic/run.html). + +- The repo adopts [hydra](https://github.com/facebookresearch/hydra), so you can easily log your experiments and overwrite parameters from the command line. More examples are provided later. +- To use more gpus or nodes, set `--nnodes` and `--nproc_per_node` correspondingly. Please refer to the [torchrun doc](https://pytorch.org/docs/stable/elastic/run.html). ### 💻 Decoder Finetuning + #### Single Temporal Semantic Segmentation Take HLSBurnScars dataset, RemoteCLIP Encoder and Upernet Segmentation Decoder as example: + ``` torchrun --nnodes=1 --nproc_per_node=1 pangaea/run.py \ --config-name=train \ @@ -137,6 +145,7 @@ torchrun --nnodes=1 --nproc_per_node=1 pangaea/run.py \ ``` If you want to overwrite some parameters (e.g. turn off wandbe, and changing batch size and the path to the dataset): + ``` torchrun --nnodes=1 --nproc_per_node=1 pangaea/run.py \ --config-name=train \ @@ -188,7 +197,7 @@ To overwrite parameters, please check the Single Temporal Semantic Segmentation #### Single Temporal Regression -The regression decoder (e.g. `configs/decoder/reg_upernet.yaml`) and the regression task (e.g. `configs/task/regression.yaml`) configs should be used. +The regression decoder (e.g. `configs/decoder/reg_upernet.yaml`) and the regression task (e.g. `configs/task/regression.yaml`) configs should be used. e.g. Prithvi encoder on BioMassters ``` @@ -206,7 +215,7 @@ To overwrite parameters, please check the Single Temporal Semantic Segmentation #### Multi-Temporal Regression -The multi-temporal regression decoder (e.g. `configs/decoder/reg_upernet_mt_ltae.yaml` or `configs/decoder/reg_upernet_mt_linear.yaml`) and the regression task (e.g. `configs/task/regression.yaml`) configs should be used. +The multi-temporal regression decoder (e.g. `configs/decoder/reg_upernet_mt_ltae.yaml` or `configs/decoder/reg_upernet_mt_linear.yaml`) and the regression task (e.g. `configs/task/regression.yaml`) configs should be used. e.g. Prithvi encoder on BioMassters ``` @@ -227,6 +236,7 @@ To overwrite parameters, please check the Single Temporal Semantic Segmentation It is enough to add `finetune=True` to the command line. For example, for single-temporal semantic segmentation: + ``` torchrun --nnodes=1 --nproc_per_node=1 pangaea/run.py \ --config-name=train \ @@ -239,10 +249,11 @@ torchrun --nnodes=1 --nproc_per_node=1 pangaea/run.py \ finetune=True ``` -### 💻 Fully Supervised Baseline +### 💻 Fully Supervised Baseline -The repo supports also training fully supervised baselines (e.g. UNet). To run these, follow the same command line rules as for other models. Keep in mind that setting finetune=True is necessary since this fully supervised approach trains the model from scratch. +The repo supports also training fully supervised baselines (e.g. UNet). To run these, follow the same command line rules as for other models. Keep in mind that setting finetune=True is necessary since this fully supervised approach trains the model from scratch. An example for single temporal semantic segmentation is provided (Sen1Floods11 dataset): + ``` torchrun --nnodes=1 --nproc_per_node=1 pangaea/run.py \ --config-name=train \ @@ -254,6 +265,7 @@ torchrun --nnodes=1 --nproc_per_node=1 pangaea/run.py \ task=segmentation \ finetune=True ``` + For the moment, there is no multi-temporal baseline supported. ## 🔧 Customization @@ -266,7 +278,7 @@ Refer to: [Adding a new downstream dataset](.github/CONTRIBUTING.md#adding-a-new Refer to: [Adding a new geospatial foundation model](.github/CONTRIBUTING.md#adding-a-new-geospatial-foundation-model) -## 🏃 Evaluation +## 🏃 Evaluation An evaluation step is always run after the training. @@ -277,26 +289,26 @@ torchrun pangaea/run.py --config-name=test ckpt_dir=path_to_ckpt_dir ``` ## ✏️ Contributing + We appreciate all contributions. Please refer to [Contributing Guidelines](.github/CONTRIBUTING.md). ## ⚠️ Warnings Some features are under construction: - - the automatic download is working for all the datasets and models' weights but, respectively, **Five Billion Pixels**, **BioMassters**, and **GFM**. +- the automatic download is working for all the datasets and models' weights but, respectively, **Five Billion Pixels**, **BioMassters**, and **GFM**. ## 🧮 Some first results A pre-print is coming soon... Stay tuned! | Encoder | Dataset | Epochs | mIoU | -|---------|---------------|--------|--------| +| ------- | ------------- | ------ | ------ | | Prithvi | MADOS | 80 | 53.455 | | Prithvi | HLSBurnScars | 80 | 86.208 | | Prithvi | Sen1Floods11 | 80 | 87.217 | | Prithvi | AI4SmallFarms | 80 | 33.796 | - ## 📝 Citation If you use this software in your work, please cite: