Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update README.md #45

Merged
merged 1 commit into from
Sep 18, 2024
Merged

Update README.md #45

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
101 changes: 61 additions & 40 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,42 +1,47 @@
[![Tests](https://github.com/yurujaja/geofm-bench/actions/workflows/python-test.yml/badge.svg)](https://github.com/yurujaja/geofm-bench/actions/workflows/python-test.yml)

## Introduction
(TBD)

### engines
In engines, basic modules in the training pipeline are defined including data_preprocessor, trainer and evaluator.
1. data_preprocessor replaced the previous adaptation.py, i.e., selects the bands needed by an encoder and pads unavailable bands with zeros, and different augmentations.
2. trainer now support mixed precision/distributed training and print training stats and metrics in real time.
3. evaluator can be called independently and evaluate a model also in distributed way and compute per class metrics.
4. see run.py for how to assemble these modules and concatenate them

### datasets
1. The implementations are simplified and standardized (I try my best).
2. Dataset metas are read from configs, including newly added classes (name), ignore_index, and so on.
3.Mados, sen1floods, hlsburnscars, xView2, biomasster are supported by this branch currently.
4. To add (register) a new dataset implementation, use the decorator @DATASET_REGISTRY.register().

### foundation_models
1. Remove all irrelevant modules and functions used in pre-training. Only keep the essential modules in encoders for extracting features.
2. Support multi-stage output that may be needed by segmentors, specified by output layers in encoder config.
3. All the encoder should work properly.
4. To add (register) a new encoder implementation, use the decorator @ENCODER_REGISTRY.register().

### segmentors
1. Now the UperNet implementation is based on mmsegmentation, which is more likely correct: https://github.com/open-mmlab/mmsegmentation/tree/main
2. We can copypaste more segmentors later.
3. To add (register) a new encoder implementation, use the decorator @SEGMENTOR_REGISTRY.register().
4. So far, we have UPerNet for unitemporal semantic segmentation, UPerNetCD for change detection and MTUPerNet for multitemporal semantic segmentation
5. for multi-temporal, L-TAE and linear projection are supported

All of these parameters can also be set in the run config file.

To use more gpus or nodes, set `--nnodes` and `--nproc_per_node` correspondingly, see:
https://pytorch.org/docs/stable/elastic/run.html

To use mixed precision training, specify either `--fp16` for float16 and or `--bf16` for bfloat16

For fine-tuning instead of linear probing, specify `--finetune`.
## 📚 Introduction

While geospatial foundation models (GFMs) have proliferated rapidly, their evaluations remain inconsistent and narrow. Existing works often utilize suboptimal downstream datasets (e.g., EuroSAT) and tasks (e.g., land cover classification), which constrain comparability and real-world usability. Additionally, a lack of diversity in evaluation protocols, including image resolution and sensor types, further complicates the extensive assessments of GFM performance. To bridge this gap, we propose a standardized evaluation protocol that incorporates a wide-ranging selection of datasets, tasks, resolutions, and sensor types, establishing a robust and widely applicable benchmark for GFMs.

In this repo, you can find the code to benchmark GFMs. For the moment we included several GFMs that present different approach. We look forward to adding new models and datasets.

For the moment, we support the following **models**:

| | Paper | GitHub | Keywords |
|:-----------:|:-----:|:------:|:--------:|
| SSL4EOS12 | | | |
| Scale-MAE | | | |
| SatlasNet | | | |
| GFM | | | |
| SpectralGPT | | | |
| DOFA | | | |
| CROMA | | | |
| Prithvi | | | |
| RemoteCLIP | | | |

And the following **datasets**:

| | Paper | Download | Domain | Task | Sensors | Location |
|:-------------------:|:-----:|:--------:|:------:|:----:|---------|----------|
| HLS Burn Scars | | | | | | |
| MADOS | | | | | | |
| PASTIS | | | | | | |
| Sen1Floods11 | | | | | | |
| xView2 | | | | | | |
| Five Billion Pixels | | | | | | |
| DynamicEarthNet | | | | | | |
| CropTypeMapping | | | | | | |
| SpaceNet7 | | | | | | |
| AI4SmallFarms | | | | | | |
| BioMassters | | | | | | |

The repository supports the following **tasks**:
- unitemporal semantic segmentation
- multi-temporal semantic segmentation
- unitemporal regression
- multi-temporal regression
- change detection

## 🛠️ Setup
Clone the repository:
Expand Down Expand Up @@ -98,7 +103,7 @@ torchrun --nnodes=1 --nproc_per_node=1 run.py \
--num_workers 4 --eval_interval 1 --use_wandb
```

**Multi Temporal Change Detection**
**Change Detection**
```
torchrun ...
```
Expand All @@ -113,25 +118,41 @@ torchrun ...
```
torchrun ...
```

To use more gpus or nodes, set `--nnodes` and `--nproc_per_node` correspondingly, see:
https://pytorch.org/docs/stable/elastic/run.html

To use mixed precision training, specify either `--fp16` for float16 and or `--bf16` for bfloat16

## 🏃 Evaluation
Indicate the `eval_dir` where the checkpoints and configurations are stored.
```
torchrun --nnodes=1 --nproc_per_node=1 run.py --batch_size 1 --eval_dir work-dir/the-folder-where-your-exp-is-saved
```


## ✏️ Contributing
We appreciate all contributions to improve xxx. Please refer to [Contributing Guidelines](.github/CONTRIBUTING.md)

## ⚠️ Warnings

Some features are under construction:
- the automatic download is working for all the datasets and models' weights but, respectively, **Five Billion Pixels** and **GFM**.


## Some numbers
## 🧮 Some first results

A pre-print is coming soon... Stay tuned to read it

| Encoder | Dataset | Epochs | mIoU |
|---------|--------------|--------|--------|
| Prithvi | MADOS | 80 | 53.455 |
| Prithvi | HLSBurnScars | 80 | 86.208 |
| Prithvi | Sen1Floods11 | 80 | 87.217 |

Please note:

## 💡 Acknowledgements

## ©️ License

## 📝 Citing