Merge pull request #64 from yurujaja/update-satlasnet-info

Update satlasnet info

README.md

@@ -9,7 +9,7 @@ While geospatial foundation models (GFMs) have proliferated rapidly, their evalu
 To bridge this gap, we propose a standardized evaluation protocol that incorporates a wide-ranging selection of datasets, tasks, resolutions, and sensor types, establishing a robust and widely applicable benchmark for GFMs.
 
 
-<img src="figures/geofmbenchmark.png" alt="PANGAEA: a diverse benchmark for geospatial foundation models" width="80%">
+<img src=".github/geofmbenchmark.png" alt="PANGAEA: a diverse benchmark for geospatial foundation models" width="90%">
 
 
 In this repo, you can find the code to benchmark GFMs. For the moment we included several GFMs that present different approach. We look forward to adding new models and datasets.
@@ -152,22 +152,35 @@ torchrun --nnodes=1 --nproc_per_node=1 pangaea/run.py \
 ```
 
 #### Multi-Temporal Semantic Segmentation
+- Multi-temporal decoder config (e.g. `configs/decoder/seg_upernet_mt_ltae.yaml` if you want to use `ltae` as a strategy to combine multi-temporal info) should be used. 
+- In addition, in the dataset config, indicate the number of time frames, e.g., `multi_temporal: 6`
 
-Multi-temporal decoder config (e.g. `configs/decoder/seg_upernet_mt_ltae.yaml` if you want to use `ltae` as a strategy to combine multi-temporal info) should be used. e.g. Prithvi encoder on CropTypeMapping
-In addition, in the dataset config, indicate the number of time frames, e.g., `multi_temporal: 6`
-
+An example of using SSL4EO-DINO on CropTypeMapping is as below
 ```
 torchrun --nnodes=1 --nproc_per_node=1 pangaea/run.py \
    --config-name=train \
    dataset=croptypemapping \
-   encoder=prithvi \
+   encoder=ssl4eo_dino \
    decoder=seg_upernet_mt_ltae \
    preprocessing=seg_resize \
    criterion=cross_entropy \
    task=segmentation
 ```
 
-To overwrite parameters, please check the Single Temporal Semantic Segmentation example
+To use SatlasNet encoder, the `configs/encoder/satlasnet_mi.yaml` is required
+```
+torchrun --nnodes=1 --nproc_per_node=1 pangaea/run.py \
+   --config-name=train \
+   dataset=croptypemapping \
+   encoder=satlasnet_mi \
+   decoder=seg_upernet_mt_ltae decoder.multi_temporal_strategy=null \
+   preprocessing=seg_resize \
+   criterion=cross_entropy \
+   task=segmentation
+```
+
+
+To overwrite parameters, please check the Single Temporal Semantic Segmentation example.
 
 #### Change Detection
 
@@ -184,12 +197,12 @@ torchrun --nnodes=1 --nproc_per_node=1 pangaea/run.py \
    task=change_detection
 ```
 
-To overwrite parameters, please check the Single Temporal Semantic Segmentation example
+To overwrite parameters, please check the Single Temporal Semantic Segmentation example.
 
 #### Single Temporal Regression
 
 The regression decoder (e.g. `configs/decoder/reg_upernet.yaml`) and the regression task (e.g. `configs/task/regression.yaml`) configs should be used. 
-e.g. Prithvi encoder on BioMassters
+E.g. Prithvi encoder on BioMassters
 
 ```
 torchrun --nnodes=1 --nproc_per_node=1 pangaea/run.py \
@@ -201,13 +214,14 @@ torchrun --nnodes=1 --nproc_per_node=1 pangaea/run.py \
    criterion=cross_entropy \
    task=regression
 ```
-
-To overwrite parameters, please check the Single Temporal Semantic Segmentation example
+To use SatlasNet encoder, the `configs/encoder/satlasnet_si.yaml` is required.
+To overwrite parameters, please check the Single Temporal Semantic Segmentation example.
 
 #### Multi-Temporal Regression
 
 The multi-temporal regression decoder (e.g. `configs/decoder/reg_upernet_mt_ltae.yaml` or `configs/decoder/reg_upernet_mt_linear.yaml`) and the regression task (e.g. `configs/task/regression.yaml`) configs should be used. 
-e.g. Prithvi encoder on BioMassters
+
+Take Prithvi encoder on BioMassters as example:
 
 ```
 torchrun --nnodes=1 --nproc_per_node=1 pangaea/run.py \
@@ -220,7 +234,8 @@ torchrun --nnodes=1 --nproc_per_node=1 pangaea/run.py \
    task=regression
 ```
 
-To overwrite parameters, please check the Single Temporal Semantic Segmentation example
+To use SatlasNet encoder, please refer to the multi-temporal semantic segmentation example.
+To overwrite parameters, please check the Single Temporal Semantic Segmentation example.
 
 ### 💻 End-to-end Finetuning
 

configs/encoder/satlasnet.yaml → configs/encoder/satlasnet_mi.yaml

@@ -2,7 +2,7 @@
 _target_: pangaea.encoders.satlasnet_encoder.SatlasNet_Encoder
 encoder_weights: null
 download_url: null
-model_identifier: Sentinel2_SwinB_MI_MS #Sentinel2_SwinB_SI_RGB   #
+model_identifier: Sentinel2_SwinB_MI_MS # Multi-Image Multi-Spectral 
 fpn: False
 input_size: 128
 output_dim: 1024

configs/encoder/satlasnet_si.yaml

@@ -0,0 +1,25 @@
+
+_target_: pangaea.encoders.satlasnet_encoder.SatlasNet_Encoder
+encoder_weights: null
+download_url: null
+model_identifier: Sentinel2_SwinB_SI_MS  # Single Image Multi-Spectral 
+fpn: False
+input_size: 128
+output_dim: 1024
+
+input_bands:
+  optical:
+    # - B4
+    # - B3
+    # - B2
+
+    #MS Bands 
+    - B2
+    - B3
+    - B4
+    - B5
+    - B6
+    - B7
+    - B8
+    - B11
+    - B12

pangaea/encoders/ssl4eo_mae_encoder.py

@@ -197,6 +197,7 @@ def __init__(
         input_size: int,
         input_bands: dict[str, list[str]],
         output_layers: int | list[int],
+        download_url: str,
         embed_dim: int = 1024,
         patch_size: int = 16,
         in_chans: int = 3,
@@ -217,6 +218,7 @@ def __init__(
             num_heads=num_heads,
             mlp_ratio=mlp_ratio,
             norm_layer=norm_layer,
+            download_url=download_url,
         )
 
         self.model_name = "ssl4eo_mae_sar"