CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Prediction

Introduction

This is an official release of the paper CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Prediction.

CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Prediction,
Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Xiangtai Li, Wentao Liu, Chen Change Loy
Bibetex

TODO

• Code and models of CLIPSelf
• Code and models of F-ViT
• Support F-ViT under the ovdet repo using MMDetection3.x

Installation

This project is adapted from OpenCLIP-v2.16.0. Run the following command to install the package

pip install -e . -v

Data Preparation

The main experiments are conducted using images from COCO and LVIS datasets. Please prepare datasets and organize them like the following:

CLIPSelf/
├── data
    ├── coco
        ├── annotations
            ├── instances_train2017.json  # the box annotations are not used
            ├── panoptic_val2017.json
            ├── panoptic_val2017     # panoptic masks
        ├── train2017
        ├── val2017
        ├── coco_pseudo_4764.json    # to run RegionCLIP
        ├── coco_proposals.json      # to run CLIPSelf with region proposals
    ├── lvis_v1
        ├── annotations
            ├── lvis_v1_train.json  # the box annotations are not used
        ├── train2017    # the same with coco
        ├── val2017      # the same with coco

For CLIPSelf with region proposals or RegionCLIP that uses region-text pairs, obtain coco_pseudo_4764.json or coco_proposals.json from Drive. Put the json files under data/coco.

Run

Original Models

To run CLIPSelf, first obtain the original models from EVA-02-CLIP, and put them under checkpoints/ like the following:

CLIPSelf/
├── checkpoints
    ├── EVA02_CLIP_B_psz16_s8B.pt
    ├── EVA02_CLIP_L_336_psz14_s6B.pt
    

Training and Testing

We provide the scripts to train CLIPSelf and RegionCLIP under scripts/, they are summarized as follows:

#	Model	Method	Proposals	Training Data	Script	Checkpoint
1	ViT-B/16	CLIPSelf	-	COCO	script	model
2	ViT-B/16	CLIPSelf	+	COCO	script	model
3	ViT-B/16	RegionCLIP	+	COCO	script	model
4	ViT-L/14	CLIPSelf	-	COCO	script	model
5	ViT-L/14	CLIPSelf	+	COCO	script	model
6	ViT-L/14	RegionCLIP	+	COCO	script	model
7	ViT-B/16	CLIPSelf	-	LVIS	script	model
8	ViT-L/14	CLIPSelf	-	LVIS	script	model

For example, if we want to refine ViT-B/16 by CLIPSelf using only image patches on COCO, simply run:

bash scripts/train_clipself_coco_image_patches_eva_vitb16.sh    # 1

We also provide the checkpoints of the listed experiments above in Drive. And they can be organized as follows:

CLIPSelf/
├── checkpoints
    ├── eva_vitb16_coco_clipself_patches.pt     # 1
    ├── eva_vitb16_coco_clipself_proposals.pt   # 2
    ├── eva_vitb16_coco_regionclip.pt           # 3
    ├── eva_vitl14_coco_clipself_patches.pt     # 4
    ├── eva_vitl14_coco_clipself_proposals.pt   # 5
    ├── eva_vitl14_coco_regionclip.pt           # 6
    ├── eva_vitb16_lvis_clipself_patches.pt     # 7
    ├── eva_vitl14_lvis_clipself_patches.pt     # 8

To evaluate a ViT-B/16 model, run:

bash scripts/test_eva_vitb16_macc_boxes_masks.sh name_of_the_test path/to/checkpoint.pt

To evaluate a ViT-L/14 model, run:

bash scripts/test_eva_vitl14_macc_boxes_masks.sh name_of_the_test path/to/checkpoint.pt

F-ViT

Go to the folder CLIPSelf/F-ViT and follow the instructions in this README.

License

This project is licensed under NTU S-Lab License 1.0.

Citation

@article{wu2023clipself,
    title={CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Prediction},
    author={Size Wu and Wenwei Zhang and Lumin Xu and Sheng Jin and Xiangtai Li and Wentao Liu and Chen Change Loy},
    journal={arXiv preprint arXiv:2310.01403},
    year={2023}
}

Acknowledgement

We thank OpenCLIP, EVA-CLIP and MMDetection for their valuable code bases.