Skip to content

Latest commit

 

History

History
95 lines (72 loc) · 3.87 KB

README.md

File metadata and controls

95 lines (72 loc) · 3.87 KB
Raw

Installation

Please refer to this README.

Data Preparation

Please refer to this README.

Usage

Obtain CLIP Checkpoints

We use CLIP's text encoder (ViT-B/32) for Detic. Obtain the state_dict of the model from GoogleDrive and put it under checkpoints.

OV-COCO

Training

  1. To pre-train the detector only on the detection data of base categories, run
cd CLIM/ovdet
bash tools/dist_train.sh \
     configs/detic/ov_coco/faster_rcnn_r50_caffe_c4_90k_ovcoco.py 8 \
     --work-dir your/output/directory/detic_coco_base

Rename the checkpoint of the trained model as detic_coco_base.pth and put it under checkpoints. We also provide this checkpoint detic_coco_base.pth in Google Drive.

2.1 To fine-tune the detector with caption data (no tags), run

cd CLIM/ovdet
bash tools/dist_train.sh \
     configs/detic/ov_coco/detic_no_tags_clim_faster_rcnn_r50_caffe_c4_45k.py 8 \
     --work-dir your/output/directory/detic_coco_cap_no_tags_clim

2.2 To fine-tune the detector using caption loss and image tag loss, run

cd CLIM/ovdet
bash tools/dist_train.sh \
     configs/detic/ov_coco/detic_w_tags_clim_faster_rcnn_r50_caffe_c4_45k.py 8 \
     --work-dir your/output/directory/detic_coco_cap_w_tags_clim

Testing

We have provided the following checkpoints in Google Drive.

OV-COCO Losses Novel AP50 Config Download
1 Caption 32.3 config model
2 Caption & Tag 35.4 config model

For example, to evaluate the model trained with caption loss and tag loss, run

cd CLIM/ovdet
bash tools/dist_test.sh \
     configs/detic/ov_coco/detic_w_tags_clim_faster_rcnn_r50_caffe_c4_45k.py \
     checkpoints/detic_coco_cap_w_tags_clim.pth \
     8 --work-dir your/output/directory/detic_coco_cap_w_tags_clim

OV-LVIS

Training

First obtain the checkpoint trained on base categories and put it under checkpoints/. Then run

cd CLIM/ovdet
bash tools/dist_train.sh \
     configs/detic/ov_lvis/detic_clim_centernet2_r50_fpn_4x_lvis-base_cc3m-lvis.py 8 \
     --work-dir your/output/directory/detic_lvis_cap_w_tags_clim

Testing

We have provided the following checkpoint.

OV-LVIS Losses mask APr Config Download
1 Caption & Tag 21.8 config model

For example, to evaluate the model trained on LVIS-base and CC3M, run

cd CLIM/ovdet
bash tools/dist_test.sh \
     configs/detic/ov_lvis/detic_clim_centernet2_r50_fpn_4x_lvis-base_cc3m-lvis.py \
     patch/to/the/checkpoint.pth \
     8 --work-dir your/output/directory/detic_lvis_cap_w_tags_clim