CaiT

Going deeper with Image Transformers, arxiv

PaddlePaddle training/validation code and pretrained models for CaiT.

The official pytorch implementation is here.

This implementation is developed by PaddleViT.

CaiT Model Overview

Update

• Update (2022-07-12): Model weights trained from scratch using PaddleViT is updated.
• Update (2022-03-17): Code is refactored and bugs are fixed.
• Update (2021-09-27): More weights are uploaded.
• Update (2021-08-11): Code is released and ported weights are uploaded.

Models Zoo

Model	Acc@1	Acc@5	#Params	FLOPs	Image Size	Crop_pct	Interpolation	Link
cait_xxs24_224	78.38	94.32	11.9M	2.2G	224	1.0	bicubic	google/baidu
cait_xxs36_224	79.75	94.88	17.2M	33.1G	224	1.0	bicubic	google/baidu
cait_xxs24_384	80.97	95.64	11.9M	6.8G	384	1.0	bicubic	google/baidu
cait_xxs36_384	82.20	96.15	17.2M	10.1G	384	1.0	bicubic	google/baidu
cait_s24_224	83.45	96.57	46.8M	8.7G	224	1.0	bicubic	google/baidu
cait_xs24_384	84.06	96.89	26.5M	15.1G	384	1.0	bicubic	google/baidu
cait_s24_384	85.05	97.34	46.8M	26.5G	384	1.0	bicubic	google/baidu
cait_s36_384	85.45	97.48	68.1M	39.5G	384	1.0	bicubic	google/baidu
cait_m36_384	86.06	97.73	270.7M	156.2G	384	1.0	bicubic	google/baidu
cait_m48_448	86.49	97.75	355.8M	287.3G	448	1.0	bicubic	google/baidu

*The results are evaluated on ImageNet2012 validation set.

Model weights trained from scratch using PaddleViT

Model	Acc@1	Acc@5	#Params	FLOPs	Image Size	Crop_pct	Interpolation	Link	Log
cait_xxs24_224	78.24	96.26	11.9M	2.2G	224	1.0	bicubic	google/baidu	google/baidu

Data Preparation

ImageNet2012 dataset is used in the following file structure:

│imagenet/
├──train_list.txt
├──val_list.txt
├──train/
│  ├── n01440764
│  │   ├── n01440764_10026.JPEG
│  │   ├── n01440764_10027.JPEG
│  │   ├── ......
│  ├── ......
├──val/
│  ├── n01440764
│  │   ├── ILSVRC2012_val_00000293.JPEG
│  │   ├── ILSVRC2012_val_00002138.JPEG
│  │   ├── ......
│  ├── ......

• train_list.txt: list of relative paths and labels of training images. You can download it from: google/baidu
• val_list.txt: list of relative paths and labels of validation images. You can download it from: google/baidu

Usage

To use the model with pretrained weights, download the .pdparam weight file and change related file paths in the following python scripts. The model config files are located in ./configs/.

For example, assume weight file is downloaded in ./cait_xxs24_224.pdparams, to use the cait_xxs24_224 model in python:

from config import get_config
from cait import build_cait as build_model
# config files in ./configs/
config = get_config('./configs/cait_xxs24_224.yaml')
# build model
model = build_model(config)
# load pretrained weights
model_state_dict = paddle.load('./cait_xxs24_224.pdparams')
model.set_state_dict(model_state_dict)

Evaluation

To evaluate CaiT model performance on ImageNet2012, run the following script using command line:

sh run_eval_multi.sh

or

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
python main_multi_gpu.py \
-cfg='./configs/cait_xxs24_224.yaml' \
-dataset='imagenet2012' \
-batch_size=256 \
-data_path='/dataset/imagenet' \
-eval \
-pretrained='./cait_xxs24_224.pdparams' \
-amp

Note: if you have only 1 GPU, change device number to CUDA_VISIBLE_DEVICES=0 would run the evaluation on single GPU.

Training

To train the CaiT model on ImageNet2012, run the following script using command line:

sh run_train_multi.sh

or

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
python main_multi_gpu.py \
-cfg='./configs/cait_xxs24_224.yaml' \
-dataset='imagenet2012' \
-batch_size=256 \
-data_path='/dataset/imagenet' \
-amp

Note: it is highly recommanded to run the training using multiple GPUs / multi-node GPUs.

Finetuning

To finetune the CaiT model on ImageNet2012, run the following script using command line:

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
python main_multi_gpu.py \
-cfg='./configs/cait_xxs24_384.yaml' \
-dataset='imagenet2012' \
-batch_size=16 \
-data_path='/dataset/imagenet' \
-pretrained='./cait_xxs24_224.pdparams' \
-amp

Note: use -pretrained argument to set the pretrained model path, you may also need to modify the hyperparams defined in config file.

Reference

@InProceedings{Touvron_2021_ICCV,
    author    = {Touvron, Hugo and Cord, Matthieu and Sablayrolles, Alexandre and Synnaeve, Gabriel and J\'egou, Herv\'e},
    title     = {Going Deeper With Image Transformers},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2021},
    pages     = {32-42}
}