Cannot Use Pretrain Weights for VGG16

[omid-reza]

Hello,

First of all, I want to appreciate your quick support.

I have used VGG16 config that is in extra branch. When i want to use pretrained model like WEIGHTS: "models/model_final_84107b.pkl", however, I get the below warning and it does not use it:

WARNING [12/24 12:27:24 fvcore.common.checkpoint]: Skip loading parameter 'proposal_generator.rpn_head.conv.weight' to the model due to incompatible shapes: (2048, 2048, 3, 3) in the checkpoint but (512, 512, 3, 3) in the model! You might want to double check if this is expected.
WARNING [12/24 12:27:24 fvcore.common.checkpoint]: Skip loading parameter 'proposal_generator.rpn_head.conv.bias' to the model due to incompatible shapes: (2048,) in the checkpoint but (512,) in the model! You might want to double check if this is expected.
WARNING [12/24 12:27:24 fvcore.common.checkpoint]: Skip loading parameter 'proposal_generator.rpn_head.objectness_logits.weight' to the model due to incompatible shapes: (15, 2048, 1, 1) in the checkpoint but (15, 512, 1, 1) in the model! You might want to double check if this is expected.
WARNING [12/24 12:27:24 fvcore.common.checkpoint]: Skip loading parameter 'proposal_generator.rpn_head.anchor_deltas.weight' to the model due to incompatible shapes: (60, 2048, 1, 1) in the checkpoint but (60, 512, 1, 1) in the model! You might want to double check if this is expected.
WARNING [12/24 12:27:24 fvcore.common.checkpoint]: Skip loading parameter 'roi_heads.box_head.fc1.weight' to the model due to incompatible shapes: (1024, 100352) in the checkpoint but (1024, 25088) in the model! You might want to double check if this is expected.
WARNING [12/24 12:27:24 fvcore.common.checkpoint]: Skip loading parameter 'roi_heads.box_predictor.cls_score.weight' to the model due to incompatible shapes: (81, 1024) in the checkpoint but (11, 1024) in the model! You might want to double check if this is expected.
WARNING [12/24 12:27:24 fvcore.common.checkpoint]: Skip loading parameter 'roi_heads.box_predictor.cls_score.bias' to the model due to incompatible shapes: (81,) in the checkpoint but (11,) in the model! You might want to double check if this is expected.
WARNING [12/24 12:27:24 fvcore.common.checkpoint]: Skip loading parameter 'roi_heads.box_predictor.bbox_pred.weight' to the model due to incompatible shapes: (320, 1024) in the checkpoint but (40, 1024) in the model! You might want to double check if this is expected.
WARNING [12/24 12:27:24 fvcore.common.checkpoint]: Skip loading parameter 'roi_heads.box_predictor.bbox_pred.bias' to the model due to incompatible shapes: (320,) in the checkpoint but (40,) in the model! You might want to double check if this is expected.

My base config file is:

_BASE_: "detectron2/Base-RCNN-DilatedC5.yaml"
MODEL:
  WEIGHTS: "models/model_final_84107b.pkl"
  MASK_ON: False
INPUT:
  MIN_SIZE_TRAIN: (800, 832, 864, 896, 928, 960, 992, 1024)
  MIN_SIZE_TRAIN_SAMPLING: "choice"
  MIN_SIZE_TEST: 1024
  MAX_SIZE_TRAIN: 2048
  MAX_SIZE_TEST: 2048
SOLVER:
  BASE_LR: 0.06
  STEPS: (1600,)
  MAX_ITER: 2000
  IMS_PER_BATCH: 48
  IMS_PER_GPU: 2
  BACKWARD_AT_END: False
  AMP:
    ENABLED: True
  WARMUP_ITERS: 100
  WARMUP_FACTOR: 0.01
  CHECKPOINT_PERIOD: 500
VIS_PERIOD: 100
TEST:
  EVAL_PERIOD: 100

I i use the same detectron2/Base-RCNN-DilatedC5.yaml that we have in the extra branch.

Thanks.

[justinkay]

Hi @omid-reza since you've closed this I guess you figured it out -- this is expected behavior, since the VGG weights are only for the backbone, so the RPN and ROI heads have to be initialized from random weights.

[omid-reza]

Thanks. I figured out the issue with that.

However, when i want to used a pre-trained model, which is trained on cityspaces (VDitB), and train my own baseline model with it, i get the following error:

        size mismatch for roi_heads.box_predictor.cls_score.weight: copying a param with shape torch.Size([2, 1024]) from checkpoint, the shape in current model is torch.Size([11, 1024]).       
        size mismatch for roi_heads.box_predictor.cls_score.bias: copying a param with shape torch.Size([2]) from checkpoint, the shape in current model is torch.Size([11]).
        size mismatch for roi_heads.box_predictor.bbox_pred.weight: copying a param with shape torch.Size([4, 1024]) from checkpoint, the shape in current model is torch.Size([40, 1024]).       
        size mismatch for roi_heads.box_predictor.bbox_pred.bias: copying a param with shape torch.Size([4]) from checkpoint, the shape in current model is torch.Size([40]).

My config file is

_BASE_: "../Base-RCNN-VitDetB-cityscapes.yaml"
MODEL:
  ROI_HEADS:
    NUM_CLASSES: 10
DATASETS:
  TRAIN: ("eds_dom1_train",)
  TEST: ("eds_dom1_test", "eds_dom2_test",)
  BATCH_CONTENTS: ("labeled_strong", )
EMA:
  ENABLED: True
SOLVER:
  STEPS: (11999,)
  MAX_ITER: 12000
  CHECKPOINT_PERIOD: 3000
TEST:
  EVAL_PERIOD: 1250
OUTPUT_DIR: "output/eds1t2cityspaces/eds1t2_vitdetb_baseline_strongaug_ema/"

Base-RCNN-VitDetB-cityscapes.yaml:

_BASE_: "./Base-RCNN-FPN.yaml"
MODEL:
  BACKBONE:
    NAME: "build_vitdet_b_backbone"
  WEIGHTS: "output/cityscapes/cityscapes_vitdetb_baseline_strongaug_ema/cityscapes_foggy_val_model_best.pth"
## See detectron2/configs/common/models/mask_rcnn_vitdet.py
  ROI_BOX_HEAD:
    NORM: "LN"
    CONV_DIM: 256
    NUM_CONV: 4
    FC_DIM: 1024
    NUM_FC: 1
  RPN:
    CONV_DIMS: [-1, -1]
  PIXEL_MEAN: [123.675, 116.28, 103.53]
  PIXEL_STD: [58.395, 57.12, 57.375]
INPUT:
  FORMAT: "RGB"
SOLVER:
  IMS_PER_BATCH: 96
  IMS_PER_GPU: 1
  STEPS: (3200,)
  MAX_ITER: 4000
  OPTIMIZER: "ADAMW"
  BASE_LR: 0.0002
VIT:
  USE_ACT_CHECKPOINT: True

The only thing that i changed in the base config file is the value for WEIGHTS.

[omid-reza]

I figured out that when i used your provided final model that works (there will be warning that we discussed about it before), but when i train the model on my own, it raises an error.

[justinkay]

Can you give some more details about the error you encounter?