Error for video SFT

[yanlai00]

I got the error "AttributeError: 'VisionCrossAttentionLayer' object has no attribute 'pos_embed_0'" when doing supervised fine-tuning on a subset of the video dataset you provided, with the LongVU_Llama3_2_3B_img checkpoint, with 1 node of 8 GPUs. Any insights on how to resolve this is appreciated. Thanks.

[xiaoqian-shen]

Hi, you need to change the token from 576 to 144 as written in README here. Since we use maximum 144 tokens to represent each video frame.

[zzzz123-0708]

Hi, you need to change the token from 576 to 144 as written in README here. Since we use maximum 144 tokens to represent each video frame.
The default config for the model I downloaded is 144. Do I need to change it to 576 when performing inference？

[xiaoqian-shen]

For video model, please set as 144 for training and inference, while for image model, please set as 576.

[yanlai00]

Thanks for your response! I noticed that in the provided llama-3.2-1B video model, the token length is set to 576. Is this intentional?

[xiaoqian-shen]

@yanlai00 Thanks for raising this concern. It should be 144. We have corrected it in LongVU_Llama3_2_1B.