Question about the retriever training

[szhang42]

Hello,

I have questions for the retriever training. I first use the provided nq reader base to generate the cross attention score json file for Natural Question training json using the below code.

python test_reader.py
--model_path ./pretrained_models/nq_reader_base
--eval_data ./nq_train.json
--per_gpu_batch_size 1
--n_context 100
--name likelihood_debug_4
--checkpoint_dir checkpoint
--write_crossattention_scores \

Then, with the above generated cross-attention score json file as the training file, I loaded provided retriever checkpoint for NQ and continued the training on the Natural Question training dataset using the provided code below. The performance continues to decrease as I finetune this longer. If possible, could you please let me know your thoughts on this? Thanks very much!

python train_retriever.py
--lr 1e-4
--optim adamw
--scheduler linear
--train_data nqtrain_data_wtih_crossattention.json
--eval_data nqdev_data.json
--n_context 100
--total_steps 20000
--scheduler_steps 30000 \

[xionghao132]

@szhang42 Hi,do you solve this problem for the retriever training?