How to train different model (InternLM-Math) on ReProver dataset other than google/byt5-small?

[realharryhero]

Hello,

Training the model on google/byt5-small works! There are other models which have been fine-tuned on other math-related subjects previously like InternLM-Math, so I'd like to get that model trained as a tactic generator (without retrieval), via the command python generation/main.py fit --config generation/confs/cli_lean4_novel_premises.yaml --trainer.logger.name train_generator_novel_premises --trainer.logger.save_dir logs/train_generator_novel_premises.

In generation/confs/cli_lean4_novel_premises.yaml, I changed the line with model_name to model_name=internlm/internlm2-math-plus-1_8b. When running the model, I obtain an error; the log is below:

Log

(ReProver) mfan@localhost:~/ReProver$ python generation/main.py fit --config generation/confs/cli_lean4_novel_premises.yaml --trainer.logger.name train_generator_novel_premises --trainer.logger.save_dir logs/train_generator_novel_premisesa2    # LeanDojo 
Benchmark 4, `novel_premises` split
[2024-08-01 20:38:08,589] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
2024-08-01 20:38:09.158 | INFO     | __main__:main:19 - PID: 25271
Seed set to 3407
The repository for internlm/internlm2-math-plus-1_8b contains custom code which must be executed to correctly load the model. You can inspect the repository content at https://hf.co/internlm/internlm2-math-plus-1_8b.
You can avoid this prompt in future by passing the argument `trust_remote_code=True`.

Do you wish to run the custom code? [y/N] y
You are using a model of type internlm2 to instantiate a model of type t5. This is not supported for all configurations of models and can yield errors.
Some weights of T5ForConditionalGeneration were not initialized from the model checkpoint at internlm/internlm2-math-plus-1_8b and are newly initialized: ['decoder.block.0.layer.0.SelfAttention.k.weight', 'decoder.block.0.layer.0.SelfAttention.o.weight', 'decoder.block.0.layer.0.SelfAttention.q.weight', 'decoder.block.0.layer.0.SelfAttention.relative_attention_bias.weight', 'decoder.block.0.layer.0.SelfAttention.v.weight', 'decoder.block.0.layer.0.layer_norm.weight', 'decoder.block.0.layer.1.EncDecAttention.k.weight', 'decoder.block.0.layer.1.EncDecAttention.o.weight', 'decoder.block.0.layer.1.EncDecAttention.q.weight', 'decoder.block.0.layer.1.EncDecAttention.v.weight', 'decoder.block.0.layer.1.layer_norm.weight', 'decoder.block.0.layer.2.DenseReluDense.wi.weight', 'decoder.block.0.layer.2.DenseReluDense.wo.weight', 'decoder.block.0.layer.2.layer_norm.weight', 'decoder.block.1.layer.0.SelfAttention.k.weight', 'decoder.block.1.layer.0.SelfAttention.o.weight', 'decoder.block.1.layer.0.SelfAttention.q.weight', 'decoder.block.1.layer.0.SelfAttention.v.weight', 'decoder.block.1.layer.0.layer_norm.weight', 'decoder.block.1.layer.1.EncDecAttention.k.weight', 'decoder.block.1.layer.1.EncDecAttention.o.weight', 'decoder.block.1.layer.1.EncDecAttention.q.weight', 'decoder.block.1.layer.1.EncDecAttention.v.weight', 'decoder.block.1.layer.1.layer_norm.weight', 'decoder.block.1.layer.2.DenseReluDense.wi.weight', 'decoder.block.1.layer.2.DenseReluDense.wo.weight', 'decoder.block.1.layer.2.layer_norm.weight', 'decoder.block.2.layer.0.SelfAttention.k.weight', 'decoder.block.2.layer.0.SelfAttention.o.weight', 'decoder.block.2.layer.0.SelfAttention.q.weight', 'decoder.block.2.layer.0.SelfAttention.v.weight', 'decoder.block.2.layer.0.layer_norm.weight', 'decoder.block.2.layer.1.EncDecAttention.k.weight', 'decoder.block.2.layer.1.EncDecAttention.o.weight', 'decoder.block.2.layer.1.EncDecAttention.q.weight', 'decoder.block.2.layer.1.EncDecAttention.v.weight', 'decoder.block.2.layer.1.layer_norm.weight', 'decoder.block.2.layer.2.DenseReluDense.wi.weight', 'decoder.block.2.layer.2.DenseReluDense.wo.weight', 'decoder.block.2.layer.2.layer_norm.weight', 'decoder.block.3.layer.0.SelfAttention.k.weight', 'decoder.block.3.layer.0.SelfAttention.o.weight', 'decoder.block.3.layer.0.SelfAttention.q.weight', 'decoder.block.3.layer.0.SelfAttention.v.weight', 'decoder.block.3.layer.0.layer_norm.weight', 'decoder.block.3.layer.1.EncDecAttention.k.weight', 'decoder.block.3.layer.1.EncDecAttention.o.weight', 'decoder.block.3.layer.1.EncDecAttention.q.weight', 'decoder.block.3.layer.1.EncDecAttention.v.weight', 'decoder.block.3.layer.1.layer_norm.weight', 'decoder.block.3.layer.2.DenseReluDense.wi.weight', 'decoder.block.3.layer.2.DenseReluDense.wo.weight', 'decoder.block.3.layer.2.layer_norm.weight', 'decoder.block.4.layer.0.SelfAttention.k.weight', 'decoder.block.4.layer.0.SelfAttention.o.weight', 'decoder.block.4.layer.0.SelfAttention.q.weight', 'decoder.block.4.layer.0.SelfAttention.v.weight', 'decoder.block.4.layer.0.layer_norm.weight', 'decoder.block.4.layer.1.EncDecAttention.k.weight', 'decoder.block.4.layer.1.EncDecAttention.o.weight', 'decoder.block.4.layer.1.EncDecAttention.q.weight', 'decoder.block.4.layer.1.EncDecAttention.v.weight', 'decoder.block.4.layer.1.layer_norm.weight', 'decoder.block.4.layer.2.DenseReluDense.wi.weight', 'decoder.block.4.layer.2.DenseReluDense.wo.weight', 'decoder.block.4.layer.2.layer_norm.weight', 'decoder.block.5.layer.0.SelfAttention.k.weight', 'decoder.block.5.layer.0.SelfAttention.o.weight', 'decoder.block.5.layer.0.SelfAttention.q.weight', 'decoder.block.5.layer.0.SelfAttention.v.weight', 'decoder.block.5.layer.0.layer_norm.weight', 'decoder.block.5.layer.1.EncDecAttention.k.weight', 'decoder.block.5.layer.1.EncDecAttention.o.weight', 'decoder.block.5.layer.1.EncDecAttention.q.weight', 'decoder.block.5.layer.1.EncDecAttention.v.weight', 'decoder.block.5.layer.1.layer_norm.weight', 'decoder.block.5.layer.2.DenseReluDense.wi.weight', 'decoder.block.5.layer.2.DenseReluDense.wo.weight', 'decoder.block.5.layer.2.layer_norm.weight', 'decoder.embed_tokens.weight', 'decoder.final_layer_norm.weight', 'encoder.block.0.layer.0.SelfAttention.k.weight', 'encoder.block.0.layer.0.SelfAttention.o.weight', 'encoder.block.0.layer.0.SelfAttention.q.weight', 'encoder.block.0.layer.0.SelfAttention.relative_attention_bias.weight', 'encoder.block.0.layer.0.SelfAttention.v.weight', 'encoder.block.0.layer.0.layer_norm.weight', 'encoder.block.0.layer.1.DenseReluDense.wi.weight', 'encoder.block.0.layer.1.DenseReluDense.wo.weight', 'encoder.block.0.layer.1.layer_norm.weight', 'encoder.block.1.layer.0.SelfAttention.k.weight', 'encoder.block.1.layer.0.SelfAttention.o.weight', 'encoder.block.1.layer.0.SelfAttention.q.weight', 'encoder.block.1.layer.0.SelfAttention.v.weight', 'encoder.block.1.layer.0.layer_norm.weight', 'encoder.block.1.layer.1.DenseReluDense.wi.weight', 'encoder.block.1.layer.1.DenseReluDense.wo.weight', 'encoder.block.1.layer.1.layer_norm.weight', 'encoder.block.10.layer.0.SelfAttention.k.weight', 'encoder.block.10.layer.0.SelfAttention.o.weight', 'encoder.block.10.layer.0.SelfAttention.q.weight', 'encoder.block.10.layer.0.SelfAttention.v.weight', 'encoder.block.10.layer.0.layer_norm.weight', 'encoder.block.10.layer.1.DenseReluDense.wi.weight', 'encoder.block.10.layer.1.DenseReluDense.wo.weight', 'encoder.block.10.layer.1.layer_norm.weight', 'encoder.block.11.layer.0.SelfAttention.k.weight', 'encoder.block.11.layer.0.SelfAttention.o.weight', 'encoder.block.11.layer.0.SelfAttention.q.weight', 'encoder.block.11.layer.0.SelfAttention.v.weight', 'encoder.block.11.layer.0.layer_norm.weight', 'encoder.block.11.layer.1.DenseReluDense.wi.weight', 'encoder.block.11.layer.1.DenseReluDense.wo.weight', 'encoder.block.11.layer.1.layer_norm.weight', 'encoder.block.12.layer.0.SelfAttention.k.weight', 'encoder.block.12.layer.0.SelfAttention.o.weight', 'encoder.block.12.layer.0.SelfAttention.q.weight', 'encoder.block.12.layer.0.SelfAttention.v.weight', 'encoder.block.12.layer.0.layer_norm.weight', 'encoder.block.12.layer.1.DenseReluDense.wi.weight', 'encoder.block.12.layer.1.DenseReluDense.wo.weight', 'encoder.block.12.layer.1.layer_norm.weight', 'encoder.block.13.layer.0.SelfAttention.k.weight', 'encoder.block.13.layer.0.SelfAttention.o.weight', 'encoder.block.13.layer.0.SelfAttention.q.weight', 'encoder.block.13.layer.0.SelfAttention.v.weight', 'encoder.block.13.layer.0.layer_norm.weight', 'encoder.block.13.layer.1.DenseReluDense.wi.weight', 'encoder.block.13.layer.1.DenseReluDense.wo.weight', 'encoder.block.13.layer.1.layer_norm.weight', 'encoder.block.14.layer.0.SelfAttention.k.weight', 'encoder.block.14.layer.0.SelfAttention.o.weight', 'encoder.block.14.layer.0.SelfAttention.q.weight', 'encoder.block.14.layer.0.SelfAttention.v.weight', 'encoder.block.14.layer.0.layer_norm.weight', 'encoder.block.14.layer.1.DenseReluDense.wi.weight', 'encoder.block.14.layer.1.DenseReluDense.wo.weight', 'encoder.block.14.layer.1.layer_norm.weight', 'encoder.block.15.layer.0.SelfAttention.k.weight', 'encoder.block.15.layer.0.SelfAttention.o.weight', 'encoder.block.15.layer.0.SelfAttention.q.weight', 'encoder.block.15.layer.0.SelfAttention.v.weight', 'encoder.block.15.layer.0.layer_norm.weight', 'encoder.block.15.layer.1.DenseReluDense.wi.weight', 'encoder.block.15.layer.1.DenseReluDense.wo.weight', 'encoder.block.15.layer.1.layer_norm.weight', 'encoder.block.16.layer.0.SelfAttention.k.weight', 'encoder.block.16.layer.0.SelfAttention.o.weight', 'encoder.block.16.layer.0.SelfAttention.q.weight', 'encoder.block.16.layer.0.SelfAttention.v.weight', 'encoder.block.16.layer.0.layer_norm.weight', 'encoder.block.16.layer.1.DenseReluDense.wi.weight', 'encoder.block.16.layer.1.DenseReluDense.wo.weight', 'encoder.block.16.layer.1.layer_norm.weight', 'encoder.block.17.layer.0.SelfAttention.k.weight', 'encoder.block.17.layer.0.SelfAttention.o.weight', 'encoder.block.17.layer.0.SelfAttention.q.weight', 'encoder.block.17.layer.0.SelfAttention.v.weight', 'encoder.block.17.layer.0.layer_norm.weight', 'encoder.block.17.layer.1.DenseReluDense.wi.weight', 'encoder.block.17.layer.1.DenseReluDense.wo.weight', 'encoder.block.17.layer.1.layer_norm.weight', 'encoder.block.18.layer.0.SelfAttention.k.weight', 'encoder.block.18.layer.0.SelfAttention.o.weight', 'encoder.block.18.layer.0.SelfAttention.q.weight', 'encoder.block.18.layer.0.SelfAttention.v.weight', 'encoder.block.18.layer.0.layer_norm.weight', 'encoder.block.18.layer.1.DenseReluDense.wi.weight', 'encoder.block.18.layer.1.DenseReluDense.wo.weight', 'encoder.block.18.layer.1.layer_norm.weight', 'encoder.block.19.layer.0.SelfAttention.k.weight', 'encoder.block.19.layer.0.SelfAttention.o.weight', 'encoder.block.19.layer.0.SelfAttention.q.weight', 'encoder.block.19.layer.0.SelfAttention.v.weight', 'encoder.block.19.layer.0.layer_norm.weight', 'encoder.block.19.layer.1.DenseReluDense.wi.weight', 'encoder.block.19.layer.1.DenseReluDense.wo.weight', 'encoder.block.19.layer.1.layer_norm.weight', 'encoder.block.2.layer.0.SelfAttention.k.weight', 'encoder.block.2.layer.0.SelfAttention.o.weight', 'encoder.block.2.layer.0.SelfAttention.q.weight', 'encoder.block.2.layer.0.SelfAttention.v.weight', 'encoder.block.2.layer.0.layer_norm.weight', 'encoder.block.2.layer.1.DenseReluDense.wi.weight', 'encoder.block.2.layer.1.DenseReluDense.wo.weight', 'encoder.block.2.layer.1.layer_norm.weight', 'encoder.block.20.layer.0.SelfAttention.k.weight', 'encoder.block.20.layer.0.SelfAttention.o.weight', 'encoder.block.20.layer.0.SelfAttention.q.weight', 'encoder.block.20.layer.0.SelfAttention.v.weight', 'encoder.block.20.layer.0.layer_norm.weight', 'encoder.block.20.layer.1.DenseReluDense.wi.weight', 'encoder.block.20.layer.1.DenseReluDense.wo.weight', 'encoder.block.20.layer.1.layer_norm.weight', 'encoder.block.21.layer.0.SelfAttention.k.weight', 'encoder.block.21.layer.0.SelfAttention.o.weight', 'encoder.block.21.layer.0.SelfAttention.q.weight', 'encoder.block.21.layer.0.SelfAttention.v.weight', 'encoder.block.21.layer.0.layer_norm.weight', 'encoder.block.21.layer.1.DenseReluDense.wi.weight', 'encoder.block.21.layer.1.DenseReluDense.wo.weight', 'encoder.block.21.layer.1.layer_norm.weight', 'encoder.block.22.layer.0.SelfAttention.k.weight', 'encoder.block.22.layer.0.SelfAttention.o.weight', 'encoder.block.22.layer.0.SelfAttention.q.weight', 'encoder.block.22.layer.0.SelfAttention.v.weight', 'encoder.block.22.layer.0.layer_norm.weight', 'encoder.block.22.layer.1.DenseReluDense.wi.weight', 'encoder.block.22.layer.1.DenseReluDense.wo.weight', 'encoder.block.22.layer.1.layer_norm.weight', 'encoder.block.23.layer.0.SelfAttention.k.weight', 'encoder.block.23.layer.0.SelfAttention.o.weight', 'encoder.block.23.layer.0.SelfAttention.q.weight', 'encoder.block.23.layer.0.SelfAttention.v.weight', 'encoder.block.23.layer.0.layer_norm.weight', 'encoder.block.23.layer.1.DenseReluDense.wi.weight', 'encoder.block.23.layer.1.DenseReluDense.wo.weight', 'encoder.block.23.layer.1.layer_norm.weight', 'encoder.block.3.layer.0.SelfAttention.k.weight', 'encoder.block.3.layer.0.SelfAttention.o.weight', 'encoder.block.3.layer.0.SelfAttention.q.weight', 'encoder.block.3.layer.0.SelfAttention.v.weight', 'encoder.block.3.layer.0.layer_norm.weight', 'encoder.block.3.layer.1.DenseReluDense.wi.weight', 'encoder.block.3.layer.1.DenseReluDense.wo.weight', 'encoder.block.3.layer.1.layer_norm.weight', 'encoder.block.4.layer.0.SelfAttention.k.weight', 'encoder.block.4.layer.0.SelfAttention.o.weight', 'encoder.block.4.layer.0.SelfAttention.q.weight', 'encoder.block.4.layer.0.SelfAttention.v.weight', 'encoder.block.4.layer.0.layer_norm.weight', 'encoder.block.4.layer.1.DenseReluDense.wi.weight', 'encoder.block.4.layer.1.DenseReluDense.wo.weight', 'encoder.block.4.layer.1.layer_norm.weight', 'encoder.block.5.layer.0.SelfAttention.k.weight', 'encoder.block.5.layer.0.SelfAttention.o.weight', 'encoder.block.5.layer.0.SelfAttention.q.weight', 'encoder.block.5.layer.0.SelfAttention.v.weight', 'encoder.block.5.layer.0.layer_norm.weight', 'encoder.block.5.layer.1.DenseReluDense.wi.weight', 'encoder.block.5.layer.1.DenseReluDense.wo.weight', 'encoder.block.5.layer.1.layer_norm.weight', 'encoder.block.6.layer.0.SelfAttention.k.weight', 'encoder.block.6.layer.0.SelfAttention.o.weight', 'encoder.block.6.layer.0.SelfAttention.q.weight', 'encoder.block.6.layer.0.SelfAttention.v.weight', 'encoder.block.6.layer.0.layer_norm.weight', 'encoder.block.6.layer.1.DenseReluDense.wi.weight', 'encoder.block.6.layer.1.DenseReluDense.wo.weight', 'encoder.block.6.layer.1.layer_norm.weight', 'encoder.block.7.layer.0.SelfAttention.k.weight', 'encoder.block.7.layer.0.SelfAttention.o.weight', 'encoder.block.7.layer.0.SelfAttention.q.weight', 'encoder.block.7.layer.0.SelfAttention.v.weight', 'encoder.block.7.layer.0.layer_norm.weight', 'encoder.block.7.layer.1.DenseReluDense.wi.weight', 'encoder.block.7.layer.1.DenseReluDense.wo.weight', 'encoder.block.7.layer.1.layer_norm.weight', 'encoder.block.8.layer.0.SelfAttention.k.weight', 'encoder.block.8.layer.0.SelfAttention.o.weight', 'encoder.block.8.layer.0.SelfAttention.q.weight', 'encoder.block.8.layer.0.SelfAttention.v.weight', 'encoder.block.8.layer.0.layer_norm.weight', 'encoder.block.8.layer.1.DenseReluDense.wi.weight', 'encoder.block.8.layer.1.DenseReluDense.wo.weight', 'encoder.block.8.layer.1.layer_norm.weight', 'encoder.block.9.layer.0.SelfAttention.k.weight', 'encoder.block.9.layer.0.SelfAttention.o.weight', 'encoder.block.9.layer.0.SelfAttention.q.weight', 'encoder.block.9.layer.0.SelfAttention.v.weight', 'encoder.block.9.layer.0.layer_norm.weight', 'encoder.block.9.layer.1.DenseReluDense.wi.weight', 'encoder.block.9.layer.1.DenseReluDense.wo.weight', 'encoder.block.9.layer.1.layer_norm.weight', 'encoder.embed_tokens.weight', 'encoder.final_layer_norm.weight', 'lm_head.weight', 'shared.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
2024-08-01 20:38:15.739 | INFO     | common:__init__:200 - Building the corpus from data/leandojo_benchmark_4/corpus.jsonl
The repository for internlm/internlm2-math-plus-1_8b contains custom code which must be executed to correctly load the model. You can inspect the repository content at https://hf.co/internlm/internlm2-math-plus-1_8b.
You can avoid this prompt in future by passing the argument `trust_remote_code=True`.

Do you wish to run the custom code? [y/N] y
2024-08-01 20:38:31.299 | INFO     | generation.datamodule:__init__:147 - Without retrieval data
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
HPU available: False, using: 0 HPUs
[rank: 0] Seed set to 3407
initializing deepspeed distributed: GLOBAL_RANK: 0, MEMBER: 1/1
wandb: Currently logged in as: greathero (greathero1user). Use `wandb login --relogin` to force relogin
wandb: WARNING Path logs/train_generator_novel_premisesa2/wandb/ wasn't writable, using system temp directory.
wandb: WARNING Path logs/train_generator_novel_premisesa2/wandb/ wasn't writable, using system temp directory
wandb: Tracking run with wandb version 0.17.5
wandb: Run data is saved locally in /tmp/wandb/run-20240801_203832-9yqjm8lw
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run train_generator_novel_premises
wandb: ⭐️ View project at https://wandb.ai/greathero1user/lightning_logs
wandb: 🚀 View run at https://wandb.ai/greathero1user/lightning_logs/runs/9yqjm8lw
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 118517/118517 [00:00<00:00, 624909.74it/s]
2024-08-01 20:38:46.073 | INFO     | generation.datamodule:_load_data:60 - 246714 examples loaded
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2000/2000 [00:00<00:00, 494932.33it/s]
2024-08-01 20:38:46.138 | INFO     | generation.datamodule:_load_data:60 - 6240 examples loaded
Enabling DeepSpeed BF16. Model parameters and inputs will be cast to `bfloat16`.
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
2024-08-01 20:38:46.206 | INFO     | common:get_optimizers:392 - Optimizing with FusedAdam
Using /home/mfan/.cache/torch_extensions/py311_cu121 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /home/mfan/.cache/torch_extensions/py311_cu121/fused_adam/build.ninja...
/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/utils/cpp_extension.py:1967: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
  warnings.warn(
Building extension module fused_adam...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module fused_adam...
Time to load fused_adam op: 0.043921470642089844 seconds

  | Name         | Type                       | Params | Mode 
--------------------------------------------------------------------
0 | generator    | T5ForConditionalGeneration | 932 M  | eval 
1 | top1_acc_val | TopkAccuracy               | 0      | train
--------------------------------------------------------------------
932 M     Trainable params
0         Non-trainable params
932 M     Total params
3,731.395 Total estimated model params size (MB)
wandb: logging graph, to disable use `wandb.watch(log_graph=False)`
2024-08-01 20:38:52.919 | INFO     | generation.model:on_fit_start:162 - Logging to logs/train_generator_novel_premisesa2
Epoch 0:   0%|                                                                                                                                                                                                                      | 0/246714 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/home/mfan/ReProver/generation/main.py", line 25, in <module>
    main()
  File "/home/mfan/ReProver/generation/main.py", line 20, in main
    cli = CLI(RetrievalAugmentedGenerator, GeneratorDataModule)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/cli.py", line 394, in __init__
    self._run_subcommand(self.subcommand)
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/cli.py", line 701, in _run_subcommand
    fn(**fn_kwargs)
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py", line 543, in fit
    call._call_and_handle_interrupt(
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/trainer/call.py", line 43, in _call_and_handle_interrupt
    return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 105, in launch
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py", line 579, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py", line 986, in _run
    results = self._run_stage()
              ^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py", line 1030, in _run_stage
    self.fit_loop.run()
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/loops/fit_loop.py", line 205, in run
    self.advance()
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/loops/fit_loop.py", line 363, in advance
    self.epoch_loop.run(self._data_fetcher)
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/loops/training_epoch_loop.py", line 140, in run
    self.advance(data_fetcher)
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/loops/training_epoch_loop.py", line 250, in advance
    batch_output = self.automatic_optimization.run(trainer.optimizers[0], batch_idx, kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 190, in run
    self._optimizer_step(batch_idx, closure)
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 268, in _optimizer_step
    call._call_lightning_module_hook(
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/trainer/call.py", line 159, in _call_lightning_module_hook
    output = fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/core/module.py", line 1308, in optimizer_step
    optimizer.step(closure=optimizer_closure)
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/core/optimizer.py", line 153, in step
    step_output = self._strategy.optimizer_step(self._optimizer, closure, **kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/strategies/ddp.py", line 270, in optimizer_step
    optimizer_output = super().optimizer_step(optimizer, closure, model, **kwargs)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/strategies/strategy.py", line 238, in optimizer_step
    return self.precision_plugin.optimizer_step(optimizer, model=model, closure=closure, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/plugins/precision/deepspeed.py", line 129, in optimizer_step
    closure_result = closure()
                     ^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 144, in __call__
    self._result = self.closure(*args, **kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 129, in closure
    step_output = self._step_fn()
                  ^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 317, in _training_step
    training_step_output = call._call_strategy_hook(trainer, "training_step", *kwargs.values())
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/trainer/call.py", line 311, in _call_strategy_hook
    output = fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/strategies/strategy.py", line 389, in training_step
    return self._forward_redirection(self.model, self.lightning_module, "training_step", *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/strategies/strategy.py", line 640, in __call__
    wrapper_output = wrapper_module(*args, **kwargs)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/deepspeed/utils/nvtx.py", line 18, in wrapped_fn
    ret_val = func(*args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1846, in forward
    loss = self.module(*inputs, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/strategies/strategy.py", line 633, in wrapped_forward
    out = method(*_args, **_kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/ReProver/generation/model.py", line 118, in training_step
    loss = self(
           ^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/ReProver/generation/model.py", line 107, in forward
    return self.generator(
           ^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
    decoder_input_ids = self._shift_right(labels)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/transformers/models/t5/modeling_t5.py", line 876, in _shift_right
    raise ValueError(
ValueError: self.model.config.decoder_start_token_id has to be defined. In T5 it is usually set to the pad_token_id. See T5 docs for more information.
[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/mfan/ReProver/generation/main.py", line 25, in <module>
[rank0]:     main()
[rank0]:   File "/home/mfan/ReProver/generation/main.py", line 20, in main
[rank0]:     cli = CLI(RetrievalAugmentedGenerator, GeneratorDataModule)
[rank0]:           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/cli.py", line 394, in __init__
[rank0]:     self._run_subcommand(self.subcommand)
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/cli.py", line 701, in _run_subcommand
[rank0]:     fn(**fn_kwargs)
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py", line 543, in fit
[rank0]:     call._call_and_handle_interrupt(
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/trainer/call.py", line 43, in _call_and_handle_interrupt
[rank0]:     return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 105, in launch
[rank0]:     return function(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py", line 579, in _fit_impl
[rank0]:     self._run(model, ckpt_path=ckpt_path)
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py", line 986, in _run
[rank0]:     results = self._run_stage()
[rank0]:               ^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py", line 1030, in _run_stage
[rank0]:     self.fit_loop.run()
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/loops/fit_loop.py", line 205, in run
[rank0]:     self.advance()
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/loops/fit_loop.py", line 363, in advance
[rank0]:     self.epoch_loop.run(self._data_fetcher)
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/loops/training_epoch_loop.py", line 140, in run
[rank0]:     self.advance(data_fetcher)
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/loops/training_epoch_loop.py", line 250, in advance
[rank0]:     batch_output = self.automatic_optimization.run(trainer.optimizers[0], batch_idx, kwargs)
[rank0]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 190, in run
[rank0]:     self._optimizer_step(batch_idx, closure)
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 268, in _optimizer_step
[rank0]:     call._call_lightning_module_hook(
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/trainer/call.py", line 159, in _call_lightning_module_hook
[rank0]:     output = fn(*args, **kwargs)
[rank0]:              ^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/core/module.py", line 1308, in optimizer_step
[rank0]:     optimizer.step(closure=optimizer_closure)
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/core/optimizer.py", line 153, in step
[rank0]:     step_output = self._strategy.optimizer_step(self._optimizer, closure, **kwargs)
[rank0]:                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/strategies/ddp.py", line 270, in optimizer_step
[rank0]:     optimizer_output = super().optimizer_step(optimizer, closure, model, **kwargs)
[rank0]:                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/strategies/strategy.py", line 238, in optimizer_step
[rank0]:     return self.precision_plugin.optimizer_step(optimizer, model=model, closure=closure, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/plugins/precision/deepspeed.py", line 129, in optimizer_step
[rank0]:     closure_result = closure()
[rank0]:                      ^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 144, in __call__
[rank0]:     self._result = self.closure(*args, **kwargs)
[rank0]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
[rank0]:     return func(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 129, in closure
[rank0]:     step_output = self._step_fn()
[rank0]:                   ^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 317, in _training_step
[rank0]:     training_step_output = call._call_strategy_hook(trainer, "training_step", *kwargs.values())
[rank0]:                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/trainer/call.py", line 311, in _call_strategy_hook
[rank0]:     output = fn(*args, **kwargs)
[rank0]:              ^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/strategies/strategy.py", line 389, in training_step
[rank0]:     return self._forward_redirection(self.model, self.lightning_module, "training_step", *args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/strategies/strategy.py", line 640, in __call__
[rank0]:     wrapper_output = wrapper_module(*args, **kwargs)
[rank0]:                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/deepspeed/utils/nvtx.py", line 18, in wrapped_fn
[rank0]:     ret_val = func(*args, **kwargs)
[rank0]:               ^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1846, in forward
[rank0]:     loss = self.module(*inputs, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/strategies/strategy.py", line 633, in wrapped_forward
[rank0]:     out = method(*_args, **_kwargs)
[rank0]:           ^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/ReProver/generation/model.py", line 118, in training_step
[rank0]:     loss = self(
[rank0]:            ^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/ReProver/generation/model.py", line 107, in forward
[rank0]:     return self.generator(
[rank0]:            ^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
[rank0]:     decoder_input_ids = self._shift_right(labels)
[rank0]:                         ^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/transformers/models/t5/modeling_t5.py", line 876, in _shift_right
[rank0]:     raise ValueError(
[rank0]: ValueError: self.model.config.decoder_start_token_id has to be defined. In T5 it is usually set to the pad_token_id. See T5 docs for more information.
wandb: 🚀 View run train_generator_novel_premises at: https://wandb.ai/greathero1user/lightning_logs/runs/9yqjm8lw
wandb: ⭐️ View project at: https://wandb.ai/greathero1user/lightning_logs
wandb: Synced 6 W&B file(s), 0 media file(s), 2 artifact file(s) and 0 other file(s)
wandb: Find logs at: /tmp/wandb/run-20240801_203832-9yqjm8lw/logs
wandb: WARNING The new W&B backend becomes opt-out in version 0.18.0; try it out with `wandb.require("core")`! See https://wandb.me/wandb-core for more information.

This seems to be as in model.py, the model is defined in line 87 by self.generator = T5ForConditionalGeneration.from_pretrained(model_name), which uses a T5ForConditionalGeneration model. The InternLM model is an AutoModelForCausalLM, so I imported AutoModelForCausalLM from transformers and set self.generator = AutoModelForCausalLM.from_pretrained(model_name).

When I ran it again, though, I recieved the following error:

Log

(ReProver) mfan@localhost:~/ReProver$ python generation/main.py fit --config generation/confs/cli_lean4_novel_premises.yaml --trainer.logger.name train_generator_novel_premises --trainer.logger.save_dir logs/train_generator_novel_premisesa1    # LeanDojo 
Benchmark 4, `novel_premises` split
[2024-08-01 20:33:35,891] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
2024-08-01 20:33:36.453 | INFO     | __main__:main:19 - PID: 24580
Seed set to 3407
The repository for internlm/internlm2-math-plus-1_8b contains custom code which must be executed to correctly load the model. You can inspect the repository content at https://hf.co/internlm/internlm2-math-plus-1_8b.
You can avoid this prompt in future by passing the argument `trust_remote_code=True`.

Do you wish to run the custom code? [y/N] y
The repository for internlm/internlm2-math-plus-1_8b contains custom code which must be executed to correctly load the model. You can inspect the repository content at https://hf.co/internlm/internlm2-math-plus-1_8b.
You can avoid this prompt in future by passing the argument `trust_remote_code=True`.

Do you wish to run the custom code? [y/N] y
The repository for internlm/internlm2-math-plus-1_8b contains custom code which must be executed to correctly load the model. You can inspect the repository content at https://hf.co/internlm/internlm2-math-plus-1_8b.
You can avoid this prompt in future by passing the argument `trust_remote_code=True`.

Do you wish to run the custom code? [y/N] y
2024-08-01 20:33:43.670 | INFO     | common:__init__:200 - Building the corpus from data/leandojo_benchmark_4/corpus.jsonl
The repository for internlm/internlm2-math-plus-1_8b contains custom code which must be executed to correctly load the model. You can inspect the repository content at https://hf.co/internlm/internlm2-math-plus-1_8b.
You can avoid this prompt in future by passing the argument `trust_remote_code=True`.

Do you wish to run the custom code? [y/N] y
2024-08-01 20:33:59.706 | INFO     | generation.datamodule:__init__:147 - Without retrieval data
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
HPU available: False, using: 0 HPUs
[rank: 0] Seed set to 3407
initializing deepspeed distributed: GLOBAL_RANK: 0, MEMBER: 1/1
wandb: Currently logged in as: greathero (greathero1user). Use `wandb login --relogin` to force relogin
wandb: WARNING Path logs/train_generator_novel_premisesa1/wandb/ wasn't writable, using system temp directory.
wandb: WARNING Path logs/train_generator_novel_premisesa1/wandb/ wasn't writable, using system temp directory
wandb: Tracking run with wandb version 0.17.5
wandb: Run data is saved locally in /tmp/wandb/run-20240801_203400-i9bmwlso
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run train_generator_novel_premises
wandb: ⭐️ View project at https://wandb.ai/greathero1user/lightning_logs
wandb: 🚀 View run at https://wandb.ai/greathero1user/lightning_logs/runs/i9bmwlso
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 118517/118517 [00:00<00:00, 619523.45it/s]
2024-08-01 20:34:14.703 | INFO     | generation.datamodule:_load_data:60 - 246714 examples loaded
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2000/2000 [00:00<00:00, 523339.45it/s]
2024-08-01 20:34:14.768 | INFO     | generation.datamodule:_load_data:60 - 6240 examples loaded
Enabling DeepSpeed BF16. Model parameters and inputs will be cast to `bfloat16`.
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
2024-08-01 20:34:14.837 | INFO     | common:get_optimizers:392 - Optimizing with FusedAdam
Using /home/mfan/.cache/torch_extensions/py311_cu121 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /home/mfan/.cache/torch_extensions/py311_cu121/fused_adam/build.ninja...
/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/utils/cpp_extension.py:1967: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
  warnings.warn(
Building extension module fused_adam...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/3] /usr/local/cuda/bin/nvcc --generate-dependencies-with-compile --dependency-output multi_tensor_adam.cuda.o.d -DTORCH_EXTENSION_NAME=fused_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/deepspeed/ops/csrc/includes -I/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/deepspeed/ops/csrc/adam -isystem /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/include -isystem /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/include/TH -isystem /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /home/mfan/miniconda3/envs/ReProver/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -O3 -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -lineinfo --use_fast_math -gencode=arch=compute_86,code=sm_86 -gencode=arch=compute_86,code=compute_86 -DBF16_AVAILABLE -U__CUDA_NO_BFLOAT16_OPERATORS__ -U__CUDA_NO_BFLOAT162_OPERATORS__ -U__CUDA_NO_BFLOAT16_CONVERSIONS__ -std=c++17 -c /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/deepspeed/ops/csrc/adam/multi_tensor_adam.cu -o multi_tensor_adam.cuda.o 
[2/3] c++ -MMD -MF fused_adam_frontend.o.d -DTORCH_EXTENSION_NAME=fused_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/deepspeed/ops/csrc/includes -I/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/deepspeed/ops/csrc/adam -isystem /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/include -isystem /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/include/TH -isystem /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /home/mfan/miniconda3/envs/ReProver/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -O3 -std=c++17 -g -Wno-reorder -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DBF16_AVAILABLE -c /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/deepspeed/ops/csrc/adam/fused_adam_frontend.cpp -o fused_adam_frontend.o 
[3/3] c++ fused_adam_frontend.o multi_tensor_adam.cuda.o -shared -L/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/lib -lc10 -lc10_cuda -ltorch_cpu -ltorch_cuda -ltorch -ltorch_python -L/usr/local/cuda/lib64 -lcudart -o fused_adam.so
Loading extension module fused_adam...
Time to load fused_adam op: 16.982542037963867 seconds

  | Name         | Type                 | Params | Mode 
--------------------------------------------------------------
0 | generator    | InternLM2ForCausalLM | 1.9 B  | eval 
1 | top1_acc_val | TopkAccuracy         | 0      | train
--------------------------------------------------------------
1.9 B     Trainable params
0         Non-trainable params
1.9 B     Total params
7,556.440 Total estimated model params size (MB)
wandb: logging graph, to disable use `wandb.watch(log_graph=False)`
2024-08-01 20:34:40.258 | INFO     | generation.model:on_fit_start:162 - Logging to logs/train_generator_novel_premisesa1
Epoch 0:   0%|                                                                                                                                                                                                                      | 0/246714 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/home/mfan/ReProver/generation/main.py", line 25, in <module>
    main()
  File "/home/mfan/ReProver/generation/main.py", line 20, in main
    cli = CLI(RetrievalAugmentedGenerator, GeneratorDataModule)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/cli.py", line 394, in __init__
    self._run_subcommand(self.subcommand)
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/cli.py", line 701, in _run_subcommand
    fn(**fn_kwargs)
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py", line 543, in fit
    call._call_and_handle_interrupt(
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/trainer/call.py", line 43, in _call_and_handle_interrupt
    return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 105, in launch
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py", line 579, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py", line 986, in _run
    results = self._run_stage()
              ^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py", line 1030, in _run_stage
    self.fit_loop.run()
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/loops/fit_loop.py", line 205, in run
    self.advance()
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/loops/fit_loop.py", line 363, in advance
    self.epoch_loop.run(self._data_fetcher)
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/loops/training_epoch_loop.py", line 140, in run
    self.advance(data_fetcher)
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/loops/training_epoch_loop.py", line 250, in advance
    batch_output = self.automatic_optimization.run(trainer.optimizers[0], batch_idx, kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 190, in run
    self._optimizer_step(batch_idx, closure)
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 268, in _optimizer_step
    call._call_lightning_module_hook(
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/trainer/call.py", line 159, in _call_lightning_module_hook
    output = fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/core/module.py", line 1308, in optimizer_step
    optimizer.step(closure=optimizer_closure)
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/core/optimizer.py", line 153, in step
    step_output = self._strategy.optimizer_step(self._optimizer, closure, **kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/strategies/ddp.py", line 270, in optimizer_step
    optimizer_output = super().optimizer_step(optimizer, closure, model, **kwargs)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/strategies/strategy.py", line 238, in optimizer_step
    return self.precision_plugin.optimizer_step(optimizer, model=model, closure=closure, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/plugins/precision/deepspeed.py", line 129, in optimizer_step
    closure_result = closure()
                     ^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 144, in __call__
    self._result = self.closure(*args, **kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 129, in closure
    step_output = self._step_fn()
                  ^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 317, in _training_step
    training_step_output = call._call_strategy_hook(trainer, "training_step", *kwargs.values())
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/trainer/call.py", line 311, in _call_strategy_hook
    output = fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/strategies/strategy.py", line 389, in training_step
    return self._forward_redirection(self.model, self.lightning_module, "training_step", *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/strategies/strategy.py", line 640, in __call__
    wrapper_output = wrapper_module(*args, **kwargs)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/deepspeed/utils/nvtx.py", line 18, in wrapped_fn
    ret_val = func(*args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1846, in forward
    loss = self.module(*inputs, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/strategies/strategy.py", line 633, in wrapped_forward
    out = method(*_args, **_kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/ReProver/generation/model.py", line 118, in training_step
    loss = self(
           ^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/ReProver/generation/model.py", line 107, in forward
    return self.generator(
           ^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/.cache/huggingface/modules/transformers_modules/internlm/internlm2-math-plus-1_8b/2131e4c8da9a520ceec7c48fe740722a835882ff/modeling_internlm2.py", line 1074, in forward
    loss = loss_fct(shift_logits, shift_labels)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/nn/modules/loss.py", line 1185, in forward
    return F.cross_entropy(input, target, weight=self.weight,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/nn/functional.py", line 3086, in cross_entropy
    return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: Expected input batch_size (79) to match target batch_size (22).
[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/mfan/ReProver/generation/main.py", line 25, in <module>
[rank0]:     main()
[rank0]:   File "/home/mfan/ReProver/generation/main.py", line 20, in main
[rank0]:     cli = CLI(RetrievalAugmentedGenerator, GeneratorDataModule)
[rank0]:           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/cli.py", line 394, in __init__
[rank0]:     self._run_subcommand(self.subcommand)
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/cli.py", line 701, in _run_subcommand
[rank0]:     fn(**fn_kwargs)
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py", line 543, in fit
[rank0]:     call._call_and_handle_interrupt(
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/trainer/call.py", line 43, in _call_and_handle_interrupt
[rank0]:     return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 105, in launch
[rank0]:     return function(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py", line 579, in _fit_impl
[rank0]:     self._run(model, ckpt_path=ckpt_path)
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py", line 986, in _run
[rank0]:     results = self._run_stage()
[rank0]:               ^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py", line 1030, in _run_stage
[rank0]:     self.fit_loop.run()
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/loops/fit_loop.py", line 205, in run
[rank0]:     self.advance()
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/loops/fit_loop.py", line 363, in advance
[rank0]:     self.epoch_loop.run(self._data_fetcher)
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/loops/training_epoch_loop.py", line 140, in run
[rank0]:     self.advance(data_fetcher)
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/loops/training_epoch_loop.py", line 250, in advance
[rank0]:     batch_output = self.automatic_optimization.run(trainer.optimizers[0], batch_idx, kwargs)
[rank0]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 190, in run
[rank0]:     self._optimizer_step(batch_idx, closure)
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 268, in _optimizer_step
[rank0]:     call._call_lightning_module_hook(
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/trainer/call.py", line 159, in _call_lightning_module_hook
[rank0]:     output = fn(*args, **kwargs)
[rank0]:              ^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/core/module.py", line 1308, in optimizer_step
[rank0]:     optimizer.step(closure=optimizer_closure)
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/core/optimizer.py", line 153, in step
[rank0]:     step_output = self._strategy.optimizer_step(self._optimizer, closure, **kwargs)
[rank0]:                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/strategies/ddp.py", line 270, in optimizer_step
[rank0]:     optimizer_output = super().optimizer_step(optimizer, closure, model, **kwargs)
[rank0]:                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/strategies/strategy.py", line 238, in optimizer_step
[rank0]:     return self.precision_plugin.optimizer_step(optimizer, model=model, closure=closure, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/plugins/precision/deepspeed.py", line 129, in optimizer_step
[rank0]:     closure_result = closure()
[rank0]:                      ^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 144, in __call__
[rank0]:     self._result = self.closure(*args, **kwargs)
[rank0]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
[rank0]:     return func(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 129, in closure
[rank0]:     step_output = self._step_fn()
[rank0]:                   ^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 317, in _training_step
[rank0]:     training_step_output = call._call_strategy_hook(trainer, "training_step", *kwargs.values())
[rank0]:                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/trainer/call.py", line 311, in _call_strategy_hook
[rank0]:     output = fn(*args, **kwargs)
[rank0]:              ^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/strategies/strategy.py", line 389, in training_step
[rank0]:     return self._forward_redirection(self.model, self.lightning_module, "training_step", *args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/strategies/strategy.py", line 640, in __call__
[rank0]:     wrapper_output = wrapper_module(*args, **kwargs)
[rank0]:                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/deepspeed/utils/nvtx.py", line 18, in wrapped_fn
[rank0]:     ret_val = func(*args, **kwargs)
[rank0]:               ^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1846, in forward
[rank0]:     loss = self.module(*inputs, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/strategies/strategy.py", line 633, in wrapped_forward
[rank0]:     out = method(*_args, **_kwargs)
[rank0]:           ^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/ReProver/generation/model.py", line 118, in training_step
[rank0]:     loss = self(
[rank0]:            ^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/ReProver/generation/model.py", line 107, in forward
[rank0]:     return self.generator(
[rank0]:            ^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/.cache/huggingface/modules/transformers_modules/internlm/internlm2-math-plus-1_8b/2131e4c8da9a520ceec7c48fe740722a835882ff/modeling_internlm2.py", line 1074, in forward
[rank0]:     loss = loss_fct(shift_logits, shift_labels)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/nn/modules/loss.py", line 1185, in forward
[rank0]:     return F.cross_entropy(input, target, weight=self.weight,
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/nn/functional.py", line 3086, in cross_entropy
[rank0]:     return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: ValueError: Expected input batch_size (79) to match target batch_size (22).
wandb: 🚀 View run train_generator_novel_premises at: https://wandb.ai/greathero1user/lightning_logs/runs/i9bmwlso
wandb: ⭐️ View project at: https://wandb.ai/greathero1user/lightning_logs
wandb: Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
wandb: Find logs at: /tmp/wandb/run-20240801_203400-i9bmwlso/logs
wandb: WARNING The new W&B backend becomes opt-out in version 0.18.0; try it out with `wandb.require("core")`! See https://wandb.me/wandb-core for more information.

Maybe this seems to be as there's some training code which uses the property that byt5-small is a T5 model. Thank you!

[yangky11]

The training script in this codebase is tailored for encoder-decoder Transformers like ByT5, whereas more recent models are typically decoder-only.

I would preprocess LeanDojo's data using something like preprocess.py and then train the model using off-the-shelf tools like LLaMA-Factory or torchtune