All instructions was written for Christophari.
For contest: you can obtain checkpoints from aws s3.
| Section | Description |
|---|---|
| ruGPT3Large fintune on essays | Examples of finetuning ruGPT3Large model for generating school essays. |
| ruGPT2Large fintune on essays | Examples of finetuning ruGPT2Large model for generating school essays. |
Finetune ruGPT3Large for school essays generation.
We prepare data with the following format:
{"text": "Тема: С какой целью В.А. Жуковский вносит русские фольклорные мотивы в традиционный балладный сюжет? (по балладе «Светлана»)\nСочинение: ..."}
For run finetuning download ruGPT3Large checkpoint and unpack to /home/jovyan/ruGPT3Large:
tar -zxvf ruGPT3Large.tar.gz
Download data to /home/jovyan/data. Data you can obtain here: train and valid.
Run script for pretrain: bash ./examples/pretrain_ruGPT3Large_essay.sh.
We obtain around 8 perplexity on valid set. Sample of generation you can see here
You can download pretrained checkpoint here.
Finetune ruGPT2Large for school essays generation.
We prepare data with the following format (raw text):
<s>Тема: С какой целью В.А. Жуковский вносит русские фольклорные мотивы в традиционный балладный сюжет? (по балладе «Светлана»)\nСочинение: ...</s>
<s>Тема: ...
For run finetuning download ruGPT2Large checkpoint and unpack to /home/jovyan/gpt2_large_bbpe_v50:
tar -zxvf gpt2_large_bbpe_v50.tar.gz
Download data to /home/jovyan/data. Data you can obtain here: train and valid.
Run script for pretrain: bash ./examples/pretrain_ruGPT2Large_essay.sh.
We obtain around 3 perplexity on valid set. Sample of generation you can see here
You can download pretrained checkpoint here.