All instructions was written for Christophari.
For contest: you can obtain checkpoints from aws s3.
| Section | Description |
|---|---|
| ruGPT3Large finetune on essays | Examples of finetuning ruGPT3Large model for generating school essays. |
| ruGPT2Large finetune on essays | Examples of finetuning ruGPT2Large model for generating school essays. |
| ruGPT3Small finetune on essays | Examples of finetuning ruGPT3Small model for generating school essays in colab. |
| ruGPT3Large generate | Examples of generate with ruGPT3Large model in colab. |
Finetune ruGPT3Large for school essays generation.
We prepare data with the following format:
{"text": "Тема: С какой целью В.А. Жуковский вносит русские фольклорные мотивы в традиционный балладный сюжет? (по балладе «Светлана»)\nСочинение: ..."}
For run finetuning download ruGPT3Large checkpoint and unpack to /home/jovyan/ruGPT3Large:
tar -zxvf ruGPT3Large.tar.gz
Download data to /home/jovyan/data. Data you can obtain here: train and valid.
Run script for pretrain: bash ./examples/pretrain_ruGPT3Large_essay.sh.
We obtain around 8 perplexity on valid set. Sample of generation you can see here
You can download pretrained checkpoint here.
Finetune ruGPT2Large for school essays generation.
We prepare data with the following format (raw text):
<s>Тема: С какой целью В.А. Жуковский вносит русские фольклорные мотивы в традиционный балладный сюжет? (по балладе «Светлана»)\nСочинение: ...</s>
<s>Тема: ...
For run finetuning download ruGPT2Large checkpoint and unpack to /home/jovyan/gpt2_large_bbpe_v50:
tar -zxvf gpt2_large_bbpe_v50.tar.gz
Download data to /home/jovyan/data. Data you can obtain here: train and valid.
Run script for pretrain: bash ./examples/pretrain_ruGPT2Large_essay.sh.
We obtain around 3 perplexity on valid set. Sample of generation you can see here
You can download pretrained checkpoint here.
Try example of finetune on essays and generation in colab
Try example of generation in colab