This code is adapted from DIRE.
-
Run
pip install -r requirements.txt
to install the required packages. -
Download the preprocessed data along with training-test splits from this link, and put them in
data/preprocessed_data
. -
Create a symbolic link in the
src
folder by runningln -s data ./src/data
.
To pretrain the BERT encoder and decoder from scratch, run
python bert_pretrain.py [-decoder]
To train the DIRECT model from scratch, first pretrain the BERT encoder and decoder. Then run
python main.py -train
To evaluate a trained DIRECT model, assuming it is saved at src/saved_checkpoints/direct.pth
, run
python main.py -restore -name direct [-val] [-top_k 1] [-approx] [-conf_piece] [-short_only]
Running the above evaluation dumps the predictions to src/predictions/<fname>.pkl
. To evaluate these predictions with other metrics like Top-5 accuracy, Jaccard distance and Character Error Rate, run
python top5_analysis.py -fname <fname>.pkl
Model | Accuracy(%) | Top-5 Accuracy (%) | CER | Jaccard Dist |
---|---|---|---|---|
DIRE | 35.8 | 41.5 | .664 | .537 |
DIRECT | 42.8 | 49.3 | .663 | .501 |
Improvement | 20% | 19% | .2% | 6.5% |