Skip to content

Latest commit

 

History

History
20 lines (16 loc) · 1.03 KB

superglue.md

File metadata and controls

20 lines (16 loc) · 1.03 KB
Raw

SuperGLUE Benchmark Submission Formatter

jiant supports generating submission files for SuperGLUE. To generate test predictions, use the --write_test_preds flag in runscript.py when running your workflow. This will generate a test_preds.p file in the specified output directory. To convert test_preds.p to the required GLUE submission format, use the following command:

python benchmark_submission_formatter.py \
    --benchmark SUPERGLUE \
    --input_base_path $INPUT_BASE_PATH \
    --output_path $OUTPUT_BASE_PATH

where $INPUT_BASE_PATH contains the task folder(s) output by runscript.py. Alternatively, a subset of tasks can be formatted using:

python benchmark_submission_formatter.py \
    --benchmark SUPERGLUE \
    --tasks cola mrpc \
    --input_base_path $INPUT_BASE_PATH \
    --output_path $OUTPUT_BASE_PATH