This repo contains python3 code for the paper Investigating Novel Verb Learning in BERT: Selectional Preference Classes and Alternation-Based Syntactic Generalization, by Tristan Thrush, Ethan Wilcox, and Roger Levy.
Setup:
pip install -r requirements.txt
Experiments:
These will take a while unless you have a GPU.
The code produces a png with a graph for each experiment. Blue-colored results represent the correct judgement with a significant p-value. Red-colored results represent the incorrect judgement with a significant p-value. There are also generated csv files for each experiment.
3 Selectional Preferences:
(unseen in-class vs. unseen out-class; the critical section 3 experiment)
python main.py --save_name='prediction' --data='prediction_experiments.txt'
(unseen in-class vs seen in-class)
python main.py --save_name='prediction_seen_vs_unseen_in_class' --data='prediction_experiments_seen_vs_unseen_in_class.txt'
(unseen out-class vs seen in-class)
python main.py --save_name='prediction_seen_vs_unseen_out_class' --data='prediction_experiments_seen_vs_unseen_out_class.txt'
4.1 Psycholinguistic Levin Alternations Assessment
python main.py --use_levin_prediction_parser --save_name='levin_prediction' --data='levin_prediction_experiments.txt'
4.2 Classificaiton Levin Alternations Assessment
(high frequency)
python main.py --save_name='linear_similarity' --similarity_experiments_metric='linear_classifier' --data='similarity_experiments.txt'
(difficult distractors)
python main.py --save_name='linear_similarity_distractor' --similarity_experiments_metric='linear_classifier' --data='distractor_similarity_experiments.txt'