Skip to content

Latest commit

 

History

History
122 lines (89 loc) · 3.87 KB

README.md

File metadata and controls

122 lines (89 loc) · 3.87 KB
Raw

Camel-depeval

Compare two CoNLL-X files or directories, to obtain the tokenization F-score and POS tag accuracy, as well as the LAS, UAS, and label scores.

Since comparison usually occurs between gold and parsed files, the two files/directories will be differentiated using gold and parsed keywords. In other words, you do not need to have gold and parsed files to compare; any two will do.

The tree alignment part of the code uses ced_word_alignment.

Note: the evaluator is also CoNLL-U compatible.

Methodology

  1. Two files or directories are passed to the evaluator. If two directories are passed, the directories must have matching file names.
  2. The files are read, and the trees every two files are compared.
  3. Align trees using ced_word_alignment
    • involves inserting null alignment tokens
  4. The evaluation scores are then calulated
    • tokenization f-score is calculated on all aligned tokens, while the remaining metrics are calulated after removing insertions (null alignment tokens added to the gold tree)

Assumptions

Since ced_word_alignment is used, the second and third assumptions are the same.

  • No words are added to either the parsed or gold files.
  • No changes to the word order.
  • Text is in the same script and encoding.

Contents

  • align_trees.py aligns trees using the ced_word_alignment algorithm
  • class_conllx used to read CoNLL-X files
  • classes dataclasses used throughout the code
  • conllx_counts gets different statistics after comparing 2 CoNLL-X files
  • conllx_scores calculates scores given counts
  • evaluate_conllx_driver main script
  • handle_args simplifies use of the argparse library
  • requirements.txt necessary dependencies needed to run the scripts.
  • ced_word_alignment/ the ced alignment library
  • README.md this document.

Requirements

  • Python 3.8 and above.

To use, you need to first install the necessary dependencies by running the following command:

pip install -r requirements.txt

Usage

usage: evaluate_conllx_driver.py [-h] [-g] [-p] [-gd] [-pd]

This script takes 2 CoNLL-X files or 2 directories of CoNLL-X files and evaluates the scores.

required arguments:
  -g , --gold          the gold CoNLL-X file
  -p , --parsed        the parsed CoNLL-X file

or:
  -gd , --gold_dir     the gold directory containing CoNLL-X files
  -pd , --parsed_dir   the parsed directory containing CoNLL-X files

Examples

The sentences used are taken from CamelTB_1001_introduction_1.conllx and CamelTB_1001_night_1_1.conllx (data can be obtained from The Camel Treebank.

Sample 1:

The toknization is the same, and so the F_score is 100%, and the insertion/deletion counts are both 0. 

python evaluate_conllx_driver.py -g samples/sample_1_gold.conllx -p samples/sample_1_parsed.conllx
tokenization_f_score 100.0
tokenization_precision 100.0
tokenization_recall 100.0
pos 81.6
uas 55.3
label 65.8
las 44.7
insertion_count 0
deletion_count 0

Sample 2:

python evaluate_conllx_driver.py -g samples/sample_2_gold.conllx -p samples/sample_2_parsed.conllx
tokenization_f_score 90.
tokenization_precision 90.
tokenization_recall 90.
pos 86.
uas 65.
label 75.
las 57.
insertion_count 2
deletion_count 2

License

conllx_evaluator is available under the MIT license. See the LICENSE file for more info.