Code

add token labelling#21

Code

Merged

jettjaniak merged 29 commits intomaindelphi-suite/delphi:mainfrom 12-categorize-tokensdelphi-suite/delphi:12-categorize-tokensCopy head branch name to clipboardFeb 17, 2024

+874-1

Commits

Commits on Feb 16, 2024

• add token labelling

  joshuawe

  committed
  c8e516b

• add explanation function

  joshuawe

  committed
  f57210c

• add notebook

  joshuawe

  committed
  32d498a

• test

  joshuawe

  committed
  e9f5c11

• swtich off dependency labels + add spacy to requirements

  joshuawe

  committed
  c10d5c1

• small improvements

  joshuawe

  committed
  ab3be19

• improve notebook explanation

  joshuawe

  committed
  3c08947

• fix errors

  joshuawe

  committed
  6bf1c56

• add notebook

  joshuawe

  committed
  70c337d

• test

  joshuawe

  committed
  1fcdd35

• swtich off dependency labels + add spacy to requirements

  joshuawe

  committed
  bd3be77

• small improvements

  joshuawe

  committed
  e3013db

• improve notebook explanation

  joshuawe

  committed
  90f2dbb

• fix errors

  joshuawe

  committed
  57689f4

• complete UPOS tags for token labels

  joshuawe

  committed
  197364d

• add tests

  joshuawe

  committed
  cdef0d6

• update requirements for delphi tokenizer

  joshuawe

  committed
  2c49e2e

• added token label script

  joshuawe

  committed
  535a0c0

• add the files containing token information/labels

  joshuawe

  committed
  48f7f6a

• small enhancements suggested for PR

  joshuawe

  committed
  210a3da

• rebasing

  joshuawe

  committed
  6a4a42d

• improve optional downloading of spacy language model

  joshuawe

  committed
  fcf4ba6

• bugfix: handle tokens empty string ''

  joshuawe

  committed
  f234ec5

• add argparse for label_all_tokens.py script

  joshuawe

  committed
  4047be4

• add tokenized dicts

  joshuawe

  committed
  3c4a1a4

• update notebook

  joshuawe

  committed
  87e18b3

• undo __init__

  joshuawe

  committed
  ef0f2e4

• change spacy model from "trf" to "sm"

  joshuawe

  committed
  5af8a6f

• bug fix

  joshuawe

  committed
  c292da7