Checker info - SEGM and XPOSTAG fields #18

epageperron · 2018-01-16T15:15:19Z

As of now, these are the rules concerning the SEGM and XPOSTAG field of the CDLI-CoNLL format. It might still change since we are still running into problems wile annotating, problems that require making decisions on the rules.

SEGM
containts the lemma which is composed of a dictionary word and its sense, appended and in square brackets. eg : udu[sheep] or dab[seize]
For all word types except verbs, there will only be suffixed morphemes, no prefixed ones.
All morphemes except the first element are composed of a dash, followed by the morpheme.
Only in the case of verbs, the first prefix will be without a dash. eg : i[-n]-dab[seize][-ø]
every morpheme can be enclosed in [], or nor.
There are also rules concerning the "slots" for morpheme but since we are not noting them we will not check for the order at this time, but we should open a backlog issue to that effect, since we want to democratize the usage of the tool, checking the possible order of morphemes would be an asset for inexperienced annotators.

XPOSTAG
This field will display the ETCSRI/ORACC POS tag OR the named entity tag instead of the lemma in the SEGM field.
For all word types except verbs, there will only be suffixed morphological tags, no prefixed ones.
All morph tags morphemes except the first element are composed of a period, followed by the morpheme.
Only verbs can have prefixes. eg : FIN.3-SG-H-A.V.3-SG-P
Tags can contain dashes, they are not meaningful in this context since the checker should use a map of morphemes and morph tags.

epageperron · 2018-04-04T00:52:11Z

tags list : https://docs.google.com/spreadsheets/d/1y0_y9HDQNwH0VqDCjjYuUpFsugw4GEybu6Pu01I_D9c/edit#gid=0

jayanthkmr · 2018-04-18T01:03:05Z

XPOSTAG:
if it is V, there can be prefixes and if not V, then no prefixes.
POS TAGS - pos tag.form regex (form may have dashes)
Check if pos tag is in the inventory.
SEGM field has dashes and XPOSTAGS has periods.

https://github.com/cdli-gh/CDLI-CoNLL-to-CoNLLU-Converter/blob/master/cdliconll2conllu/mapping.json#L25

https://docs.google.com/spreadsheets/d/1Is7MGG0h8h0vfHj9C9mnWOD2utPeuvm1ZeYb1dsaejg/edit#gid=0

only ORACC

SEGM
Main lemma followed by [sense in square brackets]

jayanthkmr · 2018-04-19T02:17:29Z

TAG lists - https://docs.google.com/spreadsheets/d/1Is7MGG0h8h0vfHj9C9mnWOD2utPeuvm1ZeYb1dsaejg/edit#gid=0

jayanthkmr · 2018-05-01T02:15:32Z

Almost done here - https://github.com/cdli-gh/morphology-pre-annotation-tool/tree/wiplast

epageperron · 2018-05-05T17:34:37Z

Can you please allow underscores in the "sense" so in the square brackets following a lemma

Error: The segm bisagdubak[filing_basket] in line number 3 in file /alty/data/mtaac_gold_corpus/morph/to_dict/P340861.conll does not follow the format "^(([a-z0-9]+)((-[a-z0-9]+)|([-[a-z0-9]+]))-)?[A-Za-z0-9()]+[[a-z0-9]]((-[a-z0-9]+)|([-[a-z0-9]+])|([-ø]))*$".

Use underscore in SEGM sense.

epageperron · 2018-05-05T17:35:05Z

same for the tilde :
Error: The segm bitum[~administration] in line number 5 in file /alty/data/mtaac_gold_corpus/morph/to_dict/P340861.conll does not follow the format "^(([a-z0-9]+)((-[a-z0-9]+)|([-[a-z0-9]+]))-)?[A-Za-z0-9()]+[[a-z0-9]]((-[a-z0-9]+)|([-[a-z0-9]+])|([-ø]))*$".

[~*] - tilda start is valid (not anywhere in between)

epageperron · 2018-05-05T17:36:20Z

morphemes and tags attached to nouns can also be attached to verbs :
Error: The xpostag MID.V.3-SG-S.TERM in line number 11 in file /alty/data/mtaac_gold_corpus/morph/to_dict/P202745.conll has a suffix TERM not in verb postag suffix list ['1-SG-A', '1-SG-S', '1-SG-P', '2-SG-A', '2-SG-S', '2-SG-P', '3-SG-S', '3-SG-P', '3-SG-A', '3-SG-S-OB', '1-PL-A', '1-PL-S', '1-PL', '2-PL-A', '2-PL-S', '2-PL', '3-PL-S', '3-PL-P', '3-PL', '3-PL-A', 'PF', 'SUB', 'PLEN'].

Noun tags can also be verb suffixes. So, corresponding morphemes are also part of verb form.

epageperron · 2018-05-05T18:10:30Z

Error: The xpostag FIN.3-SG-H-A.V.3-SG-P in line number 22 in file /alty/data/mtaac_gold_corpus/morph/to_dict/P102311.conll has a prefix 3-SG-H-A not in verb postag prefix list ['NEG', 'MOD1', 'ANT', 'MOD2', 'MOD4', 'MOD6', 'MOD7', 'MOD3', 'MOD5', 'COOR', 'VEN', 'VEN', 'MID', 'FIN', 'FIN-LI', 'FIN-L2', 'FIN', 'FIN-L2', 'FIN'].

This is correct, a verb can have more than one prefix :

Check in https://docs.google.com/spreadsheets/d/1y0_y9HDQNwH0VqDCjjYuUpFsugw4GEybu6Pu01I_D9c/edit#gid=0

Verbs can display prefixes from these categories:

Modal prefixes
Conjugation prefixes
Dimentional infixes (one pronoun, one or more cases)
Pronominal element before base

Add Pronominal element before base into prefix list.
75 - 81 Dimensional infixes pronoun can only have one but 84-114 can be multiple times.
There is order from modal to elemental.

epageperron · 2018-05-05T18:13:26Z

in text :
o.2.4 gu4-e-us2-sa gueusa[ox-following][-a] V.SUB _ _ _

Error: The segm V.SUB in line number 9 in file /alty/data/mtaac_gold_corpus/morph/to_dict/P102311.conll does not follow the format "^(([a-z0-9]+)((-[a-z0-9]+)|([-[a-z0-9]+]))-)?[A-Za-z0-9()]+[[a-z0-9]]((-[a-z0-9]+)|([-[a-z0-9]+])|([-ø]))*$".

so V.SUB is the XPOSTAG column no?

Is the dash inside the sense maybe the problem but error gives other information? If it's the case please permit - inside sense

Permit dash in the middle of the segm.

epageperron · 2018-05-05T18:14:24Z

Error: The xpostag _ in line number 9 in file /alty/data/mtaac_gold_corpus/morph/to_dict/P102311.conll does not have a base postag out of "['AJ', 'AV', 'NU', 'CNJ', 'DET', 'J', 'N', 'PRP', 'DN', 'EN', 'GN', 'MN', 'PN', 'RN', 'SN', 'TN', 'WN', 'AN', 'CN', 'FN', 'LN', 'ON', 'QN', 'YN', 'V', 'V-PL', 'V-PF', 'V-RDP']".

Maybe give a warning, not error, saying that segm or xpostag hasn't been filled if you find an underscore?

Tab missing. Error in file.

epageperron · 2018-05-05T18:19:08Z

Error: The segm us[follow]-'a in line number 15 in file /alty/data/mtaac_gold_corpus/morph/to_dict/P102315.conll does not follow the format "^(([a-z0-9]+)((-[a-z0-9]+)|([-[a-z0-9]+]))-)?[A-Za-z0-9()]+[[a-z0-9]]((-[a-z0-9]+)|([-[a-z0-9]+])|([-ø]))*$".

the normalized morpheme 'a exists, just that sheets doesn't like apostrophes

Allow apostrophes.

epageperron · 2018-05-05T18:20:52Z

Error: The segm Geme'enlila[1] in line number 126 in file /alty/data/mtaac_gold_corpus/morph/to_dict/P142753.conll does not follow the format "^(([a-z0-9]+)((-[a-z0-9]+)|([-[a-z0-9]+]))-)?[A-Za-z0-9()]+[[a-z0-9]]((-[a-z0-9]+)|([-[a-z0-9]+])|([-ø]))*$".

apostrophes in lemmata are OK

Allow apostrophes in lemmata.

epageperron · 2018-05-05T18:26:19Z

Error: The xpostag NU.3-SG-POSS.GEN in line number 31 in file /alty/data/mtaac_gold_corpus/morph/to_dict/P480072.conll has a suffix 3-SG-POSS not in noun postag suffix list ['L1', 'L2-NH', 'GEN', '3-PL-POSS', '3-SG-H-POSS', '3-SG-NH-POSS', 'DEM2', 'COM', 'PL', 'ERG', 'DAT-NH', 'L3-NH', 'DEM', 'PL', 'ADV', 'EQU', '1-SG-POSS', '1-PL-POSS', 'L4', 'ABS', 'DAT-H', 'L2-H', 'L3-H', 'TERM', 'ABL', '2-SG-POSS', '2-PL-POSS', 'COP-1-SG', 'COP-2-SG', 'COP-3-SG', 'COP-1-PL', 'COP-2-PL', 'COP-3-PL', 'EXCEPT'].

you can add NU (numerals) in the list of POS that can see suffixes appended

Error in pos tag.

epageperron · 2018-05-05T18:27:18Z

Forward slash is acceptable in lemma

Error: The segm 1/3(disz)[one] in line number 7 in file /alty/data/mtaac_gold_corpus/morph/to_dict/P480072.conll does not follow the format "^(([a-z0-9]+)((-[a-z0-9]+)|([-[a-z0-9]+]))-)?[A-Za-z0-9()]+[[a-z0-9]]((-[a-z0-9]+)|([-[a-z0-9]+])|([-ø]))*$".

Allow forward slash in lemmata.

jayanthkmr · 2018-05-05T19:24:07Z

Thanks @epageperron for your extensive testing. I will start adding the fixes one by one.

epageperron · 2018-05-05T22:56:09Z

@ is also fine in lemma ( my browser crashed so I don't have the example anymore but something like disz@t

Allow @ in lemmata

jayanthkmr · 2018-06-01T01:08:54Z

From Jinyan:

For the errors of the following list, verb postag consists of prefix, infix and suffix, and infix includes dimensional infixes (pronouns and cases), like 3-PL (I corrected 3-PL-H in the following texts) and pronominal elements, like 3-SG-H-A. Please refer to:

https://docs.google.com/spreadsheets/d/1y0_y9HDQNwH0VqDCjjYuUpFsugw4GEybu6Pu01I_D9c/edit#gid=0

Error: The xpostag MID.DAT.V.SUB in line number 210 in file /alty/data/mtaac_gold_corpus/morph/to_dict/P106438.conll has a prefix DAT not in verb postag prefix list ['NEG', 'MOD1', 'ANT', 'MOD2', 'MOD4', 'MOD6', 'MOD7', 'MOD3', 'MOD5', 'COOR', 'VEN', 'VEN', 'MID', 'FIN', 'FIN-LI', 'FIN-L2', 'FIN', 'FIN-L2', 'FIN'].

Error: The xpostag MID.DAT.V.SUB in line number 253 in file /alty/data/mtaac_gold_corpus/morph/to_dict/P106438.conll has a prefix DAT not in verb postag prefix list ['NEG', 'MOD1', 'ANT', 'MOD2', 'MOD4', 'MOD6', 'MOD7', 'MOD3', 'MOD5', 'COOR', 'VEN', 'VEN', 'MID', 'FIN', 'FIN-LI', 'FIN-L2', 'FIN', 'FIN-L2', 'FIN'].

Error: The xpostag VEN.3-PL-H.DAT.3-SG-H-A.V.3-SG-P in line number 26 in file /alty/data/mtaac_gold_corpus/morph/to_dict/P453803.conll has a prefix 3-PL-H not in verb postag prefix list ['NEG', 'MOD1', 'ANT', 'MOD2', 'MOD4', 'MOD6', 'MOD7', 'MOD3', 'MOD5', 'COOR', 'VEN', 'VEN', 'MID', 'FIN', 'FIN-LI', 'FIN-L2', 'FIN', 'FIN-L2', 'FIN'].

Error: The xpostag VEN.3-PL-H.DAT.3-SG-H-A.V.3-SG-P in line number 26 in file /alty/data/mtaac_gold_corpus/morph/to_dict/P453803.conll has a prefix DAT not in verb postag prefix list ['NEG', 'MOD1', 'ANT', 'MOD2', 'MOD4', 'MOD6', 'MOD7', 'MOD3', 'MOD5', 'COOR', 'VEN', 'VEN', 'MID', 'FIN', 'FIN-LI', 'FIN-L2', 'FIN', 'FIN-L2', 'FIN'].

Error: The xpostag VEN.3-PL-H.DAT.3-SG-H-A.V.3-SG-P in line number 26 in file /alty/data/mtaac_gold_corpus/morph/to_dict/P453803.conll has a prefix 3-SG-H-A not in verb postag prefix list ['NEG', 'MOD1', 'ANT', 'MOD2', 'MOD4', 'MOD6', 'MOD7', 'MOD3', 'MOD5', 'COOR', 'VEN', 'VEN', 'MID', 'FIN', 'FIN-LI', 'FIN-L2', 'FIN', 'FIN-L2', 'FIN'].

Error: The xpostag VEN.3-PL-H.DAT.3-SG-H-A.V.3-SG-P in line number 29 in file /alty/data/mtaac_gold_corpus/morph/to_dict/P459210.conll has a prefix 3-PL-H not in verb postag prefix list ['NEG', 'MOD1', 'ANT', 'MOD2', 'MOD4', 'MOD6', 'MOD7', 'MOD3', 'MOD5', 'COOR', 'VEN', 'VEN', 'MID', 'FIN', 'FIN-LI', 'FIN-L2', 'FIN', 'FIN-L2', 'FIN'].

Error: The xpostag VEN.3-PL-H.DAT.3-SG-H-A.V.3-SG-P in line number 29 in file /alty/data/mtaac_gold_corpus/morph/to_dict/P459210.conll has a prefix DAT not in verb postag prefix list ['NEG', 'MOD1', 'ANT', 'MOD2', 'MOD4', 'MOD6', 'MOD7', 'MOD3', 'MOD5', 'COOR', 'VEN', 'VEN', 'MID', 'FIN', 'FIN-LI', 'FIN-L2', 'FIN', 'FIN-L2', 'FIN'].

Error: The xpostag VEN.3-PL-H.DAT.3-SG-H-A.V.3-SG-P in line number 29 in file /alty/data/mtaac_gold_corpus/morph/to_dict/P459210.conll has a prefix 3-SG-H-A not in verb postag prefix list ['NEG', 'MOD1', 'ANT', 'MOD2', 'MOD4', 'MOD6', 'MOD7', 'MOD3', 'MOD5', 'COOR', 'VEN', 'VEN', 'MID', 'FIN', 'FIN-LI', 'FIN-L2', 'FIN', 'FIN-L2', 'FIN'].

Error: The xpostag MID.DAT.V.3-SG-S.SUB in line number 13 in file /alty/data/mtaac_gold_corpus/morph/to_dict/P458899.conll has a prefix DAT not in verb postag prefix list ['NEG', 'MOD1', 'ANT', 'MOD2', 'MOD4', 'MOD6', 'MOD7', 'MOD3', 'MOD5', 'COOR', 'VEN', 'VEN', 'MID', 'FIN', 'FIN-LI', 'FIN-L2', 'FIN', 'FIN-L2', 'FIN'].

For the error of RDP, RDP is included in the stem of the verb. Please refer to:

https://docs.google.com/spreadsheets/d/1y0_y9HDQNwH0VqDCjjYuUpFsugw4GEybu6Pu01I_D9c/edit#gid=0

Error: The xpostag V.RDP.SUB.GENABL in line number 16 in file /alty/data/mtaac_gold_corpus/morph/to_dict/P459210.conll has a suffix RDP not in verb postag suffix list ['1-SG-A', '1-SG-S', '1-SG-P', '2-SG-A', '2-SG-S', '2-SG-P', '3-SG-S', '3-SG-P', '3-SG-A', '3-SG-S-OB', '1-PL-A', '1-PL-S', '1-PL', '2-PL-A', '2-PL-S', '2-PL', '3-PL-S', '3-PL-P', '3-PL', '3-PL-A', 'PF', 'SUB', 'PLEN'].

Handle RDP as part of speech.

jayanthkmr · 2018-06-01T01:40:55Z

Files to check:
P340861.conll
P202745.conll
P102311.conll
P102315.conll
P142753.conll
P480072.conll

epageperron · 2018-06-19T01:43:26Z

What should I be looking for in those files exactly?

jayanthkmr · 2018-06-19T04:17:19Z

That note was for me :)

wangj619 · 2018-08-24T17:34:53Z

Error: The segm 1/2(disz)[one] in line number 6 in file /Users/user/annotation_assistant/scr/data/mtaac_gold_corpus/morph/processed/P339103.conll does not follow the format "^(([a-z0-9]+)((-[a-z0-9]+)|([-[a-z0-9]+]))-)?[A-Za-z0-9()]+[[a-z0-9]]((-[a-z0-9]+)|([-[a-z0-9]+])|([-ø]))*$".

This error appears again? Has forward slash been acceptable in lemma?

wangj619 · 2018-08-24T18:07:20Z

Error: The segm igi4(disz)[one]gal[fraction] in line number 23 in file /Users/user/annotation_assistant/scr/data/mtaac_gold_corpus/morph/processed/P339103.conll does not follow the format "^(([a-z0-9]+)((-[a-z0-9]+)|([-[a-z0-9]+]))-)?[A-Za-z0-9()]+[[a-z0-9]]((-[a-z0-9]+)|([-[a-z0-9]+])|([-ø]))*$".

This is correct. This lemma is actually a compound lemma, can it work for the computer?

@wangj619 It passes for "igi4(disz)[one]-gal[-fraction] "
@jayanthjaiswal Actually, [fraction] is the meaning of the whole lemma, not the last part 'gal'. And the number 4(disz)[one] is embedded into it.

jayanthkmr · 2018-09-05T00:14:38Z

All +1 are now fixed @epageperron @khoidt @wangj619 The segm ones are fixed now.

jayanthkmr self-assigned this Jan 18, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Checker info - SEGM and XPOSTAG fields #18

Checker info - SEGM and XPOSTAG fields #18

epageperron commented Jan 16, 2018 •

edited

Loading

epageperron commented Apr 4, 2018

jayanthkmr commented Apr 18, 2018

jayanthkmr commented Apr 19, 2018

jayanthkmr commented May 1, 2018

epageperron commented May 5, 2018 •

edited by jayanthkmr

Loading

epageperron commented May 5, 2018 •

edited by jayanthkmr

Loading

epageperron commented May 5, 2018 •

edited by jayanthkmr

Loading

epageperron commented May 5, 2018 •

edited by jayanthkmr

Loading

epageperron commented May 5, 2018 •

edited by jayanthkmr

Loading

epageperron commented May 5, 2018 •

edited by jayanthkmr

Loading

epageperron commented May 5, 2018 •

edited by jayanthkmr

Loading

epageperron commented May 5, 2018 •

edited by jayanthkmr

Loading

epageperron commented May 5, 2018 •

edited by jayanthkmr

Loading

epageperron commented May 5, 2018 •

edited by jayanthkmr

Loading

jayanthkmr commented May 5, 2018

epageperron commented May 5, 2018 •

edited by jayanthkmr

Loading

jayanthkmr commented Jun 1, 2018 •

edited

Loading

jayanthkmr commented Jun 1, 2018

epageperron commented Jun 19, 2018

jayanthkmr commented Jun 19, 2018

wangj619 commented Aug 24, 2018

wangj619 commented Aug 24, 2018 •

edited

Loading

jayanthkmr commented Sep 5, 2018 •

edited

Loading

Checker info - SEGM and XPOSTAG fields #18

Checker info - SEGM and XPOSTAG fields #18

Comments

epageperron commented Jan 16, 2018 • edited Loading

epageperron commented Apr 4, 2018

jayanthkmr commented Apr 18, 2018

jayanthkmr commented Apr 19, 2018

jayanthkmr commented May 1, 2018

epageperron commented May 5, 2018 • edited by jayanthkmr Loading

epageperron commented May 5, 2018 • edited by jayanthkmr Loading

epageperron commented May 5, 2018 • edited by jayanthkmr Loading

epageperron commented May 5, 2018 • edited by jayanthkmr Loading

epageperron commented May 5, 2018 • edited by jayanthkmr Loading

epageperron commented May 5, 2018 • edited by jayanthkmr Loading

epageperron commented May 5, 2018 • edited by jayanthkmr Loading

epageperron commented May 5, 2018 • edited by jayanthkmr Loading

epageperron commented May 5, 2018 • edited by jayanthkmr Loading

epageperron commented May 5, 2018 • edited by jayanthkmr Loading

jayanthkmr commented May 5, 2018

epageperron commented May 5, 2018 • edited by jayanthkmr Loading

jayanthkmr commented Jun 1, 2018 • edited Loading

jayanthkmr commented Jun 1, 2018

epageperron commented Jun 19, 2018

jayanthkmr commented Jun 19, 2018

wangj619 commented Aug 24, 2018

wangj619 commented Aug 24, 2018 • edited Loading

jayanthkmr commented Sep 5, 2018 • edited Loading

epageperron commented Jan 16, 2018 •

edited

Loading

epageperron commented May 5, 2018 •

edited by jayanthkmr

Loading

epageperron commented May 5, 2018 •

edited by jayanthkmr

Loading

epageperron commented May 5, 2018 •

edited by jayanthkmr

Loading

epageperron commented May 5, 2018 •

edited by jayanthkmr

Loading

epageperron commented May 5, 2018 •

edited by jayanthkmr

Loading

epageperron commented May 5, 2018 •

edited by jayanthkmr

Loading

epageperron commented May 5, 2018 •

edited by jayanthkmr

Loading

epageperron commented May 5, 2018 •

edited by jayanthkmr

Loading

epageperron commented May 5, 2018 •

edited by jayanthkmr

Loading

epageperron commented May 5, 2018 •

edited by jayanthkmr

Loading

epageperron commented May 5, 2018 •

edited by jayanthkmr

Loading

jayanthkmr commented Jun 1, 2018 •

edited

Loading

wangj619 commented Aug 24, 2018 •

edited

Loading

jayanthkmr commented Sep 5, 2018 •

edited

Loading