Event-based segmentation

[sarkipo]

From May 2023 meeting minutes:

TS: explore the possibility of event-based segmentation. That would eliminate the need for HIAT-based FSM segmentation and allow more flexible transcription conventions
@ts Note: current transcription conventions vary to some extent between INEL corpora but the core is documented in https://doi.org/10.14232/wpcl.2020.5 (with a summary of symbols in the end)

[sarkipo]

(I thought there was already an issue on that but haven't found one)

[berndmoos]

First attempt:

• New segmentation option INEL_EVENT_BASED in PE
• Same option in wizard for COMA and EXAKT

This segmentation works, not by FSM, but by an XSL stylesheet. The general approach is:

• Build an "ordinary" segmented transcription, i.e. without additional segmentation
• Copy the event segmentation, and for each event
  • decide if it's a non-word (starting with double parentheses) or a word
  • parse the event content accordingly (into ts, ats, nts)
• Group leaves into ts with "INEL:u" according to the ref tier
  This seems to work (for the Dolgan corpus), but it is anything but fast (about 20 minutes for all 17 Dolgan transcripts).
  The whole setup is analagous to HIAT, i.e. we have utterances, words, etc., they only have "INEL:" as prefix instead of "HIAT:"

To do: take care of segmentation errors. There are two options:

• Decide that there can never be segmentation errors (because the XSL transformation will never fail)
• Define segmentation errors in a separate process

[berndmoos]

it is anything but fast

It can take up to two hours for gigantic (INEL) transcripts, so it needs to be implemented differently

[berndmoos]

Please also add
: (colon)
; (semicolon)
“ (left double quotation mark)
” (right double quotation mark)

Pending decision on hyphens...

[sarkipo]

Also word-external:
« U+00AB (LEFT-POINTING DOUBLE ANGLE QUOTATION MARK)
» U+00BB (RIGHT-POINTING DOUBLE ANGLE QUOTATION MARK)
‐ U+2010 HYPHEN
‑ U+2011 NON-BREAKING HYPHEN

On the contrary, the usual
- U+002D HYPHEN-MINUS
should be word-internal. @git-debase-verbose has just seen that

<ts e="T223" id="tx.w222" n="INEL:w" s="T222">Dʼɨllara</ts>
<nts id="nts_tx.e222_2" n="INEL:ip">-</nts>
<ts e="T223" id="tx.w222" n="INEL:w" s="T222">kunnere</ts>

makes Tsakorpus unhappy.

[berndmoos]

Different implementation (no XSL) now takes seconds instead of minutes. To do / decide: What will count as a segmentation error? Or make the algorithm accept everything?

[berndmoos]

((xxx)). -- there can be utterance terminators after the closing parentheses

[sarkipo]

Not necessarily utterance terminators, since it can be in the middle of an utterance. Rather just any punctuation, e.g.
((…)), – or ((…)),. But terminators are perhaps 97% of all cases when something follows. (Also complex ones like ((…))?”).

[git-debase-verbose]

In https://github.com/Exmaralda-Org/exmaralda/blob/master/src/org/exmaralda/partitureditor/jexmaralda/segment/InelEventBasedSegmentation.java:

The exception type at line 105 ("Word characters after double closing round parentheses...") is created with null instead of tierID - so when I later call FSMException.getTierID on it, the result is null as well. Could you take a look on it?

[sarkipo]

Hi Thomas, could you please check whether (your version of) the new segmentation throws an error on events like
“((…))
(punctuation preceding double brackets)?
Ours does, although in principle it shouldn't. It also still has Tier ID: null in some errors, namely
Word characters after double closing round parentheses in event: ((… beginning at T311. Tier ID: null
(this is right to be an error, just the null is wrong)

Or maybe the jar is not the latest one?

[berndmoos]

This one works for me.

Guess I need to check the jar, then...

[berndmoos]

The last one at https://exmaralda.org/files/prevDL/webjar.html was from August, 19th. I don't think the last changes were after this, but to make sure, I updated the jar. Please try again. If it doesn't work, I would need a more complete example to reproduce the error.

[sarkipo]

Just the double brackets ((…)) work fine, and ones followed by other punctuation, but not the ones preceded by quotes:

[sarkipo]

KXN_1978_ThreeShamans_flk.zip

[berndmoos]

Okay, but that is as intended: double round brackets can be optionally followed by puncutation characters, but not preceded. Is that a request for change then?

[sarkipo]

Yes, please. Just because of cases like this, which I think are no worse than following punctuation. Or are there reasons to disallow that?

[sarkipo]

We only have very few cases with these quotes, but it seems to say what it says, unintelligible fragment within direct speech.

[berndmoos]

Or are there reasons to disallow that?

Not really, I'm just worried that new rules will break old ones... I tried a fix, there is a new Windows version and a new web jar.

[git-debase-verbose]

Thank you, these errors are no more with the new jar. Besides, combinations such as somewordhere((…)). got marked as erroneous, which they are.

[git-debase-verbose]

Would it be possible to add the forward slash "/" to the list of word-external punctuation marks in the segmentator? We use slashes to (a) mark line breaks in poetic texts and to (b) occasionally provide alternative variants of some part of a sentence.

Currently the segmentator throws errors ("Non-permissible sequence of word and punctuation characters in event: kanʼixubiʔ // ".) if there are slashes following whitespace. If a slash immediately follows a word, i.e. "kanʼixubiʔ/", it will be treated as a part of the word by the segmentator, which is not the intended behavior.

[berndmoos]

Adding or removing characters from word external punctuation is not a big deal. I have added the forward slash now and made a new Windows preview and web jar. If you want to see for yourselves: the characters are defined here:

exmaralda/src/org/exmaralda/partitureditor/jexmaralda/segment/InelEventBasedSegmentation.java

Line 40 in 651d82b

String[] WORD_EXTERNAL_PUNCUTATION = {

[git-debase-verbose]

Thank you. Do I take it right that it would be perfectly fine if I commit any further changes to the external punctuation list myself?

[berndmoos]

Yes, go ahead:-) But please do it in a branch and send a merge request

[sarkipo]

Hi Thomas,
The "Generate corpus statistics" function in Coma does not seem to see the # INEL counts, reports zeroes everywhere.

[berndmoos]

Okay, I made this a separate issue and will check:

#497