Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow to define the scale of an annotation label #349

Open
johann-petrak opened this issue Apr 24, 2023 · 2 comments
Open

Allow to define the scale of an annotation label #349

johann-petrak opened this issue Apr 24, 2023 · 2 comments

Comments

@johann-petrak
Copy link

johann-petrak commented Apr 24, 2023

It should be possible to define the scale used for each annotation label. This should be one of:

For single choice (scalar) labels:

  • nominal: values are either equal or unequal
  • ordinal: values can be equal or bigger/smaller
  • interval: values can be subtracted or added in addition to what is possible for ordinal
  • ratio: values can be multiplied and divided in addition to what is possible for interval
  • hierarchical: if a scalar label comes from a hierarchy, there are metrics that specifically consider the distance of values within the hierarchy
  • [not a label]: some fields are actually not labels where anything like IAA should get calculated, e.g. comments, reasons, confidences

There will also be the following issues related to this:

  • multiple choice labels where the user can select an arbitrary subset of all the possible values
    • this would be one example of a non-scalar label
  • having more than one field express something about a single annotated concept, e.g. label value and label confidence
@davidwilby
Copy link
Contributor

Relates to #27

@johann-petrak
Copy link
Author

Feel free to merge into #27 if that makes more sense - though defining/configuring the scale of a scalar label could also be relevant in general and is not necessarily related to IAA only.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants