Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rename rater_agreement_pearson() to rater_agreement() #16

Merged
merged 1 commit into from
Aug 30, 2024

Conversation

hagenw
Copy link
Member

@hagenw hagenw commented Aug 28, 2024

Closes #15

Renames audpsychometric.rater_agreement_pearson() to audpsychometric.rater_agreement(), to have the option to use another correlation measure at a later stage, e.g. by adding an association_type argument.

image

@hagenw hagenw requested a review from ChristianGeng August 28, 2024 13:47
Copy link
Member

@ChristianGeng ChristianGeng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am starting to think that there is even more generality to this:

  • the association measure is one thing, and by providing a more general name suggests that we could increase the flexibility on that end.

  • However, there is even more: there is the variable average_ratings_other_raters that is hardwired to use the mean value. The mean is a kind of "gold standard" in the old terminology. But could it not be that we want the mode, or even the ewe?

Apart from that: using standard association measures like pearson runs against the spirit of reliability coefficients that account for something like guessing behavior on the annotator side in the case of non-numerical data. For numerical data there is the icc family of coefficient that deal with these problems - by using aov decompositions that decompose variance into its components (between items, between raters), with different decompositions answering different questions.

Possibly we should deal with these generalizations later?
For now I would approve.

@hagenw hagenw merged commit 2e2e9fe into main Aug 30, 2024
7 checks passed
@hagenw hagenw deleted the rater-agreement branch August 30, 2024 07:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Rename rater_agreement_pearson()?
2 participants