[FEATURE-REQUEST] Bias measurements for token-classification supertask.

[AJDERS]

Is your feature request related to a problem? Please describe.
Introduce a new metric bias and corresponding evaluation-dataset which should quantify the intrinsic bias of different token-classification-models. The different demographical features for which bias should be quantified are (at least):

• Gender
  • Male
  • Female
  • Transgender
  • Non-binary
• Ethnicity
  • Caucasian
  • Non-caucasian
• Religion
  • Christianity
  • Islam
  • Judaism
• Sexuality
  • Heterosexuality
  • Homosexuality
  • Bisexuality

Describe the solution you'd like
This could be a synthetically produced dataset created from an already existing dataset. As an example for religion the dataset could include a set of sentences which includes institutions, names and places which has some kind of religious context which are then varied, and an indicator for which demographic the sentence belongs to, i.e.

("Islam", "Hun er præst i den lokale kirke"),
("Christianity", "Hun er imam i den lokale moske")
("Judaism", "Hun er rabbiner i den lokale synagoge")
...

This is largely an open question, so give it a go!