Add METEOR metric

[stancld]

🚀 Feature

Add another NLP metric - METEOR (Lavie and Agarval, 2007).

Motivation

METEOR is another metric used for the machine translation evaluation similarly to BLEU, however, it demonstrates a higher correlation with human judgements of translation quality.

Pitch

To support the METEOR metric will be likely to require nltk dependency. The nltk package is also, however, needed for the ROUGE metric. METEOR thus should not bring any new dependency.

Additional context

I will be happy to start working on this feature next week if desired to be added :)

[SkafteNicki]

@stancld sounds great :]

[Borda]

@stancld do want to take it on your own or you can promote it on slack #new_contributions and then supervise it 🐰

[stancld]

@Borda I've been working on that. :]

I've almost finished the implementation resembling the behaviour of the METEOR metric from nltk package. Once ready, I'll send a PR.

The aforementioned implementation is, however, based on the original paper (Lavie and Agarval, 2007) and thus is quite obsolete.

There is also a new version of this metric based on the paper of Denkowski and Lavie, 2014. I suggest doing this upgrade in another PR because it seems the only implementation of this metric is written in Java (see Meteor 1.5) (except for this python wrapper which directly utilizes the Java implementation) so it may take some extra time to implement it. (In general, I am trying to write the current implementation to be adaptable for newer versions)

[mattdias96]

Hey! This would be super useful to me, and I would love to contribute to make this work. Is there any way I can help?

[Borda]

@mattdias96 great! @stancld what do you think? 🐿️

[stancld]

Hey @mattdias96, do you mean the METEOR version 1.5? The thing is there's only Java implementation (I don't know Java 😬 ) + it'd be necessary to port several source files (here I'm not sure about copyrights, but I think this can be discussed w/ PL team, right @Borda?). In general, I expect it'd require large amounts of work to port this metric, but yeah, it'd be really cool to have it here.

I'd be a bit reluctant to add METEOR 1.0 used in some other python libraries, as this version is not used when referring to METEOR metric in papers, I guess.

[maliozer]

Did you finish and add this to the torchmetrics library ?

[stancld]

@maliozer Unfortunately not. Also, I think it would be pretty time-consuming to add the metric and I can't see it fitting into my schedule atm.

[stancld]

@maliozer However, feel free to give a try to add the metric if you want to :] It's currently not assigned.

[ThilinaTLM]

Does anyone working on this?
@stancld @maliozer

[stancld]

Hi @ThilinaTLM, unfortunately, I'm not aware of anyone working on this issue right now. If you wanna give a try, I'll be glad to help! .]