Fix/multiclass recall macro avg ignore index

[rittik9]

What does this PR do?

Fixes #2441

Details

• Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure to update the docs?
• Did you write any new necessary tests?

Did you have fun?

Yes

Issue:

• The root of the problem seems to be that the ignore_index information is not being properly propagated to the final averaging step i.e. the _adjust_weights_safe_divide function doesn't know that which class should be ignored.

Solution:

• To address this issue, I updated the code to ensure that the ignore_index information is preserved throughout the entire process, making sure it is correctly passed through all intermediate steps up to the final averaging stage i.e. _adjust_weights_safe_divide function .
• Updated the _adjust_weights_safe_divide function to accept an additional ignore_index parameter, which is passed through the _precision_recall_reduce function, called in the compute method of the MulticlassRecall class. This change adjusts the weights in the _adjust_weights_safe_divide function, setting the weight of the ignored class to 0.

---

📚 Documentation preview 📚: https://torchmetrics--2710.org.readthedocs.build/en/2710/

[Borda]

looks good, can we add also test for this case...

[rittik9]

looks good, can we add also test for this case...

Sure

pending on adding test

[rittik9]

@Borda What do I have to modify?

[Borda]

@rittik9 mind checking the changed docstest values and whether it is correct?

[JeroenMandersloot]

Any update on when this PR could be merged? It would really help if we could update from the 0.9.3 version once this fix is merged.

[Borda]

Any update on when this PR could be merged? It would really help if we could update from the 0.9.3 version once this fix is merged.

the tests/doctests need to be fixed, are you interested in submitting a suggestion on what else needs to be fixed/chnaged?

[Borda]

Seems we are already testing various ignore_index with reference metric so if we had it wrong this did not pass already... it is possible that we also have a bug in the reference metric?
cc: @SkafteNicki

[Borda]

looking to the code and the ignore index is already applied in _multilabel_stat_scores_format which reduces the preds/target size the same way as the reference metric so calling it with null weights in fact ignores additional index

[rittik9]

The problem is we are using sklearn's recall_score as a reference for our unittests. So even if in _reference_sklearn_precision_recall_multiclass() function we are using remove_ignore_index function for removing those predictions whose real values are ignore_index class before passing it to recall_score function, it does not matter. Because whenever average='macro' sklearn's recall_score will always return mean cosidering the total no. of classes (as we are passing all the classes in recall_score() function's labels argument). That is the reason why unittests failed in the first place. I think we need to fix the unittests to take care of ignore_index using sklearn's recall_score() function's labels argument. I've prepared a notebook for explanation. cc:@Borda.

[DimitrisMantas]

Just to chime in, I think this issue is present in pretty much all metrics that make use of _adjust_weights_safe_divide.

I see this PR fixes, some of them, but others, such as JaccardIndex are left as is.