Clarification on Retrieval Metrics

[varadgunjal]

@Cuberick-Orion

Thank you for the great work!

I'm just starting out surveying image retrieval literature for usage with pretrained vision-language models and had a clarification question regarding the evaluation metrics.

Recall_subset@K as designed in this paper makes sense to me because the cardinality of the subset (in this case, 5, since we're excluding the image itself) is the total number of relevant items for each query. This lines up with the definition of Recall@K within recommender systems (for eg. https://www.pinecone.io/learn/offline-evaluation/).

My question relates to the Recall@1, 10, 50 numbers reported in Table 3. How were these numbers calculated? What is the value of the denominator when calculating Recall since we don't know the true number of False negatives? In section 4 you mention that this is what is commonly reported in other work by setting K to a large value and all images in
D {IR, IT} are considered as negative. Does that mean when calculating Recall@50, you just count the false negatives as everything outside the 6-member subset of the query image?