Precision / recall reproducibility

[sangyun884]

Was the reproducibility of these new metrics checked with the official repo's results? I tested with my mode-collapsed generative model that produces nearly the same images as below:

and got these results:

inception_score_mean: 1.12824
inception_score_std: 0.0006825597
kernel_inception_distance_mean: 0.239309
kernel_inception_distance_std: 0.002847237
precision: 4e-05
recall: 0.99084
f_score: 7.999677e-05

We can see that precision is nearly zero and recall is close to 1. Recall is supposed to measure the diversity of generated samples; it should be close to zero in this case. Also, it seems that the car lies on the true data manifold, meaning that precision should be close to one. Results seem to be flipped.

I used 50000 generated samples. This is the command I used:

fidelity --prc --isc --kid --input1 ${dir}/${iteration}-50k/samples --input2 cifar10-train --gpu 0 | tee ${dir}/${iteration}-50k/fidelity.txt

[caohengyuan]

@sangyun884 I think when calculating precision and recall, this repo treats input1 as the real images and input2 as the generated images. This can be found in metric_prc.py line [58-61]. I think dist_nn_1 should be calculated with real image features. I not sure whether i'm right. It would be very nice of you to check these codes @toshas .