Skip to content

Latest commit

 

History

History
89 lines (83 loc) · 6.29 KB

CHANGELOG.md

File metadata and controls

89 lines (83 loc) · 6.29 KB

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[Unreleased]

[0.3.0] - 2021-06-08

Added

  • API
    • calculate_metrics
      • ppl: Calculate PPL (Perceptual Path Length)
      • ppl_epsilon: Interpolation step size in PPL
      • ppl_reduction: Reduction type to apply to the per-sample output values
      • ppl_sample_similarity: Name of the sample similarity to use in PPL metric computation
      • ppl_sample_similarity_resize: Force samples to this size when computing similarity, unless set to None
      • ppl_sample_similarity_dtype: Check samples are of compatible dtype when computing similarity, unless set to None.
      • ppl_discard_percentile_lower: Removes the lower percentile of samples before reduction
      • ppl_discard_percentile_higher: Removes the higher percentile of samples before reduction
      • ppl_z_interp_mode: Noise interpolation mode in PPL
      • input1_model_z_type: Type of noise accepted by the input1 generator model
      • input1_model_z_size: Dimensionality of noise accepted by the input1 generator model
      • input1_model_num_classes: Number of classes for conditional generation (0 for unconditional) accepted by the input1 generator model
      • input1_model_num_samples: Number of samples to draw from input1 generator model, when it is provided as a path to ONNX model. This option affects the following metrics: ISC, FID, KID
      • input2_model_z_type: Type of noise accepted by the input2 generator model
      • input2_model_z_size: Dimensionality of noise accepted by the input2 generator model
      • input2_model_num_classes: Number of classes for conditional generation (0 for unconditional) accepted by the input2 generator model
      • input2_model_num_samples: Number of samples to draw from input2 generator model, when it is provided as a path to ONNX model. This option affects the following metrics: ISC, FID, KID
  • Command line
    • --ppl: Calculate PPL (Perceptual Path Length)
    • --ppl-epsilon: Interpolation step size in PPL
    • --ppl-reduction: Reduction type to apply to the per-sample output values
    • --ppl-sample-similarity: Name of the sample similarity to use in PPL metric computation
    • --ppl-sample-similarity-resize: Force samples to this size when computing similarity, unless set to None
    • --ppl-sample-similarity-dtype: Check samples are of compatible dtype when computing similarity, unless set to None.
    • --ppl-discard-percentile-lower: Removes the lower percentile of samples before reduction
    • --ppl-discard-percentile-higher: Removes the higher percentile of samples before reduction
    • --ppl-z-interp-mode: Noise interpolation mode in PPL
    • --input1-model-z-type: Type of noise accepted by the input1 generator model
    • --input1-model-z-size: Dimensionality of noise accepted by the input1 generator model
    • --input1-model-num-classes: Number of classes for conditional generation (0 for unconditional) accepted by the input1 generator model
    • --input1-model-num-samples: Number of samples to draw from input1 generator model, when it is provided as a path to ONNX model. This option affects the following metrics: ISC, FID, KID
    • --input2-model-z-type: Type of noise accepted by the input2 generator model
    • --input2-model-z-size: Dimensionality of noise accepted by the input2 generator model
    • --input2-model-num-classes: Number of classes for conditional generation (0 for unconditional) accepted by the input2 generator model
    • --input2-model-num-samples: Number of samples to draw from input2 generator model, when it is provided as a path to ONNX model. This option affects the following metrics: ISC, FID, KID
  • Support generative model modules as inputs to all metrics
  • ONNX and PTH (JIT) model loading via command line functionality to support framework-agnostic metrics calculation
  • Noise source types and latent vector interpolation methods can now be registered and dispatched similar to registered inputs
  • Registered inputs: stl10-train, stl10-test, stl10-unlabeled
  • Registered noise source types: normal, uniform_0_1, unit
  • Registered latent vector interpolation methods: lerp, slerp_any, slerp
  • Example SNGAN training and evaluation script (examples/sngan_cifar10.py)
  • Test for LPIPS fidelity as compared to StyleGAN PyTorch implementation
  • Test for feature extraction layer
  • Unrecognized command line arguments warning
  • Added ReadTheDocs documentation

Changed

  • API
    • First input positional argument of calculate_metrics is now expected as a value to kwarg input1
    • Second input (optional) positional argument of calculate_metrics is now expected as a value to kwarg argument input2
    • cache_input1_name renamed to input1_cache_name
    • cache_input2_name renamed to input2_cache_name
    • rng_seed default value from 2020 to 2021
  • Command line
    • First input positional argument is now expected as a value to the key --input1
    • Second input (optional) positional argument is now expected as a value to the key --input2
    • --datasets-downloaded renamed to --no-datasets-download
    • --samples-alphanumeric renamed to --no-samples-shuffle
    • --cache-input1-name renamed to --input1-cache-name
    • --cache-input2-name renamed to --input2-cache-name
    • --rng-seed default value from 2020 to 2021
  • Change torch.save to an atomic saving operation in all functions of the caching layer, which makes it safe to use torch-fidelity in multiprocessing environment, such as a compute cluster with a shared file system.

Fixed

  • #15: Fix '64', '192', and '768' feature layers usage in all metrics
  • #8: Fix a missing exception for when KID subset size is larger than the number of samples in one of the inputs
  • Fix a missing check that the elements of inputs are actually instances of torch.Tensor

[0.2.0] - 2020-05-05

Added

  • Initial release with Inception Score (ISC), Frechet Inception Distance (FID), and Kernel Inception Distance (KID) metrics
  • Precision unit tests for all three metrics
  • Command line tool and Python API