Skip to content

Latest commit

 

History

History
77 lines (37 loc) · 3.93 KB

README.md

File metadata and controls

77 lines (37 loc) · 3.93 KB
Raw

InvertibleNetworks.jl

CI

Building blocks for invertible neural networks in the Julia programming language.

Installation

] dev https://github.com/slimgroup/InvertibleNetworks.jl

Building blocks

  • 1x1 Convolutions using Householder transformations (example)

  • Residual block (example)

  • Invertible coupling layer from Dinh et al. (2017) (example)

  • Invertible hyperbolic layer from Lensink et al. (2019) (example)

  • Invertible coupling layer from Putzky and Welling (2019) (example)

  • Invertible recursive coupling layer HINT from Kruse et al. (2020) (example)

  • Activation normalization (Kingma and Dhariwal, 2018) (example)

  • Various activation functions (Sigmoid, ReLU, leaky ReLU, GaLU)

  • Objective and misfit functions (mean squared error, log-likelihood)

  • Dimensionality manipulation: squeeze/unsqueeze (column, patch, checkerboard), split/cat

  • Squeeze/unsqueeze using the wavelet transform

Applications

  • Invertible recurrent inference machines (Putzky and Welling, 2019) (generic example)

  • Generative models with maximum likelihood via the change of variable formula (example)

  • Glow: Generative flow with invertible 1x1 convolutions (Kingma and Dhariwal, 2018) (generic example, source)

To Do

  • GPU support

Acknowledgments

This package uses functions from NNlib.jl, Flux.jl and Wavelets.jl

References

  • Yann Dauphin, Angela Fan, Michael Auli and David Grangier, "Language modeling with gated convolutional networks", Proceedings of the 34th International Conference on Machine Learning, 2017. https://arxiv.org/pdf/1612.08083.pdf

  • Laurent Dinh, Jascha Sohl-Dickstein and Samy Bengio, "Density estimation using Real NVP", International Conference on Learning Representations, 2017, https://arxiv.org/abs/1605.08803

  • Diederik P. Kingma and Prafulla Dhariwal, "Glow: Generative Flow with Invertible 1x1 Convolutions", Conference on Neural Information Processing Systems, 2018. https://arxiv.org/abs/1807.03039

  • Keegan Lensink, Eldad Haber and Bas Peters, "Fully Hyperbolic Convolutional Neural Networks", arXiv Computer Vision and Pattern Recognition, 2019. https://arxiv.org/abs/1905.10484

  • Patrick Putzky and Max Welling, "Invert to learn to invert", Advances in Neural Information Processing Systems, 2019. https://arxiv.org/abs/1911.10914

  • Jakob Kruse, Gianluca Detommaso, Robert Scheichl and Ullrich Köthe, "HINT: Hierarchical Invertible Neural Transport for Density Estimation and Bayesian Inference", arXiv Statistics and Machine Learning, 2020. https://arxiv.org/abs/1905.10687

Author