Christian Horvat and Jean-Pascal Pfister 2022
We introduce a new method, named inflation-deflation, to learn a density supported on a low-dimensional manifold. The idea is to exploit the ability of standard Normalizing Flows (NFs) to learn any density p(x) supported on entire embedding space. For that, we
- add noise to the data-manifold (inflation step),
- train a standard NF (denoted as F in the figure above) to learn the inflated distribution,
- scale the learned density to approximate p(x) (deflation step),
- find conditions on the type of noise and on the manifold such that the approximation is exact (main Theorem).
Crucially, for step 4. the noise must be added in the manifolds normal space. Thus, the manifold must be known beforehand. However, we show that a standard Gaussian can be well used to approximate a Gaussian in the normal space whenever the manifold dimension d is much smaller than the dimensionality of the embedding space.
The inflation-deflation method ties in with the Manifold Flow, introduced by Johann Brehmer and Kyle Cramner, to use NFs to learn densities supported on a low-dimensional manifold.
We have shown that our method performes well on a wide range of manifolds:
| manifold | data dimension | manifold dimension | argument to --dataset |
argument to --latent_distribution |
|---|---|---|---|---|
| Sphere | 3 | 2 | sphere | mixture, correlated |
| Torus | 3 | 2 | torus | mixture, correlated |
| Hyperboloid | 3 | 2 | hyperboloid | mixture, correlated, unimodal |
| Thin spiral | 2 | 1 | thin_spiral | exponential |
| Swiss Roll | 3 | 2 | swiss_roll | mixture, correlated |
| Hyperboloid-Sphere | 3 | 2 | spheroid | mixture, correlated |
| Stiefel, SO(2) | 4 | 1 | stiefel | mixture |
To use the model for your data, you need to create a simulator (see experiments/datasets), and add it to experiments/datasets/init.py. If you have problems with that, please don't hesitate to contact us. For the experiments on MNSIT, we used the original implementation of the Manifold Flow, see experiments/MNIST for the necessary additions. Everything necessary to re-plot the figures from the paper can be found in experiments/plotting.
The configs for the sphere can be found in experiments/configs. To train the model on a cluster, please consider experiments/cluster. Experiments on other manifolds can be adapted appropriate.
We thank Johann Brehmer and Kyle Cramner for publishing their implementation of the Manifold Flow. Our code is partially based on their implementation. As standard NF, we have mainly used the Block Neural Autoregressive Flow which was re-implemented here and served as inspiration for our main script. We thank the community for fostering the open-source culture.