Day 4 - Autoencoders.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Autoencoders and Variational Autoencoders (VAEs)\n",
    "\n",
    "## Motivation\n",
    "\n",
    "- most real world data is unlabelled, and we would like to be able to learn from it and use it\n",
    "- a lot of data in inputs is not useful. How can we find the importance within it?\n",
    "\n",
    "When we see things, we don't consciously percieve the wavelength of light hitting each of our individual cells. This is the very high dimensional, raw input data that your eyes recieve, but this information is too complex for us to use for planning and decision making etc. Instead, what we do is perform abstraction from it - we build up more meaningful, general and vague descriptions of the environment.\n",
    "We percieve and would describe the scene using a few high level *features* such as: ' I see a face', 'It is smiling', 'It is nearby' rather than the millions of raw wavelengths. We somehow manage to map the high dimensional, low level wavelength data into much lower dimensional, high level concepts. The variables that we choose to represent the data in this lower dimensional space are called **latent variables**. \n",
    "\n",
    "## What is an autoencoder?\n",
    "\n",
    "An autoencoder is a unsupervised machine learning model that can learn a latent representation of input data, that is a lower dimensional higher level \n",
    "An autoencoder learns to do so by passing the input through a neural network which attempts to output the same thing (or slightly different) as it is given as an input, whilst passing the data through a low dimensional bottleneck in between. We can determine a loss by taking the BCE loss between the input and its reconstruction at the output. This means that training requires no labels!\n",
    "\n",
    "<img src=\"autoencoder.png\">\n",
    "\n",
    "A lower dimensional representation is much more easily interpretable than the higher dimensional input, as there is just less information to comprehend.\n",
    "Having a trained autoencoder means that we have found a way to represent our data in a lower dimension with minimal loss.\n",
    "Our inputs can be reconstructed \n",
    "\n",
    "Note that \n",
    "- the reconstruction of the input will be lossy, as information must be thrown away to compress the input down to a lower dimension\n",
    "- the weights learned by the model are specific to the data it has been trained on. There is nothing preventing  a lot of latent space never being sampled for training. As such, the latent space may be sparse and can not be sampled from to produce an outpput. This prevents autoencoders being used as generative models.\n",
    "- the diverging part of the model can not robustly be used to generate data, because the latent space may be filled with gaps where that model has not learnt to reconstruct any data from. \n",
    "- the latent representations for similar inouts/inputs may not be nearby in the latent space.\n",
    "\n",
    "## (Beta) Variational Autoencoders\n",
    "\n",
    "Variational autencoders are an upgrade on autoencoders. They ensure that the latent space is filled, and that similar outputs occupy nearby regions in the latent space. They do this by minimising the difference between how the latent variables are distributed compared wth some prior (initially expected) distribution, $p(z)$. We usually take our prior distribution over $z$ to be a unit Gaussian - a reasonable assumption, as lots of things in real life are distributed normally.\n",
    "So each latent variable now is forced to be distributed normally around zero, with unit variance. This \n",
    "\n",
    "## Beta Variational Autencoders\n",
    "\n",
    "So for VAEs, we have two terms in the loss function: the reconstruction loss which makes sure that we get back a similar image to our input. And the KL-divergence, which ensures that the latent space which a realistic output can be generated from is continuous, and that similar examples are nearby in the latent space. \n",
    "Beta-VAEs put another hyperparamete (Beta) "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}