9_GAN_Harder.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 9. GAN\n",
    "This section will be an exercise. Surprisingly, you can build GAN fairly easily just by using the concepts we learned so far.\n",
    "\n",
    "## Preparation\n",
    "Read section 1, 2, 3 of the original [GAN paper](https://arxiv.org/pdf/1406.2661.pdf). Then, follow the next 7 steps to implement GAN.\n",
    "\n",
    "To summarie, we have the following problem setup:\n",
    "- $x$: data with distribution $p_{data}$\n",
    "- $p_g$: distribution trained by the generator\n",
    "- $z$: prior input noise variables\n",
    "- $p_z$: prior of $z$\n",
    "- $G(z;\\theta_G)$: generator neural network with parameter $\\theta_G$\n",
    "- $D(z;\\theta_D)$: discriminator neural network with parameter $\\theta_D$\n",
    "\n",
    "The goal for $D,G$ are the following:\n",
    "- $D$: $max_D V(D) = E_{x\\sim p_{data}}(x)[logD(x)] + E_{z\\sim p_z(z)}[log(1-D(G(z))]$\n",
    "- $G$: $min_G V(G) = E_{z\\sim p_z(z)}[log(1-D(G(z))]$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting MNIST_data/train-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/train-labels-idx1-ubyte.gz\n",
      "Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n"
     ]
    }
   ],
   "source": [
    "import matplotlib\n",
    "matplotlib.use('TkAgg')\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "\n",
    "# Import MNIST data\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "mnist = input_data.read_data_sets('MNIST_data', one_hot=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We pre-selected the hyper parameters for you this time. You usually need to tune this yourself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Training Params\n",
    "num_steps = 500000\n",
    "batch_size = 128\n",
    "learning_rate = 0.0002"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The hidden dimensions of the generator and the discriminator are also prespecified."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Network Params\n",
    "image_dim = 784 # 28*28 pixels\n",
    "gen_hidden_dim = 256\n",
    "disc_hidden_dim = 256\n",
    "noise_dim = 100 # Noise data points"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.reset_default_graph() # Clearing all tensors before this"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Implement generator and discriminator.\n",
    "\n",
    "Let both G and D be a 1-hidden layer fully connected neural network. Use ReLU for activation function for the hidden layer. For out layer, you should know what to use :)\n",
    "Since G and D are both 1-hidden layer fully connected NN, we need \n",
    "- weights for hidden layer and out layer\n",
    "- bias for hidden layer and out layer\n",
    "\n",
    "for each. Use `tf.layers.dense`.\n",
    "\n",
    "Becareful about the dimensions of the layers:\n",
    "- $G$ takes in noise and generates an image.\n",
    "- $D$ takes in an image and outputs a probability of the image being real."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Define the inputs to generator and discriminator.\n",
    "- Input to G: (batch size) $\\times$ ??\n",
    "- Input to D: (batch size) $\\times$ ??\n",
    "\n",
    "Think about what ?? should be."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Input noise to G and generate images.\n",
    "This should be a one linear."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Input real and fake images to D and get predictions.\n",
    "For D, you should have two inputs: real data and fake data. The latter is the output of $G$. For the latter, set `reuse=True`. I won't go into detail about it, but basically, you are reusing the samve variables in the above `discriminator` function and so you want to make them reusable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Define the objective.\n",
    "Expectation should be approximated using the sample mean. As a reminder, they are:\n",
    "- $D$: $max_D V(D) = E_{x\\sim p_{data}}(x)[logD(x)] + E_{z\\sim p_z(z)}[log(1-D(G(z))]$\n",
    "- $G$: $min_G V(G) = E_{z\\sim p_z(z)}[log(1-D(G(z))]$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Minimize (or maximize) the objective.\n",
    "Adam optimizer is recommended. We should have two optimizers for D and G. Be careful to only take the gradient with respect to the variables to optimize. Namely\n",
    "- $V(D)$: weights and biases of D\n",
    "- $V(G)$: weights and biases of G\n",
    "\n",
    "We provided the code for extracting these variables from the computation graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Train the model.\n",
    "For each iteration, take some batch of MNIST. Generate a prior noise $z$ by `np.random.uniform(-1., 1., size=[batch_size, noise_dim])`. Feed the batch data and prior noise to the model to update the objective.\n",
    "\n",
    "After some epochs of training, for each noise generated, get the output $x$ by the generator and plot it using matplotlib. This time we prepared the code for you but read through it to understand it. Then, change the variable names if they are different from yours."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7. [Optional] use TensorBoard to check the computation graph and loss.\n",
    "You might want to read about [variable sharing](https://www.tensorflow.org/versions/r1.1/programmers_guide/variable_scope) and [variable scope](https://stackoverflow.com/questions/35919020/whats-the-difference-of-name-scope-and-a-variable-scope-in-tensorflow).\n",
    "\n",
    "This might be a bit more involved than the previous steps..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, this was probably the hardest section so far since there's less hand holding. But if you could complete this exercise, this means that you can build reasonably sophisticated neural network models in TensorFlow! Look back and see how far you got :)\n",
    "\n",
    "Check [this](https://github.com/tensorflow/models/blob/master/research/gan/tutorial.ipynb) out if you're more interested in GANs.\n",
    "\n",
    "Thanks for completing this workshop. If you liked it, please `star` this repo, so that more and more people can learn about TensorFlow! Feedback is always welcome!\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:tensorflow_workshop]",
   "language": "python",
   "name": "conda-env-tensorflow_workshop-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}