Notes on Deep Learning

from the Author

These are the notes that I left working through Nielson's neural Net and Deep Learning book. You can find a table of contents of this repo below.

Table of Contents

Chapter 1: Intro to Deep Learning

• 001 - sigmoid function
• 002 - training a single perceptron
• 003 - use perceptrons to target arbitrary function
• 004 - optimize batch training

Chapter 2: Intro to Tensorflow

• 005 - Tensorflow Intro
• 006 - Tensorflow Softmax Regression
• 007 - tensorflow API exploration

Chapter 3: Advanced Tensorflow with GPU AWS Instance and PyCharm Remote Interpreter.

• MNIST Logistic Regression
• MNIST Logistic Regression with L2 Regularization
• MNIST 1 Hidden Layer with Perceptron

Project 1: Doing Particle Simulation with Tensorflow

• Annealing A 2-Dimensional Electron Ensemble with Tensorflow

Fun Highlights (Reverse Chronological Order)

some of the figures can be found scattered in the folder (I believe in a flat folder structure).

Particle Simulation with Tensorflow! (classical many body simulation for my quantum computing research)

It turned out that not needing to write the Jacobian of your equations of motion is a huge time saver in doing particle simulations.

Here is a 2D classical many body simulator I wrote for my quantum computing research. In my lab, I am building a new type of qubits by traping single electrons on the surface of super fluild helium. You can read more about our progress in this paper from PRX.

In this new experiment, we want to construct a very small electro-static trap so that we can couple a microwave mirror to the dipole of a single electron. To understand where electrons are likely to go, I need to build a simple electro-static simulation.

link to repo

Projecting MNIST into a 2-Dimensional Deep Feature Space

It turned out that you can constrict the feature space of a convolutional neural network, and project the MNIST dataset onto a 2-dimensional plane!

This is my attempt at reproducing the work from Yandong Wei's paper (link see project readme (WIP)).

MNIST ConvNet with TensorFlow

My first attempt at building a convolutional neural network with tensorflow.

This example does:

• uses different GPUs for training and evaluation (manual device placement)
• persist network parameters in check files (session saving and restore)
• pushes loss and accuracy to summary, which can be visualized by tensorboard (summary and tensorboard)

A simple toy example

This one below shows how a simple network can be trained to emulate a given target function. Implemented with numpy without the help of tensorflow.

Todos (9/9/2016):

• MNIST Perceptron logging and visualization with tensorboard tensorboard doc [2.0]
• LeNet training ConvNet doc [1.0]
• LeNet++ training [1.0]
• Adversarial Hardened LeNet++ [1.0]
• Adversarial Test of Hardened LeNet++ [1.0]
• L2 Regularization with Logistic Regression [1.0]

More Deep Neural Net Learnings

• Feedforward Neural Network (Multilayer Perceptron)
• Deep Feedforward Neural Network (Multilayer Perceptron with 2 Hidden Layers O.o)
• Convolutional Neural Network
• Denoising Autoencoder
• Recurrent Neural Network (LSTM)
• Word2vec
• TensorBoard
• Save and restore net

Done:

• work on optimize batch training. (numpy neural net)
• add summary MNIST example with Tensorflow
• multi-GPU setup tensorflow doc [0.5 - 1.0]
• CFAR Example [4.0]

More Useful Links:

• Useful examples: @Aymericdamien's TensorFlow-Example
• More useful examples: @nlintz's TensorFlow-Tutorials