Skip to content

Latest commit

 

History

History
82 lines (54 loc) · 1.72 KB

README.md

File metadata and controls

82 lines (54 loc) · 1.72 KB
Raw

Neural Network from scratch

It's really challenging!!!

I'm just feeling that: When neural network goes deep into code, you have to go back to mathematics.

Implement a neural network framework from scratch, and train with 2 examples:

  • MNIST Classifier
  • Time Series Prediction

Neural Network Framework

  • Back Propagation Algorithm
  • Model: Sequential(seq)
  • Layers: dense(fc), conv2d, flatten, maxpooling2d, dropout
  • Activations: relu, sigmoid, softmax
  • Optimizor & Training
    • Initial learning rate 0.001, set to 5e-5 after loss < 1.0
    • SGD(with momentum)
    • Gradients Clipping
    • l2 Regularization

Examples 1. MNIST Classifier

Netwrok Architecture

  • Input:(None, 28, 28, 1)
  • Conv2D(relu)
  • Flatten
  • Dropout(0.5)
  • Full-Connect(softmax)

Performance

85% accuracy after 25w samples with a simple network above.

Actually I had achieved 90% accuracy, but somehow I lost it. It's obviously that, complex model with high accuracy and low loss is harder to train, especially in Python.

Env

  • Ubuntu 16.04.2 LTS
  • Python 2.7.12(Python 3.6 Also works)

Scripts

  • train MNIST

python train_mnist.py

  • test MNIST

python test_mnist.py --weight ./weights/weights-252160-0.797545-0.8575

  • inference hand written digits

python inference_mnist.py --dir ./datasets/pics --weight ./weights/weights-252160-0.797545-0.8575

Examples 2. RNN Example

Training dataset goes like this:

     X             y
1 0 0 ... 0      1  0
0 0 0 ... 0      0  1
0 0 0 ... 0      0  1
1 0 0 ... 0      1  0
1 0 0 ... 0      1  0

Pattern is obviously, right?

Network Architecture

  • Input:(None, TIME_STEPS, INPUT_DIM)
  • RNN(RNN_UNITS)
  • Full-Connect(2, with softmax)

Scripts

python train_rnn.py