notes.org

Speed Optimizing

• Parallelize
• Use cuda with nvidia cuda compiler
  • https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html
  • https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html

Optimization Algorithms

• See Extensions and Variants (wikipedia)
• Adaptive Gradient Algorithm (AdaGrad)
• Root Mean Square Propagation (RMSProp)
• The algorithms that have a perparameter learning rate must have that rate saved to enable training on an already saved net, create a seperate cn_alloc_net_from_file function with this functionality