Skip to content

Latest commit

 

History

History
592 lines (479 loc) · 60.5 KB

README.md

File metadata and controls

592 lines (479 loc) · 60.5 KB
Raw

awesome-deep-reinforcement-learning

Curated list for Deep Reinforcement Learning (DRL): software frameworks, models, datasets, gyms, baselines...

To accomplish this, includes general Machine Learning (ML), Neural Networks (NN) and Deep Neural Networks (DNN) with many vision examples, and Reinforcement Learning (RL) with videogames/robotics examples. Some alternative Evolutionary Algorithms (EA) with similar objectives included too.

General Machine Learning (ML)

General ML Software Frameworks

General ML Books

  • Jake VanderPlas, "Python Data Science Handbook", 2017. safari

Neural Networks (NN) and Deep Neural Networks (DNN)

NN/DNN Software Frameworks

  • Overview: presentation (permalink).
  • Docker images with several pre-installed software frameworks: 1, 2, 3.
  • Projects to port trained models from one software framework to another: 1

Attempling to order software frameworks by popularity (in practice should look at more aspects such as last updates, forks, etc):

NN/DNN Models

Image Object Segmentation, Localization, Detection Models

Overviews: 1. Taxonomy: 1.

Image Segmentation Models

Image Detection Models

  • YOLO (2015). Joseph Redmond et Al; U Washington, Allen AI, FAIR; "You Only Look Once: Unified, Real-Time Object Detection"; arxiv. Variants: YOLO9000, YOLO v3... Darknet, ports: tensorflow.
  • SSD (2015). Wei Liu et Al; UNC, Zoox, Google, et Al; "SSD: Single Shot MultiBox Detector"; arxiv. caffe
  • OverFeat (2015). Pierre Sermanet, David Eigen, Xiang Zhang, Michael Mathieu, Rob Fergus, and Yann LeCun; NYU; "OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks"; arxiv.
  • R-CNN (2013). Ross Girshick et Al; Berkeley; "Rich feature hierarchies for accurate object detection and semantic segmentation"; arxiv. Variants (summary): Fast R-CNN, Faster R-CNN, Mask R-CNN.

Image Classification Models

Overviews: 1, 2, 3

  • EfficientNets (2019). Mingxing Tan and Quoc V. Le; Google; "EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks"; arxiv.
  • MobileNets (2017). Andrew Howard et Al; Google; "MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications"; arxiv.
  • DenseNets (2017). Gao Huang et Al; "Densely Connected Convolutional Networks"; arxiv. torch includes links to ports.
  • ResNet (2015). Kaiming He et Al; Microsoft Research; "Deep Residual Learning for Image Recognition"; arxiv. Introduces "Residual Blocks" via "Skip Connections" (some cite similarities with GRUs), and additionally uses heavy batch normalization. Variants: ResNet50, ResNet101, ResNet152 (correspond to number of layers). 25.5 million parameters.
  • VGGNet (Sept 2014). Karen Simonyan, Andrew Zisserman; Visual Geometry Group (Oxford); "Very Deep Convolutional Networks for Large-Scale Image Recognition"; arxiv. Input: 224x224x3. Conv/pool and fully connected. Variants: VGG11, VGG13, VGG16, VGG19 (correspond to number of layers); with batch normalization. 138 million parameters; trained on 4 Titan Black GPUs for 2-3 weeks.
  • GoogLeNet/InceptionV1 (Sept 2014). Christian Szegedy et Al; Google, UNC; "Going Deeper with Convolutions"; arxiv. 22 layer deep CNN. Only 4-7 million parameters, via smaller convs. A more aggressive cropping approach than that of Krizhevsky. Batch normalization, image distortions, RMSprop. Uses 9 novel "Inception modules" (at each layer of a traditional ConvNet, you have to make a choice of whether to have a pooling operation or a conv operation as well as the choice of filter size; an Inception module performa all these operations in parallel), and no fully connected. Trained on CPU (estimated as weeks via GPU) implemented in DistBelief (closed-source predecessor of TensorFlow). Variants (summary): v1, v2, v4, resnet v1, resnet v2; v9 (slides). Also see Xception (2017) paper.
  • NIN (2013). Min Lin et Al; NUSingapore; "Network In Network"; arxiv. Provides inspiration for GoogLeNet.
  • ZFNet (2013). Matthew D Zeiler and Rob Fergus; NYU; "Visualizing and Understanding Convolutional Networks"; doi, arxiv. Similar to AlexNet, with well-justified finer tuning and visualization (namely Deconvolutional Network).
  • AlexNet (2012). Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton; SuperVision (UToronto); "ImageNet Classification with Deep Convolutional Neural Networks"; doi. In 224x224 (227x227?) color patches (and their horizontal reflections) from 256x256 color images; 5 conv, maxpool, 3 full; ReLU; SVD with momentum; dropout and data augmentation. 60-61 million parameters, split into 2 pipelines to enable 5-6 day GTX 580 GPU training (while CPU data augmentation).
  • LeNet-5 (1998). Yann LeCun et Al; ATT now at Facebook AI Research; "Gradient-based learning applied to document recognition"; doi. In 32x32 grayscale; 7 layer (conv, pool, full...). 60 thousand parameters.

Graph/Manifold/Network Convolutional Models

Generative Models

Tutorial: pytorch

  • Auto-Regressive Generative Models: PixelRNN, PixelCNN++... ref
  • Deep Dream. caffe
  • Style Transfer:
    • Tutorial: tensorflow
    • Fujun Luan et Al (2018), "Deep Painterly Harmonization"; arxiv. torch+matlab
    • Deep Photo Style Transfer (2017). Fujun Luan et Al, "Deep Photo Style Transfer"; arxiv. torch+matlab
    • Neuralart (2015). Leon A. Gatys et Al; "A Neural Algorithm of Artistic Style"; arxiv. Uses base+style+target as inputs and optimizes for target via BFGS. tensorflow, torch, keras 1 2 3 4
  • GANs:
    • hindupuravinash/the-gan-zoo
    • BigGAN (2018); "Large Scale GAN Training for High Fidelity Natural Image Synthesis"; arxiv. pytorch
    • Terro Karas et Al (2018); NVIDIA; "Progressive Growing of GANs for Improved Quality, Stability, and Variation"; arxiv. tensorflow
    • CANs (2017). Ahmed Elgammal et Al; Berkeley; "CAN: Creative Adversarial Networks, Generating "Art" by Learning About Styles and Deviating from Style Norms"; arxiv. tensorflow
    • CycleGAN (2017). Jun-Yan Zhu et Al; Berkeley; "Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks". torch and migrated to pytorch.
    • DCGAN (2015). Alec Radford, Luke Metz, Soumith Chintala; Indico Research, Facebook AI Research; "Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks"; arxiv.
    • GAN (2014). Ian J. Goodfellow et Al; Université de Montréal; "Generative Adversarial Nets"; arxiv.
  • Audio synthesis
    • FTTNet (2018). Zeyu Jin et Al; "FFTNet: a Real-Time Speaker-Dependent Neural Vocoder". pytorch
    • WaveNet (2016). Aäron van den Oord et Al; DeepMind; "WaveNet: A Generative Model for Raw Audio"; arxiv. wikipedia.

Recurrent Models

Can be trained via Back Propagation Through Time (BPTT). Also see Connectionist Temporal Classification (CTC). Cells include: SimpleRNN (commonly has TanH activation as second derivative decays slowly to 0), Gated Recurrent Units (GRU), Long short-term memory (LSTM), ConvLSTM2D, LSTM with peephole connection; keras.

  • Recurrent Neural Networks (RNN).
  • Bidirectional RNN.
  • Stateful RNN.

Word Embedding Models

  • BERT
  • ELMo
  • GloVe (2014). Jeffrey Pennington et Al; Stanford; "GloVe: Global Vectors for Word Representation".
  • word2vec (2013). Tomas Mikolov et Al; Google; "Distributed Representations of Words and Phrases and their Compositionality".

More Models

  • Regression Networks (essentialy same, remove last activation and use some loss such as MSE rather than binary/categorical cross-entropy).
  • Autoencoders (AE), Variational Autoencoders (VAE), Denoising Autoencoders.
    • Tutorials: keras, keras
    • Yunchen Pu et Al; "Variational Autoencoder for Deep Learning of Images, Labels and Captions"; arxiv.
  • Memory Networks. Use "Memory Units".
  • Capsule Networks. Use "Capsules". wikipedia
  • Echo-state networks.
  • Restricted Boltzmann Machine (RBM).
  • AutoML.

NN/DNN Datasets

Lists of lists before citing the classics:

Image Classification

  • MNIST: Handwritten digits, set of 70000 28x28 images, is a subset of a larger set available from NIST (and centered from its 32x32). Also see 2018's Kuzushiji-MNIST.
  • ImageNet: Project organized according to the WordNet hierarchy (22000 categories). Includes SIFT features, bounding boxes, attributes. Currently over 14 million images, 21841 cognitive synonyms (synsets) indexed, goal of +1000 images per synset.
    • ImageNet Large Visual Recognition Challenge (ILSVRC): Goal of 1000 categories using +100000 test images. E.g. LS-LOC
  • PASCAL VOC (Visual Object Classes)
  • CIFAR-10: 60000 32x32 colour images (selected from MIT TinyImages) in 10 classes, with 6000 images per class
  • CIFAR-100: 60000 32x32 colour images (selected from MIT TinyImages) in 100 classes containing 600 images per class, grouped into 20 superclasses
  • MIT MM Stimuli: Massive Memory (MM) Stimuli contains Unique Objects, State Pairs, State x Color Pairs...

Image Detection

  • SVHN (Street View House Numbers)
  • HICO (Humans Interacting with Common Objects)
  • Visual Genome: Includes structured image concepts to language

Image Segmentation

  • COCO (Common Objects in Context): 2014, 2015, 2017. Includes classes and annotations.

Motion

Text

Signal Separation

Swiping (mobile text entry gesture typing)

NN/DNN Benchmarks

NN/DNN Pretrained Models

NN/DNN Techniques Misc

  • Layers: Dense (aka Fully Connected), Convolutional (1D/2D/3D... keras, advanced: upsampling (e.g. in GANs), dilated causal (aka atrous)(e.g. in WaveNet)), Pooling (aka SubSampling)(1D/2D/3D)(Max, Average, Global Max, Global Average, Average with learnable weights per feature map... keras), Normalisation. Note: Keras implements activation functions, dropout, etc as layers.
  • Weight initialization: pretrained (see above section), zeros, ones, constant, normal random, uniform random, truncated normal, variance scaling, orthogonal, identity, normal/uniform as done by Yann LeCun, normal/uniform as done by Xavier Glorot, normal/uniform as done by Kaiming He. keras, StackExchange
  • Activation functions: Linear, Sigmoid, Hard Sigmoid, Logit, Hyperbolic tangent (TanH), SoftSign, Rectified Linear Unit (ReLU), Leaky ReLU (LeakyReLU or LReLU), Parametrized or Parametric ReLU (PReLU), Thresholded ReLU (Thresholded ReLU), Exponential Linear Unit (ELU), Scaled ELU (SELU), SoftPlus, SoftMax, Swish. wikipedia, keras, keras (advanced), ref.
  • Regularization techniques (reduce overfitting and/or control the complexity of model; may be applied to kernel (weight matrix), to bias vector, or to activity (activation of the layer output)): L1(lasso)/L2(ridge)/ElasticNet(L1/L2)/Maxnorm regularization (keras), dropout, batch and weight normalization, Local Response Normalisation (LRN), data augmentation (image distortions, scale jittering...), early stopping, gradient checking.
  • Optimizers: keras, ref
    • Gradient descent variants: Batch gradient descent, Stochastic gradient descent (SGD), Mini-batch gradient descent.
    • Gradient descent optimization algorithms: Momentum, Nesterov accelerated gradient, Adagrad, Adadelta, RMSprop, Adam, AdaMax, Nadam, AMSGrad, Eve.
    • Parallelizing and distributing SGD: Hogwild!, Downpour SGD, Delay-tolerant Algorithms for SGD, TensorFlow, Elastic Averaging SGD.
    • Additional strategies for optimizing SGD: Shuffling and Curriculum Learning, Batch normalization, Early Stopping, Gradient noise.
    • Broyden-Fletcher-Goldfarb-Shanno (BFGS)
    • Gradient-free: facebookresearch/nevergrad
  • Error/loss functions: keras
    • Accuracy used for classification problems: binary accuracy (mean accuracy rate across all predictions for binary classification problems), categorical accuracy (mean accuracy rate across all predictions for multiclass classification problems), sparse categorical accuracy (useful for sparse targets), top k categorical accuracy (success when the target class is within the top k predictions provided).
    • Error loss (measures the difference between the values predicted and the values actually observed, can be used for regression): mean square error (MSE), root square error (RMSE), mean absolute error (MAE), mean absolute percentage error (MAPE), mean squared logarithmic error (MSLE).
    • Hinge: hinge loss, squared hinge loss, categorical hinge.
    • Class loss, used to calculate the cross-entropy for classification problems: binary cross-entropy (binary classification), categorical cross-entropy (multi-class classification), sparse categorical cross-entropy. wikipedia
    • Logarithm of the hyperbolic cosine of the prediction error (logcosh), kullback leibler divergence, poisson, cosine proximity.
  • Metric functions: usually same type as error/loss functions, but used for evaluationg rather than training. keras
  • Cross-validation: hold-out, stratified k-fold. wikipedia.
  • Transfer learning. tensorflow, keras

NN/DNN Visualization and Explanation

Reinforcement Learning (RL) and Deep Reinforcement Learning (DRL)

RL/DRL Algorithms

RL/DRL algorithm classification adapted from Reinforcement Learning Specialization

Classification of RL algorithms adapted from Reinforcement Learning Specialization (Martha & Adam White, from University of Alberta and Alberta Machine Intelligence Institute, on Coursera, 2019-20). Note that another major separation is off/on policy RL algorithms. DRL methods would fit into function approximators.

+-- Tablular Methods
|   +-- Average Reward (e.g. for Continuing Tasks a.k.a. Infinite Horizon Case)
|   |   +-- Continuous Action Space
|   |   |   +-- Gaussian Actor-Critic
|   |   +-- Discrete Action Space
|   |       +-- Softmax Actor-Critic
|   |       +-- Differential Semi-Gradient SARSA
|   +-- Not using Average Reward (e.g. for Episodic Tasks a.k.a. Finite Horizon Case)
|       +-- Learn at each time step
|       |   +-- Control Problem
|       |   |   +-- Expected SARSA
|       |   |   +-- Q-Learning
|       |   |   +-- SARSA
|       |   +-- Not a Control Problem
|       |       +-- Semi-Gradient TD
|       +-- Not learn at each time step
|           +-- Gradient Monte Carlo
+-- Function Approximator Methods
    +-- Access to a model (model-based, part 1/2)
    |   +-- Control Problem
    |   |   +-- Value Iteration
    |   |   +-- Policy Iteration
    |   +-- Not a Control Problem
    |   |   +-- Iterative Policy Evaluation
    +-- No access to a model
        +-- Will learn a model (model-based, part 2/2)
        |   +-- Q-Planning
        |   +-- Dyna-Q+
        |   +-- Dyna-Q
        +-- Model-free
            +-- Learn at each time step
            |   +-- Control Problem
            |   |   +-- Q-Learning
            |   |   +-- Expected SARSA
            |   |   +-- SARSA
            |   +-- Not a Control Problem
            |       +-- TD
            +-- Not learn at each time step
                +-- Control Problem
                |   +-- eplsilon-soft Monte Carlo
                |   +-- Exploring starts Monte Carlo
                +-- Not a Control Problem
                    +-- Off-Policy Monte Carlo
                    +-- Monte Carlo Prediction

DRL algorithm classification adapted from CS285 at UC Berkeley

DRL algorithm classification adapted from Deep Reinforcement Learning CS 285 at UC Berkeley, Sergey Levine, Fall 2020, Lecture 4.

  1. Policy Gradients
  2. Value-based
  3. Actor-critic
  4. Model-based RL

RL/DRL algorithm classification from OpenAI Spinning Up

Just a random misc RL/DRL algorithms and techniques

REINFORCE (on-policy policy gradient; Williams, 1992), Deep Q-Network (DQN), Expected-SARSA, True Online Temporal-Difference (TD), Double DQN, Truncated Natural Policy Gradient (TNPG), Trust Region Policy Optimization (TRPO), Reward-Weighted Regression, Relative Entropy Policy Search (REPS), Cross Entropy Method (CEM), Advantage-Actor-Critic (A2C), Asynchronous Advantage Actor-Critic (A3C), Actor-critic with Experience Replay (ACER), Actor Critic using Kronecker-Factored Trust Region (ACKTR), Generative Adversarial Imitation Learning (GAIL), Hindsight Experience Replay (HER), Proximal Policy Optimization (PPO, PPO1, PPO2), Ape-X Distributed Prioritized Experience Replay, Continuous DQN (CDQN or NAF), Dueling network DQN (Dueling DQN), Deep SARSA, Multi-Agent Deep Deterministic Policy Gradient (MADDPG), Deep Deterministic Policy Gradient (DDPG), Truncated Quantile Critics (TQC), Quantile Regression DQN (QR-DQN).

Cool image

RL/DRL Algorithm Implementations and Software Frameworks

Attempting to order by popularity (in practice should look at more aspects such as last updates, forks, etc):

Lower level:

Specific to Model-based:

For specific algorithms (e.g. original paper implementations):

Tutorials/education (typically from lower level):

Comparison:

RL/DRL Environments

Multi-agent:

RL/DRL Benchmarking

With datasets for offline reinforcement learning:

With low-cost robots:

Reproducible:

Metrics/benchmarks:

Other:

  • deepmind/bsuite "collection of carefully-designed experiments that investigate core capabilities of a reinforcement learning (RL) agent" GitHub stars GitHub last commit

RL/DRL Books

RL

Evolutionary Algorithms (EA)

Only accounting those with same objective as RL.

Misc Tools

Similar pages