Final_Report .ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MACHINE LEARNING CAPSTONE PROJECT:  \n",
    "### TOXIC COMMENT CLASSIFICATION \n",
    "\n",
    "1. Import necessary files <br/>\n",
    "2. Read the train.csv file <br/>\n",
    "3. List the various fields in our dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(95851, 8)\n"
     ]
    }
   ],
   "source": [
    "#Read the csv file into dataframe df\n",
    "df = pd.read_csv(\"train.csv\")\n",
    "print(df.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "id                int64\n",
      "comment_text     object\n",
      "toxic             int64\n",
      "severe_toxic      int64\n",
      "obscene           int64\n",
      "threat            int64\n",
      "insult            int64\n",
      "identity_hate     int64\n",
      "dtype: object\n"
     ]
    }
   ],
   "source": [
    "#List the fields in our dataframe\n",
    "print(df.dtypes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hence we have a sufficiently large dataset consistly of 95851 samples. Each sample contains 8 fields. <br/>\n",
    "**It was observed that running train_test_split on the heavy preprocessed dataframe sometimes resulted in system going out of memory. Hence to avoid such cases, one extra line of code was added. The df.reindex code will shuffle the indices initially, so that later splitting dataset into training and testing will give fairer results.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# below line causes shuffling of indices, to avoid using train_test_split later\n",
    "df = df.reindex(np.random.permutation(df.index))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Separate the comment field data and outcome labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0    Nonsense?  kiss off, geek. what I said is true...\n",
      "1    \"\\n\\n Please do not vandalize pages, as you di...\n",
      "2    \"\\n\\n \"\"Points of interest\"\" \\n\\nI removed the...\n",
      "3    Asking some his nationality is a Racial offenc...\n",
      "4    The reader here is not going by my say so for ...\n",
      "Name: comment_text, dtype: object\n"
     ]
    }
   ],
   "source": [
    "comment = df['comment_text']\n",
    "print(comment.head())\n",
    "comment = comment.as_matrix()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   toxic  severe_toxic  obscene  threat  insult  identity_hate\n",
      "0      1             0        0       0       0              0\n",
      "1      0             0        0       0       0              0\n",
      "2      0             0        0       0       0              0\n",
      "3      0             0        0       0       0              0\n",
      "4      0             0        0       0       0              0\n"
     ]
    }
   ],
   "source": [
    "label = df[['toxic', 'severe_toxic' , 'obscene' , 'threat' , 'insult' , 'identity_hate']]\n",
    "print(label.head())\n",
    "label = label.as_matrix()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let us find out the frequency of occurence of multilabelled data \n",
    "- ct1 counts samples having atleast one label\n",
    "- ct2 counts samples having 2 or more than 2 labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "9790\n",
      "5957\n"
     ]
    }
   ],
   "source": [
    "ct1,ct2 = 0,0\n",
    "for i in range(label.shape[0]):\n",
    "    ct = np.count_nonzero(label[i])\n",
    "    if ct :\n",
    "        ct1 = ct1+1\n",
    "    if ct>1 :\n",
    "        ct2 = ct2+1\n",
    "print(ct1)\n",
    "print(ct2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Visualisations\n",
    "### Let us analyse the no. of comments having lengths varying from 0 to 1200"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "average length of comment: 395.342\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZ4AAAEKCAYAAAAiizNaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3X+0HWV97/H3h4QfEYEESHNpAiVIGi9CRUAIBe1RNARqCXUhQl0SuJT0FrRY6a2h/ohCvUpbtNJaJEtiggv5UQSJAsYY2dYfi5+ChAAxBxBJLhBKIHBAkMD3/jHfDUM4OWdOcvbss3c+r7Vm7ZnvPDP7edhwvszMM8+jiMDMzKwuW7W7AmZmtmVx4jEzs1o58ZiZWa2ceMzMrFZOPGZmVisnHjMzq1VLE4+kMyXdLWm5pI9lbGdJSyStzM9xGZekCyT1SrpL0gGl88zK8islzSrFD5S0LI+5QJJa2R4zM9t8LUs8kvYFTgMOBt4KvE/S3sAcYGlETAGW5jbAUcCUXGYDF+Z5dgbmAofkueY2k1WWOa103IxWtcfMzIZHK694/idwc0Q8FxHrgR8D7wdmAguzzELg2FyfCVwShZuAsZJ2A44ElkTE2oh4ElgCzMh9O0bETVG8BXtJ6VxmZjZCjW7hue8GPi9pF+C3wNHAbcCEiHgkyzwKTMj1icDDpeNXZWyg+Kp+4q8jaTbFVRTbbbfdgXvsscemt2oEe/nll9lqq+59bOf2dTa3r3P96le/+u+IGD9c52tZ4omIeyWdB/wAeBa4E3hpgzIhqeVj9kTEPGAewNSpU2PFihWt/sq2aDQa9PT0tLsaLeP2dTa3r3NJemg4z9fS9BwRF0fEgRHxTuBJ4FfAY3mbjPxck8VXA7uXDp+UsYHik/qJm5nZCNbqXm2/l597UDzf+RawCGj2TJsFXJvri4CTsnfbNGBd3pJbDEyXNC47FUwHFue+pyVNy95sJ5XOZWZmI1Qrn/EAfDuf8bwInBERT0n6InClpFOBh4Djs+z1FM+BeoHngFMAImKtpHOBW7PcORGxNtdPBxYAY4AbcjEzsxGspYknIt7RT+wJ4Ih+4gGcsZHzzAfm9xO/Ddh382tqZmZ16c4uGGZmNmI58ZiZWa2ceMzMrFZOPGZmVisnHjMzq5UTj5mZ1cqJx8zMauXEY2ZmtXLiMTOzWjnxmJlZrZx4zMysVk48ZmZWKyceMzOrlROPmZnVyonHzMxq1eoZSP9W0nJJd0u6TNJ2kiZLullSr6QrJG2TZbfN7d7cv2fpPGdnfIWkI0vxGRnrlTSnlW0xM7Ph0bLEI2ki8DfAQRGxLzAKOAE4D/hyROwNPAmcmoecCjyZ8S9nOSTtk8e9BZgB/IekUZJGAV8FjgL2AU7MsmZmNoK1+lbbaGCMpNHAG4BHgHcDV+X+hcCxuT4zt8n9R0hSxi+PiBci4kGKqbEPzqU3Ih6IiN8Bl2dZMzMbwVqWeCJiNfAvwG8oEs464HbgqYhYn8VWARNzfSLwcB67PsvvUo5vcMzG4mZmNoKNbtWJJY2juAKZDDwF/CfFrbLaSZoNzAYYP348jUajHdVoub6+vq5tG7h9nc7ts6aWJR7gPcCDEfE4gKSrgcOAsZJG51XNJGB1ll8N7A6syltzOwFPlOJN5WM2Fn+NiJgHzAOYOnVq9PT0bHbjRqJGo0G3tg3cvk7n9llTK5/x/AaYJukN+azmCOAe4EbguCwzC7g21xflNrn/RxERGT8he71NBqYAtwC3AlOyl9w2FB0QFrWwPWZmNgxadsUTETdLugr4BbAeuIPiquM64HJJ/5ixi/OQi4FvSuoF1lIkEiJiuaQrKZLWeuCMiHgJQNJHgMUUPebmR8TyVrXHzMyGRytvtRERc4G5G4QfoOiRtmHZ54EPbOQ8nwc+30/8euD6za+pmZnVxSMXmJlZrZx4zMysVk48ZmZWKyceMzOrlROPmZnVyonHzMxq5cRjZma1cuIxM7NaOfGYmVmtnHjMzKxWTjxmZlYrJx4zM6uVE4+ZmdXKicfMzGrlxGNmZrVy4jEzs1q1LPFImirpztLytKSPSdpZ0hJJK/NzXJaXpAsk9Uq6S9IBpXPNyvIrJc0qxQ+UtCyPuSCn2DYzsxGsZYknIlZExP4RsT9wIPAccA0wB1gaEVOApbkNcBQwJZfZwIUAknammMX0EIqZS+c2k1WWOa103IxWtcfMzIZHXbfajgDuj4iHgJnAwowvBI7N9ZnAJVG4CRgraTfgSGBJRKyNiCeBJcCM3LdjRNwUEQFcUjqXmZmNUKNr+p4TgMtyfUJEPJLrjwITcn0i8HDpmFUZGyi+qp/460iaTXEVxfjx42k0GpvajhGtr6+va9sGbl+nc/usqeWJR9I2wDHA2Rvui4iQFK2uQ0TMA+YBTJ06NXp6elr9lW3RaDTo1raB29fp3D5rquNW21HALyLisdx+LG+TkZ9rMr4a2L103KSMDRSf1E/czMxGsDoSz4m8epsNYBHQ7Jk2C7i2FD8pe7dNA9blLbnFwHRJ47JTwXRgce57WtK07M12UulcZmY2QrX0Vpuk7YH3An9VCn8RuFLSqcBDwPEZvx44Guil6AF3CkBErJV0LnBrljsnItbm+unAAmAMcEMuZmY2grU08UTEs8AuG8SeoOjltmHZAM7YyHnmA/P7id8G7DsslTUzs1p45AIzM6uVE4+ZmdVq0MQj6QOSdsj1T0m6ujycjZmZ2VBUueL5dEQ8I+lw4D3AxeRwNmZmZkNVJfG8lJ9/CsyLiOuAbVpXJTMz62ZVEs9qSRcBHwSul7RtxePMzMxep0oCOZ7iJc4jI+IpYGfg/7S0VmZm1rWqJJ6LIuLqiFgJkCMGfLi11TIzs25VJfG8pbwhaRTF/DpmZmZDttHEI+lsSc8Af5Szhz6d22vwmGhmZraJNpp4IuILEbED8M8RsWMuO0TELhHxuikOzMzMqhh0rLaIOFvSROAPyuUj4r9aWTEzM+tOgyYeSV+kmEH0Hl59pycAJx4zMxuyKqNT/zkwNSJeaHVlzMys+1Xp1fYAsHWrK2JmZluGKonnOeBOSRdJuqC5VDm5pLGSrpJ0n6R7JR0qaWdJSyStzM9xWVZ57l5Jd5UHIpU0K8uvlDSrFD9Q0rI85oKcidTMzEawKolnEXAu8HPg9tJSxVeA70fEm4G3AvcCc4ClETEFWJrbAEcBU3KZTQ5EKmlnYC5wCHAwMLeZrLLMaaXjZlSsl5mZtUmVXm0LJY0B9oiIFVVPLGkn4J3AyXme3wG/kzQT6MliC4EG8AlgJnBJzkR6U14t7ZZllzSnu5a0BJghqQHsGBE3ZfwS4Fg8/bWZ2YhWpVfbnwH/QjEi9WRJ+wPnRMQxgxw6GXgc+Iakt1JcJZ0JTMhhdwAeBSbk+kTg4dLxqzI2UHxVP/H+2jCb4iqK8ePH02g0Bql6Z+rr6+vatoHb1+ncPmuq0qvtsxS3uBoAEXGnpL0qnvsA4KMRcbOkr/DqbTXyXCEphlTjTRAR84B5AFOnTo2enp5Wf2VbNBoNurVt4PZ1OrfPmqo843kxItZtEHu5wnGrgFURcXNuX0WRiB7LW2jk55rcvxrYvXT8pIwNFJ/UT9zMzEawKolnuaS/AEZJmiLp3yg6GgwoIh4FHpY0NUNHULyEugho9kybxavjvi0CTsrebdOAdXlLbjEwXdK47FQwHVic+56WNC17s52Ex5AzMxvxqtxq+yjwSeAF4DKKRHBuxfN/FLhU0jYU7wOdQpHsrpR0KvAQxXw/ANcDRwO9FF24TwGIiLWSzgVuzXLnNDsaAKcDC4AxFJ0K3LHAzGyEq9Kr7TmKxPPJoZ48Iu4EDupn1xH9lA3gjI2cZz4wv5/4bcC+Q62XmZm1T5VebQcB/wDsyWsHCf2j1lXLzMy6VZVbbZdSTHW9jGqdCszMzDaqSuJ5PCIWtbwmZma2RaiSeOZK+jrF8DavjFAdEVe3rFZmZta1qiSeU4A3U4xQ3bzVFoATj5mZDVmVxPP2iJg6eDEzM7PBVXmB9OeS9ml5TczMbItQ5YpnGsV8PA9SPOMRxWs37k5tZmZDViXxeI4bMzMbNlVGLngox0jbfYPyD7WsVmZm1rWqjFxwLsVkbvdT9GYjP9/dumqZmVm3qnKr7XjgTTmDqJmZ2Wap0qvtbmBsqytiZmZbhipXPF8A7pB0N68duWCwqa/NzMxep0riWQichwcJNTOzYVDlVttzEXFBRNwYET9uLlVOLunXkpZJulPSbRnbWdISSSvzc1zGJekCSb2S7pJ0QOk8s7L8SkmzSvED8/y9eayG2H4zM6tZlcTzE0lfkHSopAOayxC+410RsX9ENCeEmwMsjYgpFAOPzsn4UcCUXGYDF0KRqIC5wCHAwRSDlo7LYy4ETisd53eOzMxGuCq32t6Wn9NKsc3pTj0T6Mn1hUAD+ETGL8mZSG+SNFbSbll2SXO6a0lLgBmSGsCOEXFTxi8BjsXTX5uZjWhVXiB912acP4AfSArgooiYB0yIiEdy/6PAhFyfCDxcOnZVxgaKr+on/jqSZlNcRTF+/HgajcZmNGnk6uvr69q2gdvX6dw+a6ryAulOFLe63pmhHwPnRMS6Cuc/PCJWS/o9YImk+8o7IyIyKbVUJrx5AFOnTo2enp5Wf2VbNBoNurVt4PZ1OrfPmqo845kPPEPxIunxwNPAN6qcPCJW5+ca4BqKZzSP5S008nNNFl9NMSxP06SMDRSf1E/czMxGsCqJ500RMTciHsjlc8Begx0kaXtJOzTXgekUL6MuApo902YB1+b6IuCk7N02DViXt+QWA9MljctOBdOBxbnvaUnTsjfbSaVzmZnZCFWlc8FvJR0eET8FkHQY8NsKx00ArskezqOBb0XE9yXdClwp6VSKgUaPz/LXA0cDvcBzFDOfEhFrc7y4W7PcOc2OBsDpwAJgDEWnAncsMDMb4aoknr8GFuazHoAnKQYNHVBEPAC8tZ/4E8AR/cQDOGMj55pPcctvw/htwL6D1cXMzEaOKr3a7gTeKmnH3H665bUyM7OuVaVX2/8F/ikinsrtccBZEfGpVleuFX774kvsOee6dlejJRbM2L7dVTAzG1SVzgVHNZMOQEQ8SfEsxszMbMiqJJ5RkrZtbkgaA2w7QHkzM7ONqtK54FJgqaTmuzunUAx1Y2ZmNmRVOhecJ+mXwHsydG5ELG5ttczMrFtVueIhIr4PfL/FdTEzsy1AlWc8ZmZmw8aJx8zMarXRxCNpaX6eV191zMys2w30jGc3SX8MHCPpcuA100pHxC9aWjMzM+tKAyWezwCfpphu4Esb7NucGUjNzGwLttHEExFXAVdJ+nREnFtjnczMrItVeY/nXEnH8OoMpI2I+F5rq2VmZt1q0F5tkr4AnAnck8uZOXComZnZkFXpTv2nwHsjYn7OizMDeF/VL5A0StIdkr6X25Ml3SypV9IVkrbJ+La53Zv79yyd4+yMr5B0ZCk+I2O9kuZUrZOZmbVP1fd4xpbWd9poqf6dCdxb2j4P+HJE7E0xqdypGT8VeDLjX85ySNoHOAF4C0XS+49MZqOArwJHAfsAJ2ZZMzMbwaokni8Ad0haIGkhcDvw+SonlzSJ4orp67ktit5wV2WRhcCxuT6TVwcfvQo4IsvPBC6PiBci4kGKqbEPzqU3Ih6IiN8Bl2dZMzMbwap0LrhMUgN4e4Y+ERGPVjz/vwJ/D+yQ27sAT0XE+txeBUzM9YnAw/md6yWty/ITgZtK5ywf8/AG8UP6q4Sk2cBsgF13Hc9n9lvfX7GO19fXR6PRaHc1Wsbt62xunzVVHST0EWDRUE4s6X3Amoi4XVLPJtRt2ETEPGAewB577R3nL6vU7I6zYMb29PT0tLsaLdNoNNy+Dub2WVMr/wIfRjHqwdHAdsCOwFeAsZJG51XPJGB1ll8N7A6skjSa4lnSE6V4U/mYjcXNzGyEatkgoRFxdkRMiog9KToH/CgiPgTcCByXxWYB1+b6otwm9/8oIiLjJ2Svt8nAFOAW4FZgSvaS2ya/Y0hXZWZmVr8Br3iy59jyiHjzMH7nJ4DLJf0jcAdwccYvBr4pqRdYS5FIiIjlkq6keIdoPXBGRLyU9fsIsBgYBcyPiOXDWE8zM2uBARNPRLyU78nsERG/2dQviYgG0Mj1Byh6pG1Y5nngAxs5/vP005MuIq4Hrt/UepmZWf2qPOMZByyXdAvwbDMYEce0rFZmZta1qiSeT7e8FmZmtsWo8h7PjyX9ATAlIn4o6Q0Uz1TMzMyGrMogoadRjCRwUYYmAt9pZaXMzKx7VelOfQbFOzlPA0TESuD3WlkpMzPrXlUSzws5FhoA+XJntK5KZmbWzaoknh9L+gdgjKT3Av8JfLe11TIzs25VJfHMAR4HlgF/RfHezKdaWSkzM+teVXq1vZzTIdxMcYttRQ5lY2ZmNmSDJh5Jfwp8DbgfEDBZ0l9FxA2trpyZmXWfKi+Qng+8KyJ6ASS9CbgOcOIxM7Mhq/KM55lm0kkPAM+0qD5mZtblNnrFI+n9uXqbpOuBKyme8XyAYkoCMzOzIRvoVtufldYfA/4k1x8HxrSsRmZm1tU2mngi4pQ6K2JmZluGKmO1TZb0JUlXS1rUXCoct52kWyT9UtJySZ8rne9mSb2SrsjZQ8kZRq/I+M2S9iyd6+yMr5B0ZCk+I2O9kuZsyj8AMzOrV5Vebd+hmB30u8DLQzj3C8C7I6JP0tbATyXdAHwc+HJEXC7pa8CpwIX5+WRE7C3pBOA84IOS9qGYjfQtwO8DP5T0h/kdXwXeC6wCbpW0KCLuGUIdzcysZlUSz/MRccFQT5wvmfbl5ta5BPBu4C8yvhD4LEXimZnrUIyG/e+SlPHLI+IF4MGcGrs5g2lvzmiKpMuzrBOPmdkIViXxfEXSXOAHFFcxAETELwY7UNIo4HZgb4qrk/uBpyJifRZZRTHNAvn5cJ57vaR1wC4Zv6l02vIxD28QP2Qj9ZgNzAbYddfxfGa/9f0V63h9fX00Go12V6Nl3L7O5vZZU5XEsx/wYYorleattuaVy4Ai4iVgf0ljgWuAN29iPTdLRMwD5gHssdfecf6yKs3uPAtmbE9PT0+7q9EyjUbD7etgbp81VfkL/AFgr/LUCEMVEU9JuhE4FBgraXRe9UwCVmex1cDuwKqcemEn4IlSvKl8zMbiZmY2QlUZueBuYOxQTyxpfF7pIGkMRSeAe4EbgeOy2Czg2lxflNvk/h/lc6JFwAnZ620yMAW4heIl1inZS24big4Ig/a2MzOz9qpyxTMWuE/Srbz2Gc8xgxy3G7Awn/NsBVwZEd+TdA9wuaR/BO6g6DFHfn4zOw+spUgkRMRySVdSdBpYD5yRt/CQ9BFgMTAKmB8Ry6s02szM2qdK4pm7KSeOiLuAt/UTf4BXe6WV489T3Nbr71yfBz7fT/x6ivmBzMysQ1SZj+fHdVTEzMy2DFXm43mGohcbwDYU7+M8GxE7trJiZmbWnapc8ezQXC+90DmtlZUyM7PuVaVX2yui8B3gyEELm5mZ9aPKrbb3lza3Ag4Cnm9ZjczMrKtV6dVWnpdnPfBritttZmZmQ1blGY/n5TEzs2Ez0NTXnxnguIiIc1tQHzMz63IDXfE8209se4p5c3YBnHjMzGzIBpr6+vzmuqQdgDOBU4DLgfM3dpyZmdlABnzGI2lnihlDP0QxadsBEfFkHRUzM7PuNNAznn8G3k8xj81+EdG3sbJmZmZVDXTFcxbFaNSfAj5ZDFoAgCg6F3jInBFm2ep1nDznunZXo2UWzNi+3VUws2Ew0DOeIY1qYGZmVoWTi5mZ1apliUfS7pJulHSPpOWSzsz4zpKWSFqZn+MyLkkXSOqVdJekA0rnmpXlV0qaVYofKGlZHnOBSvcDzcxsZGrlFc964KyI2IdiNOszJO0DzAGWRsQUYGluAxxFMa31FGA2cCG80rNuLnAIxQRyc5vJKsucVjpuRgvbY2Zmw6BliSciHomIX+T6M8C9wESKcd4WZrGFwLG5PhO4JEfAvgkYK2k3ipGwl0TE2uzKvQSYkft2jIibIiKAS0rnMjOzEarKIKGbTdKeFNNg3wxMiIhHctejwIRcnwg8XDpsVcYGiq/qJ97f98+muIpi113H85n91m96Y0awCWPgrC5tG0BfXx+NRqPd1WgZt6+zdXv7hlPLE4+kNwLfBj4WEU+XH8NEREiKjR48TCJiHsX7SOyx195x/rJa8m3tztpvPd3aNii6U/f09LS7Gi3TaDTcvg7W7e0bTi3t1SZpa4qkc2lEXJ3hx/I2Gfm5JuOrgd1Lh0/K2EDxSf3EzcxsBGtlrzYBFwP3RsSXSrsWAc2eabOAa0vxk7J32zRgXd6SWwxMlzQuOxVMBxbnvqclTcvvOql0LjMzG6FaeV/mMODDwDJJd2bsH4AvAldKOhV4CDg+910PHA30As9RDEhKRKyVdC5wa5Y7JyLW5vrpwAJgDHBDLmZmNoK1LPFExE8phtfpzxH9lA/gjI2caz4wv5/4bcC+m1FNMzOrmUcuMDOzWjnxmJlZrZx4zMysVk48ZmZWKyceMzOrlROPmZnVyonHzMxq5cRjZma1cuIxM7NaOfGYmVmtnHjMzKxWTjxmZlYrJx4zM6uVE4+ZmdXKicfMzGrVyhlI50taI+nuUmxnSUskrczPcRmXpAsk9Uq6S9IBpWNmZfmVkmaV4gdKWpbHXJCzkJqZ2QjXyiueBcCMDWJzgKURMQVYmtsARwFTcpkNXAhFogLmAocABwNzm8kqy5xWOm7D7zIzsxGoZYknIv4LWLtBeCawMNcXAseW4pdE4SZgrKTdgCOBJRGxNiKeBJYAM3LfjhFxU85ceknpXGZmNoLV/YxnQkQ8kuuPAhNyfSLwcKncqowNFF/VT9zMzEa40e364ogISVHHd0maTXELj113Hc9n9ltfx9fWbsIYOKtL2wawZu06/u3Sa9tdjZaZvNMoGo1Gu6vRMn19fW6fAfUnnsck7RYRj+TtsjUZXw3sXio3KWOrgZ4N4o2MT+qnfL8iYh4wD2CPvfaO85e1Ld+21Fn7radb2wbd374FM7anp6en3dVomUaj4fYZUP+ttkVAs2faLODaUvyk7N02DViXt+QWA9MljctOBdOBxbnvaUnTsjfbSaVzmZnZCNay/32UdBnF1cquklZR9E77InClpFOBh4Djs/j1wNFAL/AccApARKyVdC5wa5Y7JyKaHRZOp+g5Nwa4IRczMxvhWpZ4IuLEjew6op+yAZyxkfPMB+b3E78N2Hdz6mhmZvXzyAVmZlYrJx4zM6uVE4+ZmdXKicfMzGrlxGNmZrVy4jEzs1p172vgZh1m2ep1nDznunZXo2UWzNi+3VWwEcJXPGZmVisnHjMzq5UTj5mZ1cqJx8zMauXOBWZWC3eesCZf8ZiZWa2ceMzMrFa+1WZmNgy6/VbicPIVj5mZ1arjE4+kGZJWSOqVNKfd9TEzs4F1dOKRNAr4KnAUsA9woqR92lsrMzMbSEcnHuBgoDciHoiI3wGXAzPbXCczMxuAIqLdddhkko4DZkTEX+b2h4FDIuIjG5SbDczOzX2Bu2utaH12Bf673ZVoIbevs7l9nWtqROwwXCfbInq1RcQ8YB6ApNsi4qA2V6klurlt4PZ1Orevc0m6bTjP1+m32lYDu5e2J2XMzMxGqE5PPLcCUyRNlrQNcAKwqM11MjOzAXT0rbaIWC/pI8BiYBQwPyKWD3LYvNbXrG26uW3g9nU6t69zDWvbOrpzgZmZdZ5Ov9VmZmYdxonHzMxqtcUknm4YWkfS7pJulHSPpOWSzsz4zpKWSFqZn+MyLkkXZJvvknRAe1swOEmjJN0h6Xu5PVnSzdmGK7ITCZK2ze3e3L9nO+tdhaSxkq6SdJ+keyUd2mW/3d/mv5d3S7pM0nad/PtJmi9pjaS7S7Eh/16SZmX5lZJmtaMt/dlI+/45//28S9I1ksaW9p2d7Vsh6chSfOh/WyOi6xeKjgf3A3sB2wC/BPZpd702oR27AQfk+g7AryiGCvonYE7G5wDn5frRwA2AgGnAze1uQ4U2fhz4FvC93L4SOCHXvwb8da6fDnwt108Armh33Su0bSHwl7m+DTC2W347YCLwIDCm9Lud3Mm/H/BO4ADg7lJsSL8XsDPwQH6Oy/Vx7W7bAO2bDozO9fNK7dsn/25uC0zOv6ejNvVva9sbX9M/4EOBxaXts4Gz212vYWjXtcB7gRXAbhnbDViR6xcBJ5bKv1JuJC4U72EtBd4NfC//I/7v0n8Ir/yOFD0ZD8310VlO7W7DAG3bKf8wa4N4t/x2E4GH8w/s6Pz9juz03w/Yc4M/zEP6vYATgYtK8deUa/eyYfs22PfnwKW5/pq/mc3fb1P/tm4pt9qa/1E0rcpYx8pbE28DbgYmRMQjuetRYEKud1q7/xX4e+Dl3N4FeCoi1ud2uf6vtC33r8vyI9Vk4HHgG3kr8euStqdLfruIWA38C/Ab4BGK3+N2uuf3axrq79VRv+MG/hfFVRwMc/u2lMTTVSS9Efg28LGIeLq8L4r/7ei4PvKS3gesiYjb212XFhlNcVvjwoh4G/Asxa2aV3TqbweQzzpmUiTY3we2B2a0tVIt1sm/12AkfRJYD1zaivNvKYmna4bWkbQ1RdK5NCKuzvBjknbL/bsBazLeSe0+DDhG0q8pRhl/N/AVYKyk5ovO5fq/0rbcvxPwRJ0VHqJVwKqIuDm3r6JIRN3w2wG8B3gwIh6PiBeBqyl+0275/ZqG+nt12u+IpJOB9wEfyuQKw9y+LSXxdMXQOpIEXAzcGxFfKu1aBDR7y8yiePbTjJ+UPW6mAetKtwlGlIg4OyImRcSeFL/PjyLiQ8CNwHFZbMO2Ndt8XJYfsf/3GRGPAg9LmpqhI4B76ILfLv0GmCbpDfnvabN9XfH7lQz191oMTJc0Lq8Kp2dsRJI0g+J29zER8Vxp1yLghOyNOBmYAtzCpv5tbffDrRofoh1N0QvsfuCT7a7PJrbhcIpL+7uAO3M5muLe+FJgJfBDYOcsL4qJ8u4HlgEHtbsNFdvZw6u92vbKf8F7gf8Ets34drndm/vadv7NAAAEUklEQVT3ane9K7Rrf+C2/P2+Q9HLqWt+O+BzwH0U0458k6IHVMf+fsBlFM+rXqS4Yj11U34vimclvbmc0u52DdK+XopnNs2/L18rlf9ktm8FcFQpPuS/rR4yx8zMarWl3GozM7MRwonHzMxq5cRjZma1cuIxM7NaOfGYmVmtnHis40nqa/H5T5b0+6XtX0vadTPOd1mO/vu3w1PD+knaX9LR7a6HdaaOnvrarCYnU7yb8v8290SS/gfw9ojYe3PP1Wb7AwcB17e7ItZ5fMVjXUnSeEnflnRrLodl/LM5D0lD0gOS/qZ0zKdzXpGf5lXJ30k6juIP7KWS7pQ0Jot/VNIvJC2T9OZ+vn87Sd/I/XdIelfu+gEwMc/1jg2OmZBzoPwylz/O+MdVzHFzt6SPZWzPnDdlgaRfSbpU0nsk/SznfTm41N6Fkn4i6SFJ75f0T1mv7+cQTEg6UNKPJd0uaXFpWJiGpPMk3ZLf8458Q/0c4IPZjg9K+pNcvzPbu8Ow/ZjWfdr99qwXL5u7AH39xL4FHJ7re1AMMwTwWeDnFG/V70oxPtjWwNsp3tTejmKuo5XA3+UxDV77JvqvgY/m+unA1/v5/rOA+bn+ZoohZbZj4GHor6AY+BWKeU52Ag6keBN+e+CNwHKKUcn3pBjEcT+K/4G8HZhP8Qb9TOA7pfb+NNv4VuA58q1z4Brg2Nz3c2B8xj9YqnsDOD/XjwZ+mOsnA/9eqvt3gcNy/Y3kVAhevPS3+Fabdav3APsUw4YBsKOKUb0BrouIF4AXJK2hGNr+MODaiHgeeF7Sdwc5f3OA1tuB9/ez/3Dg3wAi4j5JDwF/CDzdT9mmdwMn5TEvAeskHQ5cExHPAki6GngHxXhYD0bEsowvB5ZGREhaRpGYmm6IiBczPgr4fsab5aYC+wJL8p/XKIqhVPpra/m8ZT8DviTpUuDqiFg1QDttC+fEY91qK2BaJpJX5B/WF0qhl9i0/w6a59jU44dDuR0vl7Zf5rV1egEgIl6W9GJExAblBCyPiEMH+Z6NtjUivijpOoqrop9JOjIi7htqg2zL4Gc81q1+AHy0uSFp/0HK/wz4s3w280aKYeGbnqG4/TYUPwE+lN/9hxS3+1YMcsxS4K/zmFGSdsrzHJujPm9PMSvkT4ZYl8GsAMZLOjS/e2tJbxnkmNf8M5H0pohYFhHnUYxY/LrnXmZNTjzWDd4gaVVp+TjwN8BB2W35HuB/D3SCiLiV4vbVXRSzLi6jmBUTYAHwtQ06FwzmP4Ct8vbWFcDJeXtvIGcC78pjbqeYu/4X+f23UMw2+/WIuKNiHSqJiN9RTE1wnqRfUjzr+uNBDruR4lbmnZI+CHwsOz/cRTHa8Q0DH25bMo9ObZYkvTEi+iS9AfgvYHb+4TezYeRnPGavmidpH4reZwuddMxaw1c8ZmZWKz/jMTOzWjnxmJlZrZx4zMysVk48ZmZWKyceMzOr1f8HUobU69x9hKsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10fc59b38>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = [len(comment[i]) for i in range(comment.shape[0])]\n",
    "\n",
    "print('average length of comment: {:.3f}'.format(sum(x)/len(x)) )\n",
    "bins = [1,200,400,600,800,1000,1200]\n",
    "plt.hist(x, bins=bins)\n",
    "plt.xlabel('Length of comments')\n",
    "plt.ylabel('Number of comments')       \n",
    "plt.axis([0, 1200, 0, 90000])\n",
    "plt.grid(True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Number of comments classified as toxic,severe_toxic,....etc depending on their lengths"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZcAAAEKCAYAAADenhiQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3Xucl3P+//HHyySD0kmblP3V2ozOUynpQAcqfSMsclibZNu1DmsPqaxjWDllaZGWbGwxNiKHRYtZOaRUQympCCVKpZoSHV6/P673jE81h8/U5zMzn0/P++123ea63tfp9Z7P9Hl1Xdf7er/N3REREUmk/So6ABERST9KLiIiknBKLiIiknBKLiIiknBKLiIiknBKLiIiknBJTS5m9gcz+9DM5pvZ42aWaWaNzexdM1tiZjlmVjVse0BYXhLWN4o5zohQvsjMeiczZhER2XtJSy5m1gC4AjjG3VsAGcA5wG3A3e7+c2AdMDjsMhhYF8rvDtthZs3Cfs2BPsD9ZpaRrLhFRGTvJfu2WBXgQDOrAhwErAR6AJPD+gnAaWG+f1gmrO9pZhbKn3D37939U2AJ0CHJcYuIyF6okqwDu/sKM7sT+Bz4DngFmA186+7bwmbLgQZhvgHwRdh3m5mtB+qE8hkxh47dp5CZDQGGAGRmZrb76U9/mvA6VRY7duxgv/3S93GZ6pfa0rl+6Vw3gI8//vgbd6+biGMlLbmYWS2iq47GwLfAv4luayWFu48DxgFkZWX5okWLknWqCpebm0u3bt0qOoykUf1SWzrXL53rBmBmnyXqWMlMwScCn7r7anffCjwNdAZqhttkAA2BFWF+BXAEQFhfA1gTW17EPiIiUgklM7l8DnQ0s4PCs5OewALgdeDMsM1A4NkwPzUsE9a/5lGvmlOBc0JrssZAE2BmEuMWEZG9lMxnLu+a2WRgDrANmEt02+oF4AkzuzmUPRx2eRh4zMyWAGuJWojh7h+a2ZNEiWkbcKm7b09W3CIisveSllwA3P164Ppdij+hiNZe7r4FOKuY49wC3JLwAEUk6bZu3cry5cvZsmVLRYey12rUqMHChQsrOoy9lpmZScOGDdl///2Tdo6kJhcRkeXLl1O9enUaNWpEdIc8dW3cuJHq1atXdBh7xd1Zs2YNy5cvp3Hjxkk7T/q2qRORSmHLli3UqVMn5RNLujAz6tSpk/QrSSUXEUk6JZbKpTw+DyUXERFJOCUXESlfZomd4vDtt99y//3371G47733HldcccUe7bsvU3IRkbS3N8nlmGOO4d57701wROlPyUVE0t7w4cNZunQp2dnZDB06lKFDh9KiRQtatmxJTk4OAFOmTKFnz564OytXruSoo47iq6++Ijc3l379+gGQn5/PoEGDaNmyJa1ateKpp56qyGpVamqKLCJpb9SoUcyfP5+8vDyeeuopxo4dy/vvv88333xD+/btOf744zn99NN56qmnuO+++3jppZe48cYbOeyww/joo48Kj3P77bdTo0YN5s2bB8C6desqqkqVnq5cRGSf8uabb3LuueeSkZFBvXr1OOGEE5g1axYAY8aM4dZbb+WAAw7g3HPP3W3f3NxcLr300sLlWrVqlVvcqUbJRUQkWL58Ofvttx9ff/01O3bsqOhwUpqSi4ikverVq7Nx40YAunbtSk5ODtu3b2f16tW88cYbdOjQgW3btnHRRRfx+OOP07RpU0aPHr3bcbp37859991XuKzbYsVTchGR8uWe2CkOderUoXPnzrRo0YJ33nmHVq1a0bp1a3r06MHtt9/OYYcdxl//+le6du1Kly5dGD16NA899NBu/YgNHTqUdevW0aJFC1q3bs3rr7+ejN9QWtADfRHZJ0yaNGmn5TvuuGOn5euuu65wvnr16oUP8ps2bVo4QFi1atWYMGECUjpduYiISMIpuYiISMIpuYiISMIpuYiISMIpuYiISMIlLbmYWZaZ5cVMG8zsSjOrbWbTzGxx+FkrbG9mdq+ZLTGzD8ysbcyxBobtF5vZwGTFLCIiiZG0psjuvgjIBjCzDGAFMAUYDrzq7qPMbHhYHgacDDQJ07HAA8CxZlYbuB44BnBgtplNdXe9vSSSguzGxA5U5dfH966LlK/yui3WE1jq7p8B/YGChuITgNPCfH/gUY/MAGqaWX2gNzDN3deGhDIN6FNOcYuIJE1ubi5vv/32Hu9/3XXX8d///jeBESVOeb1EeQ7weJiv5+4rw/xXQL0w3wD4Imaf5aGsuHIRkUpl27ZtVKkS/9dqbm4u1apVo1OnTnt0vpEjR+7RfuUh6cnFzKoCpwIjdl3n7m5mCbmmNbMhwBCAunXrkpubm4jDVkr5+fmqXwrb1+pXo0aNwn69kiGeY2/atImBAwfy5Zdfsn37dq666ip+9rOfcfXVV7Np0yZq167N2LFj2bBhA0OGDCmM/7PPPmPAgAHMmDGDuXPnMmLECDZv3ly4/WGHHUbfvn1p2bIlM2bM4Mwzz+Tcc8/lyiuv5Isvov8T33bbbXTs2HG3mD777DMeeOABMjIyePTRR7njjjto0KABl156KWvWrOHQQw/l/vvv54gjjuCcc87h1FNP5bzzzmP8+PG89dZbPPzww/z2t7+lT58+nHbaacyePZthw4axefNmqlatynPPPUf16tWL/Z1s2bIlqX+H5XHlcjIwx92/Dstfm1l9d18ZbnutCuUrgCNi9msYylYA3XYpz931JO4+DhgHkJWV5QXdNaSj3NxcVL/Uta/Vb+HChSV+ye2teI79yiuv8NOf/pSXX34ZgPXr13PyySfz7LPPUrduXXJycrj11lsZP34827dv55tvvqFx48a88MILnHvuuWRmZjJ8+HAmTpxI48aNd9o+IyMDgDlz5gBw3nnnMXToULp06cLnn39O7969d+ujDKBFixZccsklVKtWjT//+c8AnHLKKVx00UUMHDiQ8ePHc/XVV/PMM88wfvx4OnfuTLNmzbjvvvuYMWMG1atXZ//99+fAAw/kgAMO4KKLLiInJ4f27duzYcMGDjrooBKvojIzM2nTpk2Zf9/xKo/kci4/3hIDmAoMBEaFn8/GlF9mZk8QPdBfHxLQy8BfC1qVAb0o4ipIRKQ4LVu25E9/+hPDhg2jX79+1KpVi/nz53PSSScBsH37durXrw/A2WefTU5ODsOHDycnJ4ecnBwWLVrE/Pnz6d+/P/vtt99O2wMMGDCgcP6///0vCxYsKFzesGED+fn5VKtWrdQ433nnHZ5++mkALrjgAq666ioA6tWrx8iRI+nevTtTpkyhdu3aO+23aNEi6tevT/v27QE45JBD9uTXlFBJTS5mdjBwEvCbmOJRwJNmNhj4DDg7lL8I9AWWAJuBQQDuvtbMbgJmhe1GuvvaZMYtIunlqKOOYs6cObz44otcc8019OjRg+bNm/POO+/stu2AAQM466yzOOOMMzAzmjRpwrx582jevDmvvPJKkVdKBx98cOH8jh07mDFjBpmZmQmtw7x586hTpw5ffvllQo+bLElNLu6+CaizS9kaotZju27rwKW7lod144HxyYhRRMpXRTQd/vLLL6lduza//OUvqVmzJvfffz+rV6/mnXfe4bjjjmPr1q18/PHHNG/enCOPPJKMjAxuuummwiuSrKwsVq9ezbvvvsuJJ5640/a76tWrF2PGjGHo0KEA5OXlkZ2dXWRc1atXZ8OGDYXLnTp14oknnuCCCy5g4sSJdO3aFYCZM2fyn//8h7lz53LCCSfQq1cvGjduXLhfVlYWK1euZNasWbRv356NGzdy4IEHlqlxQaLpDX0RSXvz5s2jQ4cOZGdnc+ONNzJy5EgmT57MsGHDaN26NdnZ2Ts1CR4wYAD/+te/OPvs6MZK1apVmTx5Mtdff32R28e69957ee+992jVqhXNmjVj7NixxcZ1yimnMGXKFLKzs5k+fTpjxozhkUceoVWrVjz22GPcc889fP/99/z6179m/PjxHH744dx1111cdNFFeMxYNlWrViUnJ4fLL7+c1q1bc9JJJ7Fly5YE/fb2jHmcg+2kkqysLF+0aFFFh5E0+9oD4XSzr9Vv4cKFNG3atOICSqCNGzcmtXFCeSrqczGz2e5+TCKOrysXERFJOI1EKSKSZI888gj33HPPTmWdO3fmvvvuq6CIkk/JRUQkyQYNGsSgQYMqOoxypdtiIiKScEouIiKScEouIiKScEouIlKuzBI77ally5bRokWLxFVMdqLkIiIiCafkIiL7hNGjR9OiRQtatGjB3/72NyAaf+X888+nadOmnHnmmWzevBmA4cOH06xZM1q1alXYY/HXX3/NeeedR+vWrWndunXhG/r/+te/Ct/+/81vfsP27dsBqFatGn/5y19o3bo1HTt25Ouvo47hV69ezS9+8Qvat29P+/bteeutt8r7V1EulFxEJO3Nnj2bRx55hHfffZcZM2bwj3/8g3Xr1rFo0SJ+97vfsXDhQg455BDuv/9+1qxZw5QpU/jwww/54IMPuOaaawC44oor6Ny5M++//z5z5syhefPmLFy4kJycHN566y3y8vLIyMhg4sSJQDSGTMeOHXn//fc5/vjj+cc//gHA73//e/7whz8wa9YsnnrqKS6++OIK+70kk95zEZG09+abb3L66acX9l58xhlnMH36dI444gg6d+4MwC9/+UvuvfderrzySjIzMxk8eDD9+vWjX79+ALz22muFLz1mZGRQo0YNHnvsMWbPnl3Y1f13333HT37yEyDq76tg33bt2jFt2jRg77rkTyVKLiKyz7JdWgSYGVWqVGHmzJm8+uqrTJ48mb///e+89tprRe7v7gwcOJBbb711t3X7779/4fEzMjLYtm0bkLwu+Ssb3RYTkbTXtWtXnnnmGTZv3symTZuYMmUKXbt25fPPPy8c02XSpEl06dKF/Px81q9fT9++fbn77rt5//33AejZsycPPfQQEA0utn79enr27MnkyZNZtSoaUHft2rV89tlnJcZS0CV/gby8vGRUucIpuYhIuXJP7BSPtm3bcuGFF9KhQweOPfZYLr74YmrVqkVWVhb33XcfTZs2Zd26dVxyySVs3LiRfv360apVK7p06cLo0aMBuOeee5g+fTotW7akXbt2LFiwgGbNmnHzzTfTq1cvWrVqxUknncTKlStLjKUsXfKnMnW5n4L2tS7b082+Vj91uV85qct9ERFJOfvWA/3SXudNw6s4EZGKkNQrFzOraWaTzewjM1toZseZWW0zm2Zmi8PPWmFbM7N7zWyJmX1gZm1jjjMwbL/YzAYmM2YREdl7yb4tdg/wkrsfDbQGFgLDgVfdvQnwalgGOBloEqYhwAMAZlYbuB44FugAXF+QkEREpHJKWnIxsxrA8cDDAO7+g7t/C/QHJoTNJgCnhfn+wKMemQHUNLP6QG9gmruvdfd1wDSgT7LiFhGRvZfMZy6NgdXAI2bWGpgN/B6o5+4FbfW+AuqF+QbAFzH7Lw9lxZXvxMyGEF3xULduXXJzc3eP6M47S464qH0qofz8/KLrlyZUv9S2a/1q1KjBxo0bKy6gBNq+fXva1GXLli1J/TtMZnKpArQFLnf3d83sHn68BQaAu7uZJeQpuruPA8ZB1BS5yKae3buXdpBEhJJ0+1pT1nSzr9Vv4cKFuzTf3Yt+8otU+r/bb7/9lkmTJvG73/2O3Nxc7rzzTp5//vkyn6m0psi5ublUrVqVTp06lfnY5S0zM5M2bdok7fil3hYzs7PMrHqYv8bMno592F6C5cByd383LE8mSjZfh9tdhJ+rwvoVwBEx+zcMZcWVi4jE5dtvv+X+++8v0z4FvRuXRW5ubmFvyfu6eJ65XOvuG82sC3Ai0TOUB0rbyd2/Ar4ws6xQ1BNYAEwFClp8DQSeDfNTgV+FVmMdgfXh9tnLQC8zqxUe5PcKZSIicRk+fDhLly4lOzuboUOHkp+fz5lnnsnRRx/N+eefT8HL5I0aNWLYsGG0bduWf//73yxdupQ+ffrQrl07unbtyscffwzAc889x7HHHkubNm048cQT+frrr1m2bBljx47l7rvvJjs7m+nTp1dklStcPLfFCtL3/wHj3P0FM7s5zuNfDkw0s6rAJ8AgooT2pJkNBj4Dzg7bvgj0BZYAm8O2uPtaM7sJmBW2G+nua+M8v4gIo0aNYv78+eTl5ZGbm0v//v358MMPOfzww+ncuTNvvfUWXbp0AaBOnTrMmTMHiPoTGzt2LE2aNOHdd9/lj3/8I//73//o0qULM2bMwMx46KGHuP3227nrrrv47W9/S7Vq1QrHgNmXxZNcVpjZg8BJwG1mdgBxtjJz9zygqK4EehaxrQOXFnOc8cD4eM4pIlKaDh060LBhQwCys7NZtmxZYXIZMGAAEDVMePvttznrrLMK9/vuu+8AWL58OQMGDGDlypX88MMPNG7cuJxrUPnFkyTOJroN1Ts0Ja4NDE1qVCIiSXTAAQcUzsd2hw8UjvmyY8cOatasSV5eXuH03nvvAXD55Zdz2WWXMW/ePB588EG2bNlSvhVIAfEklwfd/Wl3XwwQnoNckNywREQSp3r16mVuQnzIIYfQuHFj/v3vfwPR2C3z5s0DYP369TRoEL0RMWHChMJ99uQ86Sqe5NI8dsHMMoB2yQlHRNKfJ3gqXZ06dejcuTMtWrRg6ND4b7xMnDiRhx9+mNatW9O8eXNeeOEFAG644QbOOuss2rVrx6GHHlq4/SmnnMKUKVP0QJ8SnrmY2QjgauBAM9tQUAz8QHifREQkVUyaNKnI8r///e+F88uWLdtpXePGjXnppZcKlwuuSvr370///v13O9ZRRx3FBx98kIBoU1+xVy7ufqu7VwfucPdDwlTd3eu4+4hyjFFERFJMqa3F3H2EmTUA/l/s9u7+RjIDExGR1FVqcjGzUcA5RC9AFrzz4oCSi4iIFCme91xOB7Lc/ftkByMiIukhntZinwD7JzsQERFJH/FcuWwG8szsVaDw6sXdr0haVCIiktLiSS5TwyQisvcmJbjL/fNKf9elU6dOCe2teNmyZfTr16+wv7Ivv/ySvn37Juz46SCe1mITzOxA4KfuvqgcYhIRSahkdoNf0C2MksvO4hnP5RQgD3gpLGebma5kRCRlVKtWDfhxILOiutsfPnw4zZo1o1WrVoW9Gl944YVMnjy58Dj169ff6bg//PAD1113HTk5OWRnZ5OTk1NONar84rktdgPQAciFqKdjM/tZEmMSEUmauXPn7tbdftOmTZkyZQofffQRZsa3334b17GqVq3KyJEjee+993Z601/iay221d3X71K2IxnBiIgkW0F3+/vtt19hd/s1atQgMzOTwYMH8/TTT3PQQQdVdJgpL57k8qGZnQdkmFkTMxsDaBxPEUlJRXW3X6VKFWbOnMmZZ57J888/T58+fQCoUqUKO3ZE/5fesWMHP/zwQ4XEnIriSS6XE/WM/D3wOLABuDKZQYmIlKf8/HzWr19P3759ufvuu3n//feBaNjj2bNnAzB16lS2bt26277qZr9o8bQW2wz8JUwiInsnjqbD5W3jxo3079+fLVu24O6MHj0agF//+tf079+f1q1b06dPn8KBxGJ1796dUaNGkZ2dzYgRIwpHstzXxdO32DFEXe83YueOK1vFse8yYCNRn2Tb3P0YM6sN5ITjLQPOdvd1ZmbAPUBfohc3L3T3OeE4A4FrwmFvdvcJiIjEKT8/H4Bu3brRrVu3wvLYh/AzZ87cbb969eoxY8aMwuVrrom+hho1asT8+fMBqF27NrNmzUpG2CktntZiE4mGNZ7Hnj3I7+7u38QsDwdedfdRZjY8LA8DTgaahOlY4AHg2JCMrgeOIeowc7aZTXX3dXsQi4iIlIN4kstqd0/key39gW5hfgJRE+dhofxRjxqdzzCzmmZWP2w7zd3XApjZNKAP0fMfERGphOJJLteb2UPArn2LPR3Hvg68YmYOPOju44B67r4yrP8KqBfmGwBfxOy7PJQVV74TMxsCDAGoW7cuubm5u0dz550lR1vUPpVQfn5+0fVLE6pfatu1fjVq1EibB97bt29Pm7ps2bIlqX+H8SSXQcDRRD0jF9wWcyCe5NLF3VeY2U+AaWb2UexKd/eQePZaSFzjALKysjz2vmqh7t1LO0giQkm6greM05Xql9p2rd/ChQupXr16xQWUQBs3bkybumRmZtKmTZukHT+e5NLe3bP25ODuviL8XGVmU4je9P/azOq7+8pw22tV2HwFcETM7g1D2Qp+vI1WUJ67J/GIiEj5iOc9l7fNrFlZD2xmB5tZ9YJ5oBcwn6iH5YFhs4HAs2F+KvAri3QE1ofbZy8DvcyslpnVCsd5uazxiIhI+YnnyqUj0XgunxI9czGiO1qlNUWuB0yJWhhTBZjk7i+Z2SzgSTMbDHwGnB22f5GoGfISoqbIg4hOtNbMbgIK2vqNLHi4LyKp5wYS2+X+DaTG7ex9TTxXLn2Imgf3Ak4B+oWfJXL3T9y9dZiau/stoXyNu/d09ybufmJBovDIpe5+pLu3dPf3Yo413t1/HqZH9qSiIrLv6tSpU5Hlu/Z6XBZ5eXm8+OKLhctTp05l1KhRADzzzDMsWLBgj47bqFEjvvnmm9I3LCaOyqLU5OLunxF1+VIDqBMziYikhGSM57Lrl/qpp57K8OHDgb1LLnsbR2URz3guNwEfAPcCd4WplDa9IiKVR8F4Lu7OZZddRlZWFieeeCKrVq0q3Gb27NmccMIJtGvXjt69e7NyZfTGRLdu3Rg2bBgdOnSgTZs2TJ8+vchxXP75z39y2WWX8fbbbzN16lSGDh1KdnY2S5cupW3btoXnWbx48U7LRRkzZgxt27alZcuWfPRR1Mh25syZHHfccbRp04ZOnTqxaNGiIuPYtGkTF110UWG8zz77bInnSpZ4boudDRzp7t3cvXuYeiQ7MBGRRJsyZQqLFi1iwYIFPProo4VXNFu3buXyyy9n8uTJzJ49m4suuoi//OXH7hS3bdvGzJkzGTVqFDfeeGPhOC4DBgwgLy9vp/7EOnXqxKmnnsodd9xBXl4eRx55JDVq1CAvLw+ARx55hEGDBpUY56GHHsqcOXO45JJLuDO8n3f00Uczffp05s6dy8iRI7n66quLjOOWW26hR48ezJw5k9dff52hQ4eyadOmRP8qSxXPA/35QE1+bDIsIpKS3njjDc4991wyMjI4/PDD6dEj+n/yokWLmD9/PieddBIQvSwZO+rkGWecAUCbNm1YtmxZmc978cUX88gjjzB69GhycnKK7McsVsH52rVrx9NPR68Url+/noEDB7J48WLMrMgemgFeeeUVpk6dWpiUtmzZwueff07Tpk3LHPfeiCe53ArMNbP57PyG/qlJi0pEpBy5O82bN+edd94pcn3BGDAF47+U1S9+8QtuvPFGevToQbt27ahTp+TH1kWd79prr6V79+5MmTKFZcuWFfsirrvz1FNPkZW1R68nJkw8yWUCcBt73nGliEihimw6fPzxx/Pggw8ycOBAVq1axeuvv855551HVlYWq1ev5p133uG4445j69atfPzxxzRv3rzYY5U0jsuu6zIzM+nduzeXXHIJDz/88B7Fvn79eho0iHq++uc//1nsuXr37s2YMWMYM2YMZsbcuXOT+iZ+ceJ55rLZ3e9199fd/X8FU9IjExFJsNNPP50mTZrQrFkzfvWrX3HccccBULVqVSZPnsywYcNo3bo12dnZpbYw6969OwsWLCh8kB7rnHPO4Y477qBNmzYsXboUgPPPP5/99tuPXr167VHsV111FSNGjKBNmzY7XT3tGse1117L1q1badWqFc2bN+faa6/do/PtLfNS+tMys9FEt8OmsvNtsTnJDW3PZWVl+aJFi3ZfYaW8vKW+xSoF1S+1FdW3WHnf70+Wvelb7M4772T9+vXcdNNNCY5qzxT1uZjZbHc/JhHHj+e2WMH1VMeYMgfUYkxEJA6nn346S5cu5bXXXqvoUMpNPMMcl9KVsIiIlGTKlCm7lZ1++ul8+umnO5Xddttt9O7du7zCSqp4hjmuQTQS5PGh6H9E/XutT2ZgIpI+3B0r7bb0PqaohFNeSnsckgjxPNAfD2wkepnybKKuYNS/l4jEJTMzkzVr1pTLF5qUzt1Zs2YNmZmZST1PPM9cjnT3X8Qs32hmeckKSETSS8OGDVm+fDmrV6+u6FD22pYtW5L+pVweMjMzadiwYVLPEU9y+c7Murj7mwBm1hn4LqlRiUja2H///WncuHFFh5EQubm5FfLOSCqKJ7lcAkwIz14A1gEXJi0iERFJefG0FssDWpvZIWF5Q9KjEhGRlBZPl/t/NbOa7r7B3TeE4YZvLo/gREQkNcXTWuxkd/+2YMHd1xENRywiIlKkeJJLhpkdULBgZgcCB5Sw/U7MLMPM5prZ82G5sZm9a2ZLzCzHzKqG8gPC8pKwvlHMMUaE8kVmlh5vGImIpLF4kstE4FUzG2xmg4FpRD0lx+v3wMKY5duAu93950SNAwaH8sHAulB+d9gOM2sGnAM0B/oA95tZRhnOLyIi5azU5OLutwE3A03DdJO73x7Pwc2sIfB/wENh2Yj6JJscNpkAnBbm+/Nj0poM9Azb9weecPfv3f1TYAnQIZ7zi4hIxYinKTLu/hLw0h4c/2/AVUBBN6J1gG/dvaC/6OVAgzDfAPginG+bma0P2zcAZsQcM3afQmY2BBgCULduXXJzc3ePJozMVqyi9qmE8vPzi65fmlD9Uls61y+d65ZocSWXPWFm/YBV7j7bzLol6zwF3H0cMA6iLveL7NK8eyl9cKZI9xT7Wpft6Ub1S13pXLdES1pyAToDp5pZXyATOAS4B6hpZlXC1UtDYEXYfgVwBLDczKoANYA1MeUFYvcREZFKqNhnLmb2avh5254c2N1HuHtDd29E9ED+NXc/H3gdODNsNhB4NsxPDcuE9a951NPdVOCc0JqsMdAEmLknMYmISPko6cqlvpl1Irr6eALYqb/svRiJchjwRHgRcy5QMKD0w8BjZrYEWEuUkHD3D83sSWABsA241N237+G5RUSkHJSUXK4DriW6DTV6l3VlGonS3XOB3DBAGiTwAAAT30lEQVT/CUW09nL3LcBZxex/C3BLvOcTEZGKVWxycffJwGQzu9bdK8egzyIikhLi6bjyJjM7lR9Hosx19+eTG5aIiKSyeDquvJXoLfsFYfq9mf012YGJiEjqiqcp8v8B2e6+A8DMJhA9iL86mYGJiEjqiqdvMYCaMfM1it1KRESE+K5cbgXmmtnrRM2RjweGJzUqERFJafE80H/czHKB9qFomLt/ldSoREQkpcXbceVKojfl91lmxa9LkS7JRETKTbzPXEREROKm5CIiIglXYnIJQxR/VF7BiIhIeigxuYQOIheZ2U/LKR4REUkD8TzQrwV8aGYzgU0Fhe5+atKiEhGRlBZPcrk26VGku0klNDU7T03NRCT9xPOey//M7P8BTdz9v2Z2EJCR/NBERCRVxdNx5a+BycCDoagB8EwygxIRkdQWT1PkS4HOwAYAd18M/CSZQYmISGqLJ7l87+4/FCyYWRWikShFRESKFE9y+Z+ZXQ0caGYnAf8GnittJzPLNLOZZva+mX1oZjeG8sZm9q6ZLTGzHDOrGsoPCMtLwvpGMccaEcoXmVnvPamoiIiUn3iSy3BgNTAP+A3wInBNHPt9D/Rw99ZANtDHzDoCtwF3u/vPgXXA4LD9YGBdKL87bIeZNQPOAZoDfYD7zUwNCkREKrFSk0sYJGwCcBNwIzDBvfSuGj2SHxb3D5MDPYgaCBCOe1qY7x+WCet7mpmF8ifc/Xt3/xRYAnSIo24iIlJBSm2KbGb/B4wFlhKN59LYzH7j7v+JY98MYDbwc+C+cIxv3X1b2GQ5Ueszws8vANx9m5mtB+qE8hkxh43dJ/ZcQ4AhAHXr1iU3N3f3gO68s+SAi9onjl1L2C2SuTc77y4/P7/o+qUJ1S+1pXP90rluiRbPS5R3Ad3dfQmAmR0JvACUmlxC9zHZZlYTmAIcvRexlnauccA4gKysLO/WrdvuG3XvXtpBil1V0q6lXsdNKmHnbmVvG5Gbm0uR9UsTql9qS+f6pXPdEi2eZy4bCxJL8AmwsSwncfdvgdeB44CaocUZQENgRZhfARwBhS3SagBrYsuL2EdERCqhYpOLmZ1hZmcA75nZi2Z2oZkNJGopNqu0A5tZ3XDFgpkdCJwELCRKMmeGzQYCz4b5qWGZsP618GxnKnBOaE3WGGgCzCxjPUVEpByVdFvslJj5r4ETwvxq4MA4jl0fmBCeu+wHPOnuz5vZAuAJM7sZmAs8HLZ/GHjMzJYAa4laiOHuH5rZk8ACYBtwabjdJiIilVSxycXdB+3Ngd39A6BNEeWfUERrL3ffApxVzLFuAW7Zm3hERKT8xNNarDFwOdAodnt1uS8iIsWJp7XYM0S3rJ4DdiQ3HBERSQfxJJct7n5v0iMREZG0EU9yucfMrgdeIerSBQB3n5O0qEREJKXFk1xaAhcQddtScFusoBsXERGR3cSTXM4Cfhbb7b6IiEhJ4nlDfz5QM9mBiIhI+ojnyqUm8JGZzWLnZy5qiiwiIkWKJ7lcn/QoREQkrZSaXNz9f+URiIiIpI943tDfSNQ6DKAq0aBfm9z9kGQGJiIiqSueK5fqBfMxI0N2TGZQIiKS2uJpLVYoDF38DNA7SfGIiEgaiOe22Bkxi/sBxwBbkhaRiIikvHhai8WO67INWEZ0a0xERKRI8Txz2atxXUREZN9TbHIxs+tK2M/d/aYkxCMiImmgpCuXTUWUHQwMBuoASi4iIlKkYluLuftdBRMwDjgQGAQ8AfystAOb2RFm9rqZLTCzD83s96G8tplNM7PF4WetUG5mdq+ZLTGzD8ysbcyxBobtF5vZwL2ss4iIJFmJTZFDIrgZ+IDoKqetuw9z91VxHHsb8Cd3b0b0XsylZtYMGA686u5NgFfDMsDJQJMwDQEeKIiBqAuaY4EOwPUFCUlERCqnYpOLmd0BzAI2Ai3d/QZ3Xxfvgd19ZcGAYu6+EVgINCBqaTYhbDYBOC3M9wceDe/SzABqmll9ondqprn72nD+aUCfslRSRETKl7l70SvMdhD1gryNH7t/ATCiB/pxd/9iZo2AN4AWwOfuXjOUG7DO3Wua2fPAKHd/M6x7FRgGdAMy3f3mUH4t8J2737nLOYYQXfFQt27ddk8++eTugcyeXXKg7doVu6qkXUvYLbK2hJ1rl7bz7vLz86lWrVqZ90sVql9qS+f6pXPdALp37z7b3Y9JxLGKfaDv7mV6e784ZlYNeAq40t03RPmk8BxuZkVntzJy93FEz4bIysrybt267b5R9+6lHaTYVSXtWsJukUkl7Nyt7NXPzc2lyPqlCdUvtaVz/dK5bomWkARSHDPbnyixTHT3p0Px1+F2F+FnwfObFcARMbs3DGXFlYuISCWVtOQSbnk9DCx099Exq6YCBS2+BgLPxpT/KrQa6wisd/eVwMtALzOrFR7k9wplIiJSScXT/cue6gxcAMwzs7xQdjUwCnjSzAYDnwFnh3UvAn2BJcBmombPuPtaM7uJqHEBwEh3X5vEuEVEZC8lLbmEB/NWzOqeRWzvwKXFHGs8MD5x0YmISDIl9ZmLiIjsm5RcREQk4ZRcREQk4ZRcREQk4ZLZWkzKmxXXfoI43vQUEUkcXbmIiEjCKbmIiEjCKbmIiEjCKbmIiEjCKbmIiEjCKbmIiEjCKbmIiEjCKbmIiEjCKbmIiEjCKbmIiEjCKbmIiEjCKbmIiEjCKbmIiEjCJS25mNl4M1tlZvNjymqb2TQzWxx+1grlZmb3mtkSM/vAzNrG7DMwbL/YzAYmK14REUmcZF65/BPos0vZcOBVd28CvBqWAU4GmoRpCPAARMkIuB44FugAXF+QkEREpPJKWnJx9zeAtbsU9wcmhPkJwGkx5Y96ZAZQ08zqA72Bae6+1t3XAdPYPWGJiEglU97PXOq5+8ow/xVQL8w3AL6I2W55KCuuXEREKrEKG4nS3d3MEjY8opkNIbqlRt26dcnNzd19ozvvLPkgRe0Tx64l7BbJ3Judd5efn1/2+u3BeSpKsfVLE6pf6krnuiVaeSeXr82svruvDLe9VoXyFcARMds1DGUrgG67lOcWdWB3HweMA8jKyvJu3brtvlH37iVHV8JQwCXtWuoIwpNK2Llb2fNrbm4uZa5fCg1zXGz90oTql7rSuW6JVt63xaYCBS2+BgLPxpT/KrQa6wisD7fPXgZ6mVmt8CC/VygTEZFKLGlXLmb2ONFVx6Fmtpyo1dco4EkzGwx8BpwdNn8R6AssATYDgwDcfa2Z3QTMCtuNdPddGwmIiEglk7Tk4u7nFrOqZxHbOnBpMccZD4xPYGhSBLPi16XQHTURqSQq7IG+pIlJJWQlgPOUmUT2Rer+RUREEk7JRUREEk63xWLYjSXd4tHtHRGReOnKRUREEk7JRUREEk63xSR1qL20SMpQcqlgN1ByU94b9KxHRFKQbouJiEjCKbmIiEjCKbmIiEjCKbmIiEjCKbmIiEjCKbmIiEjCqSmypL2SXo+BSvKKTEoEKRI/JReRkmhIAZE9ouSSEKV8AYmI7GP0zEVERBJOVy4iKW6vHteU120/PVPa56RMcjGzPsA9QAbwkLuPquCQUkrJY9VAssarKanvNPWbJmlDnaruJiWSi5llAPcBJwHLgVlmNtXdF1RsZPsKPVMqjpJn8u3V93ZJV2ZqjJFUKZFcgA7AEnf/BMDMngD6A0ouAuztKKKVP3lW1Cip5ZU8K6J+laVH8nRNnuYpcMlmZmcCfdz94rB8AXCsu18Ws80QYEhYbAHML/dAy8+hwDcVHUQSqX6pLZ3rl851A8hy9+qJOFCqXLmUyt3HAeMAzOw9dz+mgkNKGtUvtal+qSud6wZR/RJ1rFRpirwCOCJmuWEoExGRSihVksssoImZNTazqsA5wNQKjklERIqRErfF3H2bmV0GvEzUFHm8u39Ywi7jyieyCqP6pTbVL3Wlc90ggfVLiQf6IiKSWlLltpiIiKQQJRcREUm4tEsuZtbHzBaZ2RIzG17R8ZSVmR1hZq+b2QIz+9DMfh/Ka5vZNDNbHH7WCuVmZveG+n5gZm0rtgbxMbMMM5trZs+H5cZm9m6oR05ouIGZHRCWl4T1jSoy7niYWU0zm2xmH5nZQjM7Lp0+PzP7Q/jbnG9mj5tZZip/fmY23sxWmdn8mLIyf15mNjBsv9jMBlZEXYpSTP3uCH+fH5jZFDOrGbNuRKjfIjPrHVNetu9Wd0+biehh/1LgZ0BV4H2gWUXHVcY61AfahvnqwMdAM+B2YHgoHw7cFub7Av8hes28I/BuRdchznr+EZgEPB+WnwTOCfNjgUvC/O+AsWH+HCCnomOPo24TgIvDfFWgZrp8fkAD4FPgwJjP7cJU/vyA44G2wPyYsjJ9XkBt4JPws1aYr1XRdSuhfr2AKmH+tpj6NQvfmwcAjcP3acaefLdWeMUT/Es8Dng5ZnkEMKKi49rLOj1L1KfaIqB+KKsPLArzDwLnxmxfuF1lnYjeU3oV6AE8H/6hfhPzx174ORK1EDwuzFcJ21lF16GEutUIX762S3lafH4huXwRvkSrhM+vd6p/fkCjXb58y/R5AecCD8aU77RdRU+71m+XdacDE8P8Tt+ZBZ/fnny3ptttsYI//ALLQ1lKCrcQ2gDvAvXcfWVY9RVQL8ynYp3/BlwF7AjLdYBv3X1bWI6tQ2H9wvr1YfvKqjGwGngk3PZ7yMwOJk0+P3dfAdwJfA6sJPo8ZpM+n1+Bsn5eKfU57uIioqsxSGD90i25pA0zqwY8BVzp7hti13n0X4eUbENuZv2AVe4+u6JjSZIqRLcgHnD3NsAmotsqhVL886tF1GlsY+Bw4GCgT4UGlWSp/HmVxsz+AmwDJib62OmWXNKimxgz258osUx096dD8ddmVj+srw+sCuWpVufOwKlmtgx4gujW2D1ATTMreKk3tg6F9QvrawBryjPgMloOLHf3d8PyZKJkky6f34nAp+6+2t23Ak8Tfabp8vkVKOvnlWqfI2Z2IdAPOD8kUEhg/dItuaR8NzFmZsDDwEJ3Hx2zaipQ0AJlINGzmILyX4VWLB2B9TGX85WOu49w94bu3ojo83nN3c8HXgfODJvtWr+Cep8Ztq+0/4t096+AL8wsKxT1JBoaIi0+P6LbYR3N7KDwt1pQv7T4/GKU9fN6GehlZrXC1V2vUFYpWTT44lXAqe6+OWbVVOCc0MqvMdAEmMmefLdW9IOmJDy46kvUwmop8JeKjmcP4u9CdAn+AZAXpr5E96lfBRYD/wVqh+2NaCC1pcA84JiKrkMZ6tqNH1uL/Sz8ES8B/g0cEMozw/KSsP5nFR13HPXKBt4Ln+EzRK2H0ubzA24EPiIa1uIxopZFKfv5AY8TPT/aSnTlOXhPPi+iZxdLwjSooutVSv2WED1DKfiOGRuz/V9C/RYBJ8eUl+m7Vd2/iIhIwqXbbTEREakElFxERCThlFxERCThlFxERCThlFxERCThlFwkJZhZfpKPf6GZHR6zvMzMDt2L4z0eepz9Q2IiLH9mlm1mfSs6DklNKTHMsUg5uJDovY0v9/ZAZnYY0N7df763x6pg2cAxwIsVHYikHl25SMoys7pm9pSZzQpT51B+QxjDItfMPjGzK2L2uTaMSfFmuLr4s5mdSfQlOtHM8szswLD55WY2x8zmmdnRRZw/08weCevnmln3sOoVoEE4Vtdd9qkXxs94P0ydQvkfLRofZb6ZXRnKGoUxN/5pZh+b2UQzO9HM3gpjhnSIqe8EM5tuZp+Z2RlmdnuI66XQnRBm1s7M/mdms83s5ZjuTXLN7DYzmxnO0zW8hT0SGBDqMcDMTgjzeaG+1RP2YUr6qei3RzVpimcC8osomwR0CfM/JeoyB+AG4G2iN8cPJerLan+gPdHbyJlEY+UsBv4c9sll57etlwGXh/nfAQ8Vcf4/AePD/NFEXaNkUnL35jlEnZFCNEZGDaAd0dveBwPVgA+JesNuRNSpYEui/wjOBsYTvSXeH3gmpr5vhjq2BjYT3qwGpgCnhXVvA3VD+YCY2HOBu8J8X+C/Yf5C4O8xsT8HdA7z1Qhd7GvSVNSk22KSyk4EmkVdXAFwiEW9SQO84O7fA9+b2SqiLtM7A8+6+xZgi5k9V8rxCzoNnQ2cUcT6LsAYAHf/yMw+A44CNhSxbYEewK/CPtuB9WbWBZji7psAzOxpoCtR302fuvu8UP4h8Kq7u5nNI0o+Bf7j7ltDeQbwUigv2C4LaAFMC7+vDKIuQYqqa+xxY70FjDazicDT7r68hHrKPk7JRVLZfkDHkCwKhS/P72OKtrNnf+sFx9jT/RMhth47YpZ3sHNM3wO4+w4z2+ruvst2Bnzo7seVcp5i6+ruo8zsBaKrm7fMrLe7f1TWCsm+Qc9cJJW9AlxesGBm2aVs/xZwSnhWUo2ou/ECG4lulZXFdOD8cO6jiG7NLSpln1eBS8I+GWZWIxzntNDT8MFEIwNOL2MspVkE1DWz48K59zez5qXss9PvxMyOdPd57n4bUS+5uz2HEimg5CKp4iAzWx4z/RG4AjgmNPldAPy2pAO4+yyiW00fEI28N49oZESAfwJjd3mgX5r7gf3Cragc4MJwK64kvwe6h31mE41DPiecfybRqKMPufvcOGOIi7v/QNTl/W1m9j7Rs6dOpez2OtFtxzwzGwBcGRocfEDUw+5/St5d9mXqFVn2KWZWzd3zzewg4A1gSPhyF5EE0jMX2deMM7NmRK26JiixiCSHrlxERCTh9MxFREQSTslFREQSTslFREQSTslFREQSTslFREQS7v8DSF794t2NXtYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10b72f780>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "y = np.zeros(label.shape)\n",
    "for ix in range(comment.shape[0]):\n",
    "    l = len(comment[ix])\n",
    "    if label[ix][0] :\n",
    "        y[ix][0] = l\n",
    "    if label[ix][1] :\n",
    "        y[ix][1] = l\n",
    "    if label[ix][2] :\n",
    "        y[ix][2] = l\n",
    "    if label[ix][3] :\n",
    "        y[ix][3] = l\n",
    "    if label[ix][4] :\n",
    "        y[ix][4] = l\n",
    "    if label[ix][5] :\n",
    "        y[ix][5] = l\n",
    "\n",
    "labelsplt = ['toxic','severe_toxic','obscene','threat','insult','identity_hate']\n",
    "color = ['red','green','blue','yellow','orange','chartreuse']        \n",
    "plt.hist(y,bins = bins,label = labelsplt,color = color)\n",
    "plt.axis([0, 1200, 0, 8000])\n",
    "plt.xlabel('Length of comments')\n",
    "plt.ylabel('Number of comments') \n",
    "plt.legend()\n",
    "plt.grid(True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove excessive length comments\n",
    "Some very large length comments can be seen, in our dataset. These pose serious problems like adding excessively more words to the training dataset, causing training time to increase and accuracy to decrease!<br/>\n",
    "Hence, a threshold of 400 characters will be created and only comments which have length smaller than 400 will be used further."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "comments = []\n",
    "labels = []\n",
    "\n",
    "for ix in range(comment.shape[0]):\n",
    "    if len(comment[ix])<=400:\n",
    "        comments.append(comment[ix])\n",
    "        labels.append(label[ix])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "labels = np.asarray(labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "69627\n"
     ]
    }
   ],
   "source": [
    "print(len(comments))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hence, after removing comments longer than 400 characters, we are still left with more than 69000 comments, which seems enough for training purposes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preprocessing \n",
    "Preprocessing involved the following steps, but these will be performed in a slightly different manner:\n",
    "- Removing Punctuations and other special characters\n",
    "- Splitting the comments into individual words\n",
    "- Removing Stop Words\n",
    "- Stemming and Lemmatising\n",
    "- Applying Count Vectoriser\n",
    "- Splitting dataset into Training and Testing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing a string containing all punctuations to be removed\n",
    "The string library contains punctuation characters. This is imported and all numbers are appended to this string. Also, we can notice that our comment_text field contains strings such as won't, didn't, etc which contain apostrophe character('). To prevent these words from being converted to wont/didnt, the character ' represented as \\' in escape sequence notation is replaced by empty character in the punctuation string. <br/>\n",
    "\n",
    "**maketrans()** returns a translation table that maps each character in the punctuation_edit into the character at the same position in the outtab string i.e. it replaces every character in the removal list with a space, since outtab contains a string with spaces."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~\n",
      "!\"#$%&()*+,-./:;<=>?@[\\]^_`{|}~0123456789\n"
     ]
    }
   ],
   "source": [
    "import string\n",
    "print(string.punctuation)\n",
    "punctuation_edit = string.punctuation.replace('\\'','') +\"0123456789\"\n",
    "print (punctuation_edit)\n",
    "outtab = \"                                         \"\n",
    "trantab = str.maketrans(punctuation_edit, outtab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Updating the list of stop words\n",
    "**Stop words** are those words that are frequently used in both written and verbal communication and thereby do not have either a positive/negative impact on our statement.E.g. is, this, us,etc. <br/>\n",
    "Single letter words if existing or created due to any preprocessing step do not convey any useful meaning and hence can be directly removed. Hence letters from b to z, will be added to the list of stop words imported directly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from stop_words import get_stop_words\n",
    "stop_words = get_stop_words('english')\n",
    "stop_words.append('')\n",
    "\n",
    "for x in range(ord('b'), ord('z')+1):\n",
    "    stop_words.append(chr(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['a', 'about', 'above', 'after', 'again', 'against', 'all', 'am', 'an', 'and', 'any', 'are', \"aren't\", 'as', 'at', 'be', 'because', 'been', 'before', 'being', 'below', 'between', 'both', 'but', 'by', \"can't\", 'cannot', 'could', \"couldn't\", 'did', \"didn't\", 'do', 'does', \"doesn't\", 'doing', \"don't\", 'down', 'during', 'each', 'few', 'for', 'from', 'further', 'had', \"hadn't\", 'has', \"hasn't\", 'have', \"haven't\", 'having', 'he', \"he'd\", \"he'll\", \"he's\", 'her', 'here', \"here's\", 'hers', 'herself', 'him', 'himself', 'his', 'how', \"how's\", 'i', \"i'd\", \"i'll\", \"i'm\", \"i've\", 'if', 'in', 'into', 'is', \"isn't\", 'it', \"it's\", 'its', 'itself', \"let's\", 'me', 'more', 'most', \"mustn't\", 'my', 'myself', 'no', 'nor', 'not', 'of', 'off', 'on', 'once', 'only', 'or', 'other', 'ought', 'our', 'ours', 'ourselves', 'out', 'over', 'own', 'same', \"shan't\", 'she', \"she'd\", \"she'll\", \"she's\", 'should', \"shouldn't\", 'so', 'some', 'such', 'than', 'that', \"that's\", 'the', 'their', 'theirs', 'them', 'themselves', 'then', 'there', \"there's\", 'these', 'they', \"they'd\", \"they'll\", \"they're\", \"they've\", 'this', 'those', 'through', 'to', 'too', 'under', 'until', 'up', 'very', 'was', \"wasn't\", 'we', \"we'd\", \"we'll\", \"we're\", \"we've\", 'were', \"weren't\", 'what', \"what's\", 'when', \"when's\", 'where', \"where's\", 'which', 'while', 'who', \"who's\", 'whom', 'why', \"why's\", 'with', \"won't\", 'would', \"wouldn't\", 'you', \"you'd\", \"you'll\", \"you're\", \"you've\", 'your', 'yours', 'yourself', 'yourselves', '', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z']\n"
     ]
    }
   ],
   "source": [
    "print (stop_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stemming and Lemmatizing\n",
    "**Stemming** is the process of converting inflected/derived words to their word stem or the root form. Basically, a large number of similar origin words are converted to the same word.E.g. words like \"stems\", \"stemmer\", \"stemming\", \"stemmed\" as based on \"stem\". This helps in achieving the training process with a better accuracy.<br/>\n",
    "**Lemmatizing** is the process of grouping together the inflected forms of a word so they can be analysed as a single item. This is quite similar to stemming in its working but differs since it depends on correctly identifying the intended part of speech and meaning of a word in a sentence, as well as within the larger context surrounding that sentence, such as neighboring sentences or even an entire document.<br/>\n",
    "The **wordnet library in nltk** will be used for this purpose. Stemmer and Lemmatizer are also imported from nltk.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import nltk\n",
    "from nltk.stem import PorterStemmer, WordNetLemmatizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package wordnet to\n",
      "[nltk_data]     /Users/deeptibaghel/nltk_data...\n",
      "[nltk_data]   Package wordnet is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#create objects for stemmer and lemmatizer\n",
    "lemmatiser = WordNetLemmatizer()\n",
    "stemmer = PorterStemmer()\n",
    "#download words from wordnet library\n",
    "nltk.download('wordnet')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We can now, loop once through all the comments applying :\n",
    "- punctuation removal\n",
    "- splitting the words by space\n",
    "- applying stemmer and lemmatizer\n",
    "- recombining the words again for further processing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for i in range(len(comments)):\n",
    "    comments[i] = comments[i].lower().translate(trantab)\n",
    "    l = []\n",
    "    for word in comments[i].split():\n",
    "        l.append(stemmer.stem(lemmatiser.lemmatize(word,pos=\"v\")))\n",
    "    comments[i] = \" \".join(l)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Applying Count Vectorizer\n",
    "Here we can finally convert our comments into a matrix of token counts, which signifies the number of times it occurs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#import required library\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "#create object supplying our custom stop words\n",
    "count_vector = CountVectorizer(stop_words=stop_words)\n",
    "#fitting it to converts comments into bag of words format\n",
    "tf = count_vector.fit_transform(comments).toarray()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(69627, 52905)\n"
     ]
    }
   ],
   "source": [
    "# print(count_vector.get_feature_names())\n",
    "print(tf.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hence from its shape we can imply that after all preprocessing we have a list of 52905 words in total.\n",
    "## Splitting dataset into training and testing\n",
    "- Since the system was going out of memory using train_test_split, I had jumbled all the indexes in the beginning itself. \n",
    "- The shuffle function defined here performs the task of assigning first 2/3rd values to train and remaining 1/3rd values to the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(23209, 52905)\n",
      "(46418, 52905)\n"
     ]
    }
   ],
   "source": [
    "def shuffle(matrix, target, test_proportion):\n",
    "    ratio = int(matrix.shape[0]/test_proportion)\n",
    "    X_train = matrix[ratio:,:]\n",
    "    X_test =  matrix[:ratio,:]\n",
    "    Y_train = target[ratio:,:]\n",
    "    Y_test =  target[:ratio,:]\n",
    "    return X_train, X_test, Y_train, Y_test\n",
    "\n",
    "X_train, X_test, Y_train, Y_test = shuffle(tf, labels,3)\n",
    "\n",
    "print(X_test.shape)\n",
    "print(X_train.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Implementation :\n",
    "### Let us define all the evaluation metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import hamming_loss\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.metrics import log_loss\n",
    "\n",
    "def evaluate_score(Y_test,predict): \n",
    "    loss = hamming_loss(Y_test,predict)\n",
    "    print(\"Hamming_loss : {}\".format(loss*100))\n",
    "    accuracy = accuracy_score(Y_test,predict)\n",
    "    print(\"Accuracy : {}\".format(accuracy*100))\n",
    "    try : \n",
    "        loss = log_loss(Y_test,predict)\n",
    "    except :\n",
    "        loss = log_loss(Y_test,predict.toarray())\n",
    "    print(\"Log_loss : {}\".format(loss))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Starting with the First Model -\n",
    "### Problem Transformation Methods :\n",
    "**These include the Binary Relevance, Label Powerset and Classifier Chain methods. Implementations of these methods is available in the scikit-multilearn library. **\n",
    "- I will be implementing the most basic method,which is the **Binary Relevance** method from scratch. It does not take into account the interdependence of labels and basically creates a separate classifier for each of the labels.\n",
    "- Scikit-multilearn library's classifier will also be imported and tested with different classifiers to observe if it gives similar results.\n",
    "\n",
    "### 1. Binary Relevance (BR) Method with MultinomialNB classifiers (from scratch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import MultinomialNB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# clf will be the list of the classifiers for all the 6 labels\n",
    "# each classifier is fit with the training data and corresponding classifier\n",
    "clf = []\n",
    "for ix in range(6):\n",
    "    clf.append(MultinomialNB())\n",
    "    clf[ix].fit(X_train,Y_train[:,ix])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(23209, 6)\n"
     ]
    }
   ],
   "source": [
    "# predict list contains the predictions, it is transposed later to get the proper shape\n",
    "predict = []\n",
    "for ix in range(6):\n",
    "    predict.append(clf[ix].predict(X_test))\n",
    "\n",
    "predict = np.asarray(np.transpose(predict))\n",
    "print(predict.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hamming_loss : 3.2796185387852415\n",
      "Accuracy : 88.29764315567236\n",
      "Log_loss : 1.9296193554647956\n"
     ]
    }
   ],
   "source": [
    "# calculate results\n",
    "evaluate_score(Y_test,predict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. BR Method with SVM classifier (from scikit-multilearn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BinaryRelevance(classifier=SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,\n",
       "  decision_function_shape='ovr', degree=3, gamma='auto', kernel='rbf',\n",
       "  max_iter=-1, probability=False, random_state=None, shrinking=True,\n",
       "  tol=0.001, verbose=False),\n",
       "        require_dense=[False, True])"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#create and fit classifier\n",
    "from skmultilearn.problem_transform import BinaryRelevance\n",
    "from sklearn.svm import SVC\n",
    "classifier = BinaryRelevance(classifier = SVC(), require_dense = [False, True])\n",
    "classifier.fit(X_train, Y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#predictions\n",
    "predictions = classifier.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hamming_loss : 4.26702285033105\n",
      "Accuracy : 88.276099788875\n",
      "Log_loss : 0.4616701016298293\n"
     ]
    }
   ],
   "source": [
    "#calculate scores\n",
    "evaluate_score(Y_test,predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. BR Method with Multinomial classifier (from scikit-multilearn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BinaryRelevance(classifier=MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True),\n",
       "        require_dense=[False, True])"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#create and fit classifier\n",
    "classifier = BinaryRelevance(classifier = MultinomialNB(), require_dense = [False, True])\n",
    "classifier.fit(X_train, Y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#predictions\n",
    "predictions = classifier.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hamming_loss : 3.2796185387852415\n",
      "Accuracy : 88.29764315567236\n",
      "Log_loss : 1.9296193554647956\n"
     ]
    }
   ],
   "source": [
    "#calculate scores\n",
    "evaluate_score(Y_test,predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. BR Method with GausseanNB classifier (from scratch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import GaussianNB\n",
    "#create and fit classifiers\n",
    "clf = []\n",
    "for ix in range(6):\n",
    "    clf.append(GaussianNB())\n",
    "    clf[ix].fit(X_train,Y_train[:,ix])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#predictions\n",
    "predict = []\n",
    "for ix in range(6):\n",
    "    predict.append(clf[ix].predict(X_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hamming_loss : 20.746262225860658\n",
      "Accuracy : 52.199577750010775\n",
      "Log_loss : 1.4227365989183884\n"
     ]
    }
   ],
   "source": [
    "#calculate scores\n",
    "predict = np.asarray(np.transpose(predict))\n",
    "evaluate_score(Y_test,predict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Classifier chain with MultinomialNB classifier (from scikit-multilearn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ClassifierChain(classifier=MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True),\n",
       "        require_dense=[True, True])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#create and fit classifier\n",
    "from skmultilearn.problem_transform import ClassifierChain\n",
    "classifier = ClassifierChain(MultinomialNB())\n",
    "classifier.fit(X_train, Y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#predictions\n",
    "predictions = classifier.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hamming_loss : 3.5647090927370133\n",
      "Accuracy : 88.25886509543712\n",
      "Log_loss : 1.506849253150214\n"
     ]
    }
   ],
   "source": [
    "#calculate scores\n",
    "evaluate_score(Y_test,predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Label Powerset with MultinomialNB classifier (from scikit-multilearn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LabelPowerset(classifier=MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True),\n",
       "       require_dense=[True, True])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#create and fit classifier\n",
    "from skmultilearn.problem_transform import LabelPowerset\n",
    "classifier = LabelPowerset(MultinomialNB())\n",
    "classifier.fit(X_train, Y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#predictions\n",
    "predictions = classifier.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hamming_loss : 3.1726198170250046\n",
      "Accuracy : 88.80606661209013\n",
      "Log_loss : 1.4765486777963348\n"
     ]
    }
   ],
   "source": [
    "#calculate scores\n",
    "evaluate_score(Y_test,predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adaptation Algorithms\n",
    "### 7. MLkNN  with k=2 (from scikit-multilearn)\n",
    "This is the adapted multi-label version of K Nearest Neighbours. Its implementation is available in the multilearn library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#create and fit classifier\n",
    "from skmultilearn.adapt import MLkNN\n",
    "classifier = MLkNN(k=2)\n",
    "classifier.fit(X_train, Y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#predictions\n",
    "predictions = classifier.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#calculate scores\n",
    "evaluate_score(Y_test,predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8. BP-MLL Neural Networks (from scratch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Activation, Dropout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_1 (Dense)              (None, 4)                 211956    \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 4)                 0         \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 6)                 30        \n",
      "=================================================================\n",
      "Total params: 211,986\n",
      "Trainable params: 211,986\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "#define model architecture\n",
    "model = Sequential()\n",
    "model.add(Dense(4, activation='relu', input_dim = X_train.shape[1]))\n",
    "model.add(Dropout(0.3))\n",
    "model.add(Dense(6, activation='softmax'))\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#compile model with all parameters set\n",
    "model.compile(optimizer='rmsprop',\n",
    "              loss='categorical_crossentropy',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "46418/46418 [==============================] - 60s 1ms/step - loss: 0.3684 - acc: 0.9419\n",
      "Epoch 2/10\n",
      "46418/46418 [==============================] - 60s 1ms/step - loss: 0.3489 - acc: 0.9901\n",
      "Epoch 3/10\n",
      "46418/46418 [==============================] - 62s 1ms/step - loss: 0.3449 - acc: 0.9906\n",
      "Epoch 4/10\n",
      "46418/46418 [==============================] - 59s 1ms/step - loss: 0.3425 - acc: 0.9888: 0s - loss: 0.3424 - acc: 0.98\n",
      "Epoch 5/10\n",
      "46418/46418 [==============================] - 59s 1ms/step - loss: 0.3424 - acc: 0.9876\n",
      "Epoch 6/10\n",
      "46418/46418 [==============================] - 58s 1ms/step - loss: 0.3424 - acc: 0.9879\n",
      "Epoch 7/10\n",
      "46418/46418 [==============================] - 57s 1ms/step - loss: 0.3414 - acc: 0.9868\n",
      "Epoch 8/10\n",
      "46418/46418 [==============================] - 60s 1ms/step - loss: 0.3404 - acc: 0.9851\n",
      "Epoch 9/10\n",
      "46418/46418 [==============================] - 61s 1ms/step - loss: 0.3402 - acc: 0.9846\n",
      "Epoch 10/10\n",
      "46418/46418 [==============================] - 61s 1ms/step - loss: 0.3395 - acc: 0.9825\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x210b649470>"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Fit using check pointer\n",
    "from keras.callbacks import ModelCheckpoint  \n",
    "\n",
    "checkpointer = ModelCheckpoint(filepath='saved_models/weights.best.myneural.h5py', \n",
    "                               verbose=1, save_best_only=True)\n",
    "model.fit(X_train, Y_train, epochs=10, batch_size=32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#predictions\n",
    "predict = model.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[  6.69890046e-01   9.70017936e-05   1.44642159e-01   1.16227047e-05\n",
      "   1.78119496e-01   7.23966770e-03]\n"
     ]
    }
   ],
   "source": [
    "print(predict[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Since the results returned by the model are in the form of probabilities, they have to be explicitly converted to either 0/1 using the round function. This is because the hamming_loss and accuracy_score cannot work on these values directly. However, log loss can compute loss directly without modifying the values.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Log_loss : 0.36017768848519677\n",
      "Hamming_loss : 13.960101684691285\n",
      "Accuracy : 29.52302985910638\n"
     ]
    }
   ],
   "source": [
    "#calculate score\n",
    "loss = log_loss(Y_test,predict)\n",
    "print(\"Log_loss : {}\".format(loss))\n",
    "predict = np.round(predict)\n",
    "loss = hamming_loss(Y_test,predict)\n",
    "print(\"Hamming_loss : {}\".format(loss*100))\n",
    "accuracy = accuracy_score(Y_test,predict)\n",
    "print(\"Accuracy : {}\".format(accuracy*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Let us try improving the BP-MLL model (Refining)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "from keras.wrappers.scikit_learn import KerasClassifier\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Activation, Dropout\n",
    "from keras import optimizers\n",
    "\n",
    "#define parameters for using in param grid\n",
    "nodes = [16, 32, 64] # number of nodes in the hidden layer\n",
    "lrs = [0.001, 0.002, 0.003] # learning rate, default = 0.001\n",
    "epochs = [10,20,30]\n",
    "batch_size = 64"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def create_model(nodes=10,lr=0.001):\n",
    "    model = Sequential()\n",
    "    model.add(Dense(nodes, activation='relu', input_dim = X_train.shape[1]))\n",
    "    model.add(Dropout(0.3))\n",
    "    model.add(Dense(6, activation='softmax'))\n",
    "    opt = optimizers.RMSprop(lr=lr)\n",
    "    model.compile(optimizer=opt,\n",
    "              loss='categorical_crossentropy',\n",
    "              metrics=['accuracy'])\n",
    "    return model\n",
    "\n",
    "model = KerasClassifier(build_fn=create_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 27 candidates, totalling 81 fits\n",
      "[CV] epochs=10, lr=0.001, nodes=16 ...................................\n",
      "Epoch 1/10\n",
      "30945/30945 [==============================] - 39s 1ms/step - loss: 0.3682 - acc: 0.9536\n",
      "Epoch 2/10\n",
      "30945/30945 [==============================] - 27s 878us/step - loss: 0.3440 - acc: 0.9794\n",
      "Epoch 3/10\n",
      "30945/30945 [==============================] - 25s 820us/step - loss: 0.3379 - acc: 0.9786\n",
      "Epoch 4/10\n",
      "30945/30945 [==============================] - 26s 837us/step - loss: 0.3324 - acc: 0.9745\n",
      "Epoch 5/10\n",
      "30945/30945 [==============================] - 26s 832us/step - loss: 0.3280 - acc: 0.9718\n",
      "Epoch 6/10\n",
      "30945/30945 [==============================] - 26s 842us/step - loss: 0.3249 - acc: 0.9688\n",
      "Epoch 7/10\n",
      "30945/30945 [==============================] - 27s 878us/step - loss: 0.3227 - acc: 0.9654\n",
      "Epoch 8/10\n",
      "30945/30945 [==============================] - 26s 838us/step - loss: 0.3202 - acc: 0.9647\n",
      "Epoch 9/10\n",
      "30945/30945 [==============================] - 26s 843us/step - loss: 0.3190 - acc: 0.9596\n",
      "Epoch 10/10\n",
      "30945/30945 [==============================] - 27s 858us/step - loss: 0.3157 - acc: 0.9602\n",
      "15473/15473 [==============================] - 14s 882us/step\n",
      "30945/30945 [==============================] - 25s 812us/step\n",
      "[CV] .................... epochs=10, lr=0.001, nodes=16, total= 5.5min\n",
      "[CV] epochs=10, lr=0.001, nodes=16 ...................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:  5.9min remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "30945/30945 [==============================] - 33s 1ms/step - loss: 0.3783 - acc: 0.9584\n",
      "Epoch 2/10\n",
      "30945/30945 [==============================] - 22s 715us/step - loss: 0.3515 - acc: 0.9802\n",
      "Epoch 3/10\n",
      "30945/30945 [==============================] - 22s 706us/step - loss: 0.3447 - acc: 0.9793\n",
      "Epoch 4/10\n",
      "30945/30945 [==============================] - 23s 731us/step - loss: 0.3407 - acc: 0.9788\n",
      "Epoch 5/10\n",
      "30945/30945 [==============================] - 23s 733us/step - loss: 0.3373 - acc: 0.9770\n",
      "Epoch 6/10\n",
      "30945/30945 [==============================] - 22s 726us/step - loss: 0.3342 - acc: 0.9748\n",
      "Epoch 7/10\n",
      "30945/30945 [==============================] - 22s 713us/step - loss: 0.3312 - acc: 0.9697\n",
      "Epoch 8/10\n",
      "30945/30945 [==============================] - 22s 695us/step - loss: 0.3303 - acc: 0.9648\n",
      "Epoch 9/10\n",
      "30945/30945 [==============================] - 22s 702us/step - loss: 0.3280 - acc: 0.9621\n",
      "Epoch 10/10\n",
      "30945/30945 [==============================] - 22s 712us/step - loss: 0.3268 - acc: 0.9588\n",
      "15473/15473 [==============================] - 11s 720us/step\n",
      "30945/30945 [==============================] - 20s 657us/step\n",
      "[CV] .................... epochs=10, lr=0.001, nodes=16, total= 4.6min\n",
      "[CV] epochs=10, lr=0.001, nodes=16 ...................................\n",
      "Epoch 1/10\n",
      "30946/30946 [==============================] - 35s 1ms/step - loss: 0.3705 - acc: 0.8698\n",
      "Epoch 2/10\n",
      "30946/30946 [==============================] - 22s 718us/step - loss: 0.3464 - acc: 0.9863\n",
      "Epoch 3/10\n",
      "30946/30946 [==============================] - 22s 703us/step - loss: 0.3407 - acc: 0.9855\n",
      "Epoch 4/10\n",
      "30946/30946 [==============================] - 22s 707us/step - loss: 0.3358 - acc: 0.9822\n",
      "Epoch 5/10\n",
      "30946/30946 [==============================] - 21s 680us/step - loss: 0.3315 - acc: 0.9808\n",
      "Epoch 6/10\n",
      "30946/30946 [==============================] - 21s 684us/step - loss: 0.3276 - acc: 0.9772\n",
      "Epoch 7/10\n",
      "30946/30946 [==============================] - 21s 678us/step - loss: 0.3268 - acc: 0.9731\n",
      "Epoch 8/10\n",
      "30946/30946 [==============================] - 23s 742us/step - loss: 0.3239 - acc: 0.9692\n",
      "Epoch 9/10\n",
      "30946/30946 [==============================] - 22s 720us/step - loss: 0.3239 - acc: 0.9666\n",
      "Epoch 10/10\n",
      "30946/30946 [==============================] - 23s 735us/step - loss: 0.3200 - acc: 0.9641\n",
      "15472/15472 [==============================] - 11s 726us/step\n",
      "30946/30946 [==============================] - 20s 648us/step\n",
      "[CV] .................... epochs=10, lr=0.001, nodes=16, total= 4.6min\n",
      "[CV] epochs=10, lr=0.001, nodes=32 ...................................\n",
      "Epoch 1/10\n",
      "30945/30945 [==============================] - 50s 2ms/step - loss: 0.3620 - acc: 0.9808\n",
      "Epoch 2/10\n",
      "30945/30945 [==============================] - 38s 1ms/step - loss: 0.3390 - acc: 0.9796\n",
      "Epoch 3/10\n",
      "30945/30945 [==============================] - 36s 1ms/step - loss: 0.3307 - acc: 0.9748\n",
      "Epoch 4/10\n",
      "30945/30945 [==============================] - 35s 1ms/step - loss: 0.3259 - acc: 0.9741\n",
      "Epoch 5/10\n",
      "30945/30945 [==============================] - 37s 1ms/step - loss: 0.3214 - acc: 0.9715\n",
      "Epoch 6/10\n",
      "30945/30945 [==============================] - 36s 1ms/step - loss: 0.3169 - acc: 0.9686\n",
      "Epoch 7/10\n",
      "30945/30945 [==============================] - 35s 1ms/step - loss: 0.3139 - acc: 0.9637\n",
      "Epoch 8/10\n",
      "30945/30945 [==============================] - 37s 1ms/step - loss: 0.3112 - acc: 0.9610\n",
      "Epoch 9/10\n",
      "30945/30945 [==============================] - 37s 1ms/step - loss: 0.3077 - acc: 0.9581\n",
      "Epoch 10/10\n",
      "30945/30945 [==============================] - 37s 1ms/step - loss: 0.3059 - acc: 0.9556\n",
      "15473/15473 [==============================] - 14s 918us/step\n",
      "30945/30945 [==============================] - 26s 836us/step\n",
      "[CV] .................... epochs=10, lr=0.001, nodes=32, total= 7.1min\n",
      "[CV] epochs=10, lr=0.001, nodes=32 ...................................\n",
      "Epoch 1/10\n",
      "30945/30945 [==============================] - 48s 2ms/step - loss: 0.3743 - acc: 0.9775\n",
      "Epoch 2/10\n",
      "30945/30945 [==============================] - 37s 1ms/step - loss: 0.3518 - acc: 0.9829\n",
      "Epoch 3/10\n",
      "30945/30945 [==============================] - 36s 1ms/step - loss: 0.3443 - acc: 0.9796\n",
      "Epoch 4/10\n",
      "30945/30945 [==============================] - 37s 1ms/step - loss: 0.3386 - acc: 0.9766\n",
      "Epoch 5/10\n",
      "30945/30945 [==============================] - 37s 1ms/step - loss: 0.3343 - acc: 0.9740\n",
      "Epoch 6/10\n",
      "30945/30945 [==============================] - 37s 1ms/step - loss: 0.3309 - acc: 0.9720\n",
      "Epoch 7/10\n",
      "30945/30945 [==============================] - 37s 1ms/step - loss: 0.3280 - acc: 0.9694\n",
      "Epoch 8/10\n",
      "30945/30945 [==============================] - 37s 1ms/step - loss: 0.3261 - acc: 0.9677\n",
      "Epoch 9/10\n",
      "30945/30945 [==============================] - 37s 1ms/step - loss: 0.3237 - acc: 0.9626\n",
      "Epoch 10/10\n",
      "30945/30945 [==============================] - 37s 1ms/step - loss: 0.3212 - acc: 0.9587\n",
      "15473/15473 [==============================] - 14s 932us/step\n",
      "30945/30945 [==============================] - 27s 861us/step\n",
      "[CV] .................... epochs=10, lr=0.001, nodes=32, total= 7.2min\n",
      "[CV] epochs=10, lr=0.001, nodes=32 ...................................\n",
      "Epoch 1/10\n",
      "30946/30946 [==============================] - 44s 1ms/step - loss: 0.3608 - acc: 0.9766\n",
      "Epoch 2/10\n",
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.3390 - acc: 0.9807\n",
      "Epoch 3/10\n",
      "30946/30946 [==============================] - 35s 1ms/step - loss: 0.3311 - acc: 0.9780\n",
      "Epoch 4/10\n",
      "30946/30946 [==============================] - 34s 1ms/step - loss: 0.3269 - acc: 0.9726\n",
      "Epoch 5/10\n",
      "30946/30946 [==============================] - 38s 1ms/step - loss: 0.3224 - acc: 0.9702\n",
      "Epoch 6/10\n",
      "30946/30946 [==============================] - 38s 1ms/step - loss: 0.3198 - acc: 0.9676\n",
      "Epoch 7/10\n",
      "30946/30946 [==============================] - 38s 1ms/step - loss: 0.3156 - acc: 0.9659\n",
      "Epoch 8/10\n",
      "30946/30946 [==============================] - 38s 1ms/step - loss: 0.3120 - acc: 0.9646\n",
      "Epoch 9/10\n",
      "30946/30946 [==============================] - 38s 1ms/step - loss: 0.3094 - acc: 0.9592\n",
      "Epoch 10/10\n",
      "30946/30946 [==============================] - 38s 1ms/step - loss: 0.3088 - acc: 0.9571\n",
      "15472/15472 [==============================] - 14s 935us/step\n",
      "30946/30946 [==============================] - 26s 855us/step\n",
      "[CV] .................... epochs=10, lr=0.001, nodes=32, total= 7.1min\n",
      "[CV] epochs=10, lr=0.001, nodes=64 ...................................\n",
      "Epoch 1/10\n",
      "30945/30945 [==============================] - 73s 2ms/step - loss: 0.3561 - acc: 0.9807\n",
      "Epoch 2/10\n",
      "30945/30945 [==============================] - 61s 2ms/step - loss: 0.3341 - acc: 0.9778\n",
      "Epoch 3/10\n",
      "30945/30945 [==============================] - 61s 2ms/step - loss: 0.3249 - acc: 0.9749\n",
      "Epoch 4/10\n",
      "30945/30945 [==============================] - 61s 2ms/step - loss: 0.3188 - acc: 0.9719\n",
      "Epoch 5/10\n",
      "30945/30945 [==============================] - 59s 2ms/step - loss: 0.3143 - acc: 0.9656\n",
      "Epoch 6/10\n",
      "30945/30945 [==============================] - 57s 2ms/step - loss: 0.3086 - acc: 0.9597\n",
      "Epoch 7/10\n",
      "30945/30945 [==============================] - 56s 2ms/step - loss: 0.3055 - acc: 0.9588\n",
      "Epoch 8/10\n",
      "30945/30945 [==============================] - 56s 2ms/step - loss: 0.3015 - acc: 0.9559\n",
      "Epoch 9/10\n",
      "30945/30945 [==============================] - 55s 2ms/step - loss: 0.2992 - acc: 0.9491\n",
      "Epoch 10/10\n",
      "30945/30945 [==============================] - 56s 2ms/step - loss: 0.2956 - acc: 0.9456\n",
      "15473/15473 [==============================] - 14s 906us/step\n",
      "30945/30945 [==============================] - 24s 769us/step\n",
      "[CV] .................... epochs=10, lr=0.001, nodes=64, total=10.7min\n",
      "[CV] epochs=10, lr=0.001, nodes=64 ...................................\n",
      "Epoch 1/10\n",
      "30945/30945 [==============================] - 63s 2ms/step - loss: 0.3643 - acc: 0.9838\n",
      "Epoch 2/10\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3427 - acc: 0.9774\n",
      "Epoch 3/10\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3337 - acc: 0.9737\n",
      "Epoch 4/10\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3264 - acc: 0.9703\n",
      "Epoch 5/10\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3204 - acc: 0.9655\n",
      "Epoch 6/10\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3175 - acc: 0.9599\n",
      "Epoch 7/10\n",
      "30945/30945 [==============================] - 52s 2ms/step - loss: 0.3131 - acc: 0.9585\n",
      "Epoch 8/10\n",
      "30945/30945 [==============================] - 52s 2ms/step - loss: 0.3099 - acc: 0.9515\n",
      "Epoch 9/10\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3073 - acc: 0.9495\n",
      "Epoch 10/10\n",
      "30945/30945 [==============================] - 54s 2ms/step - loss: 0.3056 - acc: 0.9434\n",
      "15473/15473 [==============================] - 13s 853us/step\n",
      "30945/30945 [==============================] - 24s 776us/step\n",
      "[CV] .................... epochs=10, lr=0.001, nodes=64, total= 9.8min\n",
      "[CV] epochs=10, lr=0.001, nodes=64 ...................................\n",
      "Epoch 1/10\n",
      "30946/30946 [==============================] - 62s 2ms/step - loss: 0.3546 - acc: 0.9802\n",
      "Epoch 2/10\n",
      "30946/30946 [==============================] - 53s 2ms/step - loss: 0.3317 - acc: 0.9786\n",
      "Epoch 3/10\n",
      "30946/30946 [==============================] - 52s 2ms/step - loss: 0.3230 - acc: 0.9737\n",
      "Epoch 4/10\n",
      "30946/30946 [==============================] - 53s 2ms/step - loss: 0.3166 - acc: 0.9714\n",
      "Epoch 5/10\n",
      "30946/30946 [==============================] - 54s 2ms/step - loss: 0.3120 - acc: 0.9684\n",
      "Epoch 6/10\n",
      "30946/30946 [==============================] - 54s 2ms/step - loss: 0.3084 - acc: 0.9609\n",
      "Epoch 7/10\n",
      "30946/30946 [==============================] - 53s 2ms/step - loss: 0.3045 - acc: 0.9566\n",
      "Epoch 8/10\n",
      "30946/30946 [==============================] - 53s 2ms/step - loss: 0.3018 - acc: 0.9511\n",
      "Epoch 9/10\n",
      "30946/30946 [==============================] - 54s 2ms/step - loss: 0.2986 - acc: 0.9489\n",
      "Epoch 10/10\n",
      "30946/30946 [==============================] - 50s 2ms/step - loss: 0.2965 - acc: 0.9439\n",
      "15472/15472 [==============================] - 13s 848us/step\n",
      "30946/30946 [==============================] - 26s 830us/step\n",
      "[CV] .................... epochs=10, lr=0.001, nodes=64, total= 9.8min\n",
      "[CV] epochs=10, lr=0.002, nodes=16 ...................................\n",
      "Epoch 1/10\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3601 - acc: 0.9709\n",
      "Epoch 2/10\n",
      "30945/30945 [==============================] - 23s 742us/step - loss: 0.3402 - acc: 0.9831\n",
      "Epoch 3/10\n",
      "30945/30945 [==============================] - 23s 738us/step - loss: 0.3344 - acc: 0.9774\n",
      "Epoch 4/10\n",
      "30945/30945 [==============================] - 23s 745us/step - loss: 0.3306 - acc: 0.9756\n",
      "Epoch 5/10\n",
      "30945/30945 [==============================] - 23s 735us/step - loss: 0.3290 - acc: 0.9693\n",
      "Epoch 6/10\n",
      "30945/30945 [==============================] - 23s 731us/step - loss: 0.3258 - acc: 0.9688\n",
      "Epoch 7/10\n",
      "30945/30945 [==============================] - 22s 726us/step - loss: 0.3224 - acc: 0.9659\n",
      "Epoch 8/10\n",
      "30945/30945 [==============================] - 23s 731us/step - loss: 0.3201 - acc: 0.9626\n",
      "Epoch 9/10\n",
      "30945/30945 [==============================] - 22s 711us/step - loss: 0.3189 - acc: 0.9584\n",
      "Epoch 10/10\n",
      "30945/30945 [==============================] - 23s 738us/step - loss: 0.3169 - acc: 0.9568\n",
      "15473/15473 [==============================] - 12s 798us/step\n",
      "30945/30945 [==============================] - 22s 714us/step\n",
      "[CV] .................... epochs=10, lr=0.002, nodes=16, total= 4.7min\n",
      "[CV] epochs=10, lr=0.002, nodes=16 ...................................\n",
      "Epoch 1/10\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3694 - acc: 0.9723\n",
      "Epoch 2/10\n",
      "30945/30945 [==============================] - 22s 716us/step - loss: 0.3500 - acc: 0.9833\n",
      "Epoch 3/10\n",
      "30945/30945 [==============================] - 22s 712us/step - loss: 0.3439 - acc: 0.9797\n",
      "Epoch 4/10\n",
      "30945/30945 [==============================] - 22s 710us/step - loss: 0.3407 - acc: 0.9738\n",
      "Epoch 5/10\n",
      "30945/30945 [==============================] - 22s 710us/step - loss: 0.3354 - acc: 0.9691\n",
      "Epoch 6/10\n",
      "30945/30945 [==============================] - 22s 712us/step - loss: 0.3329 - acc: 0.9694\n",
      "Epoch 7/10\n",
      "30945/30945 [==============================] - 22s 712us/step - loss: 0.3292 - acc: 0.9608\n",
      "Epoch 8/10\n",
      "30945/30945 [==============================] - 22s 717us/step - loss: 0.3286 - acc: 0.9598\n",
      "Epoch 9/10\n",
      "30945/30945 [==============================] - 22s 716us/step - loss: 0.3275 - acc: 0.9554\n",
      "Epoch 10/10\n",
      "30945/30945 [==============================] - 22s 723us/step - loss: 0.3248 - acc: 0.9513\n",
      "15473/15473 [==============================] - 12s 796us/step\n",
      "30945/30945 [==============================] - 22s 721us/step\n",
      "[CV] .................... epochs=10, lr=0.002, nodes=16, total= 4.6min\n",
      "[CV] epochs=10, lr=0.002, nodes=16 ...................................\n",
      "Epoch 1/10\n",
      "30946/30946 [==============================] - 35s 1ms/step - loss: 0.3582 - acc: 0.9780\n",
      "Epoch 2/10\n",
      "30946/30946 [==============================] - 23s 739us/step - loss: 0.3391 - acc: 0.9829\n",
      "Epoch 3/10\n",
      "30946/30946 [==============================] - 22s 723us/step - loss: 0.3343 - acc: 0.9771\n",
      "Epoch 4/10\n",
      "30946/30946 [==============================] - 22s 720us/step - loss: 0.3311 - acc: 0.9725\n",
      "Epoch 5/10\n",
      "30946/30946 [==============================] - 22s 718us/step - loss: 0.3278 - acc: 0.9690\n",
      "Epoch 6/10\n",
      "30946/30946 [==============================] - 22s 719us/step - loss: 0.3245 - acc: 0.9660\n",
      "Epoch 7/10\n",
      "30946/30946 [==============================] - 22s 720us/step - loss: 0.3220 - acc: 0.9642\n",
      "Epoch 8/10\n",
      "30946/30946 [==============================] - 22s 722us/step - loss: 0.3193 - acc: 0.9606\n",
      "Epoch 9/10\n",
      "30946/30946 [==============================] - 22s 712us/step - loss: 0.3176 - acc: 0.9598\n",
      "Epoch 10/10\n",
      "30946/30946 [==============================] - 22s 722us/step - loss: 0.3147 - acc: 0.9576\n",
      "15472/15472 [==============================] - 12s 793us/step\n",
      "30946/30946 [==============================] - 22s 723us/step\n",
      "[CV] .................... epochs=10, lr=0.002, nodes=16, total= 4.7min\n",
      "[CV] epochs=10, lr=0.002, nodes=32 ...................................\n",
      "Epoch 1/10\n",
      "30945/30945 [==============================] - 48s 2ms/step - loss: 0.3539 - acc: 0.9795\n",
      "Epoch 2/10\n",
      "30945/30945 [==============================] - 36s 1ms/step - loss: 0.3353 - acc: 0.9778\n",
      "Epoch 3/10\n",
      "30945/30945 [==============================] - 37s 1ms/step - loss: 0.3276 - acc: 0.9752\n",
      "Epoch 4/10\n",
      "30945/30945 [==============================] - 37s 1ms/step - loss: 0.3207 - acc: 0.9723\n",
      "Epoch 5/10\n",
      "30945/30945 [==============================] - 38s 1ms/step - loss: 0.3176 - acc: 0.9687\n",
      "Epoch 6/10\n",
      "30945/30945 [==============================] - 37s 1ms/step - loss: 0.3130 - acc: 0.9643\n",
      "Epoch 7/10\n",
      "30945/30945 [==============================] - 37s 1ms/step - loss: 0.3112 - acc: 0.9614\n",
      "Epoch 8/10\n",
      "30945/30945 [==============================] - 38s 1ms/step - loss: 0.3082 - acc: 0.9553\n",
      "Epoch 9/10\n",
      "30945/30945 [==============================] - 37s 1ms/step - loss: 0.3045 - acc: 0.9519\n",
      "Epoch 10/10\n",
      "30945/30945 [==============================] - 37s 1ms/step - loss: 0.3018 - acc: 0.9487\n",
      "15473/15473 [==============================] - 12s 800us/step\n",
      "30945/30945 [==============================] - 22s 713us/step\n",
      "[CV] .................... epochs=10, lr=0.002, nodes=32, total= 7.1min\n",
      "[CV] epochs=10, lr=0.002, nodes=32 ...................................\n",
      "Epoch 1/10\n",
      "30945/30945 [==============================] - 46s 1ms/step - loss: 0.3632 - acc: 0.9803\n",
      "Epoch 2/10\n",
      "30945/30945 [==============================] - 33s 1ms/step - loss: 0.3445 - acc: 0.9795\n",
      "Epoch 3/10\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3367 - acc: 0.9724\n",
      "Epoch 4/10\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3307 - acc: 0.9690\n",
      "Epoch 5/10\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3288 - acc: 0.9625\n",
      "Epoch 6/10\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3243 - acc: 0.9632\n",
      "Epoch 7/10\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3197 - acc: 0.9583\n",
      "Epoch 8/10\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3204 - acc: 0.9520\n",
      "Epoch 9/10\n",
      "30945/30945 [==============================] - 33s 1ms/step - loss: 0.3159 - acc: 0.9495\n",
      "Epoch 10/10\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3138 - acc: 0.9470\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "15473/15473 [==============================] - 13s 852us/step\n",
      "30945/30945 [==============================] - 24s 770us/step\n",
      "[CV] .................... epochs=10, lr=0.002, nodes=32, total= 6.6min\n",
      "[CV] epochs=10, lr=0.002, nodes=32 ...................................\n",
      "Epoch 1/10\n",
      "30946/30946 [==============================] - 45s 1ms/step - loss: 0.3548 - acc: 0.9814\n",
      "Epoch 2/10\n",
      "30946/30946 [==============================] - 34s 1ms/step - loss: 0.3352 - acc: 0.9780\n",
      "Epoch 3/10\n",
      "30946/30946 [==============================] - 34s 1ms/step - loss: 0.3259 - acc: 0.9722\n",
      "Epoch 4/10\n",
      "30946/30946 [==============================] - 34s 1ms/step - loss: 0.3205 - acc: 0.9696\n",
      "Epoch 5/10\n",
      "30946/30946 [==============================] - 34s 1ms/step - loss: 0.3160 - acc: 0.9651\n",
      "Epoch 6/10\n",
      "30946/30946 [==============================] - 34s 1ms/step - loss: 0.3119 - acc: 0.9607\n",
      "Epoch 7/10\n",
      "30946/30946 [==============================] - 33s 1ms/step - loss: 0.3099 - acc: 0.9561\n",
      "Epoch 8/10\n",
      "30946/30946 [==============================] - 34s 1ms/step - loss: 0.3047 - acc: 0.9522\n",
      "Epoch 9/10\n",
      "30946/30946 [==============================] - 34s 1ms/step - loss: 0.3058 - acc: 0.9479\n",
      "Epoch 10/10\n",
      "30946/30946 [==============================] - 34s 1ms/step - loss: 0.3041 - acc: 0.9500\n",
      "15472/15472 [==============================] - 13s 851us/step\n",
      "30946/30946 [==============================] - 24s 776us/step\n",
      "[CV] .................... epochs=10, lr=0.002, nodes=32, total= 6.6min\n",
      "[CV] epochs=10, lr=0.002, nodes=64 ...................................\n",
      "Epoch 1/10\n",
      "30945/30945 [==============================] - 67s 2ms/step - loss: 0.3499 - acc: 0.9750\n",
      "Epoch 2/10\n",
      "30945/30945 [==============================] - 58s 2ms/step - loss: 0.3300 - acc: 0.9731\n",
      "Epoch 3/10\n",
      "30945/30945 [==============================] - 58s 2ms/step - loss: 0.3217 - acc: 0.9677\n",
      "Epoch 4/10\n",
      "30945/30945 [==============================] - 58s 2ms/step - loss: 0.3148 - acc: 0.9635\n",
      "Epoch 5/10\n",
      "30945/30945 [==============================] - 56s 2ms/step - loss: 0.3100 - acc: 0.9568\n",
      "Epoch 6/10\n",
      "30945/30945 [==============================] - 57s 2ms/step - loss: 0.3064 - acc: 0.9474\n",
      "Epoch 7/10\n",
      "30945/30945 [==============================] - 57s 2ms/step - loss: 0.3021 - acc: 0.9409\n",
      "Epoch 8/10\n",
      "30945/30945 [==============================] - 58s 2ms/step - loss: 0.3010 - acc: 0.9378\n",
      "Epoch 9/10\n",
      "30945/30945 [==============================] - 57s 2ms/step - loss: 0.3002 - acc: 0.9291\n",
      "Epoch 10/10\n",
      "30945/30945 [==============================] - 58s 2ms/step - loss: 0.2972 - acc: 0.9269\n",
      "15473/15473 [==============================] - 14s 873us/step\n",
      "30945/30945 [==============================] - 24s 787us/step\n",
      "[CV] .................... epochs=10, lr=0.002, nodes=64, total=10.5min\n",
      "[CV] epochs=10, lr=0.002, nodes=64 ...................................\n",
      "Epoch 1/10\n",
      "30945/30945 [==============================] - 65s 2ms/step - loss: 0.3631 - acc: 0.9803\n",
      "Epoch 2/10\n",
      "30945/30945 [==============================] - 55s 2ms/step - loss: 0.3417 - acc: 0.9754\n",
      "Epoch 3/10\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3341 - acc: 0.9694\n",
      "Epoch 4/10\n",
      "30945/30945 [==============================] - 54s 2ms/step - loss: 0.3273 - acc: 0.9643\n",
      "Epoch 5/10\n",
      "30945/30945 [==============================] - 54s 2ms/step - loss: 0.3216 - acc: 0.9594\n",
      "Epoch 6/10\n",
      "30945/30945 [==============================] - 55s 2ms/step - loss: 0.3171 - acc: 0.9568\n",
      "Epoch 7/10\n",
      "30945/30945 [==============================] - 54s 2ms/step - loss: 0.3137 - acc: 0.9479\n",
      "Epoch 8/10\n",
      "30945/30945 [==============================] - 55s 2ms/step - loss: 0.3091 - acc: 0.9446\n",
      "Epoch 9/10\n",
      "30945/30945 [==============================] - 54s 2ms/step - loss: 0.3080 - acc: 0.9408\n",
      "Epoch 10/10\n",
      "30945/30945 [==============================] - 54s 2ms/step - loss: 0.3070 - acc: 0.9382\n",
      "15473/15473 [==============================] - 14s 909us/step\n",
      "30945/30945 [==============================] - 26s 826us/step\n",
      "[CV] .................... epochs=10, lr=0.002, nodes=64, total=10.0min\n",
      "[CV] epochs=10, lr=0.002, nodes=64 ...................................\n",
      "Epoch 1/10\n",
      "30946/30946 [==============================] - 65s 2ms/step - loss: 0.3508 - acc: 0.9796\n",
      "Epoch 2/10\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.3297 - acc: 0.9765\n",
      "Epoch 3/10\n",
      "30946/30946 [==============================] - 53s 2ms/step - loss: 0.3221 - acc: 0.9693\n",
      "Epoch 4/10\n",
      "30946/30946 [==============================] - 54s 2ms/step - loss: 0.3160 - acc: 0.9637\n",
      "Epoch 5/10\n",
      "30946/30946 [==============================] - 54s 2ms/step - loss: 0.3109 - acc: 0.9568\n",
      "Epoch 6/10\n",
      "30946/30946 [==============================] - 53s 2ms/step - loss: 0.3077 - acc: 0.9508\n",
      "Epoch 7/10\n",
      "30946/30946 [==============================] - 53s 2ms/step - loss: 0.3049 - acc: 0.9450\n",
      "Epoch 8/10\n",
      "30946/30946 [==============================] - 54s 2ms/step - loss: 0.3030 - acc: 0.9413\n",
      "Epoch 9/10\n",
      "30946/30946 [==============================] - 54s 2ms/step - loss: 0.2981 - acc: 0.9332\n",
      "Epoch 10/10\n",
      "30946/30946 [==============================] - 54s 2ms/step - loss: 0.2986 - acc: 0.9289\n",
      "15472/15472 [==============================] - 14s 917us/step\n",
      "30946/30946 [==============================] - 24s 765us/step\n",
      "[CV] .................... epochs=10, lr=0.002, nodes=64, total= 9.9min\n",
      "[CV] epochs=10, lr=0.003, nodes=16 ...................................\n",
      "Epoch 1/10\n",
      "30945/30945 [==============================] - 35s 1ms/step - loss: 0.3543 - acc: 0.9575\n",
      "Epoch 2/10\n",
      "30945/30945 [==============================] - 23s 743us/step - loss: 0.3372 - acc: 0.9789\n",
      "Epoch 3/10\n",
      "30945/30945 [==============================] - 23s 732us/step - loss: 0.3319 - acc: 0.9771\n",
      "Epoch 4/10\n",
      "30945/30945 [==============================] - 23s 733us/step - loss: 0.3288 - acc: 0.9730\n",
      "Epoch 5/10\n",
      "30945/30945 [==============================] - 23s 732us/step - loss: 0.3250 - acc: 0.9677\n",
      "Epoch 6/10\n",
      "30945/30945 [==============================] - 22s 707us/step - loss: 0.3217 - acc: 0.9682\n",
      "Epoch 7/10\n",
      "30945/30945 [==============================] - 22s 724us/step - loss: 0.3197 - acc: 0.9630\n",
      "Epoch 8/10\n",
      "30945/30945 [==============================] - 23s 734us/step - loss: 0.3182 - acc: 0.9570\n",
      "Epoch 9/10\n",
      "30945/30945 [==============================] - 23s 733us/step - loss: 0.3164 - acc: 0.9526\n",
      "Epoch 10/10\n",
      "30945/30945 [==============================] - 23s 734us/step - loss: 0.3177 - acc: 0.9496\n",
      "15473/15473 [==============================] - 12s 792us/step\n",
      "30945/30945 [==============================] - 23s 730us/step\n",
      "[CV] .................... epochs=10, lr=0.003, nodes=16, total= 4.7min\n",
      "[CV] epochs=10, lr=0.003, nodes=16 ...................................\n",
      "Epoch 1/10\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3635 - acc: 0.9752\n",
      "Epoch 2/10\n",
      "30945/30945 [==============================] - 19s 608us/step - loss: 0.3477 - acc: 0.9794\n",
      "Epoch 3/10\n",
      "30945/30945 [==============================] - 19s 610us/step - loss: 0.3434 - acc: 0.9747\n",
      "Epoch 4/10\n",
      "30945/30945 [==============================] - 19s 604us/step - loss: 0.3400 - acc: 0.9707\n",
      "Epoch 5/10\n",
      "30945/30945 [==============================] - 18s 585us/step - loss: 0.3377 - acc: 0.9707\n",
      "Epoch 6/10\n",
      "30945/30945 [==============================] - 19s 608us/step - loss: 0.3339 - acc: 0.9617\n",
      "Epoch 7/10\n",
      "30945/30945 [==============================] - 19s 598us/step - loss: 0.3309 - acc: 0.9575\n",
      "Epoch 8/10\n",
      "30945/30945 [==============================] - 19s 609us/step - loss: 0.3303 - acc: 0.9576\n",
      "Epoch 9/10\n",
      "30945/30945 [==============================] - 19s 600us/step - loss: 0.3297 - acc: 0.9534\n",
      "Epoch 10/10\n",
      "30945/30945 [==============================] - 19s 612us/step - loss: 0.3277 - acc: 0.9475\n",
      "15473/15473 [==============================] - 11s 689us/step\n",
      "30945/30945 [==============================] - 19s 607us/step\n",
      "[CV] .................... epochs=10, lr=0.003, nodes=16, total= 4.0min\n",
      "[CV] epochs=10, lr=0.003, nodes=16 ...................................\n",
      "Epoch 1/10\n",
      "30946/30946 [==============================] - 31s 1ms/step - loss: 0.3553 - acc: 0.9826\n",
      "Epoch 2/10\n",
      "30946/30946 [==============================] - 19s 618us/step - loss: 0.3363 - acc: 0.9818\n",
      "Epoch 3/10\n",
      "30946/30946 [==============================] - 19s 623us/step - loss: 0.3281 - acc: 0.9751\n",
      "Epoch 4/10\n",
      "30946/30946 [==============================] - 19s 622us/step - loss: 0.3245 - acc: 0.9675\n",
      "Epoch 5/10\n",
      "30946/30946 [==============================] - 19s 627us/step - loss: 0.3230 - acc: 0.9649\n",
      "Epoch 6/10\n",
      "30946/30946 [==============================] - 19s 619us/step - loss: 0.3188 - acc: 0.9581\n",
      "Epoch 7/10\n",
      "30946/30946 [==============================] - 20s 632us/step - loss: 0.3160 - acc: 0.9534\n",
      "Epoch 8/10\n",
      "30946/30946 [==============================] - 19s 621us/step - loss: 0.3147 - acc: 0.9505\n",
      "Epoch 9/10\n",
      "30946/30946 [==============================] - 20s 637us/step - loss: 0.3142 - acc: 0.9454\n",
      "Epoch 10/10\n",
      "30946/30946 [==============================] - 20s 635us/step - loss: 0.3123 - acc: 0.9435\n",
      "15472/15472 [==============================] - 11s 692us/step\n",
      "30946/30946 [==============================] - 19s 612us/step\n",
      "[CV] .................... epochs=10, lr=0.003, nodes=16, total= 4.2min\n",
      "[CV] epochs=10, lr=0.003, nodes=32 ...................................\n",
      "Epoch 1/10\n",
      "30945/30945 [==============================] - 42s 1ms/step - loss: 0.3524 - acc: 0.9763\n",
      "Epoch 2/10\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3349 - acc: 0.9775\n",
      "Epoch 3/10\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3278 - acc: 0.9733\n",
      "Epoch 4/10\n",
      "30945/30945 [==============================] - 32s 1ms/step - loss: 0.3229 - acc: 0.9684\n",
      "Epoch 5/10\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3201 - acc: 0.9621\n",
      "Epoch 6/10\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3146 - acc: 0.9584\n",
      "Epoch 7/10\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3112 - acc: 0.9537\n",
      "Epoch 8/10\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3088 - acc: 0.9480\n",
      "Epoch 9/10\n",
      "30945/30945 [==============================] - 32s 1ms/step - loss: 0.3088 - acc: 0.9463\n",
      "Epoch 10/10\n",
      "30945/30945 [==============================] - 32s 1ms/step - loss: 0.3069 - acc: 0.9414\n",
      "15473/15473 [==============================] - 13s 863us/step\n",
      "30945/30945 [==============================] - 24s 777us/step\n",
      "[CV] .................... epochs=10, lr=0.003, nodes=32, total= 6.2min\n",
      "[CV] epochs=10, lr=0.003, nodes=32 ...................................\n",
      "Epoch 1/10\n",
      "30945/30945 [==============================] - 45s 1ms/step - loss: 0.3633 - acc: 0.9725\n",
      "Epoch 2/10\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3441 - acc: 0.9799\n",
      "Epoch 3/10\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3371 - acc: 0.9718\n",
      "Epoch 4/10\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3308 - acc: 0.9659\n",
      "Epoch 5/10\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3280 - acc: 0.9571\n",
      "Epoch 6/10\n",
      "30945/30945 [==============================] - 35s 1ms/step - loss: 0.3259 - acc: 0.9502\n",
      "Epoch 7/10\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3234 - acc: 0.9452\n",
      "Epoch 8/10\n",
      "30945/30945 [==============================] - 33s 1ms/step - loss: 0.3215 - acc: 0.9444\n",
      "Epoch 9/10\n",
      "30945/30945 [==============================] - 35s 1ms/step - loss: 0.3199 - acc: 0.9354\n",
      "Epoch 10/10\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3191 - acc: 0.9344\n",
      "15473/15473 [==============================] - 13s 855us/step\n",
      "30945/30945 [==============================] - 24s 764us/step\n",
      "[CV] .................... epochs=10, lr=0.003, nodes=32, total= 6.6min\n",
      "[CV] epochs=10, lr=0.003, nodes=32 ...................................\n",
      "Epoch 1/10\n",
      "30946/30946 [==============================] - 45s 1ms/step - loss: 0.3513 - acc: 0.9797\n",
      "Epoch 2/10\n",
      "30946/30946 [==============================] - 34s 1ms/step - loss: 0.3349 - acc: 0.9799\n",
      "Epoch 3/10\n",
      "30946/30946 [==============================] - 33s 1ms/step - loss: 0.3254 - acc: 0.9713\n",
      "Epoch 4/10\n",
      "30946/30946 [==============================] - 33s 1ms/step - loss: 0.3198 - acc: 0.9646\n",
      "Epoch 5/10\n",
      "30946/30946 [==============================] - 33s 1ms/step - loss: 0.3155 - acc: 0.9591\n",
      "Epoch 6/10\n",
      "30946/30946 [==============================] - 33s 1ms/step - loss: 0.3125 - acc: 0.9549\n",
      "Epoch 7/10\n",
      "30946/30946 [==============================] - 33s 1ms/step - loss: 0.3106 - acc: 0.9427\n",
      "Epoch 8/10\n",
      "30946/30946 [==============================] - 33s 1ms/step - loss: 0.3087 - acc: 0.9396\n",
      "Epoch 9/10\n",
      "30946/30946 [==============================] - 34s 1ms/step - loss: 0.3074 - acc: 0.9336\n",
      "Epoch 10/10\n",
      "30946/30946 [==============================] - 34s 1ms/step - loss: 0.3075 - acc: 0.9259\n",
      "15472/15472 [==============================] - 13s 863us/step\n",
      "30946/30946 [==============================] - 24s 777us/step\n",
      "[CV] .................... epochs=10, lr=0.003, nodes=32, total= 6.5min\n",
      "[CV] epochs=10, lr=0.003, nodes=64 ...................................\n",
      "Epoch 1/10\n",
      "30945/30945 [==============================] - 65s 2ms/step - loss: 0.3491 - acc: 0.9751\n",
      "Epoch 2/10\n",
      "30945/30945 [==============================] - 55s 2ms/step - loss: 0.3304 - acc: 0.9719\n",
      "Epoch 3/10\n",
      "30945/30945 [==============================] - 55s 2ms/step - loss: 0.3219 - acc: 0.9677\n",
      "Epoch 4/10\n",
      "30945/30945 [==============================] - 56s 2ms/step - loss: 0.3147 - acc: 0.9589\n",
      "Epoch 5/10\n",
      "30945/30945 [==============================] - 54s 2ms/step - loss: 0.3111 - acc: 0.9520\n",
      "Epoch 6/10\n",
      "30945/30945 [==============================] - 55s 2ms/step - loss: 0.3078 - acc: 0.9498\n",
      "Epoch 7/10\n",
      "30945/30945 [==============================] - 55s 2ms/step - loss: 0.3061 - acc: 0.9427\n",
      "Epoch 8/10\n",
      "30945/30945 [==============================] - 55s 2ms/step - loss: 0.3074 - acc: 0.9406\n",
      "Epoch 9/10\n",
      "30945/30945 [==============================] - 55s 2ms/step - loss: 0.3020 - acc: 0.9376\n",
      "Epoch 10/10\n",
      "30945/30945 [==============================] - 55s 2ms/step - loss: 0.3011 - acc: 0.9271\n",
      "15473/15473 [==============================] - 14s 917us/step\n",
      "30945/30945 [==============================] - 26s 845us/step\n",
      "[CV] .................... epochs=10, lr=0.003, nodes=64, total=10.1min\n",
      "[CV] epochs=10, lr=0.003, nodes=64 ...................................\n",
      "Epoch 1/10\n",
      "30945/30945 [==============================] - 66s 2ms/step - loss: 0.3603 - acc: 0.9801\n",
      "Epoch 2/10\n",
      "30945/30945 [==============================] - 56s 2ms/step - loss: 0.3412 - acc: 0.9748\n",
      "Epoch 3/10\n",
      "30945/30945 [==============================] - 51s 2ms/step - loss: 0.3308 - acc: 0.9662\n",
      "Epoch 4/10\n",
      "30945/30945 [==============================] - 51s 2ms/step - loss: 0.3260 - acc: 0.9568\n",
      "Epoch 5/10\n",
      "30945/30945 [==============================] - 51s 2ms/step - loss: 0.3207 - acc: 0.9554\n",
      "Epoch 6/10\n",
      "30945/30945 [==============================] - 52s 2ms/step - loss: 0.3165 - acc: 0.9516\n",
      "Epoch 7/10\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3148 - acc: 0.9438\n",
      "Epoch 8/10\n",
      "30945/30945 [==============================] - 56s 2ms/step - loss: 0.3136 - acc: 0.9408\n",
      "Epoch 9/10\n",
      "30945/30945 [==============================] - 56s 2ms/step - loss: 0.3128 - acc: 0.9390\n",
      "Epoch 10/10\n",
      "30945/30945 [==============================] - 56s 2ms/step - loss: 0.3112 - acc: 0.9342\n",
      "15473/15473 [==============================] - 14s 937us/step\n",
      "30945/30945 [==============================] - 26s 846us/step\n",
      "[CV] .................... epochs=10, lr=0.003, nodes=64, total= 9.9min\n",
      "[CV] epochs=10, lr=0.003, nodes=64 ...................................\n",
      "Epoch 1/10\n",
      "30946/30946 [==============================] - 65s 2ms/step - loss: 0.3504 - acc: 0.9786\n",
      "Epoch 2/10\n",
      "30946/30946 [==============================] - 54s 2ms/step - loss: 0.3322 - acc: 0.9749\n",
      "Epoch 3/10\n",
      "30946/30946 [==============================] - 54s 2ms/step - loss: 0.3201 - acc: 0.9655\n",
      "Epoch 4/10\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.3149 - acc: 0.9567\n",
      "Epoch 5/10\n",
      "30946/30946 [==============================] - 54s 2ms/step - loss: 0.3093 - acc: 0.9545\n",
      "Epoch 6/10\n",
      "30946/30946 [==============================] - 54s 2ms/step - loss: 0.3076 - acc: 0.9505\n",
      "Epoch 7/10\n",
      "30946/30946 [==============================] - 54s 2ms/step - loss: 0.3042 - acc: 0.9391\n",
      "Epoch 8/10\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "30946/30946 [==============================] - 54s 2ms/step - loss: 0.3025 - acc: 0.9366\n",
      "Epoch 9/10\n",
      "30946/30946 [==============================] - 54s 2ms/step - loss: 0.3007 - acc: 0.9303\n",
      "Epoch 10/10\n",
      "30946/30946 [==============================] - 54s 2ms/step - loss: 0.3020 - acc: 0.9212\n",
      "15472/15472 [==============================] - 14s 935us/step\n",
      "30946/30946 [==============================] - 24s 778us/step\n",
      "[CV] .................... epochs=10, lr=0.003, nodes=64, total=10.0min\n",
      "[CV] epochs=20, lr=0.001, nodes=16 ...................................\n",
      "Epoch 1/20\n",
      "30945/30945 [==============================] - 35s 1ms/step - loss: 0.3706 - acc: 0.9596\n",
      "Epoch 2/20\n",
      "30945/30945 [==============================] - 23s 734us/step - loss: 0.3420 - acc: 0.9811\n",
      "Epoch 3/20\n",
      "30945/30945 [==============================] - 22s 700us/step - loss: 0.3362 - acc: 0.9808\n",
      "Epoch 4/20\n",
      "30945/30945 [==============================] - 23s 733us/step - loss: 0.3320 - acc: 0.9792\n",
      "Epoch 5/20\n",
      "30945/30945 [==============================] - 23s 733us/step - loss: 0.3284 - acc: 0.9772\n",
      "Epoch 6/20\n",
      "30945/30945 [==============================] - 23s 739us/step - loss: 0.3237 - acc: 0.9749\n",
      "Epoch 7/20\n",
      "30945/30945 [==============================] - 23s 738us/step - loss: 0.3222 - acc: 0.9709\n",
      "Epoch 8/20\n",
      "30945/30945 [==============================] - 23s 750us/step - loss: 0.3200 - acc: 0.9684\n",
      "Epoch 9/20\n",
      "30945/30945 [==============================] - 23s 747us/step - loss: 0.3179 - acc: 0.9669\n",
      "Epoch 10/20\n",
      "30945/30945 [==============================] - 22s 718us/step - loss: 0.3160 - acc: 0.9606\n",
      "Epoch 11/20\n",
      "30945/30945 [==============================] - 22s 724us/step - loss: 0.3133 - acc: 0.9587\n",
      "Epoch 12/20\n",
      "30945/30945 [==============================] - 23s 728us/step - loss: 0.3125 - acc: 0.9560\n",
      "Epoch 13/20\n",
      "30945/30945 [==============================] - 23s 727us/step - loss: 0.3106 - acc: 0.9552\n",
      "Epoch 14/20\n",
      "30945/30945 [==============================] - 23s 729us/step - loss: 0.3105 - acc: 0.9492\n",
      "Epoch 15/20\n",
      "30945/30945 [==============================] - 23s 729us/step - loss: 0.3086 - acc: 0.9478\n",
      "Epoch 16/20\n",
      "30945/30945 [==============================] - 23s 728us/step - loss: 0.3084 - acc: 0.9427\n",
      "Epoch 17/20\n",
      "30945/30945 [==============================] - 23s 739us/step - loss: 0.3089 - acc: 0.9461\n",
      "Epoch 18/20\n",
      "30945/30945 [==============================] - 22s 724us/step - loss: 0.3062 - acc: 0.9425\n",
      "Epoch 19/20\n",
      "30945/30945 [==============================] - 22s 722us/step - loss: 0.3033 - acc: 0.9362\n",
      "Epoch 20/20\n",
      "30945/30945 [==============================] - 22s 722us/step - loss: 0.3020 - acc: 0.9394\n",
      "15473/15473 [==============================] - 12s 802us/step\n",
      "30945/30945 [==============================] - 23s 737us/step\n",
      "[CV] .................... epochs=20, lr=0.001, nodes=16, total= 8.5min\n",
      "[CV] epochs=20, lr=0.001, nodes=16 ...................................\n",
      "Epoch 1/20\n",
      "30945/30945 [==============================] - 32s 1ms/step - loss: 0.3778 - acc: 0.9414\n",
      "Epoch 2/20\n",
      "30945/30945 [==============================] - 18s 580us/step - loss: 0.3517 - acc: 0.9800\n",
      "Epoch 3/20\n",
      "30945/30945 [==============================] - 19s 601us/step - loss: 0.3457 - acc: 0.9793\n",
      "Epoch 4/20\n",
      "30945/30945 [==============================] - 19s 606us/step - loss: 0.3419 - acc: 0.9802\n",
      "Epoch 5/20\n",
      "30945/30945 [==============================] - 19s 605us/step - loss: 0.3397 - acc: 0.9772\n",
      "Epoch 6/20\n",
      "30945/30945 [==============================] - 19s 608us/step - loss: 0.3374 - acc: 0.9751\n",
      "Epoch 7/20\n",
      "30945/30945 [==============================] - 19s 602us/step - loss: 0.3344 - acc: 0.9729\n",
      "Epoch 8/20\n",
      "30945/30945 [==============================] - 19s 606us/step - loss: 0.3313 - acc: 0.9694\n",
      "Epoch 9/20\n",
      "30945/30945 [==============================] - 19s 605us/step - loss: 0.3293 - acc: 0.9673\n",
      "Epoch 10/20\n",
      "30945/30945 [==============================] - 19s 603us/step - loss: 0.3280 - acc: 0.9665\n",
      "Epoch 11/20\n",
      "30945/30945 [==============================] - 19s 612us/step - loss: 0.3270 - acc: 0.9624\n",
      "Epoch 12/20\n",
      "30945/30945 [==============================] - 19s 612us/step - loss: 0.3251 - acc: 0.9591\n",
      "Epoch 13/20\n",
      "30945/30945 [==============================] - 19s 608us/step - loss: 0.3230 - acc: 0.9547\n",
      "Epoch 14/20\n",
      "30945/30945 [==============================] - 19s 610us/step - loss: 0.3214 - acc: 0.9527\n",
      "Epoch 15/20\n",
      "30945/30945 [==============================] - 19s 612us/step - loss: 0.3203 - acc: 0.9525\n",
      "Epoch 16/20\n",
      "30945/30945 [==============================] - 19s 616us/step - loss: 0.3199 - acc: 0.9492\n",
      "Epoch 17/20\n",
      "30945/30945 [==============================] - 19s 619us/step - loss: 0.3194 - acc: 0.9487\n",
      "Epoch 18/20\n",
      "30945/30945 [==============================] - 19s 620us/step - loss: 0.3184 - acc: 0.9430\n",
      "Epoch 19/20\n",
      "30945/30945 [==============================] - 19s 615us/step - loss: 0.3157 - acc: 0.9447\n",
      "Epoch 20/20\n",
      "30945/30945 [==============================] - 19s 617us/step - loss: 0.3167 - acc: 0.9394\n",
      "15473/15473 [==============================] - 11s 712us/step\n",
      "30945/30945 [==============================] - 19s 616us/step\n",
      "[CV] .................... epochs=20, lr=0.001, nodes=16, total= 7.2min\n",
      "[CV] epochs=20, lr=0.001, nodes=16 ...................................\n",
      "Epoch 1/20\n",
      "30946/30946 [==============================] - 32s 1ms/step - loss: 0.3643 - acc: 0.9891\n",
      "Epoch 2/20\n",
      "30946/30946 [==============================] - 19s 615us/step - loss: 0.3431 - acc: 0.9850\n",
      "Epoch 3/20\n",
      "30946/30946 [==============================] - 19s 604us/step - loss: 0.3377 - acc: 0.9829\n",
      "Epoch 4/20\n",
      "30946/30946 [==============================] - 19s 606us/step - loss: 0.3341 - acc: 0.9783\n",
      "Epoch 5/20\n",
      "30946/30946 [==============================] - 19s 601us/step - loss: 0.3296 - acc: 0.9772\n",
      "Epoch 6/20\n",
      "30946/30946 [==============================] - 19s 608us/step - loss: 0.3285 - acc: 0.9738\n",
      "Epoch 7/20\n",
      "30946/30946 [==============================] - 19s 607us/step - loss: 0.3236 - acc: 0.9717\n",
      "Epoch 8/20\n",
      "30946/30946 [==============================] - 19s 604us/step - loss: 0.3222 - acc: 0.9698\n",
      "Epoch 9/20\n",
      "30946/30946 [==============================] - 19s 601us/step - loss: 0.3192 - acc: 0.9684\n",
      "Epoch 10/20\n",
      "30946/30946 [==============================] - 19s 606us/step - loss: 0.3191 - acc: 0.9673\n",
      "Epoch 11/20\n",
      "30946/30946 [==============================] - 19s 605us/step - loss: 0.3171 - acc: 0.9632\n",
      "Epoch 12/20\n",
      "30946/30946 [==============================] - 19s 609us/step - loss: 0.3138 - acc: 0.9603\n",
      "Epoch 13/20\n",
      "30946/30946 [==============================] - 18s 591us/step - loss: 0.3149 - acc: 0.9626\n",
      "Epoch 14/20\n",
      "30946/30946 [==============================] - 18s 596us/step - loss: 0.3126 - acc: 0.9551\n",
      "Epoch 15/20\n",
      "30946/30946 [==============================] - 19s 604us/step - loss: 0.3119 - acc: 0.9566\n",
      "Epoch 16/20\n",
      "30946/30946 [==============================] - 19s 609us/step - loss: 0.3099 - acc: 0.9542\n",
      "Epoch 17/20\n",
      "30946/30946 [==============================] - 18s 597us/step - loss: 0.3092 - acc: 0.9496\n",
      "Epoch 18/20\n",
      "30946/30946 [==============================] - 18s 582us/step - loss: 0.3087 - acc: 0.9467\n",
      "Epoch 19/20\n",
      "30946/30946 [==============================] - 19s 616us/step - loss: 0.3077 - acc: 0.9408\n",
      "Epoch 20/20\n",
      "30946/30946 [==============================] - 19s 603us/step - loss: 0.3080 - acc: 0.9418\n",
      "15472/15472 [==============================] - 11s 715us/step\n",
      "30946/30946 [==============================] - 19s 618us/step\n",
      "[CV] .................... epochs=20, lr=0.001, nodes=16, total= 7.2min\n",
      "[CV] epochs=20, lr=0.001, nodes=32 ...................................\n",
      "Epoch 1/20\n",
      "30945/30945 [==============================] - 44s 1ms/step - loss: 0.3614 - acc: 0.9747\n",
      "Epoch 2/20\n",
      "30945/30945 [==============================] - 32s 1ms/step - loss: 0.3379 - acc: 0.9800\n",
      "Epoch 3/20\n",
      "30945/30945 [==============================] - 32s 1ms/step - loss: 0.3304 - acc: 0.9764\n",
      "Epoch 4/20\n",
      "30945/30945 [==============================] - 32s 1ms/step - loss: 0.3243 - acc: 0.9743\n",
      "Epoch 5/20\n",
      "30945/30945 [==============================] - 32s 1ms/step - loss: 0.3197 - acc: 0.9721\n",
      "Epoch 6/20\n",
      "30945/30945 [==============================] - 32s 1ms/step - loss: 0.3164 - acc: 0.9699\n",
      "Epoch 7/20\n",
      "30945/30945 [==============================] - 33s 1ms/step - loss: 0.3129 - acc: 0.9663\n",
      "Epoch 8/20\n",
      "30945/30945 [==============================] - 33s 1ms/step - loss: 0.3110 - acc: 0.9641\n",
      "Epoch 9/20\n",
      "30945/30945 [==============================] - 32s 1ms/step - loss: 0.3069 - acc: 0.9635\n",
      "Epoch 10/20\n",
      "30945/30945 [==============================] - 33s 1ms/step - loss: 0.3040 - acc: 0.9585\n",
      "Epoch 11/20\n",
      "30945/30945 [==============================] - 33s 1ms/step - loss: 0.3008 - acc: 0.9570\n",
      "Epoch 12/20\n",
      "30945/30945 [==============================] - 33s 1ms/step - loss: 0.3009 - acc: 0.9533\n",
      "Epoch 13/20\n",
      "30945/30945 [==============================] - 33s 1ms/step - loss: 0.2995 - acc: 0.9514\n",
      "Epoch 14/20\n",
      "30945/30945 [==============================] - 33s 1ms/step - loss: 0.2976 - acc: 0.9474\n",
      "Epoch 15/20\n",
      "30945/30945 [==============================] - 33s 1ms/step - loss: 0.2963 - acc: 0.9437\n",
      "Epoch 16/20\n",
      "30945/30945 [==============================] - 33s 1ms/step - loss: 0.2962 - acc: 0.9406\n",
      "Epoch 17/20\n",
      "30945/30945 [==============================] - 33s 1ms/step - loss: 0.2932 - acc: 0.9392\n",
      "Epoch 18/20\n",
      "30945/30945 [==============================] - 33s 1ms/step - loss: 0.2929 - acc: 0.9384\n",
      "Epoch 19/20\n",
      "30945/30945 [==============================] - 33s 1ms/step - loss: 0.2923 - acc: 0.9351\n",
      "Epoch 20/20\n",
      "30945/30945 [==============================] - 33s 1ms/step - loss: 0.2902 - acc: 0.9287\n",
      "15473/15473 [==============================] - 14s 916us/step\n",
      "30945/30945 [==============================] - 24s 780us/step\n",
      "[CV] .................... epochs=20, lr=0.001, nodes=32, total=11.9min\n",
      "[CV] epochs=20, lr=0.001, nodes=32 ...................................\n",
      "Epoch 1/20\n",
      "30945/30945 [==============================] - 42s 1ms/step - loss: 0.3700 - acc: 0.9682\n",
      "Epoch 2/20\n",
      "30945/30945 [==============================] - 31s 989us/step - loss: 0.3489 - acc: 0.9814\n",
      "Epoch 3/20\n",
      "30945/30945 [==============================] - 31s 991us/step - loss: 0.3422 - acc: 0.9787\n",
      "Epoch 4/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3350 - acc: 0.9767\n",
      "Epoch 5/20\n",
      "30945/30945 [==============================] - 31s 995us/step - loss: 0.3303 - acc: 0.9709\n",
      "Epoch 6/20\n",
      "30945/30945 [==============================] - 30s 981us/step - loss: 0.3269 - acc: 0.9667\n",
      "Epoch 7/20\n",
      "30945/30945 [==============================] - 31s 998us/step - loss: 0.3234 - acc: 0.9648\n",
      "Epoch 8/20\n",
      "30945/30945 [==============================] - 31s 991us/step - loss: 0.3210 - acc: 0.9608\n",
      "Epoch 9/20\n",
      "30945/30945 [==============================] - 31s 994us/step - loss: 0.3184 - acc: 0.9541\n",
      "Epoch 10/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3161 - acc: 0.9528\n",
      "Epoch 11/20\n",
      "30945/30945 [==============================] - 31s 999us/step - loss: 0.3166 - acc: 0.9467\n",
      "Epoch 12/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3121 - acc: 0.9400\n",
      "Epoch 13/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3131 - acc: 0.9384\n",
      "Epoch 14/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3089 - acc: 0.9363\n",
      "Epoch 15/20\n",
      "30945/30945 [==============================] - 31s 996us/step - loss: 0.3083 - acc: 0.9273\n",
      "Epoch 16/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3073 - acc: 0.9307\n",
      "Epoch 17/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3072 - acc: 0.9251\n",
      "Epoch 18/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3065 - acc: 0.9267\n",
      "Epoch 19/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3051 - acc: 0.9224\n",
      "Epoch 20/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3058 - acc: 0.9233\n",
      "15473/15473 [==============================] - 14s 879us/step\n",
      "30945/30945 [==============================] - 24s 778us/step\n",
      "[CV] .................... epochs=20, lr=0.001, nodes=32, total=11.3min\n",
      "[CV] epochs=20, lr=0.001, nodes=32 ...................................\n",
      "Epoch 1/20\n",
      "30946/30946 [==============================] - 42s 1ms/step - loss: 0.3591 - acc: 0.9844\n",
      "Epoch 2/20\n",
      "30946/30946 [==============================] - 30s 981us/step - loss: 0.3369 - acc: 0.9804\n",
      "Epoch 3/20\n",
      "30946/30946 [==============================] - 30s 962us/step - loss: 0.3284 - acc: 0.9768\n",
      "Epoch 4/20\n",
      "30946/30946 [==============================] - 30s 982us/step - loss: 0.3220 - acc: 0.9723\n",
      "Epoch 5/20\n",
      "30946/30946 [==============================] - 31s 986us/step - loss: 0.3177 - acc: 0.9710\n",
      "Epoch 6/20\n",
      "30946/30946 [==============================] - 31s 1ms/step - loss: 0.3143 - acc: 0.9665\n",
      "Epoch 7/20\n",
      "30946/30946 [==============================] - 31s 1ms/step - loss: 0.3114 - acc: 0.9625\n",
      "Epoch 8/20\n",
      "30946/30946 [==============================] - 31s 1ms/step - loss: 0.3084 - acc: 0.9617\n",
      "Epoch 9/20\n",
      "30946/30946 [==============================] - 32s 1ms/step - loss: 0.3062 - acc: 0.9609\n",
      "Epoch 10/20\n",
      "30946/30946 [==============================] - 31s 1ms/step - loss: 0.3037 - acc: 0.9549\n",
      "Epoch 11/20\n",
      "30946/30946 [==============================] - 31s 1ms/step - loss: 0.3036 - acc: 0.9508\n",
      "Epoch 12/20\n",
      "30946/30946 [==============================] - 31s 1ms/step - loss: 0.3011 - acc: 0.9471\n",
      "Epoch 13/20\n",
      "30946/30946 [==============================] - 31s 1ms/step - loss: 0.3000 - acc: 0.9469\n",
      "Epoch 14/20\n",
      "30946/30946 [==============================] - 31s 1ms/step - loss: 0.2984 - acc: 0.9406\n",
      "Epoch 15/20\n",
      "30946/30946 [==============================] - 31s 995us/step - loss: 0.2976 - acc: 0.9379\n",
      "Epoch 16/20\n",
      "30946/30946 [==============================] - 31s 989us/step - loss: 0.2954 - acc: 0.9354\n",
      "Epoch 17/20\n",
      "30946/30946 [==============================] - 31s 1ms/step - loss: 0.2938 - acc: 0.9320\n",
      "Epoch 18/20\n",
      "30946/30946 [==============================] - 31s 1ms/step - loss: 0.2943 - acc: 0.9239\n",
      "Epoch 19/20\n",
      "30946/30946 [==============================] - 30s 983us/step - loss: 0.2933 - acc: 0.9229\n",
      "Epoch 20/20\n",
      "30946/30946 [==============================] - 31s 999us/step - loss: 0.2913 - acc: 0.9210\n",
      "15472/15472 [==============================] - 13s 870us/step\n",
      "30946/30946 [==============================] - 22s 727us/step\n",
      "[CV] .................... epochs=20, lr=0.001, nodes=32, total=11.3min\n",
      "[CV] epochs=20, lr=0.001, nodes=64 ...................................\n",
      "Epoch 1/20\n",
      "30945/30945 [==============================] - 64s 2ms/step - loss: 0.3551 - acc: 0.9815\n",
      "Epoch 2/20\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3340 - acc: 0.9779\n",
      "Epoch 3/20\n",
      "30945/30945 [==============================] - 52s 2ms/step - loss: 0.3239 - acc: 0.9744\n",
      "Epoch 4/20\n",
      "30945/30945 [==============================] - 52s 2ms/step - loss: 0.3175 - acc: 0.9685\n",
      "Epoch 5/20\n",
      "30945/30945 [==============================] - 52s 2ms/step - loss: 0.3115 - acc: 0.9650\n",
      "Epoch 6/20\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3062 - acc: 0.9593\n",
      "Epoch 7/20\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3024 - acc: 0.9540\n",
      "Epoch 8/20\n",
      "30945/30945 [==============================] - 52s 2ms/step - loss: 0.2992 - acc: 0.9511\n",
      "Epoch 9/20\n",
      "30945/30945 [==============================] - 52s 2ms/step - loss: 0.2976 - acc: 0.9452\n",
      "Epoch 10/20\n",
      "30945/30945 [==============================] - 52s 2ms/step - loss: 0.2955 - acc: 0.9431\n",
      "Epoch 11/20\n",
      "30945/30945 [==============================] - 51s 2ms/step - loss: 0.2924 - acc: 0.9364\n",
      "Epoch 12/20\n",
      "30945/30945 [==============================] - 52s 2ms/step - loss: 0.2917 - acc: 0.9361\n",
      "Epoch 13/20\n",
      "30945/30945 [==============================] - 52s 2ms/step - loss: 0.2905 - acc: 0.9335\n",
      "Epoch 14/20\n",
      "30945/30945 [==============================] - 52s 2ms/step - loss: 0.2885 - acc: 0.9286\n",
      "Epoch 15/20\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.2874 - acc: 0.9275\n",
      "Epoch 16/20\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.2870 - acc: 0.9295\n",
      "Epoch 17/20\n",
      "30945/30945 [==============================] - 52s 2ms/step - loss: 0.2866 - acc: 0.9224\n",
      "Epoch 18/20\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.2875 - acc: 0.9209\n",
      "Epoch 19/20\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.2852 - acc: 0.9189\n",
      "Epoch 20/20\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.2851 - acc: 0.9132\n",
      "15473/15473 [==============================] - 14s 890us/step\n",
      "30945/30945 [==============================] - 24s 783us/step\n",
      "[CV] .................... epochs=20, lr=0.001, nodes=64, total=18.5min\n",
      "[CV] epochs=20, lr=0.001, nodes=64 ...................................\n",
      "Epoch 1/20\n",
      "30945/30945 [==============================] - 65s 2ms/step - loss: 0.3652 - acc: 0.9787\n",
      "Epoch 2/20\n",
      "30945/30945 [==============================] - 54s 2ms/step - loss: 0.3437 - acc: 0.9787\n",
      "Epoch 3/20\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3337 - acc: 0.9731\n",
      "Epoch 4/20\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3279 - acc: 0.9686\n",
      "Epoch 5/20\n",
      "30945/30945 [==============================] - 52s 2ms/step - loss: 0.3227 - acc: 0.9650\n",
      "Epoch 6/20\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3197 - acc: 0.9635\n",
      "Epoch 7/20\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3140 - acc: 0.9575\n",
      "Epoch 8/20\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3124 - acc: 0.9516\n",
      "Epoch 9/20\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3095 - acc: 0.9497\n",
      "Epoch 10/20\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3070 - acc: 0.9446\n",
      "Epoch 11/20\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3059 - acc: 0.9419\n",
      "Epoch 12/20\n",
      "30945/30945 [==============================] - 52s 2ms/step - loss: 0.3029 - acc: 0.9387\n",
      "Epoch 13/20\n",
      "30945/30945 [==============================] - 54s 2ms/step - loss: 0.3012 - acc: 0.9348\n",
      "Epoch 14/20\n",
      "30945/30945 [==============================] - 54s 2ms/step - loss: 0.3024 - acc: 0.9336\n",
      "Epoch 15/20\n",
      "30945/30945 [==============================] - 52s 2ms/step - loss: 0.3002 - acc: 0.9276\n",
      "Epoch 16/20\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.2979 - acc: 0.9207\n",
      "Epoch 17/20\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.2977 - acc: 0.9200\n",
      "Epoch 18/20\n",
      "30945/30945 [==============================] - 52s 2ms/step - loss: 0.2955 - acc: 0.9156\n",
      "Epoch 19/20\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.2949 - acc: 0.9162\n",
      "Epoch 20/20\n",
      "30945/30945 [==============================] - 52s 2ms/step - loss: 0.2945 - acc: 0.9137\n",
      "15473/15473 [==============================] - 14s 894us/step\n",
      "30945/30945 [==============================] - 24s 783us/step\n",
      "[CV] .................... epochs=20, lr=0.001, nodes=64, total=18.6min\n",
      "[CV] epochs=20, lr=0.001, nodes=64 ...................................\n",
      "Epoch 1/20\n",
      "30946/30946 [==============================] - 63s 2ms/step - loss: 0.3544 - acc: 0.9832\n",
      "Epoch 2/20\n",
      "30946/30946 [==============================] - 51s 2ms/step - loss: 0.3329 - acc: 0.9773\n",
      "Epoch 3/20\n",
      "30946/30946 [==============================] - 52s 2ms/step - loss: 0.3235 - acc: 0.9746\n",
      "Epoch 4/20\n",
      "30946/30946 [==============================] - 53s 2ms/step - loss: 0.3165 - acc: 0.9688\n",
      "Epoch 5/20\n",
      "30946/30946 [==============================] - 52s 2ms/step - loss: 0.3110 - acc: 0.9648\n",
      "Epoch 6/20\n",
      "30946/30946 [==============================] - 52s 2ms/step - loss: 0.3081 - acc: 0.9574\n",
      "Epoch 7/20\n",
      "30946/30946 [==============================] - 53s 2ms/step - loss: 0.3044 - acc: 0.9581\n",
      "Epoch 8/20\n",
      "30946/30946 [==============================] - 52s 2ms/step - loss: 0.3010 - acc: 0.9536\n",
      "Epoch 9/20\n",
      "30946/30946 [==============================] - 53s 2ms/step - loss: 0.2991 - acc: 0.9467\n",
      "Epoch 10/20\n",
      "30946/30946 [==============================] - 53s 2ms/step - loss: 0.2968 - acc: 0.9445\n",
      "Epoch 11/20\n",
      "30946/30946 [==============================] - 53s 2ms/step - loss: 0.2942 - acc: 0.9416\n",
      "Epoch 12/20\n",
      "30946/30946 [==============================] - 53s 2ms/step - loss: 0.2920 - acc: 0.9319\n",
      "Epoch 13/20\n",
      "30946/30946 [==============================] - 51s 2ms/step - loss: 0.2897 - acc: 0.9299\n",
      "Epoch 14/20\n",
      "30946/30946 [==============================] - 51s 2ms/step - loss: 0.2903 - acc: 0.9288\n",
      "Epoch 15/20\n",
      "30946/30946 [==============================] - 52s 2ms/step - loss: 0.2871 - acc: 0.9238\n",
      "Epoch 16/20\n",
      "30946/30946 [==============================] - 52s 2ms/step - loss: 0.2884 - acc: 0.9132\n",
      "Epoch 17/20\n",
      "30946/30946 [==============================] - 53s 2ms/step - loss: 0.2858 - acc: 0.9189\n",
      "Epoch 18/20\n",
      "30946/30946 [==============================] - 53s 2ms/step - loss: 0.2864 - acc: 0.9139\n",
      "Epoch 19/20\n",
      "30946/30946 [==============================] - 54s 2ms/step - loss: 0.2858 - acc: 0.9124\n",
      "Epoch 20/20\n",
      "30946/30946 [==============================] - 54s 2ms/step - loss: 0.2856 - acc: 0.9092\n",
      "15472/15472 [==============================] - 14s 893us/step\n",
      "30946/30946 [==============================] - 26s 840us/step\n",
      "[CV] .................... epochs=20, lr=0.001, nodes=64, total=18.4min\n",
      "[CV] epochs=20, lr=0.002, nodes=16 ...................................\n",
      "Epoch 1/20\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3594 - acc: 0.9672\n",
      "Epoch 2/20\n",
      "30945/30945 [==============================] - 22s 706us/step - loss: 0.3411 - acc: 0.9851\n",
      "Epoch 3/20\n",
      "30945/30945 [==============================] - 21s 693us/step - loss: 0.3360 - acc: 0.9817\n",
      "Epoch 4/20\n",
      "30945/30945 [==============================] - 22s 698us/step - loss: 0.3308 - acc: 0.9795\n",
      "Epoch 5/20\n",
      "30945/30945 [==============================] - 22s 696us/step - loss: 0.3287 - acc: 0.9748\n",
      "Epoch 6/20\n",
      "30945/30945 [==============================] - 22s 701us/step - loss: 0.3230 - acc: 0.9714\n",
      "Epoch 7/20\n",
      "30945/30945 [==============================] - 22s 698us/step - loss: 0.3230 - acc: 0.9661\n",
      "Epoch 8/20\n",
      "30945/30945 [==============================] - 22s 696us/step - loss: 0.3178 - acc: 0.9609\n",
      "Epoch 9/20\n",
      "30945/30945 [==============================] - 21s 684us/step - loss: 0.3162 - acc: 0.9599\n",
      "Epoch 10/20\n",
      "30945/30945 [==============================] - 22s 695us/step - loss: 0.3172 - acc: 0.9568\n",
      "Epoch 11/20\n",
      "30945/30945 [==============================] - 21s 679us/step - loss: 0.3138 - acc: 0.9507\n",
      "Epoch 12/20\n",
      "30945/30945 [==============================] - 21s 676us/step - loss: 0.3137 - acc: 0.9485\n",
      "Epoch 13/20\n",
      "30945/30945 [==============================] - 21s 682us/step - loss: 0.3117 - acc: 0.9425\n",
      "Epoch 14/20\n",
      "30945/30945 [==============================] - 21s 681us/step - loss: 0.3102 - acc: 0.9409\n",
      "Epoch 15/20\n",
      "30945/30945 [==============================] - 21s 680us/step - loss: 0.3100 - acc: 0.9397\n",
      "Epoch 16/20\n",
      "30945/30945 [==============================] - 21s 687us/step - loss: 0.3108 - acc: 0.9366\n",
      "Epoch 17/20\n",
      "30945/30945 [==============================] - 21s 686us/step - loss: 0.3088 - acc: 0.9373\n",
      "Epoch 18/20\n",
      "30945/30945 [==============================] - 21s 686us/step - loss: 0.3074 - acc: 0.9347\n",
      "Epoch 19/20\n",
      "30945/30945 [==============================] - 21s 691us/step - loss: 0.3052 - acc: 0.9261\n",
      "Epoch 20/20\n",
      "30945/30945 [==============================] - 21s 686us/step - loss: 0.3047 - acc: 0.9297\n",
      "15473/15473 [==============================] - 13s 838us/step\n",
      "30945/30945 [==============================] - 23s 729us/step\n",
      "[CV] .................... epochs=20, lr=0.002, nodes=16, total= 8.1min\n",
      "[CV] epochs=20, lr=0.002, nodes=16 ...................................\n",
      "Epoch 1/20\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3672 - acc: 0.9509\n",
      "Epoch 2/20\n",
      "30945/30945 [==============================] - 22s 699us/step - loss: 0.3481 - acc: 0.9826\n",
      "Epoch 3/20\n",
      "30945/30945 [==============================] - 21s 689us/step - loss: 0.3430 - acc: 0.9813\n",
      "Epoch 4/20\n",
      "30945/30945 [==============================] - 21s 665us/step - loss: 0.3405 - acc: 0.9780\n",
      "Epoch 5/20\n",
      "30945/30945 [==============================] - 20s 657us/step - loss: 0.3348 - acc: 0.9734\n",
      "Epoch 6/20\n",
      "30945/30945 [==============================] - 20s 660us/step - loss: 0.3332 - acc: 0.9697\n",
      "Epoch 7/20\n",
      "30945/30945 [==============================] - 21s 670us/step - loss: 0.3309 - acc: 0.9659\n",
      "Epoch 8/20\n",
      "30945/30945 [==============================] - 20s 660us/step - loss: 0.3285 - acc: 0.9650\n",
      "Epoch 9/20\n",
      "30945/30945 [==============================] - 21s 672us/step - loss: 0.3258 - acc: 0.9625\n",
      "Epoch 10/20\n",
      "30945/30945 [==============================] - 21s 684us/step - loss: 0.3252 - acc: 0.9596\n",
      "Epoch 11/20\n",
      "30945/30945 [==============================] - 21s 678us/step - loss: 0.3226 - acc: 0.9575\n",
      "Epoch 12/20\n",
      "30945/30945 [==============================] - 21s 673us/step - loss: 0.3249 - acc: 0.9529\n",
      "Epoch 13/20\n",
      "30945/30945 [==============================] - 20s 656us/step - loss: 0.3212 - acc: 0.9520\n",
      "Epoch 14/20\n",
      "30945/30945 [==============================] - 21s 678us/step - loss: 0.3186 - acc: 0.9450\n",
      "Epoch 15/20\n",
      "30945/30945 [==============================] - 21s 686us/step - loss: 0.3181 - acc: 0.9441\n",
      "Epoch 16/20\n",
      "30945/30945 [==============================] - 21s 682us/step - loss: 0.3164 - acc: 0.9430\n",
      "Epoch 17/20\n",
      "30945/30945 [==============================] - 21s 668us/step - loss: 0.3162 - acc: 0.9379\n",
      "Epoch 18/20\n",
      "30945/30945 [==============================] - 21s 691us/step - loss: 0.3165 - acc: 0.9360\n",
      "Epoch 19/20\n",
      "30945/30945 [==============================] - 21s 688us/step - loss: 0.3164 - acc: 0.9315\n",
      "Epoch 20/20\n",
      "30945/30945 [==============================] - 21s 681us/step - loss: 0.3149 - acc: 0.9350\n",
      "15473/15473 [==============================] - 13s 840us/step\n",
      "30945/30945 [==============================] - 23s 738us/step\n",
      "[CV] .................... epochs=20, lr=0.002, nodes=16, total= 8.0min\n",
      "[CV] epochs=20, lr=0.002, nodes=16 ...................................\n",
      "Epoch 1/20\n",
      "30946/30946 [==============================] - 34s 1ms/step - loss: 0.3571 - acc: 0.9702\n",
      "Epoch 2/20\n",
      "30946/30946 [==============================] - 21s 672us/step - loss: 0.3395 - acc: 0.9835\n",
      "Epoch 3/20\n",
      "30946/30946 [==============================] - 21s 690us/step - loss: 0.3312 - acc: 0.9802\n",
      "Epoch 4/20\n",
      "30946/30946 [==============================] - 21s 680us/step - loss: 0.3277 - acc: 0.9730\n",
      "Epoch 5/20\n",
      "30946/30946 [==============================] - 21s 667us/step - loss: 0.3232 - acc: 0.9709\n",
      "Epoch 6/20\n",
      "30946/30946 [==============================] - 21s 675us/step - loss: 0.3204 - acc: 0.9654\n",
      "Epoch 7/20\n",
      "30946/30946 [==============================] - 21s 681us/step - loss: 0.3185 - acc: 0.9610\n",
      "Epoch 8/20\n",
      "30946/30946 [==============================] - 21s 679us/step - loss: 0.3160 - acc: 0.9565\n",
      "Epoch 9/20\n",
      "30946/30946 [==============================] - 20s 657us/step - loss: 0.3140 - acc: 0.9511\n",
      "Epoch 10/20\n",
      "30946/30946 [==============================] - 21s 679us/step - loss: 0.3129 - acc: 0.9455\n",
      "Epoch 11/20\n",
      "30946/30946 [==============================] - 21s 681us/step - loss: 0.3111 - acc: 0.9455\n",
      "Epoch 12/20\n",
      "30946/30946 [==============================] - 21s 680us/step - loss: 0.3098 - acc: 0.9388\n",
      "Epoch 13/20\n",
      "30946/30946 [==============================] - 21s 684us/step - loss: 0.3106 - acc: 0.9344\n",
      "Epoch 14/20\n",
      "30946/30946 [==============================] - 21s 694us/step - loss: 0.3066 - acc: 0.9299\n",
      "Epoch 15/20\n",
      "30946/30946 [==============================] - 21s 687us/step - loss: 0.3071 - acc: 0.9290\n",
      "Epoch 16/20\n",
      "30946/30946 [==============================] - 21s 685us/step - loss: 0.3052 - acc: 0.9216\n",
      "Epoch 17/20\n",
      "30946/30946 [==============================] - 20s 646us/step - loss: 0.3047 - acc: 0.9182\n",
      "Epoch 18/20\n",
      "30946/30946 [==============================] - 21s 687us/step - loss: 0.3049 - acc: 0.9140\n",
      "Epoch 19/20\n",
      "30946/30946 [==============================] - 21s 685us/step - loss: 0.3036 - acc: 0.9122\n",
      "Epoch 20/20\n",
      "30946/30946 [==============================] - 21s 679us/step - loss: 0.3025 - acc: 0.9049\n",
      "15472/15472 [==============================] - 13s 828us/step\n",
      "30946/30946 [==============================] - 23s 733us/step\n",
      "[CV] .................... epochs=20, lr=0.002, nodes=16, total= 8.0min\n",
      "[CV] epochs=20, lr=0.002, nodes=32 ...................................\n",
      "Epoch 1/20\n",
      "30945/30945 [==============================] - 43s 1ms/step - loss: 0.3528 - acc: 0.9804\n",
      "Epoch 2/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3365 - acc: 0.9792\n",
      "Epoch 3/20\n",
      "30945/30945 [==============================] - 30s 966us/step - loss: 0.3264 - acc: 0.9744\n",
      "Epoch 4/20\n",
      "30945/30945 [==============================] - 31s 990us/step - loss: 0.3232 - acc: 0.9688\n",
      "Epoch 5/20\n",
      "30945/30945 [==============================] - 30s 985us/step - loss: 0.3184 - acc: 0.9659\n",
      "Epoch 6/20\n",
      "30945/30945 [==============================] - 29s 948us/step - loss: 0.3144 - acc: 0.9604\n",
      "Epoch 7/20\n",
      "30945/30945 [==============================] - 31s 999us/step - loss: 0.3109 - acc: 0.9595\n",
      "Epoch 8/20\n",
      "30945/30945 [==============================] - 31s 993us/step - loss: 0.3077 - acc: 0.9542\n",
      "Epoch 9/20\n",
      "30945/30945 [==============================] - 31s 995us/step - loss: 0.3071 - acc: 0.9518\n",
      "Epoch 10/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3040 - acc: 0.9461\n",
      "Epoch 11/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3020 - acc: 0.9487\n",
      "Epoch 12/20\n",
      "30945/30945 [==============================] - 32s 1ms/step - loss: 0.3014 - acc: 0.9375\n",
      "Epoch 13/20\n",
      "30945/30945 [==============================] - 32s 1ms/step - loss: 0.2996 - acc: 0.9397\n",
      "Epoch 14/20\n",
      "30945/30945 [==============================] - 32s 1ms/step - loss: 0.2984 - acc: 0.9368\n",
      "Epoch 15/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.2987 - acc: 0.9311\n",
      "Epoch 16/20\n",
      "30945/30945 [==============================] - 32s 1ms/step - loss: 0.2975 - acc: 0.9299\n",
      "Epoch 17/20\n",
      "30945/30945 [==============================] - 32s 1ms/step - loss: 0.2964 - acc: 0.9249\n",
      "Epoch 18/20\n",
      "30945/30945 [==============================] - 32s 1ms/step - loss: 0.2976 - acc: 0.9211\n",
      "Epoch 19/20\n",
      "30945/30945 [==============================] - 32s 1ms/step - loss: 0.2973 - acc: 0.9191\n",
      "Epoch 20/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.2967 - acc: 0.9159\n",
      "15473/15473 [==============================] - 14s 896us/step\n",
      "30945/30945 [==============================] - 23s 729us/step\n",
      "[CV] .................... epochs=20, lr=0.002, nodes=32, total=11.4min\n",
      "[CV] epochs=20, lr=0.002, nodes=32 ...................................\n",
      "Epoch 1/20\n",
      "30945/30945 [==============================] - 46s 1ms/step - loss: 0.3656 - acc: 0.9688\n",
      "Epoch 2/20\n",
      "30945/30945 [==============================] - 32s 1ms/step - loss: 0.3438 - acc: 0.9777\n",
      "Epoch 3/20\n",
      "30945/30945 [==============================] - 32s 1ms/step - loss: 0.3363 - acc: 0.9764\n",
      "Epoch 4/20\n",
      "30945/30945 [==============================] - 30s 973us/step - loss: 0.3315 - acc: 0.9715\n",
      "Epoch 5/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3279 - acc: 0.9696\n",
      "Epoch 6/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3245 - acc: 0.9656\n",
      "Epoch 7/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3227 - acc: 0.9587\n",
      "Epoch 8/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3189 - acc: 0.9554\n",
      "Epoch 9/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3169 - acc: 0.9503\n",
      "Epoch 10/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3156 - acc: 0.9457\n",
      "Epoch 11/20\n",
      "30945/30945 [==============================] - 31s 999us/step - loss: 0.3128 - acc: 0.9464\n",
      "Epoch 12/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3105 - acc: 0.9400\n",
      "Epoch 13/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3126 - acc: 0.9436\n",
      "Epoch 14/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3083 - acc: 0.9399\n",
      "Epoch 15/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3093 - acc: 0.9386\n",
      "Epoch 16/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3081 - acc: 0.9327\n",
      "Epoch 17/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3082 - acc: 0.9227\n",
      "Epoch 18/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3078 - acc: 0.9256\n",
      "Epoch 19/20\n",
      "30945/30945 [==============================] - 32s 1ms/step - loss: 0.3070 - acc: 0.9206\n",
      "Epoch 20/20\n",
      "30945/30945 [==============================] - 32s 1ms/step - loss: 0.3073 - acc: 0.9175\n",
      "15473/15473 [==============================] - 14s 889us/step\n",
      "30945/30945 [==============================] - 23s 735us/step\n",
      "[CV] .................... epochs=20, lr=0.002, nodes=32, total=11.5min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] epochs=20, lr=0.002, nodes=32 ...................................\n",
      "Epoch 1/20\n",
      "30946/30946 [==============================] - 46s 1ms/step - loss: 0.3526 - acc: 0.9826\n",
      "Epoch 2/20\n",
      "30946/30946 [==============================] - 32s 1ms/step - loss: 0.3335 - acc: 0.9775\n",
      "Epoch 3/20\n",
      "30946/30946 [==============================] - 31s 990us/step - loss: 0.3248 - acc: 0.9709\n",
      "Epoch 4/20\n",
      "30946/30946 [==============================] - 31s 994us/step - loss: 0.3187 - acc: 0.9674\n",
      "Epoch 5/20\n",
      "30946/30946 [==============================] - 31s 996us/step - loss: 0.3153 - acc: 0.9621\n",
      "Epoch 6/20\n",
      "30946/30946 [==============================] - 31s 1ms/step - loss: 0.3110 - acc: 0.9586\n",
      "Epoch 7/20\n",
      "30946/30946 [==============================] - 31s 1ms/step - loss: 0.3083 - acc: 0.9551\n",
      "Epoch 8/20\n",
      "30946/30946 [==============================] - 29s 951us/step - loss: 0.3062 - acc: 0.9510\n",
      "Epoch 9/20\n",
      "30946/30946 [==============================] - 31s 998us/step - loss: 0.3034 - acc: 0.9460\n",
      "Epoch 10/20\n",
      "30946/30946 [==============================] - 31s 1ms/step - loss: 0.3019 - acc: 0.9436\n",
      "Epoch 11/20\n",
      "30946/30946 [==============================] - 31s 1ms/step - loss: 0.3016 - acc: 0.9416\n",
      "Epoch 12/20\n",
      "30946/30946 [==============================] - 31s 1ms/step - loss: 0.3015 - acc: 0.9366\n",
      "Epoch 13/20\n",
      "30946/30946 [==============================] - 29s 953us/step - loss: 0.2988 - acc: 0.9309\n",
      "Epoch 14/20\n",
      "30946/30946 [==============================] - 31s 1ms/step - loss: 0.2976 - acc: 0.9307\n",
      "Epoch 15/20\n",
      "30946/30946 [==============================] - 31s 1ms/step - loss: 0.2963 - acc: 0.9247\n",
      "Epoch 16/20\n",
      "30946/30946 [==============================] - 31s 1ms/step - loss: 0.2983 - acc: 0.9230\n",
      "Epoch 17/20\n",
      "30946/30946 [==============================] - 30s 964us/step - loss: 0.2968 - acc: 0.9213\n",
      "Epoch 18/20\n",
      "30946/30946 [==============================] - 31s 1ms/step - loss: 0.2947 - acc: 0.9176\n",
      "Epoch 19/20\n",
      "30946/30946 [==============================] - 31s 1ms/step - loss: 0.2955 - acc: 0.9144\n",
      "Epoch 20/20\n",
      "30946/30946 [==============================] - 31s 1ms/step - loss: 0.2964 - acc: 0.9118\n",
      "15472/15472 [==============================] - 14s 891us/step\n",
      "30946/30946 [==============================] - 22s 715us/step\n",
      "[CV] .................... epochs=20, lr=0.002, nodes=32, total=11.3min\n",
      "[CV] epochs=20, lr=0.002, nodes=64 ...................................\n",
      "Epoch 1/20\n",
      "30945/30945 [==============================] - 66s 2ms/step - loss: 0.3503 - acc: 0.9808\n",
      "Epoch 2/20\n",
      "30945/30945 [==============================] - 54s 2ms/step - loss: 0.3307 - acc: 0.9710\n",
      "Epoch 3/20\n",
      "30945/30945 [==============================] - 54s 2ms/step - loss: 0.3183 - acc: 0.9673\n",
      "Epoch 4/20\n",
      "30945/30945 [==============================] - 54s 2ms/step - loss: 0.3126 - acc: 0.9632\n",
      "Epoch 5/20\n",
      "30945/30945 [==============================] - 54s 2ms/step - loss: 0.3076 - acc: 0.9584\n",
      "Epoch 6/20\n",
      "30945/30945 [==============================] - 56s 2ms/step - loss: 0.3030 - acc: 0.9526\n",
      "Epoch 7/20\n",
      "30945/30945 [==============================] - 57s 2ms/step - loss: 0.3022 - acc: 0.9461\n",
      "Epoch 8/20\n",
      "30945/30945 [==============================] - 54s 2ms/step - loss: 0.2989 - acc: 0.9391\n",
      "Epoch 9/20\n",
      "30945/30945 [==============================] - 54s 2ms/step - loss: 0.2962 - acc: 0.9285\n",
      "Epoch 10/20\n",
      "30945/30945 [==============================] - 54s 2ms/step - loss: 0.2947 - acc: 0.9281\n",
      "Epoch 11/20\n",
      "30945/30945 [==============================] - 54s 2ms/step - loss: 0.2943 - acc: 0.9251\n",
      "Epoch 12/20\n",
      "30945/30945 [==============================] - 54s 2ms/step - loss: 0.2941 - acc: 0.9225\n",
      "Epoch 13/20\n",
      "30945/30945 [==============================] - 54s 2ms/step - loss: 0.2941 - acc: 0.9104\n",
      "Epoch 14/20\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.2932 - acc: 0.9192\n",
      "Epoch 15/20\n",
      "30945/30945 [==============================] - 54s 2ms/step - loss: 0.2920 - acc: 0.9054\n",
      "Epoch 16/20\n",
      "30945/30945 [==============================] - 54s 2ms/step - loss: 0.2942 - acc: 0.9048\n",
      "Epoch 17/20\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.2938 - acc: 0.9031\n",
      "Epoch 18/20\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.2954 - acc: 0.8965\n",
      "Epoch 19/20\n",
      "30945/30945 [==============================] - 55s 2ms/step - loss: 0.2935 - acc: 0.8966\n",
      "Epoch 20/20\n",
      "30945/30945 [==============================] - 55s 2ms/step - loss: 0.2947 - acc: 0.8883\n",
      "15473/15473 [==============================] - 15s 990us/step\n",
      "30945/30945 [==============================] - 26s 841us/step\n",
      "[CV] .................... epochs=20, lr=0.002, nodes=64, total=19.1min\n",
      "[CV] epochs=20, lr=0.002, nodes=64 ...................................\n",
      "Epoch 1/20\n",
      "30945/30945 [==============================] - 64s 2ms/step - loss: 0.3619 - acc: 0.9792\n",
      "Epoch 2/20\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3417 - acc: 0.9735\n",
      "Epoch 3/20\n",
      "30945/30945 [==============================] - 52s 2ms/step - loss: 0.3320 - acc: 0.9696\n",
      "Epoch 4/20\n",
      "30945/30945 [==============================] - 52s 2ms/step - loss: 0.3245 - acc: 0.9659\n",
      "Epoch 5/20\n",
      "30945/30945 [==============================] - 52s 2ms/step - loss: 0.3203 - acc: 0.9582\n",
      "Epoch 6/20\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3147 - acc: 0.9580\n",
      "Epoch 7/20\n",
      "30945/30945 [==============================] - 52s 2ms/step - loss: 0.3127 - acc: 0.9508\n",
      "Epoch 8/20\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3089 - acc: 0.9429\n",
      "Epoch 9/20\n",
      "30945/30945 [==============================] - 52s 2ms/step - loss: 0.3069 - acc: 0.9376\n",
      "Epoch 10/20\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3068 - acc: 0.9350\n",
      "Epoch 11/20\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3058 - acc: 0.9329\n",
      "Epoch 12/20\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3046 - acc: 0.9270\n",
      "Epoch 13/20\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3044 - acc: 0.9193\n",
      "Epoch 14/20\n",
      "30945/30945 [==============================] - 52s 2ms/step - loss: 0.3036 - acc: 0.9152\n",
      "Epoch 15/20\n",
      "30945/30945 [==============================] - 51s 2ms/step - loss: 0.3032 - acc: 0.9140\n",
      "Epoch 16/20\n",
      "30945/30945 [==============================] - 52s 2ms/step - loss: 0.3036 - acc: 0.9099\n",
      "Epoch 17/20\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3060 - acc: 0.8989\n",
      "Epoch 18/20\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3024 - acc: 0.9024\n",
      "Epoch 19/20\n",
      "30945/30945 [==============================] - 51s 2ms/step - loss: 0.3062 - acc: 0.8947\n",
      "Epoch 20/20\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3020 - acc: 0.8902\n",
      "15473/15473 [==============================] - 14s 906us/step\n",
      "30945/30945 [==============================] - 24s 784us/step\n",
      "[CV] .................... epochs=20, lr=0.002, nodes=64, total=18.4min\n",
      "[CV] epochs=20, lr=0.002, nodes=64 ...................................\n",
      "Epoch 1/20\n",
      "30946/30946 [==============================] - 65s 2ms/step - loss: 0.3505 - acc: 0.9804\n",
      "Epoch 2/20\n",
      "30946/30946 [==============================] - 54s 2ms/step - loss: 0.3302 - acc: 0.9772\n",
      "Epoch 3/20\n",
      "30946/30946 [==============================] - 53s 2ms/step - loss: 0.3214 - acc: 0.9717\n",
      "Epoch 4/20\n",
      "30946/30946 [==============================] - 54s 2ms/step - loss: 0.3159 - acc: 0.9665\n",
      "Epoch 5/20\n",
      "30946/30946 [==============================] - 53s 2ms/step - loss: 0.3130 - acc: 0.9604\n",
      "Epoch 6/20\n",
      "30946/30946 [==============================] - 53s 2ms/step - loss: 0.3072 - acc: 0.9569\n",
      "Epoch 7/20\n",
      "30946/30946 [==============================] - 53s 2ms/step - loss: 0.3025 - acc: 0.9511\n",
      "Epoch 8/20\n",
      "30946/30946 [==============================] - 53s 2ms/step - loss: 0.3008 - acc: 0.9466\n",
      "Epoch 9/20\n",
      "30946/30946 [==============================] - 54s 2ms/step - loss: 0.2992 - acc: 0.9412\n",
      "Epoch 10/20\n",
      "30946/30946 [==============================] - 53s 2ms/step - loss: 0.2965 - acc: 0.9361\n",
      "Epoch 11/20\n",
      "30946/30946 [==============================] - 53s 2ms/step - loss: 0.2962 - acc: 0.9309\n",
      "Epoch 12/20\n",
      "30946/30946 [==============================] - 53s 2ms/step - loss: 0.2947 - acc: 0.9266\n",
      "Epoch 13/20\n",
      "30946/30946 [==============================] - 53s 2ms/step - loss: 0.2929 - acc: 0.9225\n",
      "Epoch 14/20\n",
      "30946/30946 [==============================] - 52s 2ms/step - loss: 0.2917 - acc: 0.9181\n",
      "Epoch 15/20\n",
      "30946/30946 [==============================] - 53s 2ms/step - loss: 0.2920 - acc: 0.9129\n",
      "Epoch 16/20\n",
      "30946/30946 [==============================] - 50s 2ms/step - loss: 0.2927 - acc: 0.9125\n",
      "Epoch 17/20\n",
      "30946/30946 [==============================] - 52s 2ms/step - loss: 0.2923 - acc: 0.9028\n",
      "Epoch 18/20\n",
      "30946/30946 [==============================] - 52s 2ms/step - loss: 0.2917 - acc: 0.9020\n",
      "Epoch 19/20\n",
      "30946/30946 [==============================] - 53s 2ms/step - loss: 0.2927 - acc: 0.8999\n",
      "Epoch 20/20\n",
      "30946/30946 [==============================] - 53s 2ms/step - loss: 0.2910 - acc: 0.8946\n",
      "15472/15472 [==============================] - 14s 901us/step\n",
      "30946/30946 [==============================] - 26s 846us/step\n",
      "[CV] .................... epochs=20, lr=0.002, nodes=64, total=18.6min\n",
      "[CV] epochs=20, lr=0.003, nodes=16 ...................................\n",
      "Epoch 1/20\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3534 - acc: 0.9817\n",
      "Epoch 2/20\n",
      "30945/30945 [==============================] - 21s 687us/step - loss: 0.3372 - acc: 0.9798\n",
      "Epoch 3/20\n",
      "30945/30945 [==============================] - 21s 678us/step - loss: 0.3309 - acc: 0.9734\n",
      "Epoch 4/20\n",
      "30945/30945 [==============================] - 21s 674us/step - loss: 0.3255 - acc: 0.9716\n",
      "Epoch 5/20\n",
      "30945/30945 [==============================] - 21s 695us/step - loss: 0.3237 - acc: 0.9637\n",
      "Epoch 6/20\n",
      "30945/30945 [==============================] - 21s 694us/step - loss: 0.3204 - acc: 0.9613\n",
      "Epoch 7/20\n",
      "30945/30945 [==============================] - 21s 680us/step - loss: 0.3178 - acc: 0.9557\n",
      "Epoch 8/20\n",
      "30945/30945 [==============================] - 21s 693us/step - loss: 0.3157 - acc: 0.9505\n",
      "Epoch 9/20\n",
      "30945/30945 [==============================] - 21s 694us/step - loss: 0.3148 - acc: 0.9496\n",
      "Epoch 10/20\n",
      "30945/30945 [==============================] - 21s 688us/step - loss: 0.3136 - acc: 0.9396\n",
      "Epoch 11/20\n",
      "30945/30945 [==============================] - 21s 663us/step - loss: 0.3129 - acc: 0.9343\n",
      "Epoch 12/20\n",
      "30945/30945 [==============================] - 20s 660us/step - loss: 0.3111 - acc: 0.9279\n",
      "Epoch 13/20\n",
      "30945/30945 [==============================] - 20s 660us/step - loss: 0.3107 - acc: 0.9298\n",
      "Epoch 14/20\n",
      "30945/30945 [==============================] - 20s 652us/step - loss: 0.3093 - acc: 0.9271\n",
      "Epoch 15/20\n",
      "30945/30945 [==============================] - 21s 668us/step - loss: 0.3113 - acc: 0.9260\n",
      "Epoch 16/20\n",
      "30945/30945 [==============================] - 20s 662us/step - loss: 0.3103 - acc: 0.9202\n",
      "Epoch 17/20\n",
      "30945/30945 [==============================] - 21s 684us/step - loss: 0.3081 - acc: 0.9163\n",
      "Epoch 18/20\n",
      "30945/30945 [==============================] - 21s 682us/step - loss: 0.3107 - acc: 0.9155\n",
      "Epoch 19/20\n",
      "30945/30945 [==============================] - 21s 672us/step - loss: 0.3118 - acc: 0.9152\n",
      "Epoch 20/20\n",
      "30945/30945 [==============================] - 21s 672us/step - loss: 0.3101 - acc: 0.9093\n",
      "15473/15473 [==============================] - 13s 844us/step\n",
      "30945/30945 [==============================] - 23s 730us/step\n",
      "[CV] .................... epochs=20, lr=0.003, nodes=16, total= 8.0min\n",
      "[CV] epochs=20, lr=0.003, nodes=16 ...................................\n",
      "Epoch 1/20\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3641 - acc: 0.9802\n",
      "Epoch 2/20\n",
      "30945/30945 [==============================] - 22s 711us/step - loss: 0.3487 - acc: 0.9794\n",
      "Epoch 3/20\n",
      "30945/30945 [==============================] - 22s 702us/step - loss: 0.3433 - acc: 0.9722\n",
      "Epoch 4/20\n",
      "30945/30945 [==============================] - 22s 706us/step - loss: 0.3376 - acc: 0.9664\n",
      "Epoch 5/20\n",
      "30945/30945 [==============================] - 22s 709us/step - loss: 0.3380 - acc: 0.9650\n",
      "Epoch 6/20\n",
      "30945/30945 [==============================] - 22s 709us/step - loss: 0.3356 - acc: 0.9593\n",
      "Epoch 7/20\n",
      "30945/30945 [==============================] - 22s 708us/step - loss: 0.3316 - acc: 0.9510\n",
      "Epoch 8/20\n",
      "30945/30945 [==============================] - 22s 708us/step - loss: 0.3305 - acc: 0.9474\n",
      "Epoch 9/20\n",
      "30945/30945 [==============================] - 21s 691us/step - loss: 0.3295 - acc: 0.9500\n",
      "Epoch 10/20\n",
      "30945/30945 [==============================] - 21s 685us/step - loss: 0.3283 - acc: 0.9435\n",
      "Epoch 11/20\n",
      "30945/30945 [==============================] - 21s 688us/step - loss: 0.3271 - acc: 0.9394\n",
      "Epoch 12/20\n",
      "30945/30945 [==============================] - 22s 695us/step - loss: 0.3243 - acc: 0.9380\n",
      "Epoch 13/20\n",
      "30945/30945 [==============================] - 22s 696us/step - loss: 0.3233 - acc: 0.9388\n",
      "Epoch 14/20\n",
      "30945/30945 [==============================] - 21s 678us/step - loss: 0.3234 - acc: 0.9340\n",
      "Epoch 15/20\n",
      "30945/30945 [==============================] - 22s 707us/step - loss: 0.3217 - acc: 0.9300\n",
      "Epoch 16/20\n",
      "30945/30945 [==============================] - 22s 702us/step - loss: 0.3224 - acc: 0.9288\n",
      "Epoch 17/20\n",
      "30945/30945 [==============================] - 22s 695us/step - loss: 0.3223 - acc: 0.9208\n",
      "Epoch 18/20\n",
      "30945/30945 [==============================] - 21s 680us/step - loss: 0.3230 - acc: 0.9204\n",
      "Epoch 19/20\n",
      "30945/30945 [==============================] - 22s 709us/step - loss: 0.3205 - acc: 0.9201\n",
      "Epoch 20/20\n",
      "30945/30945 [==============================] - 22s 707us/step - loss: 0.3243 - acc: 0.9182\n",
      "15473/15473 [==============================] - 13s 870us/step\n",
      "30945/30945 [==============================] - 23s 737us/step\n",
      "[CV] .................... epochs=20, lr=0.003, nodes=16, total= 8.2min\n",
      "[CV] epochs=20, lr=0.003, nodes=16 ...................................\n",
      "Epoch 1/20\n",
      "30946/30946 [==============================] - 34s 1ms/step - loss: 0.3552 - acc: 0.9835\n",
      "Epoch 2/20\n",
      "30946/30946 [==============================] - 22s 699us/step - loss: 0.3387 - acc: 0.9815\n",
      "Epoch 3/20\n",
      "30946/30946 [==============================] - 21s 686us/step - loss: 0.3324 - acc: 0.9777\n",
      "Epoch 4/20\n",
      "30946/30946 [==============================] - 21s 686us/step - loss: 0.3275 - acc: 0.9735\n",
      "Epoch 5/20\n",
      "30946/30946 [==============================] - 21s 687us/step - loss: 0.3254 - acc: 0.9686\n",
      "Epoch 6/20\n",
      "30946/30946 [==============================] - 21s 684us/step - loss: 0.3241 - acc: 0.9659\n",
      "Epoch 7/20\n",
      "30946/30946 [==============================] - 21s 679us/step - loss: 0.3196 - acc: 0.9592\n",
      "Epoch 8/20\n",
      "30946/30946 [==============================] - 22s 701us/step - loss: 0.3198 - acc: 0.9534\n",
      "Epoch 9/20\n",
      "30946/30946 [==============================] - 21s 689us/step - loss: 0.3169 - acc: 0.9469\n",
      "Epoch 10/20\n",
      "30946/30946 [==============================] - 21s 688us/step - loss: 0.3169 - acc: 0.9497\n",
      "Epoch 11/20\n",
      "30946/30946 [==============================] - 21s 689us/step - loss: 0.3141 - acc: 0.9455\n",
      "Epoch 12/20\n",
      "30946/30946 [==============================] - 21s 689us/step - loss: 0.3123 - acc: 0.9409\n",
      "Epoch 13/20\n",
      "30946/30946 [==============================] - 21s 689us/step - loss: 0.3147 - acc: 0.9307\n",
      "Epoch 14/20\n",
      "30946/30946 [==============================] - 21s 683us/step - loss: 0.3131 - acc: 0.9316\n",
      "Epoch 15/20\n",
      "30946/30946 [==============================] - 21s 681us/step - loss: 0.3107 - acc: 0.9282\n",
      "Epoch 16/20\n",
      "30946/30946 [==============================] - 21s 679us/step - loss: 0.3119 - acc: 0.9304\n",
      "Epoch 17/20\n",
      "30946/30946 [==============================] - 21s 675us/step - loss: 0.3098 - acc: 0.9249\n",
      "Epoch 18/20\n",
      "30946/30946 [==============================] - 21s 691us/step - loss: 0.3087 - acc: 0.9205\n",
      "Epoch 19/20\n",
      "30946/30946 [==============================] - 21s 689us/step - loss: 0.3105 - acc: 0.9190\n",
      "Epoch 20/20\n",
      "30946/30946 [==============================] - 21s 689us/step - loss: 0.3064 - acc: 0.9067\n",
      "15472/15472 [==============================] - 13s 851us/step\n",
      "30946/30946 [==============================] - 23s 738us/step\n",
      "[CV] .................... epochs=20, lr=0.003, nodes=16, total= 8.1min\n",
      "[CV] epochs=20, lr=0.003, nodes=32 ...................................\n",
      "Epoch 1/20\n",
      "30945/30945 [==============================] - 43s 1ms/step - loss: 0.3514 - acc: 0.9831\n",
      "Epoch 2/20\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3359 - acc: 0.9762\n",
      "Epoch 3/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3267 - acc: 0.9721\n",
      "Epoch 4/20\n",
      "30945/30945 [==============================] - ETA: 0s - loss: 0.3212 - acc: 0.967 - 31s 1ms/step - loss: 0.3217 - acc: 0.9674\n",
      "Epoch 5/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3176 - acc: 0.9609\n",
      "Epoch 6/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3143 - acc: 0.9580\n",
      "Epoch 7/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3125 - acc: 0.9501\n",
      "Epoch 8/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3081 - acc: 0.9483\n",
      "Epoch 9/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3064 - acc: 0.9406\n",
      "Epoch 10/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3067 - acc: 0.9411\n",
      "Epoch 11/20\n",
      "30945/30945 [==============================] - 32s 1ms/step - loss: 0.3052 - acc: 0.9363\n",
      "Epoch 12/20\n",
      "30945/30945 [==============================] - 32s 1ms/step - loss: 0.3056 - acc: 0.9281\n",
      "Epoch 13/20\n",
      "30945/30945 [==============================] - 32s 1ms/step - loss: 0.3050 - acc: 0.9277\n",
      "Epoch 14/20\n",
      "30945/30945 [==============================] - 32s 1ms/step - loss: 0.3028 - acc: 0.9243\n",
      "Epoch 15/20\n",
      "30945/30945 [==============================] - 32s 1ms/step - loss: 0.3017 - acc: 0.9189\n",
      "Epoch 16/20\n",
      "30945/30945 [==============================] - 32s 1ms/step - loss: 0.3015 - acc: 0.9198\n",
      "Epoch 17/20\n",
      "30945/30945 [==============================] - 32s 1ms/step - loss: 0.3017 - acc: 0.9129\n",
      "Epoch 18/20\n",
      "30945/30945 [==============================] - 30s 958us/step - loss: 0.3032 - acc: 0.9100\n",
      "Epoch 19/20\n",
      "30945/30945 [==============================] - 32s 1ms/step - loss: 0.3028 - acc: 0.9031\n",
      "Epoch 20/20\n",
      "30945/30945 [==============================] - 32s 1ms/step - loss: 0.2995 - acc: 0.9008\n",
      "15473/15473 [==============================] - 14s 889us/step\n",
      "30945/30945 [==============================] - 22s 724us/step\n",
      "[CV] .................... epochs=20, lr=0.003, nodes=32, total=11.5min\n",
      "[CV] epochs=20, lr=0.003, nodes=32 ...................................\n",
      "Epoch 1/20\n",
      "30945/30945 [==============================] - 46s 1ms/step - loss: 0.3612 - acc: 0.9805\n",
      "Epoch 2/20\n",
      "30945/30945 [==============================] - 32s 1ms/step - loss: 0.3440 - acc: 0.9782\n",
      "Epoch 3/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3364 - acc: 0.9723\n",
      "Epoch 4/20\n",
      "30945/30945 [==============================] - 31s 988us/step - loss: 0.3318 - acc: 0.9677\n",
      "Epoch 5/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3265 - acc: 0.9628\n",
      "Epoch 6/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3235 - acc: 0.9592\n",
      "Epoch 7/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3213 - acc: 0.9491\n",
      "Epoch 8/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3185 - acc: 0.9408\n",
      "Epoch 9/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3181 - acc: 0.9402\n",
      "Epoch 10/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3163 - acc: 0.9360\n",
      "Epoch 11/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3157 - acc: 0.9301\n",
      "Epoch 12/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3141 - acc: 0.9258\n",
      "Epoch 13/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3154 - acc: 0.9197\n",
      "Epoch 14/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3123 - acc: 0.9160\n",
      "Epoch 15/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3141 - acc: 0.9131\n",
      "Epoch 16/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3128 - acc: 0.9051\n",
      "Epoch 17/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3115 - acc: 0.9013\n",
      "Epoch 18/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3103 - acc: 0.9000\n",
      "Epoch 19/20\n",
      "30945/30945 [==============================] - 27s 868us/step - loss: 0.3139 - acc: 0.8929\n",
      "Epoch 20/20\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3120 - acc: 0.8971\n",
      "15473/15473 [==============================] - 14s 905us/step\n",
      "30945/30945 [==============================] - 24s 782us/step\n",
      "[CV] .................... epochs=20, lr=0.003, nodes=32, total=11.4min\n",
      "[CV] epochs=20, lr=0.003, nodes=32 ...................................\n",
      "Epoch 1/20\n",
      "30946/30946 [==============================] - 43s 1ms/step - loss: 0.3532 - acc: 0.9820\n",
      "Epoch 2/20\n",
      "30946/30946 [==============================] - 31s 1ms/step - loss: 0.3339 - acc: 0.9769\n",
      "Epoch 3/20\n",
      "30946/30946 [==============================] - 31s 1ms/step - loss: 0.3254 - acc: 0.9724\n",
      "Epoch 4/20\n",
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.3198 - acc: 0.9650\n",
      "Epoch 5/20\n",
      "30946/30946 [==============================] - 37s 1ms/step - loss: 0.3150 - acc: 0.9600\n",
      "Epoch 6/20\n",
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.3115 - acc: 0.9567\n",
      "Epoch 7/20\n",
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.3130 - acc: 0.9529\n",
      "Epoch 8/20\n",
      "30946/30946 [==============================] - 35s 1ms/step - loss: 0.3099 - acc: 0.9469\n",
      "Epoch 9/20\n",
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.3082 - acc: 0.9419\n",
      "Epoch 10/20\n",
      "30946/30946 [==============================] - 35s 1ms/step - loss: 0.3077 - acc: 0.9355\n",
      "Epoch 11/20\n",
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.3043 - acc: 0.9299\n",
      "Epoch 12/20\n",
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.3032 - acc: 0.9292\n",
      "Epoch 13/20\n",
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.3041 - acc: 0.9225\n",
      "Epoch 14/20\n",
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.3039 - acc: 0.9148\n",
      "Epoch 15/20\n",
      "30946/30946 [==============================] - 35s 1ms/step - loss: 0.3034 - acc: 0.9162\n",
      "Epoch 16/20\n",
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.3026 - acc: 0.9020\n",
      "Epoch 17/20\n",
      "30946/30946 [==============================] - 37s 1ms/step - loss: 0.3026 - acc: 0.9072\n",
      "Epoch 18/20\n",
      "30946/30946 [==============================] - 38s 1ms/step - loss: 0.3009 - acc: 0.9005\n",
      "Epoch 19/20\n",
      "30946/30946 [==============================] - 37s 1ms/step - loss: 0.2988 - acc: 0.8981\n",
      "Epoch 20/20\n",
      "30946/30946 [==============================] - 39s 1ms/step - loss: 0.2978 - acc: 0.8928\n",
      "15472/15472 [==============================] - 17s 1ms/step\n",
      "30946/30946 [==============================] - 25s 804us/step\n",
      "[CV] .................... epochs=20, lr=0.003, nodes=32, total=12.8min\n",
      "[CV] epochs=20, lr=0.003, nodes=64 ...................................\n",
      "Epoch 1/20\n",
      "30945/30945 [==============================] - 75s 2ms/step - loss: 0.3511 - acc: 0.9769\n",
      "Epoch 2/20\n",
      "30945/30945 [==============================] - 62s 2ms/step - loss: 0.3313 - acc: 0.9749\n",
      "Epoch 3/20\n",
      "30945/30945 [==============================] - 61s 2ms/step - loss: 0.3223 - acc: 0.9685\n",
      "Epoch 4/20\n",
      "30945/30945 [==============================] - 62s 2ms/step - loss: 0.3155 - acc: 0.9594\n",
      "Epoch 5/20\n",
      "30945/30945 [==============================] - 63s 2ms/step - loss: 0.3093 - acc: 0.9550\n",
      "Epoch 6/20\n",
      "30945/30945 [==============================] - 61s 2ms/step - loss: 0.3061 - acc: 0.9534\n",
      "Epoch 7/20\n",
      "30945/30945 [==============================] - 62s 2ms/step - loss: 0.3044 - acc: 0.9452\n",
      "Epoch 8/20\n",
      "30945/30945 [==============================] - 62s 2ms/step - loss: 0.3027 - acc: 0.9398\n",
      "Epoch 9/20\n",
      "30945/30945 [==============================] - 61s 2ms/step - loss: 0.3000 - acc: 0.9310: 0s - loss: 0.3001 - \n",
      "Epoch 10/20\n",
      "30945/30945 [==============================] - 61s 2ms/step - loss: 0.2994 - acc: 0.9298\n",
      "Epoch 11/20\n",
      "30945/30945 [==============================] - 62s 2ms/step - loss: 0.3012 - acc: 0.9284\n",
      "Epoch 12/20\n",
      "30945/30945 [==============================] - 62s 2ms/step - loss: 0.3000 - acc: 0.9210\n",
      "Epoch 13/20\n",
      "30945/30945 [==============================] - 64s 2ms/step - loss: 0.3007 - acc: 0.9186\n",
      "Epoch 14/20\n",
      "30945/30945 [==============================] - 63s 2ms/step - loss: 0.3006 - acc: 0.9185\n",
      "Epoch 15/20\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "30945/30945 [==============================] - 68s 2ms/step - loss: 0.3018 - acc: 0.9026\n",
      "Epoch 16/20\n",
      "30945/30945 [==============================] - 66s 2ms/step - loss: 0.2999 - acc: 0.9020\n",
      "Epoch 17/20\n",
      "30945/30945 [==============================] - 65s 2ms/step - loss: 0.3019 - acc: 0.9033\n",
      "Epoch 18/20\n",
      "30945/30945 [==============================] - 63s 2ms/step - loss: 0.3028 - acc: 0.8954\n",
      "Epoch 19/20\n",
      "30945/30945 [==============================] - 63s 2ms/step - loss: 0.3009 - acc: 0.8938\n",
      "Epoch 20/20\n",
      "30945/30945 [==============================] - 67s 2ms/step - loss: 0.3023 - acc: 0.8908\n",
      "15473/15473 [==============================] - 14s 934us/step\n",
      "30945/30945 [==============================] - 26s 837us/step\n",
      "[CV] .................... epochs=20, lr=0.003, nodes=64, total=22.0min\n",
      "[CV] epochs=20, lr=0.003, nodes=64 ...................................\n",
      "Epoch 1/20\n",
      "30945/30945 [==============================] - 70s 2ms/step - loss: 0.3607 - acc: 0.9771\n",
      "Epoch 2/20\n",
      "30945/30945 [==============================] - 57s 2ms/step - loss: 0.3423 - acc: 0.9740\n",
      "Epoch 3/20\n",
      "30945/30945 [==============================] - 54s 2ms/step - loss: 0.3346 - acc: 0.9677\n",
      "Epoch 4/20\n",
      "30945/30945 [==============================] - 61s 2ms/step - loss: 0.3271 - acc: 0.9604\n",
      "Epoch 5/20\n",
      "30945/30945 [==============================] - 62s 2ms/step - loss: 0.3223 - acc: 0.9555\n",
      "Epoch 6/20\n",
      "30945/30945 [==============================] - 59s 2ms/step - loss: 0.3178 - acc: 0.9451\n",
      "Epoch 7/20\n",
      "30945/30945 [==============================] - 55s 2ms/step - loss: 0.3158 - acc: 0.9417\n",
      "Epoch 8/20\n",
      "30945/30945 [==============================] - 57s 2ms/step - loss: 0.3129 - acc: 0.9295\n",
      "Epoch 9/20\n",
      "30945/30945 [==============================] - 60s 2ms/step - loss: 0.3124 - acc: 0.9328\n",
      "Epoch 10/20\n",
      "30945/30945 [==============================] - 58s 2ms/step - loss: 0.3086 - acc: 0.9262\n",
      "Epoch 11/20\n",
      "30945/30945 [==============================] - 58s 2ms/step - loss: 0.3100 - acc: 0.9251\n",
      "Epoch 12/20\n",
      "30945/30945 [==============================] - 56s 2ms/step - loss: 0.3094 - acc: 0.9232\n",
      "Epoch 13/20\n",
      "30945/30945 [==============================] - 55s 2ms/step - loss: 0.3067 - acc: 0.9151\n",
      "Epoch 14/20\n",
      "30945/30945 [==============================] - 57s 2ms/step - loss: 0.3113 - acc: 0.9072\n",
      "Epoch 15/20\n",
      "30945/30945 [==============================] - 58s 2ms/step - loss: 0.3076 - acc: 0.9105\n",
      "Epoch 16/20\n",
      "30945/30945 [==============================] - 58s 2ms/step - loss: 0.3077 - acc: 0.9047\n",
      "Epoch 17/20\n",
      "30945/30945 [==============================] - 57s 2ms/step - loss: 0.3079 - acc: 0.9012\n",
      "Epoch 18/20\n",
      "30945/30945 [==============================] - 55s 2ms/step - loss: 0.3087 - acc: 0.8971\n",
      "Epoch 19/20\n",
      "30945/30945 [==============================] - 55s 2ms/step - loss: 0.3064 - acc: 0.8968\n",
      "Epoch 20/20\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3098 - acc: 0.8919\n",
      "15473/15473 [==============================] - 14s 908us/step\n",
      "30945/30945 [==============================] - 24s 788us/step\n",
      "[CV] .................... epochs=20, lr=0.003, nodes=64, total=20.1min\n",
      "[CV] epochs=20, lr=0.003, nodes=64 ...................................\n",
      "Epoch 1/20\n",
      "30946/30946 [==============================] - 65s 2ms/step - loss: 0.3489 - acc: 0.9812\n",
      "Epoch 2/20\n",
      "30946/30946 [==============================] - 53s 2ms/step - loss: 0.3297 - acc: 0.9710\n",
      "Epoch 3/20\n",
      "30946/30946 [==============================] - 54s 2ms/step - loss: 0.3224 - acc: 0.9671\n",
      "Epoch 4/20\n",
      "30946/30946 [==============================] - 54s 2ms/step - loss: 0.3166 - acc: 0.9576\n",
      "Epoch 5/20\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.3102 - acc: 0.9504\n",
      "Epoch 6/20\n",
      "30946/30946 [==============================] - 57s 2ms/step - loss: 0.3086 - acc: 0.9457\n",
      "Epoch 7/20\n",
      "30946/30946 [==============================] - 56s 2ms/step - loss: 0.3041 - acc: 0.9406\n",
      "Epoch 8/20\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.3017 - acc: 0.9320\n",
      "Epoch 9/20\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.3016 - acc: 0.9317\n",
      "Epoch 10/20\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.3029 - acc: 0.9277\n",
      "Epoch 11/20\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.3012 - acc: 0.9211\n",
      "Epoch 12/20\n",
      "30946/30946 [==============================] - 54s 2ms/step - loss: 0.3006 - acc: 0.9196\n",
      "Epoch 13/20\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.3013 - acc: 0.9125\n",
      "Epoch 14/20\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.3028 - acc: 0.9040\n",
      "Epoch 15/20\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.2992 - acc: 0.9040\n",
      "Epoch 16/20\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.3034 - acc: 0.9010\n",
      "Epoch 17/20\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.2996 - acc: 0.8967\n",
      "Epoch 18/20\n",
      "30946/30946 [==============================] - 54s 2ms/step - loss: 0.3008 - acc: 0.8943\n",
      "Epoch 19/20\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.3018 - acc: 0.8923\n",
      "Epoch 20/20\n",
      "30946/30946 [==============================] - 52s 2ms/step - loss: 0.3003 - acc: 0.8926\n",
      "15472/15472 [==============================] - 14s 908us/step\n",
      "30946/30946 [==============================] - 26s 854us/step\n",
      "[CV] .................... epochs=20, lr=0.003, nodes=64, total=19.2min\n",
      "[CV] epochs=30, lr=0.001, nodes=16 ...................................\n",
      "Epoch 1/30\n",
      "30945/30945 [==============================] - 36s 1ms/step - loss: 0.3703 - acc: 0.9533\n",
      "Epoch 2/30\n",
      "30945/30945 [==============================] - 24s 767us/step - loss: 0.3458 - acc: 0.9824\n",
      "Epoch 3/30\n",
      "30945/30945 [==============================] - 23s 748us/step - loss: 0.3388 - acc: 0.9815\n",
      "Epoch 4/30\n",
      "30945/30945 [==============================] - 23s 747us/step - loss: 0.3348 - acc: 0.9794\n",
      "Epoch 5/30\n",
      "30945/30945 [==============================] - 23s 749us/step - loss: 0.3346 - acc: 0.9779\n",
      "Epoch 6/30\n",
      "30945/30945 [==============================] - 23s 752us/step - loss: 0.3312 - acc: 0.9756\n",
      "Epoch 7/30\n",
      "30945/30945 [==============================] - 23s 753us/step - loss: 0.3293 - acc: 0.9741\n",
      "Epoch 8/30\n",
      "30945/30945 [==============================] - 23s 734us/step - loss: 0.3275 - acc: 0.9712\n",
      "Epoch 9/30\n",
      "30945/30945 [==============================] - 23s 740us/step - loss: 0.3256 - acc: 0.9688\n",
      "Epoch 10/30\n",
      "30945/30945 [==============================] - 23s 742us/step - loss: 0.3222 - acc: 0.9665\n",
      "Epoch 11/30\n",
      "30945/30945 [==============================] - 23s 735us/step - loss: 0.3208 - acc: 0.9590\n",
      "Epoch 12/30\n",
      "30945/30945 [==============================] - 23s 738us/step - loss: 0.3211 - acc: 0.9555\n",
      "Epoch 13/30\n",
      "30945/30945 [==============================] - 23s 750us/step - loss: 0.3190 - acc: 0.9587\n",
      "Epoch 14/30\n",
      "30945/30945 [==============================] - 23s 748us/step - loss: 0.3152 - acc: 0.9538\n",
      "Epoch 15/30\n",
      "30945/30945 [==============================] - 23s 744us/step - loss: 0.3161 - acc: 0.9534\n",
      "Epoch 16/30\n",
      "30945/30945 [==============================] - 23s 744us/step - loss: 0.3150 - acc: 0.9488\n",
      "Epoch 17/30\n",
      "30945/30945 [==============================] - 23s 746us/step - loss: 0.3128 - acc: 0.9461\n",
      "Epoch 18/30\n",
      "30945/30945 [==============================] - 23s 743us/step - loss: 0.3123 - acc: 0.9457\n",
      "Epoch 19/30\n",
      "30945/30945 [==============================] - 23s 730us/step - loss: 0.3095 - acc: 0.9430\n",
      "Epoch 20/30\n",
      "30945/30945 [==============================] - 23s 755us/step - loss: 0.3095 - acc: 0.9463\n",
      "Epoch 21/30\n",
      "30945/30945 [==============================] - 23s 745us/step - loss: 0.3095 - acc: 0.9434\n",
      "Epoch 22/30\n",
      "30945/30945 [==============================] - 23s 746us/step - loss: 0.3075 - acc: 0.9419\n",
      "Epoch 23/30\n",
      "30945/30945 [==============================] - 23s 738us/step - loss: 0.3072 - acc: 0.9345\n",
      "Epoch 24/30\n",
      "30945/30945 [==============================] - 23s 747us/step - loss: 0.3059 - acc: 0.9332\n",
      "Epoch 25/30\n",
      "30945/30945 [==============================] - 23s 742us/step - loss: 0.3067 - acc: 0.9308\n",
      "Epoch 26/30\n",
      "30945/30945 [==============================] - 23s 747us/step - loss: 0.3074 - acc: 0.9319\n",
      "Epoch 27/30\n",
      "30945/30945 [==============================] - 23s 743us/step - loss: 0.3062 - acc: 0.9291\n",
      "Epoch 28/30\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "30945/30945 [==============================] - 23s 752us/step - loss: 0.3055 - acc: 0.9248\n",
      "Epoch 29/30\n",
      "30945/30945 [==============================] - 23s 740us/step - loss: 0.3047 - acc: 0.9277\n",
      "Epoch 30/30\n",
      "30945/30945 [==============================] - 23s 753us/step - loss: 0.3047 - acc: 0.9231\n",
      "15473/15473 [==============================] - 13s 858us/step\n",
      "30945/30945 [==============================] - 21s 692us/step\n",
      "[CV] .................... epochs=30, lr=0.001, nodes=16, total=12.5min\n",
      "[CV] epochs=30, lr=0.001, nodes=16 ...................................\n",
      "Epoch 1/30\n",
      "30945/30945 [==============================] - 36s 1ms/step - loss: 0.3806 - acc: 0.9364\n",
      "Epoch 2/30\n",
      "30945/30945 [==============================] - 23s 737us/step - loss: 0.3550 - acc: 0.9847\n",
      "Epoch 3/30\n",
      "30945/30945 [==============================] - 23s 741us/step - loss: 0.3481 - acc: 0.9829\n",
      "Epoch 4/30\n",
      "30945/30945 [==============================] - 23s 741us/step - loss: 0.3434 - acc: 0.9805\n",
      "Epoch 5/30\n",
      "30945/30945 [==============================] - 23s 745us/step - loss: 0.3398 - acc: 0.9776\n",
      "Epoch 6/30\n",
      "30945/30945 [==============================] - 24s 765us/step - loss: 0.3361 - acc: 0.9750\n",
      "Epoch 7/30\n",
      "30945/30945 [==============================] - 24s 768us/step - loss: 0.3321 - acc: 0.9733\n",
      "Epoch 8/30\n",
      "30945/30945 [==============================] - 23s 756us/step - loss: 0.3299 - acc: 0.9693\n",
      "Epoch 9/30\n",
      "30945/30945 [==============================] - 23s 759us/step - loss: 0.3277 - acc: 0.9693\n",
      "Epoch 10/30\n",
      "30945/30945 [==============================] - 23s 748us/step - loss: 0.3267 - acc: 0.9660\n",
      "Epoch 11/30\n",
      "30945/30945 [==============================] - 23s 740us/step - loss: 0.3251 - acc: 0.9658\n",
      "Epoch 12/30\n",
      "30945/30945 [==============================] - 23s 745us/step - loss: 0.3246 - acc: 0.9624\n",
      "Epoch 13/30\n",
      "30945/30945 [==============================] - 23s 745us/step - loss: 0.3225 - acc: 0.9614\n",
      "Epoch 14/30\n",
      "30945/30945 [==============================] - 23s 739us/step - loss: 0.3209 - acc: 0.9567\n",
      "Epoch 15/30\n",
      "30945/30945 [==============================] - 23s 750us/step - loss: 0.3200 - acc: 0.9572\n",
      "Epoch 16/30\n",
      "30945/30945 [==============================] - 23s 752us/step - loss: 0.3184 - acc: 0.9559\n",
      "Epoch 17/30\n",
      "30945/30945 [==============================] - 23s 740us/step - loss: 0.3176 - acc: 0.9514\n",
      "Epoch 18/30\n",
      "30945/30945 [==============================] - 23s 737us/step - loss: 0.3161 - acc: 0.9489\n",
      "Epoch 19/30\n",
      "30945/30945 [==============================] - 23s 744us/step - loss: 0.3158 - acc: 0.9496\n",
      "Epoch 20/30\n",
      "30945/30945 [==============================] - 23s 731us/step - loss: 0.3148 - acc: 0.9470\n",
      "Epoch 21/30\n",
      "30945/30945 [==============================] - 22s 719us/step - loss: 0.3140 - acc: 0.9468\n",
      "Epoch 22/30\n",
      "30945/30945 [==============================] - 22s 723us/step - loss: 0.3150 - acc: 0.9448\n",
      "Epoch 23/30\n",
      "30945/30945 [==============================] - 22s 722us/step - loss: 0.3132 - acc: 0.9404\n",
      "Epoch 24/30\n",
      "30945/30945 [==============================] - 23s 728us/step - loss: 0.3134 - acc: 0.9391\n",
      "Epoch 25/30\n",
      "30945/30945 [==============================] - 23s 730us/step - loss: 0.3130 - acc: 0.9386\n",
      "Epoch 26/30\n",
      "30945/30945 [==============================] - 22s 718us/step - loss: 0.3105 - acc: 0.9331\n",
      "Epoch 27/30\n",
      "30945/30945 [==============================] - 23s 736us/step - loss: 0.3125 - acc: 0.9342\n",
      "Epoch 28/30\n",
      "30945/30945 [==============================] - 23s 747us/step - loss: 0.3112 - acc: 0.9329\n",
      "Epoch 29/30\n",
      "30945/30945 [==============================] - 23s 731us/step - loss: 0.3134 - acc: 0.9312\n",
      "Epoch 30/30\n",
      "30945/30945 [==============================] - 23s 744us/step - loss: 0.3116 - acc: 0.9337\n",
      "15473/15473 [==============================] - 14s 906us/step\n",
      "30945/30945 [==============================] - 21s 691us/step\n",
      "[CV] .................... epochs=30, lr=0.001, nodes=16, total=12.5min\n",
      "[CV] epochs=30, lr=0.001, nodes=16 ...................................\n",
      "Epoch 1/30\n",
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.3684 - acc: 0.9716\n",
      "Epoch 2/30\n",
      "30946/30946 [==============================] - 22s 724us/step - loss: 0.3425 - acc: 0.9813\n",
      "Epoch 3/30\n",
      "30946/30946 [==============================] - 23s 728us/step - loss: 0.3364 - acc: 0.9801\n",
      "Epoch 4/30\n",
      "30946/30946 [==============================] - 23s 727us/step - loss: 0.3331 - acc: 0.9775\n",
      "Epoch 5/30\n",
      "30946/30946 [==============================] - 22s 707us/step - loss: 0.3292 - acc: 0.9737\n",
      "Epoch 6/30\n",
      "30946/30946 [==============================] - 22s 715us/step - loss: 0.3278 - acc: 0.9712\n",
      "Epoch 7/30\n",
      "30946/30946 [==============================] - 22s 716us/step - loss: 0.3262 - acc: 0.9677\n",
      "Epoch 8/30\n",
      "30946/30946 [==============================] - 22s 703us/step - loss: 0.3217 - acc: 0.9640\n",
      "Epoch 9/30\n",
      "30946/30946 [==============================] - 22s 710us/step - loss: 0.3198 - acc: 0.9614\n",
      "Epoch 10/30\n",
      "30946/30946 [==============================] - 22s 708us/step - loss: 0.3183 - acc: 0.9596\n",
      "Epoch 11/30\n",
      "30946/30946 [==============================] - 22s 703us/step - loss: 0.3178 - acc: 0.9537\n",
      "Epoch 12/30\n",
      "30946/30946 [==============================] - 22s 697us/step - loss: 0.3157 - acc: 0.9568\n",
      "Epoch 13/30\n",
      "30946/30946 [==============================] - 22s 717us/step - loss: 0.3148 - acc: 0.9514\n",
      "Epoch 14/30\n",
      "30946/30946 [==============================] - 23s 743us/step - loss: 0.3124 - acc: 0.9512\n",
      "Epoch 15/30\n",
      "30946/30946 [==============================] - 22s 723us/step - loss: 0.3111 - acc: 0.9488\n",
      "Epoch 16/30\n",
      "30946/30946 [==============================] - 22s 725us/step - loss: 0.3110 - acc: 0.9501\n",
      "Epoch 17/30\n",
      "30946/30946 [==============================] - 22s 699us/step - loss: 0.3100 - acc: 0.9453\n",
      "Epoch 18/30\n",
      "30946/30946 [==============================] - 22s 700us/step - loss: 0.3087 - acc: 0.9465\n",
      "Epoch 19/30\n",
      "30946/30946 [==============================] - 22s 703us/step - loss: 0.3078 - acc: 0.9453\n",
      "Epoch 20/30\n",
      "30946/30946 [==============================] - 21s 681us/step - loss: 0.3074 - acc: 0.9377\n",
      "Epoch 21/30\n",
      "30946/30946 [==============================] - 21s 687us/step - loss: 0.3063 - acc: 0.9409\n",
      "Epoch 22/30\n",
      "30946/30946 [==============================] - 21s 688us/step - loss: 0.3046 - acc: 0.9355\n",
      "Epoch 23/30\n",
      "30946/30946 [==============================] - 21s 683us/step - loss: 0.3041 - acc: 0.9329\n",
      "Epoch 24/30\n",
      "30946/30946 [==============================] - 21s 688us/step - loss: 0.3050 - acc: 0.9368\n",
      "Epoch 25/30\n",
      "30946/30946 [==============================] - 22s 704us/step - loss: 0.3039 - acc: 0.9313\n",
      "Epoch 26/30\n",
      "30946/30946 [==============================] - 22s 704us/step - loss: 0.3026 - acc: 0.9275\n",
      "Epoch 27/30\n",
      "30946/30946 [==============================] - 22s 699us/step - loss: 0.3026 - acc: 0.9225\n",
      "Epoch 28/30\n",
      "30946/30946 [==============================] - 22s 704us/step - loss: 0.3022 - acc: 0.9193\n",
      "Epoch 29/30\n",
      "30946/30946 [==============================] - 22s 701us/step - loss: 0.3030 - acc: 0.9205\n",
      "Epoch 30/30\n",
      "30946/30946 [==============================] - 22s 714us/step - loss: 0.3023 - acc: 0.9172\n",
      "15472/15472 [==============================] - 14s 925us/step\n",
      "30946/30946 [==============================] - 25s 809us/step\n",
      "[CV] .................... epochs=30, lr=0.001, nodes=16, total=12.0min\n",
      "[CV] epochs=30, lr=0.001, nodes=32 ...................................\n",
      "Epoch 1/30\n",
      "30945/30945 [==============================] - 52s 2ms/step - loss: 0.3592 - acc: 0.9794\n",
      "Epoch 2/30\n",
      "30945/30945 [==============================] - 38s 1ms/step - loss: 0.3372 - acc: 0.9800\n",
      "Epoch 3/30\n",
      "30945/30945 [==============================] - 38s 1ms/step - loss: 0.3301 - acc: 0.9756\n",
      "Epoch 4/30\n",
      "30945/30945 [==============================] - 39s 1ms/step - loss: 0.3241 - acc: 0.9742\n",
      "Epoch 5/30\n",
      "30945/30945 [==============================] - 38s 1ms/step - loss: 0.3202 - acc: 0.9705\n",
      "Epoch 6/30\n",
      "30945/30945 [==============================] - 38s 1ms/step - loss: 0.3165 - acc: 0.9682\n",
      "Epoch 7/30\n",
      "30945/30945 [==============================] - 38s 1ms/step - loss: 0.3115 - acc: 0.9636\n",
      "Epoch 8/30\n",
      "30945/30945 [==============================] - 38s 1ms/step - loss: 0.3092 - acc: 0.9614\n",
      "Epoch 9/30\n",
      "30945/30945 [==============================] - 38s 1ms/step - loss: 0.3076 - acc: 0.9607\n",
      "Epoch 10/30\n",
      "30945/30945 [==============================] - 38s 1ms/step - loss: 0.3046 - acc: 0.9568\n",
      "Epoch 11/30\n",
      "30945/30945 [==============================] - 38s 1ms/step - loss: 0.3025 - acc: 0.9541\n",
      "Epoch 12/30\n",
      "30945/30945 [==============================] - 38s 1ms/step - loss: 0.3017 - acc: 0.9512\n",
      "Epoch 13/30\n",
      "30945/30945 [==============================] - 39s 1ms/step - loss: 0.2992 - acc: 0.9471\n",
      "Epoch 14/30\n",
      "30945/30945 [==============================] - 39s 1ms/step - loss: 0.2986 - acc: 0.9430\n",
      "Epoch 15/30\n",
      "30945/30945 [==============================] - 40s 1ms/step - loss: 0.2980 - acc: 0.9431\n",
      "Epoch 16/30\n",
      "30945/30945 [==============================] - 39s 1ms/step - loss: 0.2967 - acc: 0.9419\n",
      "Epoch 17/30\n",
      "30945/30945 [==============================] - 41s 1ms/step - loss: 0.2968 - acc: 0.9347\n",
      "Epoch 18/30\n",
      "30945/30945 [==============================] - 38s 1ms/step - loss: 0.2938 - acc: 0.9355\n",
      "Epoch 19/30\n",
      "30945/30945 [==============================] - 36s 1ms/step - loss: 0.2938 - acc: 0.9360\n",
      "Epoch 20/30\n",
      "30945/30945 [==============================] - 36s 1ms/step - loss: 0.2934 - acc: 0.9286\n",
      "Epoch 21/30\n",
      "30945/30945 [==============================] - 35s 1ms/step - loss: 0.2920 - acc: 0.9283\n",
      "Epoch 22/30\n",
      "30945/30945 [==============================] - 37s 1ms/step - loss: 0.2926 - acc: 0.9247\n",
      "Epoch 23/30\n",
      "30945/30945 [==============================] - 38s 1ms/step - loss: 0.2911 - acc: 0.9210\n",
      "Epoch 24/30\n",
      "30945/30945 [==============================] - 36s 1ms/step - loss: 0.2912 - acc: 0.9160\n",
      "Epoch 25/30\n",
      "30945/30945 [==============================] - 35s 1ms/step - loss: 0.2902 - acc: 0.9179\n",
      "Epoch 26/30\n",
      "30945/30945 [==============================] - 36s 1ms/step - loss: 0.2897 - acc: 0.9111\n",
      "Epoch 27/30\n",
      "30945/30945 [==============================] - 36s 1ms/step - loss: 0.2923 - acc: 0.9107\n",
      "Epoch 28/30\n",
      "30945/30945 [==============================] - 36s 1ms/step - loss: 0.2921 - acc: 0.9074\n",
      "Epoch 29/30\n",
      "30945/30945 [==============================] - 35s 1ms/step - loss: 0.2894 - acc: 0.9030\n",
      "Epoch 30/30\n",
      "30945/30945 [==============================] - 39s 1ms/step - loss: 0.2897 - acc: 0.8993\n",
      "15473/15473 [==============================] - 16s 1ms/step\n",
      "30945/30945 [==============================] - 25s 816us/step\n",
      "[CV] .................... epochs=30, lr=0.001, nodes=32, total=20.0min\n",
      "[CV] epochs=30, lr=0.001, nodes=32 ...................................\n",
      "Epoch 1/30\n",
      "30945/30945 [==============================] - 56s 2ms/step - loss: 0.3691 - acc: 0.9816\n",
      "Epoch 2/30\n",
      "30945/30945 [==============================] - 41s 1ms/step - loss: 0.3494 - acc: 0.9783\n",
      "Epoch 3/30\n",
      "30945/30945 [==============================] - 38s 1ms/step - loss: 0.3397 - acc: 0.9770\n",
      "Epoch 4/30\n",
      "30945/30945 [==============================] - 39s 1ms/step - loss: 0.3358 - acc: 0.9745\n",
      "Epoch 5/30\n",
      "30945/30945 [==============================] - 40s 1ms/step - loss: 0.3298 - acc: 0.9702\n",
      "Epoch 6/30\n",
      "30945/30945 [==============================] - 41s 1ms/step - loss: 0.3266 - acc: 0.9671\n",
      "Epoch 7/30\n",
      "30945/30945 [==============================] - 41s 1ms/step - loss: 0.3241 - acc: 0.9630\n",
      "Epoch 8/30\n",
      "30945/30945 [==============================] - 39s 1ms/step - loss: 0.3212 - acc: 0.9624\n",
      "Epoch 9/30\n",
      "30945/30945 [==============================] - 40s 1ms/step - loss: 0.3175 - acc: 0.9575\n",
      "Epoch 10/30\n",
      "30945/30945 [==============================] - 40s 1ms/step - loss: 0.3156 - acc: 0.9548\n",
      "Epoch 11/30\n",
      "30945/30945 [==============================] - 40s 1ms/step - loss: 0.3134 - acc: 0.9549\n",
      "Epoch 12/30\n",
      "30945/30945 [==============================] - 40s 1ms/step - loss: 0.3120 - acc: 0.9480\n",
      "Epoch 13/30\n",
      "30945/30945 [==============================] - 39s 1ms/step - loss: 0.3103 - acc: 0.9466\n",
      "Epoch 14/30\n",
      "30945/30945 [==============================] - 41s 1ms/step - loss: 0.3071 - acc: 0.9398\n",
      "Epoch 15/30\n",
      "30945/30945 [==============================] - 40s 1ms/step - loss: 0.3053 - acc: 0.9393\n",
      "Epoch 16/30\n",
      "30945/30945 [==============================] - 40s 1ms/step - loss: 0.3061 - acc: 0.9336\n",
      "Epoch 17/30\n",
      "30945/30945 [==============================] - 40s 1ms/step - loss: 0.3045 - acc: 0.9348\n",
      "Epoch 18/30\n",
      "30945/30945 [==============================] - 40s 1ms/step - loss: 0.3044 - acc: 0.9303\n",
      "Epoch 19/30\n",
      "30945/30945 [==============================] - 41s 1ms/step - loss: 0.3041 - acc: 0.9289\n",
      "Epoch 20/30\n",
      "30945/30945 [==============================] - 40s 1ms/step - loss: 0.3033 - acc: 0.9271\n",
      "Epoch 21/30\n",
      "30945/30945 [==============================] - 40s 1ms/step - loss: 0.3013 - acc: 0.9266\n",
      "Epoch 22/30\n",
      "30945/30945 [==============================] - 41s 1ms/step - loss: 0.3015 - acc: 0.9235\n",
      "Epoch 23/30\n",
      "30945/30945 [==============================] - 41s 1ms/step - loss: 0.3010 - acc: 0.9190\n",
      "Epoch 24/30\n",
      "30945/30945 [==============================] - 40s 1ms/step - loss: 0.3008 - acc: 0.9109\n",
      "Epoch 25/30\n",
      "30945/30945 [==============================] - 40s 1ms/step - loss: 0.3005 - acc: 0.9092\n",
      "Epoch 26/30\n",
      "30945/30945 [==============================] - 40s 1ms/step - loss: 0.2995 - acc: 0.9160\n",
      "Epoch 27/30\n",
      "30945/30945 [==============================] - 40s 1ms/step - loss: 0.3019 - acc: 0.9118\n",
      "Epoch 28/30\n",
      "30945/30945 [==============================] - 41s 1ms/step - loss: 0.3008 - acc: 0.9105\n",
      "Epoch 29/30\n",
      "30945/30945 [==============================] - 40s 1ms/step - loss: 0.3004 - acc: 0.9065\n",
      "Epoch 30/30\n",
      "30945/30945 [==============================] - 40s 1ms/step - loss: 0.3010 - acc: 0.9010\n",
      "15473/15473 [==============================] - 17s 1ms/step\n",
      "30945/30945 [==============================] - 25s 808us/step\n",
      "[CV] .................... epochs=30, lr=0.001, nodes=32, total=21.2min\n",
      "[CV] epochs=30, lr=0.001, nodes=32 ...................................\n",
      "Epoch 1/30\n",
      "30946/30946 [==============================] - 49s 2ms/step - loss: 0.3587 - acc: 0.9812\n",
      "Epoch 2/30\n",
      "30946/30946 [==============================] - 37s 1ms/step - loss: 0.3391 - acc: 0.9802\n",
      "Epoch 3/30\n",
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.3324 - acc: 0.9789\n",
      "Epoch 4/30\n",
      "30946/30946 [==============================] - 37s 1ms/step - loss: 0.3255 - acc: 0.9783\n",
      "Epoch 5/30\n",
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.3217 - acc: 0.9725\n",
      "Epoch 6/30\n",
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.3177 - acc: 0.9710\n",
      "Epoch 7/30\n",
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.3138 - acc: 0.9669\n",
      "Epoch 8/30\n",
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.3123 - acc: 0.9621\n",
      "Epoch 9/30\n",
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.3095 - acc: 0.9595\n",
      "Epoch 10/30\n",
      "30946/30946 [==============================] - 35s 1ms/step - loss: 0.3089 - acc: 0.9561\n",
      "Epoch 11/30\n",
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.3061 - acc: 0.9519\n",
      "Epoch 12/30\n",
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.3034 - acc: 0.9499\n",
      "Epoch 13/30\n",
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.3048 - acc: 0.9446\n",
      "Epoch 14/30\n",
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.3030 - acc: 0.9418\n",
      "Epoch 15/30\n",
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.3010 - acc: 0.9367\n",
      "Epoch 16/30\n",
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.2986 - acc: 0.9351\n",
      "Epoch 17/30\n",
      "30946/30946 [==============================] - 38s 1ms/step - loss: 0.2998 - acc: 0.9360\n",
      "Epoch 18/30\n",
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.2975 - acc: 0.9352\n",
      "Epoch 19/30\n",
      "30946/30946 [==============================] - 38s 1ms/step - loss: 0.2972 - acc: 0.9281\n",
      "Epoch 20/30\n",
      "30946/30946 [==============================] - 37s 1ms/step - loss: 0.2961 - acc: 0.9254\n",
      "Epoch 21/30\n",
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.2942 - acc: 0.9244\n",
      "Epoch 22/30\n",
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.2955 - acc: 0.9215\n",
      "Epoch 23/30\n",
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.2928 - acc: 0.9177\n",
      "Epoch 24/30\n",
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.2928 - acc: 0.9144\n",
      "Epoch 25/30\n",
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.2920 - acc: 0.9110\n",
      "Epoch 26/30\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.2913 - acc: 0.9141\n",
      "Epoch 27/30\n",
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.2918 - acc: 0.9088\n",
      "Epoch 28/30\n",
      "30946/30946 [==============================] - 37s 1ms/step - loss: 0.2915 - acc: 0.9094\n",
      "Epoch 29/30\n",
      "30946/30946 [==============================] - 37s 1ms/step - loss: 0.2902 - acc: 0.9022\n",
      "Epoch 30/30\n",
      "30946/30946 [==============================] - 37s 1ms/step - loss: 0.2899 - acc: 0.9019\n",
      "15472/15472 [==============================] - 16s 1ms/step\n",
      "30946/30946 [==============================] - 25s 812us/step\n",
      "[CV] .................... epochs=30, lr=0.001, nodes=32, total=19.2min\n",
      "[CV] epochs=30, lr=0.001, nodes=64 ...................................\n",
      "Epoch 1/30\n",
      "30945/30945 [==============================] - 76s 2ms/step - loss: 0.3576 - acc: 0.9797\n",
      "Epoch 2/30\n",
      "30945/30945 [==============================] - 63s 2ms/step - loss: 0.3346 - acc: 0.9770\n",
      "Epoch 3/30\n",
      "30945/30945 [==============================] - 63s 2ms/step - loss: 0.3259 - acc: 0.9727\n",
      "Epoch 4/30\n",
      "30945/30945 [==============================] - 63s 2ms/step - loss: 0.3197 - acc: 0.9715\n",
      "Epoch 5/30\n",
      "30945/30945 [==============================] - 65s 2ms/step - loss: 0.3145 - acc: 0.9661\n",
      "Epoch 6/30\n",
      "30945/30945 [==============================] - 63s 2ms/step - loss: 0.3097 - acc: 0.9618\n",
      "Epoch 7/30\n",
      "30945/30945 [==============================] - 63s 2ms/step - loss: 0.3059 - acc: 0.9600\n",
      "Epoch 8/30\n",
      "30945/30945 [==============================] - 62s 2ms/step - loss: 0.3020 - acc: 0.9565\n",
      "Epoch 9/30\n",
      "30945/30945 [==============================] - 63s 2ms/step - loss: 0.2990 - acc: 0.9515\n",
      "Epoch 10/30\n",
      "30945/30945 [==============================] - 63s 2ms/step - loss: 0.2965 - acc: 0.9514\n",
      "Epoch 11/30\n",
      "30945/30945 [==============================] - 65s 2ms/step - loss: 0.2932 - acc: 0.9465\n",
      "Epoch 12/30\n",
      "30945/30945 [==============================] - 64s 2ms/step - loss: 0.2921 - acc: 0.9432\n",
      "Epoch 13/30\n",
      "30945/30945 [==============================] - 64s 2ms/step - loss: 0.2906 - acc: 0.9379\n",
      "Epoch 14/30\n",
      "30945/30945 [==============================] - 64s 2ms/step - loss: 0.2885 - acc: 0.9336\n",
      "Epoch 15/30\n",
      "30945/30945 [==============================] - 63s 2ms/step - loss: 0.2893 - acc: 0.9300\n",
      "Epoch 16/30\n",
      "30945/30945 [==============================] - 62s 2ms/step - loss: 0.2871 - acc: 0.9297\n",
      "Epoch 17/30\n",
      "30945/30945 [==============================] - 62s 2ms/step - loss: 0.2876 - acc: 0.9218\n",
      "Epoch 18/30\n",
      "30945/30945 [==============================] - 63s 2ms/step - loss: 0.2871 - acc: 0.9236\n",
      "Epoch 19/30\n",
      "30945/30945 [==============================] - 62s 2ms/step - loss: 0.2876 - acc: 0.9214\n",
      "Epoch 20/30\n",
      "30945/30945 [==============================] - 64s 2ms/step - loss: 0.2879 - acc: 0.9201\n",
      "Epoch 21/30\n",
      "30945/30945 [==============================] - 63s 2ms/step - loss: 0.2859 - acc: 0.9110\n",
      "Epoch 22/30\n",
      "30945/30945 [==============================] - 64s 2ms/step - loss: 0.2854 - acc: 0.9075\n",
      "Epoch 23/30\n",
      "30945/30945 [==============================] - 64s 2ms/step - loss: 0.2846 - acc: 0.9111\n",
      "Epoch 24/30\n",
      "30945/30945 [==============================] - 64s 2ms/step - loss: 0.2857 - acc: 0.9038\n",
      "Epoch 25/30\n",
      "30945/30945 [==============================] - 63s 2ms/step - loss: 0.2846 - acc: 0.9042\n",
      "Epoch 26/30\n",
      "30945/30945 [==============================] - 63s 2ms/step - loss: 0.2845 - acc: 0.9040\n",
      "Epoch 27/30\n",
      "30945/30945 [==============================] - 64s 2ms/step - loss: 0.2855 - acc: 0.8937\n",
      "Epoch 28/30\n",
      "30945/30945 [==============================] - 63s 2ms/step - loss: 0.2848 - acc: 0.8947\n",
      "Epoch 29/30\n",
      "30945/30945 [==============================] - 64s 2ms/step - loss: 0.2847 - acc: 0.8925\n",
      "Epoch 30/30\n",
      "30945/30945 [==============================] - 65s 2ms/step - loss: 0.2838 - acc: 0.8937\n",
      "15473/15473 [==============================] - 17s 1ms/step\n",
      "30945/30945 [==============================] - 30s 964us/step\n",
      "[CV] .................... epochs=30, lr=0.001, nodes=64, total=32.8min\n",
      "[CV] epochs=30, lr=0.001, nodes=64 ...................................\n",
      "Epoch 1/30\n",
      "30945/30945 [==============================] - 75s 2ms/step - loss: 0.3655 - acc: 0.9797\n",
      "Epoch 2/30\n",
      "30945/30945 [==============================] - 63s 2ms/step - loss: 0.3445 - acc: 0.9773\n",
      "Epoch 3/30\n",
      "30945/30945 [==============================] - 63s 2ms/step - loss: 0.3353 - acc: 0.9753\n",
      "Epoch 4/30\n",
      "30945/30945 [==============================] - 64s 2ms/step - loss: 0.3284 - acc: 0.9719\n",
      "Epoch 5/30\n",
      "30945/30945 [==============================] - 62s 2ms/step - loss: 0.3235 - acc: 0.9687\n",
      "Epoch 6/30\n",
      "30945/30945 [==============================] - 62s 2ms/step - loss: 0.3193 - acc: 0.9627\n",
      "Epoch 7/30\n",
      "30945/30945 [==============================] - 62s 2ms/step - loss: 0.3146 - acc: 0.9561\n",
      "Epoch 8/30\n",
      "30945/30945 [==============================] - 64s 2ms/step - loss: 0.3106 - acc: 0.9538\n",
      "Epoch 9/30\n",
      "30945/30945 [==============================] - 63s 2ms/step - loss: 0.3082 - acc: 0.9465\n",
      "Epoch 10/30\n",
      "30945/30945 [==============================] - 62s 2ms/step - loss: 0.3063 - acc: 0.9448\n",
      "Epoch 11/30\n",
      "30945/30945 [==============================] - 63s 2ms/step - loss: 0.3040 - acc: 0.9407\n",
      "Epoch 12/30\n",
      "30945/30945 [==============================] - 62s 2ms/step - loss: 0.3014 - acc: 0.9320\n",
      "Epoch 13/30\n",
      "30945/30945 [==============================] - 61s 2ms/step - loss: 0.3003 - acc: 0.9333\n",
      "Epoch 14/30\n",
      "30945/30945 [==============================] - 63s 2ms/step - loss: 0.2998 - acc: 0.9279\n",
      "Epoch 15/30\n",
      "30945/30945 [==============================] - 62s 2ms/step - loss: 0.2991 - acc: 0.9243\n",
      "Epoch 16/30\n",
      "30945/30945 [==============================] - 61s 2ms/step - loss: 0.2975 - acc: 0.9237\n",
      "Epoch 17/30\n",
      "30945/30945 [==============================] - 63s 2ms/step - loss: 0.2978 - acc: 0.9205\n",
      "Epoch 18/30\n",
      "30945/30945 [==============================] - 62s 2ms/step - loss: 0.2963 - acc: 0.9209\n",
      "Epoch 19/30\n",
      "30945/30945 [==============================] - 62s 2ms/step - loss: 0.2964 - acc: 0.9160\n",
      "Epoch 20/30\n",
      "30945/30945 [==============================] - 63s 2ms/step - loss: 0.2945 - acc: 0.9116\n",
      "Epoch 21/30\n",
      "30945/30945 [==============================] - 62s 2ms/step - loss: 0.2955 - acc: 0.9076: 1s\n",
      "Epoch 22/30\n",
      "30945/30945 [==============================] - 62s 2ms/step - loss: 0.2961 - acc: 0.9029\n",
      "Epoch 23/30\n",
      "30945/30945 [==============================] - 61s 2ms/step - loss: 0.2961 - acc: 0.9041\n",
      "Epoch 24/30\n",
      "30945/30945 [==============================] - 63s 2ms/step - loss: 0.2966 - acc: 0.9002\n",
      "Epoch 25/30\n",
      "30945/30945 [==============================] - ETA: 0s - loss: 0.2960 - acc: 0.897 - 63s 2ms/step - loss: 0.2960 - acc: 0.8974\n",
      "Epoch 26/30\n",
      "30945/30945 [==============================] - 61s 2ms/step - loss: 0.2934 - acc: 0.8973\n",
      "Epoch 27/30\n",
      "30945/30945 [==============================] - 61s 2ms/step - loss: 0.2939 - acc: 0.8922\n",
      "Epoch 28/30\n",
      "30945/30945 [==============================] - 62s 2ms/step - loss: 0.2939 - acc: 0.8933\n",
      "Epoch 29/30\n",
      "30945/30945 [==============================] - 62s 2ms/step - loss: 0.2959 - acc: 0.8901\n",
      "Epoch 30/30\n",
      "30945/30945 [==============================] - 61s 2ms/step - loss: 0.2946 - acc: 0.8894\n",
      "15473/15473 [==============================] - 18s 1ms/step\n",
      "30945/30945 [==============================] - 30s 955us/step\n",
      "[CV] .................... epochs=30, lr=0.001, nodes=64, total=32.3min\n",
      "[CV] epochs=30, lr=0.001, nodes=64 ...................................\n",
      "Epoch 1/30\n",
      "30946/30946 [==============================] - 72s 2ms/step - loss: 0.3570 - acc: 0.9778\n",
      "Epoch 2/30\n",
      "30946/30946 [==============================] - 63s 2ms/step - loss: 0.3332 - acc: 0.9796\n",
      "Epoch 3/30\n",
      "30946/30946 [==============================] - 61s 2ms/step - loss: 0.3239 - acc: 0.9756\n",
      "Epoch 4/30\n",
      "30946/30946 [==============================] - 61s 2ms/step - loss: 0.3176 - acc: 0.9696\n",
      "Epoch 5/30\n",
      "30946/30946 [==============================] - 62s 2ms/step - loss: 0.3136 - acc: 0.9649\n",
      "Epoch 6/30\n",
      "30946/30946 [==============================] - 60s 2ms/step - loss: 0.3090 - acc: 0.9633\n",
      "Epoch 7/30\n",
      "30946/30946 [==============================] - 62s 2ms/step - loss: 0.3063 - acc: 0.9573\n",
      "Epoch 8/30\n",
      "30946/30946 [==============================] - 61s 2ms/step - loss: 0.3013 - acc: 0.9549\n",
      "Epoch 9/30\n",
      "30946/30946 [==============================] - 61s 2ms/step - loss: 0.2991 - acc: 0.9525\n",
      "Epoch 10/30\n",
      "30946/30946 [==============================] - 60s 2ms/step - loss: 0.2965 - acc: 0.9467\n",
      "Epoch 11/30\n",
      "30946/30946 [==============================] - 59s 2ms/step - loss: 0.2939 - acc: 0.9417\n",
      "Epoch 12/30\n",
      "30946/30946 [==============================] - 60s 2ms/step - loss: 0.2936 - acc: 0.9417\n",
      "Epoch 13/30\n",
      "30946/30946 [==============================] - 62s 2ms/step - loss: 0.2933 - acc: 0.9399\n",
      "Epoch 14/30\n",
      "30946/30946 [==============================] - 61s 2ms/step - loss: 0.2902 - acc: 0.9302\n",
      "Epoch 15/30\n",
      "30946/30946 [==============================] - 60s 2ms/step - loss: 0.2884 - acc: 0.9270\n",
      "Epoch 16/30\n",
      "30946/30946 [==============================] - 59s 2ms/step - loss: 0.2870 - acc: 0.9290\n",
      "Epoch 17/30\n",
      "30946/30946 [==============================] - 62s 2ms/step - loss: 0.2866 - acc: 0.9243\n",
      "Epoch 18/30\n",
      "30946/30946 [==============================] - 62s 2ms/step - loss: 0.2869 - acc: 0.9213\n",
      "Epoch 19/30\n",
      "30946/30946 [==============================] - 60s 2ms/step - loss: 0.2852 - acc: 0.9168\n",
      "Epoch 20/30\n",
      "30946/30946 [==============================] - 60s 2ms/step - loss: 0.2855 - acc: 0.9138\n",
      "Epoch 21/30\n",
      "30946/30946 [==============================] - 62s 2ms/step - loss: 0.2849 - acc: 0.9078\n",
      "Epoch 22/30\n",
      "30946/30946 [==============================] - 61s 2ms/step - loss: 0.2852 - acc: 0.9068\n",
      "Epoch 23/30\n",
      "30946/30946 [==============================] - 60s 2ms/step - loss: 0.2838 - acc: 0.9087\n",
      "Epoch 24/30\n",
      "30946/30946 [==============================] - 60s 2ms/step - loss: 0.2838 - acc: 0.9037\n",
      "Epoch 25/30\n",
      "30946/30946 [==============================] - 1259s 41ms/step - loss: 0.2851 - acc: 0.9023\n",
      "Epoch 26/30\n",
      "30946/30946 [==============================] - 32621s 1s/step - loss: 0.2840 - acc: 0.8957\n",
      "Epoch 27/30\n",
      "30946/30946 [==============================] - 46s 1ms/step - loss: 0.2839 - acc: 0.9011\n",
      "Epoch 28/30\n",
      "30946/30946 [==============================] - 48s 2ms/step - loss: 0.2842 - acc: 0.8935\n",
      "Epoch 29/30\n",
      "30946/30946 [==============================] - 48s 2ms/step - loss: 0.2856 - acc: 0.8936\n",
      "Epoch 30/30\n",
      "30946/30946 [==============================] - 49s 2ms/step - loss: 0.2843 - acc: 0.8920\n",
      "15472/15472 [==============================] - 14s 919us/step\n",
      "30946/30946 [==============================] - 24s 763us/step\n",
      "[CV] ................... epochs=30, lr=0.001, nodes=64, total=593.2min\n",
      "[CV] epochs=30, lr=0.002, nodes=16 ...................................\n",
      "Epoch 1/30\n",
      "30945/30945 [==============================] - 39s 1ms/step - loss: 0.3593 - acc: 0.9760\n",
      "Epoch 2/30\n",
      "30945/30945 [==============================] - 25s 798us/step - loss: 0.3403 - acc: 0.9836\n",
      "Epoch 3/30\n",
      "30945/30945 [==============================] - 25s 798us/step - loss: 0.3345 - acc: 0.9805\n",
      "Epoch 4/30\n",
      "30945/30945 [==============================] - 25s 804us/step - loss: 0.3313 - acc: 0.9736\n",
      "Epoch 5/30\n",
      "30945/30945 [==============================] - 25s 806us/step - loss: 0.3281 - acc: 0.9696\n",
      "Epoch 6/30\n",
      "30945/30945 [==============================] - 25s 813us/step - loss: 0.3246 - acc: 0.9683\n",
      "Epoch 7/30\n",
      "30945/30945 [==============================] - 25s 814us/step - loss: 0.3230 - acc: 0.9625\n",
      "Epoch 8/30\n",
      "30945/30945 [==============================] - 25s 800us/step - loss: 0.3211 - acc: 0.9628\n",
      "Epoch 9/30\n",
      "30945/30945 [==============================] - 25s 823us/step - loss: 0.3191 - acc: 0.9577\n",
      "Epoch 10/30\n",
      "30945/30945 [==============================] - 26s 824us/step - loss: 0.3167 - acc: 0.9593\n",
      "Epoch 11/30\n",
      "30945/30945 [==============================] - 25s 816us/step - loss: 0.3163 - acc: 0.9596\n",
      "Epoch 12/30\n",
      "30945/30945 [==============================] - 28s 905us/step - loss: 0.3158 - acc: 0.9589\n",
      "Epoch 13/30\n",
      "30945/30945 [==============================] - 26s 837us/step - loss: 0.3140 - acc: 0.9548\n",
      "Epoch 14/30\n",
      "30945/30945 [==============================] - 26s 834us/step - loss: 0.3128 - acc: 0.9529\n",
      "Epoch 15/30\n",
      "30945/30945 [==============================] - 25s 819us/step - loss: 0.3125 - acc: 0.9509\n",
      "Epoch 16/30\n",
      "30945/30945 [==============================] - 26s 829us/step - loss: 0.3115 - acc: 0.9465\n",
      "Epoch 17/30\n",
      "30945/30945 [==============================] - 25s 817us/step - loss: 0.3082 - acc: 0.9461\n",
      "Epoch 18/30\n",
      "30945/30945 [==============================] - 26s 850us/step - loss: 0.3090 - acc: 0.9436\n",
      "Epoch 19/30\n",
      "30945/30945 [==============================] - 26s 852us/step - loss: 0.3072 - acc: 0.9405\n",
      "Epoch 20/30\n",
      "30945/30945 [==============================] - 26s 845us/step - loss: 0.3078 - acc: 0.9388\n",
      "Epoch 21/30\n",
      "30945/30945 [==============================] - 26s 837us/step - loss: 0.3059 - acc: 0.9350\n",
      "Epoch 22/30\n",
      "30945/30945 [==============================] - 26s 835us/step - loss: 0.3052 - acc: 0.9292\n",
      "Epoch 23/30\n",
      "30945/30945 [==============================] - 25s 823us/step - loss: 0.3057 - acc: 0.9294\n",
      "Epoch 24/30\n",
      "30945/30945 [==============================] - 26s 851us/step - loss: 0.3046 - acc: 0.9191\n",
      "Epoch 25/30\n",
      "30945/30945 [==============================] - 28s 907us/step - loss: 0.3046 - acc: 0.9202\n",
      "Epoch 26/30\n",
      "30945/30945 [==============================] - 27s 876us/step - loss: 0.3034 - acc: 0.9180\n",
      "Epoch 27/30\n",
      "30945/30945 [==============================] - 26s 835us/step - loss: 0.3040 - acc: 0.9123\n",
      "Epoch 28/30\n",
      "30945/30945 [==============================] - 26s 831us/step - loss: 0.3035 - acc: 0.9123\n",
      "Epoch 29/30\n",
      "30945/30945 [==============================] - 26s 843us/step - loss: 0.3033 - acc: 0.9109\n",
      "Epoch 30/30\n",
      "30945/30945 [==============================] - 27s 869us/step - loss: 0.3028 - acc: 0.9094\n",
      "15473/15473 [==============================] - 15s 982us/step\n",
      "30945/30945 [==============================] - 24s 761us/step\n",
      "[CV] .................... epochs=30, lr=0.002, nodes=16, total=13.9min\n",
      "[CV] epochs=30, lr=0.002, nodes=16 ...................................\n",
      "Epoch 1/30\n",
      "30945/30945 [==============================] - 39s 1ms/step - loss: 0.3678 - acc: 0.9748\n",
      "Epoch 2/30\n",
      "30945/30945 [==============================] - 29s 939us/step - loss: 0.3481 - acc: 0.9827\n",
      "Epoch 3/30\n",
      "30945/30945 [==============================] - 26s 841us/step - loss: 0.3417 - acc: 0.9795\n",
      "Epoch 4/30\n",
      "30945/30945 [==============================] - 25s 813us/step - loss: 0.3389 - acc: 0.9762\n",
      "Epoch 5/30\n",
      "30945/30945 [==============================] - 25s 818us/step - loss: 0.3353 - acc: 0.9712\n",
      "Epoch 6/30\n",
      "30945/30945 [==============================] - 25s 819us/step - loss: 0.3350 - acc: 0.9694\n",
      "Epoch 7/30\n",
      "30945/30945 [==============================] - 26s 825us/step - loss: 0.3314 - acc: 0.9624\n",
      "Epoch 8/30\n",
      "30945/30945 [==============================] - 25s 820us/step - loss: 0.3301 - acc: 0.9566\n",
      "Epoch 9/30\n",
      "30945/30945 [==============================] - 25s 818us/step - loss: 0.3306 - acc: 0.9573\n",
      "Epoch 10/30\n",
      "30945/30945 [==============================] - 25s 822us/step - loss: 0.3277 - acc: 0.9516\n",
      "Epoch 11/30\n",
      "30945/30945 [==============================] - 27s 878us/step - loss: 0.3259 - acc: 0.9444\n",
      "Epoch 12/30\n",
      "30945/30945 [==============================] - 24s 786us/step - loss: 0.3251 - acc: 0.9441\n",
      "Epoch 13/30\n",
      "30945/30945 [==============================] - 23s 752us/step - loss: 0.3237 - acc: 0.9389\n",
      "Epoch 14/30\n",
      "30945/30945 [==============================] - 23s 733us/step - loss: 0.3215 - acc: 0.9381\n",
      "Epoch 15/30\n",
      "30945/30945 [==============================] - 22s 723us/step - loss: 0.3217 - acc: 0.9361\n",
      "Epoch 16/30\n",
      "30945/30945 [==============================] - 23s 736us/step - loss: 0.3209 - acc: 0.9333\n",
      "Epoch 17/30\n",
      "30945/30945 [==============================] - 23s 743us/step - loss: 0.3195 - acc: 0.9327\n",
      "Epoch 18/30\n",
      "30945/30945 [==============================] - 23s 745us/step - loss: 0.3207 - acc: 0.9301\n",
      "Epoch 19/30\n",
      "30945/30945 [==============================] - 23s 734us/step - loss: 0.3185 - acc: 0.9292\n",
      "Epoch 20/30\n",
      "30945/30945 [==============================] - 23s 746us/step - loss: 0.3207 - acc: 0.9223\n",
      "Epoch 21/30\n",
      "30945/30945 [==============================] - 23s 746us/step - loss: 0.3202 - acc: 0.9257\n",
      "Epoch 22/30\n",
      "30945/30945 [==============================] - 23s 744us/step - loss: 0.3188 - acc: 0.9191\n",
      "Epoch 23/30\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "30945/30945 [==============================] - 25s 822us/step - loss: 0.3195 - acc: 0.9178\n",
      "Epoch 24/30\n",
      "30945/30945 [==============================] - 24s 780us/step - loss: 0.3194 - acc: 0.9173\n",
      "Epoch 25/30\n",
      "30945/30945 [==============================] - 22s 703us/step - loss: 0.3188 - acc: 0.9147\n",
      "Epoch 26/30\n",
      "30945/30945 [==============================] - 22s 721us/step - loss: 0.3193 - acc: 0.9092\n",
      "Epoch 27/30\n",
      "30945/30945 [==============================] - 23s 742us/step - loss: 0.3192 - acc: 0.9111\n",
      "Epoch 28/30\n",
      "30945/30945 [==============================] - 23s 752us/step - loss: 0.3194 - acc: 0.9019\n",
      "Epoch 29/30\n",
      "30945/30945 [==============================] - 23s 737us/step - loss: 0.3204 - acc: 0.9066\n",
      "Epoch 30/30\n",
      "30945/30945 [==============================] - 23s 736us/step - loss: 0.3190 - acc: 0.9017\n",
      "15473/15473 [==============================] - 14s 928us/step\n",
      "30945/30945 [==============================] - 22s 697us/step\n",
      "[CV] .................... epochs=30, lr=0.002, nodes=16, total=13.1min\n",
      "[CV] epochs=30, lr=0.002, nodes=16 ...................................\n",
      "Epoch 1/30\n",
      "30946/30946 [==============================] - 35s 1ms/step - loss: 0.3600 - acc: 0.9842\n",
      "Epoch 2/30\n",
      "30946/30946 [==============================] - 24s 770us/step - loss: 0.3391 - acc: 0.9810\n",
      "Epoch 3/30\n",
      "30946/30946 [==============================] - 23s 735us/step - loss: 0.3342 - acc: 0.9774\n",
      "Epoch 4/30\n",
      "30946/30946 [==============================] - 23s 743us/step - loss: 0.3297 - acc: 0.9725\n",
      "Epoch 5/30\n",
      "30946/30946 [==============================] - 24s 783us/step - loss: 0.3251 - acc: 0.9671\n",
      "Epoch 6/30\n",
      "30946/30946 [==============================] - 25s 807us/step - loss: 0.3229 - acc: 0.9669\n",
      "Epoch 7/30\n",
      "30946/30946 [==============================] - 27s 873us/step - loss: 0.3201 - acc: 0.9633\n",
      "Epoch 8/30\n",
      "30946/30946 [==============================] - 28s 890us/step - loss: 0.3180 - acc: 0.9637\n",
      "Epoch 9/30\n",
      "30946/30946 [==============================] - 26s 853us/step - loss: 0.3145 - acc: 0.9584\n",
      "Epoch 10/30\n",
      "30946/30946 [==============================] - 26s 849us/step - loss: 0.3161 - acc: 0.9579\n",
      "Epoch 11/30\n",
      "30946/30946 [==============================] - 26s 846us/step - loss: 0.3142 - acc: 0.9604\n",
      "Epoch 12/30\n",
      "30946/30946 [==============================] - 27s 864us/step - loss: 0.3126 - acc: 0.9531\n",
      "Epoch 13/30\n",
      "30946/30946 [==============================] - 27s 863us/step - loss: 0.3107 - acc: 0.9509\n",
      "Epoch 14/30\n",
      "30946/30946 [==============================] - 26s 856us/step - loss: 0.3107 - acc: 0.9516\n",
      "Epoch 15/30\n",
      "30946/30946 [==============================] - 23s 753us/step - loss: 0.3085 - acc: 0.9494\n",
      "Epoch 16/30\n",
      "30946/30946 [==============================] - 23s 756us/step - loss: 0.3089 - acc: 0.9456\n",
      "Epoch 17/30\n",
      "30946/30946 [==============================] - 24s 776us/step - loss: 0.3087 - acc: 0.9434\n",
      "Epoch 18/30\n",
      "30946/30946 [==============================] - 25s 823us/step - loss: 0.3064 - acc: 0.9389\n",
      "Epoch 19/30\n",
      "30946/30946 [==============================] - 25s 793us/step - loss: 0.3068 - acc: 0.9345\n",
      "Epoch 20/30\n",
      "30946/30946 [==============================] - 23s 730us/step - loss: 0.3060 - acc: 0.9331\n",
      "Epoch 21/30\n",
      "30946/30946 [==============================] - 22s 723us/step - loss: 0.3056 - acc: 0.9302\n",
      "Epoch 22/30\n",
      "30946/30946 [==============================] - 22s 720us/step - loss: 0.3064 - acc: 0.9309\n",
      "Epoch 23/30\n",
      "30946/30946 [==============================] - 22s 717us/step - loss: 0.3073 - acc: 0.9271\n",
      "Epoch 24/30\n",
      "30946/30946 [==============================] - 23s 728us/step - loss: 0.3031 - acc: 0.9199\n",
      "Epoch 25/30\n",
      "30946/30946 [==============================] - 23s 736us/step - loss: 0.3052 - acc: 0.9214\n",
      "Epoch 26/30\n",
      "30946/30946 [==============================] - 23s 735us/step - loss: 0.3073 - acc: 0.9184\n",
      "Epoch 27/30\n",
      "30946/30946 [==============================] - 23s 734us/step - loss: 0.3039 - acc: 0.9135\n",
      "Epoch 28/30\n",
      "30946/30946 [==============================] - 23s 751us/step - loss: 0.3062 - acc: 0.9113\n",
      "Epoch 29/30\n",
      "30946/30946 [==============================] - 24s 781us/step - loss: 0.3033 - acc: 0.9131\n",
      "Epoch 30/30\n",
      "30946/30946 [==============================] - 25s 806us/step - loss: 0.3067 - acc: 0.9145\n",
      "15472/15472 [==============================] - 15s 948us/step\n",
      "30946/30946 [==============================] - 22s 712us/step\n",
      "[CV] .................... epochs=30, lr=0.002, nodes=16, total=13.2min\n",
      "[CV] epochs=30, lr=0.002, nodes=32 ...................................\n",
      "Epoch 1/30\n",
      "30945/30945 [==============================] - 47s 2ms/step - loss: 0.3546 - acc: 0.9650\n",
      "Epoch 2/30\n",
      "30945/30945 [==============================] - 36s 1ms/step - loss: 0.3343 - acc: 0.9787\n",
      "Epoch 3/30\n",
      "30945/30945 [==============================] - 36s 1ms/step - loss: 0.3268 - acc: 0.9739\n",
      "Epoch 4/30\n",
      "30945/30945 [==============================] - 37s 1ms/step - loss: 0.3207 - acc: 0.9700\n",
      "Epoch 5/30\n",
      "30945/30945 [==============================] - 33s 1ms/step - loss: 0.3158 - acc: 0.9682\n",
      "Epoch 6/30\n",
      "30945/30945 [==============================] - 33s 1ms/step - loss: 0.3129 - acc: 0.9642\n",
      "Epoch 7/30\n",
      "30945/30945 [==============================] - 35s 1ms/step - loss: 0.3070 - acc: 0.9605\n",
      "Epoch 8/30\n",
      "30945/30945 [==============================] - 35s 1ms/step - loss: 0.3069 - acc: 0.9552\n",
      "Epoch 9/30\n",
      "30945/30945 [==============================] - 35s 1ms/step - loss: 0.3050 - acc: 0.9527\n",
      "Epoch 10/30\n",
      "30945/30945 [==============================] - 35s 1ms/step - loss: 0.3034 - acc: 0.9472\n",
      "Epoch 11/30\n",
      "30945/30945 [==============================] - 36s 1ms/step - loss: 0.3019 - acc: 0.9452\n",
      "Epoch 12/30\n",
      "30945/30945 [==============================] - 36s 1ms/step - loss: 0.3004 - acc: 0.9399\n",
      "Epoch 13/30\n",
      "30945/30945 [==============================] - 36s 1ms/step - loss: 0.2995 - acc: 0.9370\n",
      "Epoch 14/30\n",
      "30945/30945 [==============================] - 37s 1ms/step - loss: 0.2995 - acc: 0.9320\n",
      "Epoch 15/30\n",
      "30945/30945 [==============================] - 35s 1ms/step - loss: 0.2977 - acc: 0.9234\n",
      "Epoch 16/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3000 - acc: 0.9205\n",
      "Epoch 17/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.2974 - acc: 0.9201\n",
      "Epoch 18/30\n",
      "30945/30945 [==============================] - 35s 1ms/step - loss: 0.2962 - acc: 0.9134\n",
      "Epoch 19/30\n",
      "30945/30945 [==============================] - 35s 1ms/step - loss: 0.2970 - acc: 0.9145\n",
      "Epoch 20/30\n",
      "30945/30945 [==============================] - 36s 1ms/step - loss: 0.2959 - acc: 0.9054\n",
      "Epoch 21/30\n",
      "30945/30945 [==============================] - 35s 1ms/step - loss: 0.2960 - acc: 0.9036\n",
      "Epoch 22/30\n",
      "30945/30945 [==============================] - 36s 1ms/step - loss: 0.2959 - acc: 0.8992\n",
      "Epoch 23/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.2965 - acc: 0.8946\n",
      "Epoch 24/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.2940 - acc: 0.8987\n",
      "Epoch 25/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.2964 - acc: 0.8950\n",
      "Epoch 26/30\n",
      "30945/30945 [==============================] - 35s 1ms/step - loss: 0.2954 - acc: 0.8943\n",
      "Epoch 27/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.2983 - acc: 0.8937\n",
      "Epoch 28/30\n",
      "30945/30945 [==============================] - 35s 1ms/step - loss: 0.2967 - acc: 0.8873\n",
      "Epoch 29/30\n",
      "30945/30945 [==============================] - 35s 1ms/step - loss: 0.2968 - acc: 0.8805\n",
      "Epoch 30/30\n",
      "30945/30945 [==============================] - 35s 1ms/step - loss: 0.2963 - acc: 0.8781\n",
      "15473/15473 [==============================] - 15s 988us/step\n",
      "30945/30945 [==============================] - 25s 802us/step\n",
      "[CV] .................... epochs=30, lr=0.002, nodes=32, total=18.5min\n",
      "[CV] epochs=30, lr=0.002, nodes=32 ...................................\n",
      "Epoch 1/30\n",
      "30945/30945 [==============================] - 46s 1ms/step - loss: 0.3639 - acc: 0.9800\n",
      "Epoch 2/30\n",
      "30945/30945 [==============================] - 37s 1ms/step - loss: 0.3466 - acc: 0.9783\n",
      "Epoch 3/30\n",
      "30945/30945 [==============================] - 35s 1ms/step - loss: 0.3391 - acc: 0.9736\n",
      "Epoch 4/30\n",
      "30945/30945 [==============================] - 35s 1ms/step - loss: 0.3343 - acc: 0.9703\n",
      "Epoch 5/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3293 - acc: 0.9663\n",
      "Epoch 6/30\n",
      "30945/30945 [==============================] - 35s 1ms/step - loss: 0.3249 - acc: 0.9633\n",
      "Epoch 7/30\n",
      "30945/30945 [==============================] - 38s 1ms/step - loss: 0.3230 - acc: 0.9597\n",
      "Epoch 8/30\n",
      "30945/30945 [==============================] - 38s 1ms/step - loss: 0.3204 - acc: 0.9547\n",
      "Epoch 9/30\n",
      "30945/30945 [==============================] - 36s 1ms/step - loss: 0.3177 - acc: 0.9534\n",
      "Epoch 10/30\n",
      "30945/30945 [==============================] - 38s 1ms/step - loss: 0.3169 - acc: 0.9524\n",
      "Epoch 11/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3144 - acc: 0.9459\n",
      "Epoch 12/30\n",
      "30945/30945 [==============================] - 33s 1ms/step - loss: 0.3121 - acc: 0.9387\n",
      "Epoch 13/30\n",
      "30945/30945 [==============================] - 33s 1ms/step - loss: 0.3092 - acc: 0.9409\n",
      "Epoch 14/30\n",
      "30945/30945 [==============================] - 33s 1ms/step - loss: 0.3103 - acc: 0.9352\n",
      "Epoch 15/30\n",
      "30945/30945 [==============================] - 33s 1ms/step - loss: 0.3087 - acc: 0.9338\n",
      "Epoch 16/30\n",
      "30945/30945 [==============================] - 33s 1ms/step - loss: 0.3084 - acc: 0.9295\n",
      "Epoch 17/30\n",
      "30945/30945 [==============================] - 35s 1ms/step - loss: 0.3073 - acc: 0.9227\n",
      "Epoch 18/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3078 - acc: 0.9173\n",
      "Epoch 19/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3067 - acc: 0.9206\n",
      "Epoch 20/30\n",
      "30945/30945 [==============================] - 33s 1ms/step - loss: 0.3056 - acc: 0.9124\n",
      "Epoch 21/30\n",
      "30945/30945 [==============================] - 33s 1ms/step - loss: 0.3065 - acc: 0.9114\n",
      "Epoch 22/30\n",
      "30945/30945 [==============================] - 36s 1ms/step - loss: 0.3061 - acc: 0.9024\n",
      "Epoch 23/30\n",
      "30945/30945 [==============================] - 36s 1ms/step - loss: 0.3012 - acc: 0.9007\n",
      "Epoch 24/30\n",
      "30945/30945 [==============================] - 35s 1ms/step - loss: 0.3038 - acc: 0.8993\n",
      "Epoch 25/30\n",
      "30945/30945 [==============================] - 32s 1ms/step - loss: 0.3030 - acc: 0.8960\n",
      "Epoch 26/30\n",
      "30945/30945 [==============================] - 31s 1ms/step - loss: 0.3036 - acc: 0.8969\n",
      "Epoch 27/30\n",
      "30945/30945 [==============================] - 35s 1ms/step - loss: 0.3025 - acc: 0.8979\n",
      "Epoch 28/30\n",
      "30945/30945 [==============================] - 36s 1ms/step - loss: 0.3017 - acc: 0.8970\n",
      "Epoch 29/30\n",
      "30945/30945 [==============================] - 36s 1ms/step - loss: 0.3038 - acc: 0.8949\n",
      "Epoch 30/30\n",
      "30945/30945 [==============================] - 36s 1ms/step - loss: 0.3027 - acc: 0.8926\n",
      "15473/15473 [==============================] - 15s 999us/step\n",
      "30945/30945 [==============================] - 27s 883us/step\n",
      "[CV] .................... epochs=30, lr=0.002, nodes=32, total=18.3min\n",
      "[CV] epochs=30, lr=0.002, nodes=32 ...................................\n",
      "Epoch 1/30\n",
      "30946/30946 [==============================] - 51s 2ms/step - loss: 0.3534 - acc: 0.9795\n",
      "Epoch 2/30\n",
      "30946/30946 [==============================] - 37s 1ms/step - loss: 0.3335 - acc: 0.9784\n",
      "Epoch 3/30\n",
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.3252 - acc: 0.9720\n",
      "Epoch 4/30\n",
      "30946/30946 [==============================] - 37s 1ms/step - loss: 0.3202 - acc: 0.9669\n",
      "Epoch 5/30\n",
      "30946/30946 [==============================] - 37s 1ms/step - loss: 0.3157 - acc: 0.9632\n",
      "Epoch 6/30\n",
      "30946/30946 [==============================] - 34s 1ms/step - loss: 0.3133 - acc: 0.9530\n",
      "Epoch 7/30\n",
      "30946/30946 [==============================] - 34s 1ms/step - loss: 0.3094 - acc: 0.9508\n",
      "Epoch 8/30\n",
      "30946/30946 [==============================] - 33s 1ms/step - loss: 0.3069 - acc: 0.9444\n",
      "Epoch 9/30\n",
      "30946/30946 [==============================] - 34s 1ms/step - loss: 0.3069 - acc: 0.9408\n",
      "Epoch 10/30\n",
      "30946/30946 [==============================] - 34s 1ms/step - loss: 0.3023 - acc: 0.9346\n",
      "Epoch 11/30\n",
      "30946/30946 [==============================] - 33s 1ms/step - loss: 0.3026 - acc: 0.9374\n",
      "Epoch 12/30\n",
      "30946/30946 [==============================] - 32s 1ms/step - loss: 0.3023 - acc: 0.9293\n",
      "Epoch 13/30\n",
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.2992 - acc: 0.9259\n",
      "Epoch 14/30\n",
      "30946/30946 [==============================] - 34s 1ms/step - loss: 0.3001 - acc: 0.9218\n",
      "Epoch 15/30\n",
      "30946/30946 [==============================] - 35s 1ms/step - loss: 0.3017 - acc: 0.9209\n",
      "Epoch 16/30\n",
      "30946/30946 [==============================] - 35s 1ms/step - loss: 0.2981 - acc: 0.9143\n",
      "Epoch 17/30\n",
      "30946/30946 [==============================] - 38s 1ms/step - loss: 0.2976 - acc: 0.9133\n",
      "Epoch 18/30\n",
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.2984 - acc: 0.9074\n",
      "Epoch 19/30\n",
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.2974 - acc: 0.9064\n",
      "Epoch 20/30\n",
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.2970 - acc: 0.9022\n",
      "Epoch 21/30\n",
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.2972 - acc: 0.9003\n",
      "Epoch 22/30\n",
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.2958 - acc: 0.8946\n",
      "Epoch 23/30\n",
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.2990 - acc: 0.8967\n",
      "Epoch 24/30\n",
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.2977 - acc: 0.8912\n",
      "Epoch 25/30\n",
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.2978 - acc: 0.8885\n",
      "Epoch 26/30\n",
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.2995 - acc: 0.8906\n",
      "Epoch 27/30\n",
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.2978 - acc: 0.8923\n",
      "Epoch 28/30\n",
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.2968 - acc: 0.8865\n",
      "Epoch 29/30\n",
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.2954 - acc: 0.8839\n",
      "Epoch 30/30\n",
      "30946/30946 [==============================] - 36s 1ms/step - loss: 0.2958 - acc: 0.8816\n",
      "15472/15472 [==============================] - 17s 1ms/step\n",
      "30946/30946 [==============================] - 25s 814us/step\n",
      "[CV] .................... epochs=30, lr=0.002, nodes=32, total=18.8min\n",
      "[CV] epochs=30, lr=0.002, nodes=64 ...................................\n",
      "Epoch 1/30\n",
      "30945/30945 [==============================] - 73s 2ms/step - loss: 0.3522 - acc: 0.9786\n",
      "Epoch 2/30\n",
      "30945/30945 [==============================] - 60s 2ms/step - loss: 0.3324 - acc: 0.9738\n",
      "Epoch 3/30\n",
      "30945/30945 [==============================] - 62s 2ms/step - loss: 0.3228 - acc: 0.9688\n",
      "Epoch 4/30\n",
      "30945/30945 [==============================] - 62s 2ms/step - loss: 0.3155 - acc: 0.9634\n",
      "Epoch 5/30\n",
      "30945/30945 [==============================] - 59s 2ms/step - loss: 0.3111 - acc: 0.9595\n",
      "Epoch 6/30\n",
      "30945/30945 [==============================] - 55s 2ms/step - loss: 0.3061 - acc: 0.9542\n",
      "Epoch 7/30\n",
      "30945/30945 [==============================] - 59s 2ms/step - loss: 0.3020 - acc: 0.9460\n",
      "Epoch 8/30\n",
      "30945/30945 [==============================] - 65s 2ms/step - loss: 0.3013 - acc: 0.9469\n",
      "Epoch 9/30\n",
      "30945/30945 [==============================] - 61s 2ms/step - loss: 0.2991 - acc: 0.9455\n",
      "Epoch 10/30\n",
      "30945/30945 [==============================] - 60s 2ms/step - loss: 0.2968 - acc: 0.9392\n",
      "Epoch 11/30\n",
      "30945/30945 [==============================] - 59s 2ms/step - loss: 0.2961 - acc: 0.9323\n",
      "Epoch 12/30\n",
      "30945/30945 [==============================] - 61s 2ms/step - loss: 0.2934 - acc: 0.9283\n",
      "Epoch 13/30\n",
      "30945/30945 [==============================] - 57s 2ms/step - loss: 0.2946 - acc: 0.9304\n",
      "Epoch 14/30\n",
      "30945/30945 [==============================] - 57s 2ms/step - loss: 0.2959 - acc: 0.9210\n",
      "Epoch 15/30\n",
      "30945/30945 [==============================] - 57s 2ms/step - loss: 0.2945 - acc: 0.9184\n",
      "Epoch 16/30\n",
      "30945/30945 [==============================] - 59s 2ms/step - loss: 0.2938 - acc: 0.9195\n",
      "Epoch 17/30\n",
      "30945/30945 [==============================] - 60s 2ms/step - loss: 0.2947 - acc: 0.9106\n",
      "Epoch 18/30\n",
      "30945/30945 [==============================] - 63s 2ms/step - loss: 0.2932 - acc: 0.9123\n",
      "Epoch 19/30\n",
      "30945/30945 [==============================] - 64s 2ms/step - loss: 0.2924 - acc: 0.9026\n",
      "Epoch 20/30\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "30945/30945 [==============================] - 64s 2ms/step - loss: 0.2955 - acc: 0.9027\n",
      "Epoch 21/30\n",
      "30945/30945 [==============================] - 65s 2ms/step - loss: 0.2927 - acc: 0.9015\n",
      "Epoch 22/30\n",
      "30945/30945 [==============================] - 64s 2ms/step - loss: 0.2943 - acc: 0.9038\n",
      "Epoch 23/30\n",
      "30945/30945 [==============================] - 63s 2ms/step - loss: 0.2930 - acc: 0.8963\n",
      "Epoch 24/30\n",
      "30945/30945 [==============================] - 63s 2ms/step - loss: 0.2946 - acc: 0.8939\n",
      "Epoch 25/30\n",
      "30945/30945 [==============================] - 64s 2ms/step - loss: 0.2956 - acc: 0.8918\n",
      "Epoch 26/30\n",
      "30945/30945 [==============================] - 64s 2ms/step - loss: 0.2933 - acc: 0.8879\n",
      "Epoch 27/30\n",
      "30945/30945 [==============================] - 64s 2ms/step - loss: 0.2946 - acc: 0.8891\n",
      "Epoch 28/30\n",
      "30945/30945 [==============================] - 59s 2ms/step - loss: 0.2941 - acc: 0.8809\n",
      "Epoch 29/30\n",
      "30945/30945 [==============================] - 55s 2ms/step - loss: 0.2964 - acc: 0.8816\n",
      "Epoch 30/30\n",
      "30945/30945 [==============================] - 56s 2ms/step - loss: 0.2965 - acc: 0.8790\n",
      "15473/15473 [==============================] - 16s 1ms/step\n",
      "30945/30945 [==============================] - 25s 798us/step\n",
      "[CV] .................... epochs=30, lr=0.002, nodes=64, total=31.4min\n",
      "[CV] epochs=30, lr=0.002, nodes=64 ...................................\n",
      "Epoch 1/30\n",
      "30945/30945 [==============================] - 66s 2ms/step - loss: 0.3608 - acc: 0.9783\n",
      "Epoch 2/30\n",
      "30945/30945 [==============================] - 55s 2ms/step - loss: 0.3404 - acc: 0.9753\n",
      "Epoch 3/30\n",
      "30945/30945 [==============================] - 55s 2ms/step - loss: 0.3316 - acc: 0.9690\n",
      "Epoch 4/30\n",
      "30945/30945 [==============================] - 55s 2ms/step - loss: 0.3247 - acc: 0.9651\n",
      "Epoch 5/30\n",
      "30945/30945 [==============================] - 56s 2ms/step - loss: 0.3196 - acc: 0.9587\n",
      "Epoch 6/30\n",
      "30945/30945 [==============================] - 54s 2ms/step - loss: 0.3160 - acc: 0.9548\n",
      "Epoch 7/30\n",
      "30945/30945 [==============================] - 55s 2ms/step - loss: 0.3116 - acc: 0.9479\n",
      "Epoch 8/30\n",
      "30945/30945 [==============================] - 56s 2ms/step - loss: 0.3099 - acc: 0.9447\n",
      "Epoch 9/30\n",
      "30945/30945 [==============================] - 55s 2ms/step - loss: 0.3073 - acc: 0.9337\n",
      "Epoch 10/30\n",
      "30945/30945 [==============================] - 56s 2ms/step - loss: 0.3059 - acc: 0.9286\n",
      "Epoch 11/30\n",
      "30945/30945 [==============================] - 56s 2ms/step - loss: 0.3023 - acc: 0.9199\n",
      "Epoch 12/30\n",
      "30945/30945 [==============================] - 56s 2ms/step - loss: 0.3040 - acc: 0.9200\n",
      "Epoch 13/30\n",
      "30945/30945 [==============================] - 56s 2ms/step - loss: 0.3043 - acc: 0.9201\n",
      "Epoch 14/30\n",
      "30945/30945 [==============================] - 55s 2ms/step - loss: 0.3026 - acc: 0.9193\n",
      "Epoch 15/30\n",
      "30945/30945 [==============================] - 55s 2ms/step - loss: 0.3014 - acc: 0.9116\n",
      "Epoch 16/30\n",
      "30945/30945 [==============================] - 55s 2ms/step - loss: 0.3038 - acc: 0.9034\n",
      "Epoch 17/30\n",
      "30945/30945 [==============================] - 55s 2ms/step - loss: 0.3022 - acc: 0.9002\n",
      "Epoch 18/30\n",
      "30945/30945 [==============================] - 56s 2ms/step - loss: 0.3032 - acc: 0.9011\n",
      "Epoch 19/30\n",
      "30945/30945 [==============================] - 56s 2ms/step - loss: 0.3014 - acc: 0.9009\n",
      "Epoch 20/30\n",
      "30945/30945 [==============================] - 55s 2ms/step - loss: 0.3030 - acc: 0.8902\n",
      "Epoch 21/30\n",
      "30945/30945 [==============================] - 55s 2ms/step - loss: 0.3022 - acc: 0.8955\n",
      "Epoch 22/30\n",
      "30945/30945 [==============================] - 56s 2ms/step - loss: 0.3016 - acc: 0.8893\n",
      "Epoch 23/30\n",
      "30945/30945 [==============================] - 56s 2ms/step - loss: 0.3042 - acc: 0.8835\n",
      "Epoch 24/30\n",
      "30945/30945 [==============================] - 56s 2ms/step - loss: 0.3032 - acc: 0.8814\n",
      "Epoch 25/30\n",
      "30945/30945 [==============================] - 56s 2ms/step - loss: 0.3009 - acc: 0.8759\n",
      "Epoch 26/30\n",
      "30945/30945 [==============================] - 56s 2ms/step - loss: 0.3025 - acc: 0.8836\n",
      "Epoch 27/30\n",
      "30945/30945 [==============================] - 56s 2ms/step - loss: 0.3007 - acc: 0.8792\n",
      "Epoch 28/30\n",
      "30945/30945 [==============================] - 55s 2ms/step - loss: 0.3042 - acc: 0.8780\n",
      "Epoch 29/30\n",
      "30945/30945 [==============================] - 56s 2ms/step - loss: 0.3036 - acc: 0.8805\n",
      "Epoch 30/30\n",
      "30945/30945 [==============================] - 55s 2ms/step - loss: 0.3029 - acc: 0.8777\n",
      "15473/15473 [==============================] - 16s 1ms/step\n",
      "30945/30945 [==============================] - 25s 809us/step\n",
      "[CV] .................... epochs=30, lr=0.002, nodes=64, total=28.7min\n",
      "[CV] epochs=30, lr=0.002, nodes=64 ...................................\n",
      "Epoch 1/30\n",
      "30946/30946 [==============================] - 66s 2ms/step - loss: 0.3521 - acc: 0.9815\n",
      "Epoch 2/30\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.3292 - acc: 0.9752\n",
      "Epoch 3/30\n",
      "30946/30946 [==============================] - 54s 2ms/step - loss: 0.3206 - acc: 0.9707\n",
      "Epoch 4/30\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.3153 - acc: 0.9642\n",
      "Epoch 5/30\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.3089 - acc: 0.9577\n",
      "Epoch 6/30\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.3056 - acc: 0.9534\n",
      "Epoch 7/30\n",
      "30946/30946 [==============================] - 54s 2ms/step - loss: 0.3027 - acc: 0.9461\n",
      "Epoch 8/30\n",
      "30946/30946 [==============================] - 56s 2ms/step - loss: 0.2992 - acc: 0.9445\n",
      "Epoch 9/30\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.2966 - acc: 0.9364\n",
      "Epoch 10/30\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.2964 - acc: 0.9316\n",
      "Epoch 11/30\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.2946 - acc: 0.9244\n",
      "Epoch 12/30\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.2947 - acc: 0.9217\n",
      "Epoch 13/30\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.2934 - acc: 0.9202\n",
      "Epoch 14/30\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.2952 - acc: 0.9170\n",
      "Epoch 15/30\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.2936 - acc: 0.9110\n",
      "Epoch 16/30\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.2959 - acc: 0.9032\n",
      "Epoch 17/30\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.2966 - acc: 0.9048\n",
      "Epoch 18/30\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.2938 - acc: 0.8992\n",
      "Epoch 19/30\n",
      "30946/30946 [==============================] - 56s 2ms/step - loss: 0.2930 - acc: 0.8952\n",
      "Epoch 20/30\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.2944 - acc: 0.8946\n",
      "Epoch 21/30\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.2964 - acc: 0.8944\n",
      "Epoch 22/30\n",
      "30946/30946 [==============================] - 56s 2ms/step - loss: 0.2959 - acc: 0.8921\n",
      "Epoch 23/30\n",
      "30946/30946 [==============================] - 56s 2ms/step - loss: 0.2962 - acc: 0.8894\n",
      "Epoch 24/30\n",
      "30946/30946 [==============================] - 56s 2ms/step - loss: 0.2992 - acc: 0.8806\n",
      "Epoch 25/30\n",
      "30946/30946 [==============================] - 54s 2ms/step - loss: 0.2951 - acc: 0.8854\n",
      "Epoch 26/30\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.2989 - acc: 0.8746\n",
      "Epoch 27/30\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.2943 - acc: 0.8769\n",
      "Epoch 28/30\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.2967 - acc: 0.8761\n",
      "Epoch 29/30\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.2980 - acc: 0.8686\n",
      "Epoch 30/30\n",
      "30946/30946 [==============================] - 56s 2ms/step - loss: 0.2970 - acc: 0.8715\n",
      "15472/15472 [==============================] - 16s 1ms/step\n",
      "30946/30946 [==============================] - 25s 801us/step\n",
      "[CV] .................... epochs=30, lr=0.002, nodes=64, total=28.6min\n",
      "[CV] epochs=30, lr=0.003, nodes=16 ...................................\n",
      "Epoch 1/30\n",
      "30945/30945 [==============================] - 37s 1ms/step - loss: 0.3552 - acc: 0.9788\n",
      "Epoch 2/30\n",
      "30945/30945 [==============================] - 24s 791us/step - loss: 0.3379 - acc: 0.9803\n",
      "Epoch 3/30\n",
      "30945/30945 [==============================] - 24s 764us/step - loss: 0.3338 - acc: 0.9733\n",
      "Epoch 4/30\n",
      "30945/30945 [==============================] - 23s 755us/step - loss: 0.3298 - acc: 0.9698\n",
      "Epoch 5/30\n",
      "30945/30945 [==============================] - 24s 768us/step - loss: 0.3234 - acc: 0.9653\n",
      "Epoch 6/30\n",
      "30945/30945 [==============================] - 24s 762us/step - loss: 0.3198 - acc: 0.9593\n",
      "Epoch 7/30\n",
      "30945/30945 [==============================] - 24s 761us/step - loss: 0.3183 - acc: 0.9598\n",
      "Epoch 8/30\n",
      "30945/30945 [==============================] - 23s 756us/step - loss: 0.3161 - acc: 0.9526\n",
      "Epoch 9/30\n",
      "30945/30945 [==============================] - 24s 764us/step - loss: 0.3151 - acc: 0.9533\n",
      "Epoch 10/30\n",
      "30945/30945 [==============================] - 23s 753us/step - loss: 0.3122 - acc: 0.9486\n",
      "Epoch 11/30\n",
      "30945/30945 [==============================] - 23s 756us/step - loss: 0.3115 - acc: 0.9469\n",
      "Epoch 12/30\n",
      "30945/30945 [==============================] - 23s 755us/step - loss: 0.3113 - acc: 0.9472\n",
      "Epoch 13/30\n",
      "30945/30945 [==============================] - 24s 765us/step - loss: 0.3092 - acc: 0.9427\n",
      "Epoch 14/30\n",
      "30945/30945 [==============================] - 24s 763us/step - loss: 0.3089 - acc: 0.9408\n",
      "Epoch 15/30\n",
      "30945/30945 [==============================] - 24s 765us/step - loss: 0.3093 - acc: 0.9362\n",
      "Epoch 16/30\n",
      "30945/30945 [==============================] - 24s 764us/step - loss: 0.3066 - acc: 0.9349\n",
      "Epoch 17/30\n",
      "30945/30945 [==============================] - 23s 751us/step - loss: 0.3079 - acc: 0.9322\n",
      "Epoch 18/30\n",
      "30945/30945 [==============================] - 24s 769us/step - loss: 0.3054 - acc: 0.9265\n",
      "Epoch 19/30\n",
      "30945/30945 [==============================] - 23s 759us/step - loss: 0.3080 - acc: 0.9271\n",
      "Epoch 20/30\n",
      "30945/30945 [==============================] - 23s 758us/step - loss: 0.3094 - acc: 0.9220\n",
      "Epoch 21/30\n",
      "30945/30945 [==============================] - 23s 759us/step - loss: 0.3074 - acc: 0.9211\n",
      "Epoch 22/30\n",
      "30945/30945 [==============================] - 23s 744us/step - loss: 0.3068 - acc: 0.9216\n",
      "Epoch 23/30\n",
      "30945/30945 [==============================] - 24s 761us/step - loss: 0.3068 - acc: 0.9140\n",
      "Epoch 24/30\n",
      "30945/30945 [==============================] - 23s 759us/step - loss: 0.3081 - acc: 0.9069\n",
      "Epoch 25/30\n",
      "30945/30945 [==============================] - 23s 756us/step - loss: 0.3054 - acc: 0.9058\n",
      "Epoch 26/30\n",
      "30945/30945 [==============================] - 24s 760us/step - loss: 0.3078 - acc: 0.9099\n",
      "Epoch 27/30\n",
      "30945/30945 [==============================] - 23s 744us/step - loss: 0.3080 - acc: 0.9012\n",
      "Epoch 28/30\n",
      "30945/30945 [==============================] - 23s 758us/step - loss: 0.3075 - acc: 0.9003\n",
      "Epoch 29/30\n",
      "30945/30945 [==============================] - 24s 765us/step - loss: 0.3075 - acc: 0.8988\n",
      "Epoch 30/30\n",
      "30945/30945 [==============================] - 24s 760us/step - loss: 0.3090 - acc: 0.9051\n",
      "15473/15473 [==============================] - 15s 948us/step\n",
      "30945/30945 [==============================] - 22s 704us/step\n",
      "[CV] .................... epochs=30, lr=0.003, nodes=16, total=12.8min\n",
      "[CV] epochs=30, lr=0.003, nodes=16 ...................................\n",
      "Epoch 1/30\n",
      "30945/30945 [==============================] - 36s 1ms/step - loss: 0.3647 - acc: 0.9788\n",
      "Epoch 2/30\n",
      "30945/30945 [==============================] - 25s 815us/step - loss: 0.3490 - acc: 0.9827\n",
      "Epoch 3/30\n",
      "30945/30945 [==============================] - 24s 767us/step - loss: 0.3428 - acc: 0.9781\n",
      "Epoch 4/30\n",
      "30945/30945 [==============================] - 24s 764us/step - loss: 0.3387 - acc: 0.9751\n",
      "Epoch 5/30\n",
      "30945/30945 [==============================] - 24s 772us/step - loss: 0.3358 - acc: 0.9687\n",
      "Epoch 6/30\n",
      "30945/30945 [==============================] - 24s 772us/step - loss: 0.3309 - acc: 0.9692\n",
      "Epoch 7/30\n",
      "30945/30945 [==============================] - 23s 758us/step - loss: 0.3300 - acc: 0.9603\n",
      "Epoch 8/30\n",
      "30945/30945 [==============================] - 23s 756us/step - loss: 0.3284 - acc: 0.9554\n",
      "Epoch 9/30\n",
      "30945/30945 [==============================] - 24s 761us/step - loss: 0.3267 - acc: 0.9543\n",
      "Epoch 10/30\n",
      "30945/30945 [==============================] - 24s 767us/step - loss: 0.3240 - acc: 0.9447\n",
      "Epoch 11/30\n",
      "30945/30945 [==============================] - 24s 766us/step - loss: 0.3238 - acc: 0.9429\n",
      "Epoch 12/30\n",
      "30945/30945 [==============================] - 24s 769us/step - loss: 0.3221 - acc: 0.9407\n",
      "Epoch 13/30\n",
      "30945/30945 [==============================] - 24s 768us/step - loss: 0.3227 - acc: 0.9406\n",
      "Epoch 14/30\n",
      "30945/30945 [==============================] - 24s 760us/step - loss: 0.3225 - acc: 0.9332\n",
      "Epoch 15/30\n",
      "30945/30945 [==============================] - 24s 761us/step - loss: 0.3229 - acc: 0.9350\n",
      "Epoch 16/30\n",
      "30945/30945 [==============================] - 24s 763us/step - loss: 0.3233 - acc: 0.9295\n",
      "Epoch 17/30\n",
      "30945/30945 [==============================] - 24s 763us/step - loss: 0.3220 - acc: 0.9291\n",
      "Epoch 18/30\n",
      "30945/30945 [==============================] - 24s 770us/step - loss: 0.3219 - acc: 0.9171\n",
      "Epoch 19/30\n",
      "30945/30945 [==============================] - 24s 761us/step - loss: 0.3217 - acc: 0.9240\n",
      "Epoch 20/30\n",
      "30945/30945 [==============================] - 24s 770us/step - loss: 0.3239 - acc: 0.9238\n",
      "Epoch 21/30\n",
      "30945/30945 [==============================] - 24s 764us/step - loss: 0.3242 - acc: 0.9209\n",
      "Epoch 22/30\n",
      "30945/30945 [==============================] - 24s 764us/step - loss: 0.3223 - acc: 0.9182\n",
      "Epoch 23/30\n",
      "30945/30945 [==============================] - 23s 753us/step - loss: 0.3202 - acc: 0.9169\n",
      "Epoch 24/30\n",
      "30945/30945 [==============================] - 23s 741us/step - loss: 0.3213 - acc: 0.9140\n",
      "Epoch 25/30\n",
      "30945/30945 [==============================] - 24s 760us/step - loss: 0.3218 - acc: 0.9163\n",
      "Epoch 26/30\n",
      "30945/30945 [==============================] - 23s 755us/step - loss: 0.3235 - acc: 0.9107\n",
      "Epoch 27/30\n",
      "30945/30945 [==============================] - 23s 759us/step - loss: 0.3246 - acc: 0.8952\n",
      "Epoch 28/30\n",
      "30945/30945 [==============================] - 23s 754us/step - loss: 0.3231 - acc: 0.9058\n",
      "Epoch 29/30\n",
      "30945/30945 [==============================] - 25s 796us/step - loss: 0.3230 - acc: 0.9040\n",
      "Epoch 30/30\n",
      "30945/30945 [==============================] - 24s 774us/step - loss: 0.3245 - acc: 0.9030\n",
      "15473/15473 [==============================] - 15s 975us/step\n",
      "30945/30945 [==============================] - 22s 697us/step\n",
      "[CV] .................... epochs=30, lr=0.003, nodes=16, total=12.9min\n",
      "[CV] epochs=30, lr=0.003, nodes=16 ...................................\n",
      "Epoch 1/30\n",
      "30946/30946 [==============================] - 37s 1ms/step - loss: 0.3561 - acc: 0.9680\n",
      "Epoch 2/30\n",
      "30946/30946 [==============================] - 24s 776us/step - loss: 0.3396 - acc: 0.9843\n",
      "Epoch 3/30\n",
      "30946/30946 [==============================] - 23s 750us/step - loss: 0.3345 - acc: 0.9782\n",
      "Epoch 4/30\n",
      "30946/30946 [==============================] - 23s 755us/step - loss: 0.3292 - acc: 0.9704\n",
      "Epoch 5/30\n",
      "30946/30946 [==============================] - 23s 748us/step - loss: 0.3241 - acc: 0.9654\n",
      "Epoch 6/30\n",
      "30946/30946 [==============================] - 23s 748us/step - loss: 0.3206 - acc: 0.9656\n",
      "Epoch 7/30\n",
      "30946/30946 [==============================] - 24s 765us/step - loss: 0.3193 - acc: 0.9626\n",
      "Epoch 8/30\n",
      "30946/30946 [==============================] - 23s 746us/step - loss: 0.3168 - acc: 0.9569\n",
      "Epoch 9/30\n",
      "30946/30946 [==============================] - 23s 748us/step - loss: 0.3149 - acc: 0.9568\n",
      "Epoch 10/30\n",
      "30946/30946 [==============================] - 24s 767us/step - loss: 0.3144 - acc: 0.9563\n",
      "Epoch 11/30\n",
      "30946/30946 [==============================] - 23s 750us/step - loss: 0.3108 - acc: 0.9521\n",
      "Epoch 12/30\n",
      "30946/30946 [==============================] - 24s 763us/step - loss: 0.3087 - acc: 0.9476\n",
      "Epoch 13/30\n",
      "30946/30946 [==============================] - 23s 746us/step - loss: 0.3081 - acc: 0.9455\n",
      "Epoch 14/30\n",
      "30946/30946 [==============================] - 23s 748us/step - loss: 0.3091 - acc: 0.9383\n",
      "Epoch 15/30\n",
      "30946/30946 [==============================] - 23s 748us/step - loss: 0.3061 - acc: 0.9335\n",
      "Epoch 16/30\n",
      "30946/30946 [==============================] - 23s 741us/step - loss: 0.3055 - acc: 0.9350\n",
      "Epoch 17/30\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "30946/30946 [==============================] - 23s 745us/step - loss: 0.3050 - acc: 0.9294\n",
      "Epoch 18/30\n",
      "30946/30946 [==============================] - 23s 749us/step - loss: 0.3043 - acc: 0.9247\n",
      "Epoch 19/30\n",
      "30946/30946 [==============================] - 23s 743us/step - loss: 0.3011 - acc: 0.9232\n",
      "Epoch 20/30\n",
      "30946/30946 [==============================] - 23s 755us/step - loss: 0.3019 - acc: 0.9242\n",
      "Epoch 21/30\n",
      "30946/30946 [==============================] - 24s 762us/step - loss: 0.3025 - acc: 0.9211\n",
      "Epoch 22/30\n",
      "30946/30946 [==============================] - 23s 751us/step - loss: 0.2993 - acc: 0.9179\n",
      "Epoch 23/30\n",
      "30946/30946 [==============================] - 23s 754us/step - loss: 0.3013 - acc: 0.9133\n",
      "Epoch 24/30\n",
      "30946/30946 [==============================] - 23s 754us/step - loss: 0.2995 - acc: 0.9096\n",
      "Epoch 25/30\n",
      "30946/30946 [==============================] - 23s 743us/step - loss: 0.2999 - acc: 0.9058\n",
      "Epoch 26/30\n",
      "30946/30946 [==============================] - 23s 742us/step - loss: 0.2993 - acc: 0.9064\n",
      "Epoch 27/30\n",
      "30946/30946 [==============================] - 23s 739us/step - loss: 0.3015 - acc: 0.9016\n",
      "Epoch 28/30\n",
      "30946/30946 [==============================] - 23s 757us/step - loss: 0.3023 - acc: 0.9017\n",
      "Epoch 29/30\n",
      "30946/30946 [==============================] - 24s 761us/step - loss: 0.3014 - acc: 0.8974\n",
      "Epoch 30/30\n",
      "30946/30946 [==============================] - 23s 751us/step - loss: 0.3022 - acc: 0.8960\n",
      "15472/15472 [==============================] - 15s 949us/step\n",
      "30946/30946 [==============================] - 22s 699us/step\n",
      "[CV] .................... epochs=30, lr=0.003, nodes=16, total=12.7min\n",
      "[CV] epochs=30, lr=0.003, nodes=32 ...................................\n",
      "Epoch 1/30\n",
      "30945/30945 [==============================] - 46s 1ms/step - loss: 0.3515 - acc: 0.9801\n",
      "Epoch 2/30\n",
      "30945/30945 [==============================] - 35s 1ms/step - loss: 0.3358 - acc: 0.9799\n",
      "Epoch 3/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3267 - acc: 0.9718\n",
      "Epoch 4/30\n",
      "30945/30945 [==============================] - 35s 1ms/step - loss: 0.3209 - acc: 0.9682\n",
      "Epoch 5/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3180 - acc: 0.9597\n",
      "Epoch 6/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3135 - acc: 0.9578\n",
      "Epoch 7/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3116 - acc: 0.9536\n",
      "Epoch 8/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3075 - acc: 0.9519\n",
      "Epoch 9/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3073 - acc: 0.9437\n",
      "Epoch 10/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3056 - acc: 0.9415\n",
      "Epoch 11/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3046 - acc: 0.9321\n",
      "Epoch 12/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3031 - acc: 0.9357\n",
      "Epoch 13/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3046 - acc: 0.9279\n",
      "Epoch 14/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3033 - acc: 0.9263\n",
      "Epoch 15/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3037 - acc: 0.9160\n",
      "Epoch 16/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3036 - acc: 0.9136\n",
      "Epoch 17/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3037 - acc: 0.9100\n",
      "Epoch 18/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3044 - acc: 0.9127\n",
      "Epoch 19/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3021 - acc: 0.8991\n",
      "Epoch 20/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3022 - acc: 0.9069\n",
      "Epoch 21/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3040 - acc: 0.9038\n",
      "Epoch 22/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3058 - acc: 0.8990\n",
      "Epoch 23/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3070 - acc: 0.8975\n",
      "Epoch 24/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3060 - acc: 0.8867\n",
      "Epoch 25/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3058 - acc: 0.8894\n",
      "Epoch 26/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3067 - acc: 0.8874\n",
      "Epoch 27/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3069 - acc: 0.8922\n",
      "Epoch 28/30\n",
      "30945/30945 [==============================] - 35s 1ms/step - loss: 0.3084 - acc: 0.8843\n",
      "Epoch 29/30\n",
      "30945/30945 [==============================] - 35s 1ms/step - loss: 0.3060 - acc: 0.8848\n",
      "Epoch 30/30\n",
      "30945/30945 [==============================] - 35s 1ms/step - loss: 0.3086 - acc: 0.8805\n",
      "15473/15473 [==============================] - 15s 996us/step\n",
      "30945/30945 [==============================] - 25s 802us/step\n",
      "[CV] .................... epochs=30, lr=0.003, nodes=32, total=18.1min\n",
      "[CV] epochs=30, lr=0.003, nodes=32 ...................................\n",
      "Epoch 1/30\n",
      "30945/30945 [==============================] - 46s 1ms/step - loss: 0.3613 - acc: 0.9762\n",
      "Epoch 2/30\n",
      "30945/30945 [==============================] - 35s 1ms/step - loss: 0.3452 - acc: 0.9791\n",
      "Epoch 3/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3386 - acc: 0.9727\n",
      "Epoch 4/30\n",
      "30945/30945 [==============================] - 35s 1ms/step - loss: 0.3325 - acc: 0.9677\n",
      "Epoch 5/30\n",
      "30945/30945 [==============================] - 35s 1ms/step - loss: 0.3301 - acc: 0.9635\n",
      "Epoch 6/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3242 - acc: 0.9602\n",
      "Epoch 7/30\n",
      "30945/30945 [==============================] - 35s 1ms/step - loss: 0.3245 - acc: 0.9545\n",
      "Epoch 8/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3206 - acc: 0.9487\n",
      "Epoch 9/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3183 - acc: 0.9468\n",
      "Epoch 10/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3163 - acc: 0.9364\n",
      "Epoch 11/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3164 - acc: 0.9329\n",
      "Epoch 12/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3161 - acc: 0.9295\n",
      "Epoch 13/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3175 - acc: 0.9286\n",
      "Epoch 14/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3170 - acc: 0.9277\n",
      "Epoch 15/30\n",
      "30945/30945 [==============================] - 35s 1ms/step - loss: 0.3164 - acc: 0.9172\n",
      "Epoch 16/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3153 - acc: 0.9080\n",
      "Epoch 17/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3168 - acc: 0.9050\n",
      "Epoch 18/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3152 - acc: 0.9027\n",
      "Epoch 19/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3134 - acc: 0.9048\n",
      "Epoch 20/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3157 - acc: 0.9081\n",
      "Epoch 21/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3146 - acc: 0.8976\n",
      "Epoch 22/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3150 - acc: 0.8923\n",
      "Epoch 23/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3131 - acc: 0.8900\n",
      "Epoch 24/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3174 - acc: 0.8890\n",
      "Epoch 25/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3153 - acc: 0.8855\n",
      "Epoch 26/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3163 - acc: 0.8917\n",
      "Epoch 27/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3170 - acc: 0.8752\n",
      "Epoch 28/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3176 - acc: 0.8842\n",
      "Epoch 29/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3157 - acc: 0.8806\n",
      "Epoch 30/30\n",
      "30945/30945 [==============================] - 34s 1ms/step - loss: 0.3153 - acc: 0.8745\n",
      "15473/15473 [==============================] - 15s 998us/step\n",
      "30945/30945 [==============================] - 25s 808us/step\n",
      "[CV] .................... epochs=30, lr=0.003, nodes=32, total=18.1min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] epochs=30, lr=0.003, nodes=32 ...................................\n",
      "Epoch 1/30\n",
      "30946/30946 [==============================] - 46s 1ms/step - loss: 0.3526 - acc: 0.9823\n",
      "Epoch 2/30\n",
      "30946/30946 [==============================] - 34s 1ms/step - loss: 0.3338 - acc: 0.9771\n",
      "Epoch 3/30\n",
      "30946/30946 [==============================] - 33s 1ms/step - loss: 0.3271 - acc: 0.9722\n",
      "Epoch 4/30\n",
      "30946/30946 [==============================] - 33s 1ms/step - loss: 0.3207 - acc: 0.9624\n",
      "Epoch 5/30\n",
      "30946/30946 [==============================] - 33s 1ms/step - loss: 0.3190 - acc: 0.9596\n",
      "Epoch 6/30\n",
      "30946/30946 [==============================] - 33s 1ms/step - loss: 0.3156 - acc: 0.9543\n",
      "Epoch 7/30\n",
      "30946/30946 [==============================] - 33s 1ms/step - loss: 0.3119 - acc: 0.9472\n",
      "Epoch 8/30\n",
      "30946/30946 [==============================] - 34s 1ms/step - loss: 0.3103 - acc: 0.9466\n",
      "Epoch 9/30\n",
      "30946/30946 [==============================] - 33s 1ms/step - loss: 0.3051 - acc: 0.9394\n",
      "Epoch 10/30\n",
      "30946/30946 [==============================] - 33s 1ms/step - loss: 0.3050 - acc: 0.9371\n",
      "Epoch 11/30\n",
      "30946/30946 [==============================] - 33s 1ms/step - loss: 0.3037 - acc: 0.9259\n",
      "Epoch 12/30\n",
      "30946/30946 [==============================] - 33s 1ms/step - loss: 0.3017 - acc: 0.9263\n",
      "Epoch 13/30\n",
      "30946/30946 [==============================] - 33s 1ms/step - loss: 0.3012 - acc: 0.9174\n",
      "Epoch 14/30\n",
      "30946/30946 [==============================] - 33s 1ms/step - loss: 0.3009 - acc: 0.9195\n",
      "Epoch 15/30\n",
      "30946/30946 [==============================] - 33s 1ms/step - loss: 0.2998 - acc: 0.9163\n",
      "Epoch 16/30\n",
      "30946/30946 [==============================] - 33s 1ms/step - loss: 0.3008 - acc: 0.9074\n",
      "Epoch 17/30\n",
      "30946/30946 [==============================] - 33s 1ms/step - loss: 0.2994 - acc: 0.9083\n",
      "Epoch 18/30\n",
      "30946/30946 [==============================] - 33s 1ms/step - loss: 0.2965 - acc: 0.8994\n",
      "Epoch 19/30\n",
      "30946/30946 [==============================] - 33s 1ms/step - loss: 0.2986 - acc: 0.9006\n",
      "Epoch 20/30\n",
      "30946/30946 [==============================] - 33s 1ms/step - loss: 0.2973 - acc: 0.8997\n",
      "Epoch 21/30\n",
      "30946/30946 [==============================] - 33s 1ms/step - loss: 0.2975 - acc: 0.8981\n",
      "Epoch 22/30\n",
      "30946/30946 [==============================] - 33s 1ms/step - loss: 0.2993 - acc: 0.8963\n",
      "Epoch 23/30\n",
      "30946/30946 [==============================] - 33s 1ms/step - loss: 0.2981 - acc: 0.8915\n",
      "Epoch 24/30\n",
      "30946/30946 [==============================] - 33s 1ms/step - loss: 0.2980 - acc: 0.8861\n",
      "Epoch 25/30\n",
      "30946/30946 [==============================] - 33s 1ms/step - loss: 0.2997 - acc: 0.8875\n",
      "Epoch 26/30\n",
      "30946/30946 [==============================] - 33s 1ms/step - loss: 0.2978 - acc: 0.8903\n",
      "Epoch 27/30\n",
      "30946/30946 [==============================] - 33s 1ms/step - loss: 0.2988 - acc: 0.8847\n",
      "Epoch 28/30\n",
      "30946/30946 [==============================] - 33s 1ms/step - loss: 0.2985 - acc: 0.8803\n",
      "Epoch 29/30\n",
      "30946/30946 [==============================] - 33s 1ms/step - loss: 0.2992 - acc: 0.8843\n",
      "Epoch 30/30\n",
      "30946/30946 [==============================] - 33s 1ms/step - loss: 0.2982 - acc: 0.8777\n",
      "15472/15472 [==============================] - 15s 1ms/step\n",
      "30946/30946 [==============================] - 23s 752us/step\n",
      "[CV] .................... epochs=30, lr=0.003, nodes=32, total=17.6min\n",
      "[CV] epochs=30, lr=0.003, nodes=64 ...................................\n",
      "Epoch 1/30\n",
      "30945/30945 [==============================] - 67s 2ms/step - loss: 0.3507 - acc: 0.9771\n",
      "Epoch 2/30\n",
      "30945/30945 [==============================] - 57s 2ms/step - loss: 0.3321 - acc: 0.9726\n",
      "Epoch 3/30\n",
      "30945/30945 [==============================] - 55s 2ms/step - loss: 0.3241 - acc: 0.9665\n",
      "Epoch 4/30\n",
      "30945/30945 [==============================] - 57s 2ms/step - loss: 0.3158 - acc: 0.9608\n",
      "Epoch 5/30\n",
      "30945/30945 [==============================] - 57s 2ms/step - loss: 0.3106 - acc: 0.9541\n",
      "Epoch 6/30\n",
      "30945/30945 [==============================] - 57s 2ms/step - loss: 0.3080 - acc: 0.9506\n",
      "Epoch 7/30\n",
      "30945/30945 [==============================] - 57s 2ms/step - loss: 0.3058 - acc: 0.9442\n",
      "Epoch 8/30\n",
      "30945/30945 [==============================] - 57s 2ms/step - loss: 0.3035 - acc: 0.9391\n",
      "Epoch 9/30\n",
      "30945/30945 [==============================] - 57s 2ms/step - loss: 0.3020 - acc: 0.9348\n",
      "Epoch 10/30\n",
      "30945/30945 [==============================] - 57s 2ms/step - loss: 0.3013 - acc: 0.9313\n",
      "Epoch 11/30\n",
      "30945/30945 [==============================] - 57s 2ms/step - loss: 0.2984 - acc: 0.9225\n",
      "Epoch 12/30\n",
      "30945/30945 [==============================] - 57s 2ms/step - loss: 0.3002 - acc: 0.9185\n",
      "Epoch 13/30\n",
      "30945/30945 [==============================] - 57s 2ms/step - loss: 0.3022 - acc: 0.9124\n",
      "Epoch 14/30\n",
      "30945/30945 [==============================] - 56s 2ms/step - loss: 0.3014 - acc: 0.9117\n",
      "Epoch 15/30\n",
      "30945/30945 [==============================] - 56s 2ms/step - loss: 0.3047 - acc: 0.9088\n",
      "Epoch 16/30\n",
      "30945/30945 [==============================] - 57s 2ms/step - loss: 0.3038 - acc: 0.9043\n",
      "Epoch 17/30\n",
      "30945/30945 [==============================] - 57s 2ms/step - loss: 0.3033 - acc: 0.9013\n",
      "Epoch 18/30\n",
      "30945/30945 [==============================] - 57s 2ms/step - loss: 0.3017 - acc: 0.8984\n",
      "Epoch 19/30\n",
      "30945/30945 [==============================] - 57s 2ms/step - loss: 0.3030 - acc: 0.9023\n",
      "Epoch 20/30\n",
      "30945/30945 [==============================] - 57s 2ms/step - loss: 0.3079 - acc: 0.8878\n",
      "Epoch 21/30\n",
      "30945/30945 [==============================] - 57s 2ms/step - loss: 0.3040 - acc: 0.8931\n",
      "Epoch 22/30\n",
      "30945/30945 [==============================] - 57s 2ms/step - loss: 0.3037 - acc: 0.8868\n",
      "Epoch 23/30\n",
      "30945/30945 [==============================] - 58s 2ms/step - loss: 0.3031 - acc: 0.8848\n",
      "Epoch 24/30\n",
      "30945/30945 [==============================] - 56s 2ms/step - loss: 0.3044 - acc: 0.8863\n",
      "Epoch 25/30\n",
      "30945/30945 [==============================] - 58s 2ms/step - loss: 0.3043 - acc: 0.8872\n",
      "Epoch 26/30\n",
      "30945/30945 [==============================] - 56s 2ms/step - loss: 0.3062 - acc: 0.8805\n",
      "Epoch 27/30\n",
      "30945/30945 [==============================] - 57s 2ms/step - loss: 0.3048 - acc: 0.8801\n",
      "Epoch 28/30\n",
      "30945/30945 [==============================] - 57s 2ms/step - loss: 0.3034 - acc: 0.8786\n",
      "Epoch 29/30\n",
      "30945/30945 [==============================] - 58s 2ms/step - loss: 0.3047 - acc: 0.8728\n",
      "Epoch 30/30\n",
      "30945/30945 [==============================] - 56s 2ms/step - loss: 0.3074 - acc: 0.8708\n",
      "15473/15473 [==============================] - 14s 883us/step\n",
      "30945/30945 [==============================] - 23s 740us/step\n",
      "[CV] .................... epochs=30, lr=0.003, nodes=64, total=29.4min\n",
      "[CV] epochs=30, lr=0.003, nodes=64 ...................................\n",
      "Epoch 1/30\n",
      "30945/30945 [==============================] - 65s 2ms/step - loss: 0.3611 - acc: 0.9753\n",
      "Epoch 2/30\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3423 - acc: 0.9746\n",
      "Epoch 3/30\n",
      "30945/30945 [==============================] - 54s 2ms/step - loss: 0.3332 - acc: 0.9673\n",
      "Epoch 4/30\n",
      "30945/30945 [==============================] - 54s 2ms/step - loss: 0.3269 - acc: 0.9636\n",
      "Epoch 5/30\n",
      "30945/30945 [==============================] - 54s 2ms/step - loss: 0.3212 - acc: 0.9570\n",
      "Epoch 6/30\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3186 - acc: 0.9488\n",
      "Epoch 7/30\n",
      "30945/30945 [==============================] - 54s 2ms/step - loss: 0.3161 - acc: 0.9459\n",
      "Epoch 8/30\n",
      "30945/30945 [==============================] - 54s 2ms/step - loss: 0.3139 - acc: 0.9413\n",
      "Epoch 9/30\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3139 - acc: 0.9366\n",
      "Epoch 10/30\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3122 - acc: 0.9334\n",
      "Epoch 11/30\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3117 - acc: 0.9246\n",
      "Epoch 12/30\n",
      "30945/30945 [==============================] - 54s 2ms/step - loss: 0.3113 - acc: 0.9239\n",
      "Epoch 13/30\n",
      "30945/30945 [==============================] - 54s 2ms/step - loss: 0.3101 - acc: 0.9192\n",
      "Epoch 14/30\n",
      "30945/30945 [==============================] - 54s 2ms/step - loss: 0.3109 - acc: 0.9167\n",
      "Epoch 15/30\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3136 - acc: 0.9156\n",
      "Epoch 16/30\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3120 - acc: 0.9071\n",
      "Epoch 17/30\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3112 - acc: 0.9056\n",
      "Epoch 18/30\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3111 - acc: 0.8931\n",
      "Epoch 19/30\n",
      "30945/30945 [==============================] - 54s 2ms/step - loss: 0.3101 - acc: 0.8974\n",
      "Epoch 20/30\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3130 - acc: 0.8895\n",
      "Epoch 21/30\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3092 - acc: 0.8938\n",
      "Epoch 22/30\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3096 - acc: 0.8904\n",
      "Epoch 23/30\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3136 - acc: 0.8872\n",
      "Epoch 24/30\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3114 - acc: 0.8820\n",
      "Epoch 25/30\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3110 - acc: 0.8763\n",
      "Epoch 26/30\n",
      "30945/30945 [==============================] - 52s 2ms/step - loss: 0.3130 - acc: 0.8857\n",
      "Epoch 27/30\n",
      "30945/30945 [==============================] - 54s 2ms/step - loss: 0.3100 - acc: 0.8712\n",
      "Epoch 28/30\n",
      "30945/30945 [==============================] - 54s 2ms/step - loss: 0.3110 - acc: 0.8778\n",
      "Epoch 29/30\n",
      "30945/30945 [==============================] - 53s 2ms/step - loss: 0.3092 - acc: 0.8739\n",
      "Epoch 30/30\n",
      "30945/30945 [==============================] - 52s 2ms/step - loss: 0.3103 - acc: 0.8714\n",
      "15473/15473 [==============================] - 16s 1ms/step\n",
      "30945/30945 [==============================] - 25s 810us/step\n",
      "[CV] .................... epochs=30, lr=0.003, nodes=64, total=27.7min\n",
      "[CV] epochs=30, lr=0.003, nodes=64 ...................................\n",
      "Epoch 1/30\n",
      "30946/30946 [==============================] - 66s 2ms/step - loss: 0.3500 - acc: 0.9782\n",
      "Epoch 2/30\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.3314 - acc: 0.9730\n",
      "Epoch 3/30\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.3215 - acc: 0.9672\n",
      "Epoch 4/30\n",
      "30946/30946 [==============================] - 54s 2ms/step - loss: 0.3165 - acc: 0.9574\n",
      "Epoch 5/30\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.3089 - acc: 0.9505\n",
      "Epoch 6/30\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.3075 - acc: 0.9494\n",
      "Epoch 7/30\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.3031 - acc: 0.9425\n",
      "Epoch 8/30\n",
      "30946/30946 [==============================] - 54s 2ms/step - loss: 0.3020 - acc: 0.9327\n",
      "Epoch 9/30\n",
      "30946/30946 [==============================] - 54s 2ms/step - loss: 0.3018 - acc: 0.9267\n",
      "Epoch 10/30\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.2997 - acc: 0.9191\n",
      "Epoch 11/30\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.2990 - acc: 0.9197\n",
      "Epoch 12/30\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.2984 - acc: 0.9203\n",
      "Epoch 13/30\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.2979 - acc: 0.9107\n",
      "Epoch 14/30\n",
      "30946/30946 [==============================] - 56s 2ms/step - loss: 0.2976 - acc: 0.9034\n",
      "Epoch 15/30\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.2955 - acc: 0.9067\n",
      "Epoch 16/30\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.2961 - acc: 0.9030\n",
      "Epoch 17/30\n",
      "30946/30946 [==============================] - 56s 2ms/step - loss: 0.2995 - acc: 0.8987\n",
      "Epoch 18/30\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.2973 - acc: 0.8911\n",
      "Epoch 19/30\n",
      "30946/30946 [==============================] - 56s 2ms/step - loss: 0.2986 - acc: 0.8862\n",
      "Epoch 20/30\n",
      "30946/30946 [==============================] - 56s 2ms/step - loss: 0.3000 - acc: 0.8894\n",
      "Epoch 21/30\n",
      "30946/30946 [==============================] - 56s 2ms/step - loss: 0.2995 - acc: 0.8803\n",
      "Epoch 22/30\n",
      "30946/30946 [==============================] - 54s 2ms/step - loss: 0.3002 - acc: 0.8782\n",
      "Epoch 23/30\n",
      "30946/30946 [==============================] - 56s 2ms/step - loss: 0.3014 - acc: 0.8740\n",
      "Epoch 24/30\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.3013 - acc: 0.8715\n",
      "Epoch 25/30\n",
      "30946/30946 [==============================] - 55s 2ms/step - loss: 0.3024 - acc: 0.8618\n",
      "Epoch 26/30\n",
      "30946/30946 [==============================] - 56s 2ms/step - loss: 0.3032 - acc: 0.8683\n",
      "Epoch 27/30\n",
      "30946/30946 [==============================] - 56s 2ms/step - loss: 0.3019 - acc: 0.8635\n",
      "Epoch 28/30\n",
      "30946/30946 [==============================] - 56s 2ms/step - loss: 0.3032 - acc: 0.8641\n",
      "Epoch 29/30\n",
      "30946/30946 [==============================] - 56s 2ms/step - loss: 0.3030 - acc: 0.8644\n",
      "Epoch 30/30\n",
      "30946/30946 [==============================] - 56s 2ms/step - loss: 0.3010 - acc: 0.8560\n",
      "15472/15472 [==============================] - 16s 1ms/step\n",
      "30946/30946 [==============================] - 26s 831us/step\n",
      "[CV] .................... epochs=30, lr=0.003, nodes=64, total=28.6min\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done  81 out of  81 | elapsed: 1693.6min finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "46418/46418 [==============================] - 47s 1ms/step - loss: 0.3641 - acc: 0.9618\n",
      "Epoch 2/10\n",
      "46418/46418 [==============================] - 45s 965us/step - loss: 0.3442 - acc: 0.9830\n",
      "Epoch 3/10\n",
      "46418/46418 [==============================] - 44s 946us/step - loss: 0.3383 - acc: 0.9804\n",
      "Epoch 4/10\n",
      "46418/46418 [==============================] - 44s 944us/step - loss: 0.3344 - acc: 0.9750\n",
      "Epoch 5/10\n",
      "46418/46418 [==============================] - 43s 936us/step - loss: 0.3320 - acc: 0.9718\n",
      "Epoch 6/10\n",
      "46418/46418 [==============================] - 43s 934us/step - loss: 0.3297 - acc: 0.9654\n",
      "Epoch 7/10\n",
      "46418/46418 [==============================] - 43s 933us/step - loss: 0.3277 - acc: 0.9643\n",
      "Epoch 8/10\n",
      "46418/46418 [==============================] - 43s 934us/step - loss: 0.3257 - acc: 0.9574\n",
      "Epoch 9/10\n",
      "46418/46418 [==============================] - 43s 937us/step - loss: 0.3251 - acc: 0.9565\n",
      "Epoch 10/10\n",
      "46418/46418 [==============================] - 43s 937us/step - loss: 0.3240 - acc: 0.9539\n"
     ]
    }
   ],
   "source": [
    "#start fitting process\n",
    "param_grid = dict(epochs=epochs,nodes=nodes, lr=lrs)\n",
    "grid = GridSearchCV(estimator=model, param_grid=param_grid, n_jobs=1,refit=True,verbose=2)\n",
    "grid_result = grid.fit(X_train, Y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GridSearchCV(cv=None, error_score='raise',\n",
      "       estimator=<keras.wrappers.scikit_learn.KerasClassifier object at 0x7ffc79278>,\n",
      "       fit_params=None, iid=True, n_jobs=1,\n",
      "       param_grid={'epochs': [10, 20, 30], 'nodes': [16, 32, 64], 'lr': [0.001, 0.002, 0.003]},\n",
      "       pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',\n",
      "       scoring=None, verbose=2)\n"
     ]
    }
   ],
   "source": [
    "print(grid_result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best estimator : <keras.wrappers.scikit_learn.KerasClassifier object at 0x887d8e8d0>\n",
      "Best score : 0.9802662760148995\n",
      "Best params : {'epochs': 10, 'lr': 0.001, 'nodes': 16}\n"
     ]
    }
   ],
   "source": [
    "print('Best estimator : {}'.format (grid.best_estimator_))\n",
    "print('Best score : {}'.format(grid.best_score_))\n",
    "print('Best params : {}'.format(grid.best_params_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'mean_fit_time': array([  282.14273198,   412.61235968,   590.69232233,   267.92312781,\n",
      "         392.4418989 ,   593.37367829,   246.37826745,   373.27251657,\n",
      "         585.33218129,   446.11738094,   675.25052961,  1095.17514602,\n",
      "         467.25050513,   668.95570461,  1108.70000831,   472.17317764,\n",
      "         698.6985542 ,  1210.74587321,   725.23662893,  1192.23849821,\n",
      "       13149.07480772,   788.9326237 ,  1096.581803  ,  1758.84920621,\n",
      "         752.81749868,  1062.1605703 ,  1698.34982109]), 'std_fit_time': array([2.23591969e+01, 2.15047723e+00, 2.64419706e+01, 2.76638036e+00,\n",
      "       1.48957421e+01, 1.52180747e+01, 1.64775877e+01, 1.12871429e+01,\n",
      "       4.83799418e+00, 3.62796152e+01, 1.81110625e+01, 3.96805023e+00,\n",
      "       4.08766490e+00, 3.99024957e+00, 1.62677176e+01, 7.07579227e+00,\n",
      "       3.89000555e+01, 6.99882339e+01, 1.55882980e+01, 4.87303314e+01,\n",
      "       1.58591078e+04, 2.11081407e+01, 1.12746574e+01, 7.81184425e+01,\n",
      "       6.30624252e+00, 1.40923421e+01, 4.29098750e+01]), 'mean_score_time': array([12.02174298, 14.38173946, 13.46587944, 12.32335536, 12.92721065,\n",
      "       13.93557262, 11.21999733, 13.33071971, 14.39609599, 11.50784167,\n",
      "       13.76164508, 13.82472722, 12.9402113 , 13.82380398, 14.44641264,\n",
      "       13.24558075, 14.79218674, 14.20285447, 13.89610442, 16.36801004,\n",
      "       16.2751472 , 14.77849229, 15.88712605, 15.95045932, 14.8662227 ,\n",
      "       15.4829824 , 15.09711059]), 'std_score_time': array([1.16481462, 0.11500077, 0.40565325, 0.03134912, 0.37499668,\n",
      "       0.29574422, 0.73839175, 0.05551965, 0.13849248, 0.64646563,\n",
      "       0.31374801, 0.02677314, 0.08158559, 0.04038155, 0.64145047,\n",
      "       0.16964276, 1.26399535, 0.18698179, 0.4390211 , 0.22425822,\n",
      "       1.47202456, 0.33031886, 0.68195178, 0.27370412, 0.20737284,\n",
      "       0.02631901, 1.01448767]), 'param_epochs': masked_array(data=[10, 10, 10, 10, 10, 10, 10, 10, 10, 20, 20, 20, 20, 20,\n",
      "                   20, 20, 20, 20, 30, 30, 30, 30, 30, 30, 30, 30, 30],\n",
      "             mask=[False, False, False, False, False, False, False, False,\n",
      "                   False, False, False, False, False, False, False, False,\n",
      "                   False, False, False, False, False, False, False, False,\n",
      "                   False, False, False],\n",
      "       fill_value='?',\n",
      "            dtype=object), 'param_lr': masked_array(data=[0.001, 0.001, 0.001, 0.002, 0.002, 0.002, 0.003, 0.003,\n",
      "                   0.003, 0.001, 0.001, 0.001, 0.002, 0.002, 0.002, 0.003,\n",
      "                   0.003, 0.003, 0.001, 0.001, 0.001, 0.002, 0.002, 0.002,\n",
      "                   0.003, 0.003, 0.003],\n",
      "             mask=[False, False, False, False, False, False, False, False,\n",
      "                   False, False, False, False, False, False, False, False,\n",
      "                   False, False, False, False, False, False, False, False,\n",
      "                   False, False, False],\n",
      "       fill_value='?',\n",
      "            dtype=object), 'param_nodes': masked_array(data=[16, 32, 64, 16, 32, 64, 16, 32, 64, 16, 32, 64, 16, 32,\n",
      "                   64, 16, 32, 64, 16, 32, 64, 16, 32, 64, 16, 32, 64],\n",
      "             mask=[False, False, False, False, False, False, False, False,\n",
      "                   False, False, False, False, False, False, False, False,\n",
      "                   False, False, False, False, False, False, False, False,\n",
      "                   False, False, False],\n",
      "       fill_value='?',\n",
      "            dtype=object), 'params': [{'epochs': 10, 'lr': 0.001, 'nodes': 16}, {'epochs': 10, 'lr': 0.001, 'nodes': 32}, {'epochs': 10, 'lr': 0.001, 'nodes': 64}, {'epochs': 10, 'lr': 0.002, 'nodes': 16}, {'epochs': 10, 'lr': 0.002, 'nodes': 32}, {'epochs': 10, 'lr': 0.002, 'nodes': 64}, {'epochs': 10, 'lr': 0.003, 'nodes': 16}, {'epochs': 10, 'lr': 0.003, 'nodes': 32}, {'epochs': 10, 'lr': 0.003, 'nodes': 64}, {'epochs': 20, 'lr': 0.001, 'nodes': 16}, {'epochs': 20, 'lr': 0.001, 'nodes': 32}, {'epochs': 20, 'lr': 0.001, 'nodes': 64}, {'epochs': 20, 'lr': 0.002, 'nodes': 16}, {'epochs': 20, 'lr': 0.002, 'nodes': 32}, {'epochs': 20, 'lr': 0.002, 'nodes': 64}, {'epochs': 20, 'lr': 0.003, 'nodes': 16}, {'epochs': 20, 'lr': 0.003, 'nodes': 32}, {'epochs': 20, 'lr': 0.003, 'nodes': 64}, {'epochs': 30, 'lr': 0.001, 'nodes': 16}, {'epochs': 30, 'lr': 0.001, 'nodes': 32}, {'epochs': 30, 'lr': 0.001, 'nodes': 64}, {'epochs': 30, 'lr': 0.002, 'nodes': 16}, {'epochs': 30, 'lr': 0.002, 'nodes': 32}, {'epochs': 30, 'lr': 0.002, 'nodes': 64}, {'epochs': 30, 'lr': 0.003, 'nodes': 16}, {'epochs': 30, 'lr': 0.003, 'nodes': 32}, {'epochs': 30, 'lr': 0.003, 'nodes': 64}], 'split0_test_score': array([0.98351968, 0.97725069, 0.95779745, 0.98009436, 0.96833193,\n",
      "       0.95088218, 0.97544109, 0.95139921, 0.97214503, 0.97382537,\n",
      "       0.95404899, 0.93013637, 0.96426032, 0.94584114, 0.90609449,\n",
      "       0.95165773, 0.91869709, 0.90305694, 0.96005946, 0.92302721,\n",
      "       0.89995476, 0.94920184, 0.91766303, 0.90758095, 0.94842629,\n",
      "       0.93239837, 0.92606476]), 'split1_test_score': array([0.9747948 , 0.97679829, 0.96361404, 0.9777031 , 0.97240354,\n",
      "       0.96561753, 0.97608738, 0.96955988, 0.96348478, 0.97033542,\n",
      "       0.95740968, 0.94674594, 0.97621664, 0.94357914, 0.90674077,\n",
      "       0.95960706, 0.94286822, 0.92451367, 0.96975376, 0.9380857 ,\n",
      "       0.92179926, 0.94222193, 0.92515996, 0.92257481, 0.94144639,\n",
      "       0.91307439, 0.92386738]), 'split2_test_score': array([0.98248449, 0.97440538, 0.95934592, 0.97330662, 0.96904085,\n",
      "       0.96464581, 0.9663909 , 0.95314116, 0.95643744, 0.9709152 ,\n",
      "       0.94525595, 0.92638314, 0.9473242 , 0.93181231, 0.92567218,\n",
      "       0.94234747, 0.92418563, 0.87609876, 0.96063857, 0.93265253,\n",
      "       0.90324457, 0.93368666, 0.89438987, 0.90059462, 0.94034385,\n",
      "       0.91171148, 0.88139866]), 'mean_test_score': array([0.98026628, 0.97615149, 0.96025249, 0.97703477, 0.96992546,\n",
      "       0.96038175, 0.97263992, 0.95803352, 0.96402258, 0.97169202,\n",
      "       0.95223836, 0.93442199, 0.96260072, 0.94041105, 0.91283554,\n",
      "       0.95120427, 0.92858374, 0.90122366, 0.96348399, 0.93125512,\n",
      "       0.90833297, 0.94170365, 0.91240467, 0.91025033, 0.94340558,\n",
      "       0.91906157, 0.91044422]), 'std_test_score': array([0.00389199, 0.00124839, 0.00245964, 0.00281108, 0.00177604,\n",
      "       0.00672902, 0.00442646, 0.00818146, 0.00642383, 0.00152699,\n",
      "       0.00512424, 0.00884817, 0.01185346, 0.00614976, 0.00908042,\n",
      "       0.00705345, 0.01034636, 0.01980766, 0.00443977, 0.00622657,\n",
      "       0.00961651, 0.0063446 , 0.01310052, 0.0091697 , 0.00357866,\n",
      "       0.00944709, 0.02055724]), 'rank_test_score': array([ 1,  3, 11,  2,  6, 10,  4, 12,  7,  5, 13, 18,  9, 17, 22, 14, 20,\n",
      "       27,  8, 19, 26, 16, 23, 25, 15, 21, 24], dtype=int32), 'split0_train_score': array([0.98545807, 0.97676523, 0.95201163, 0.97634513, 0.96351592,\n",
      "       0.94703506, 0.97259654, 0.94364195, 0.97056067, 0.97450315,\n",
      "       0.94926482, 0.92506059, 0.95983196, 0.94231701, 0.89794797,\n",
      "       0.95401519, 0.91278074, 0.89251899, 0.95789304, 0.92008402,\n",
      "       0.88993375, 0.94819842, 0.91694943, 0.90176119, 0.95097754,\n",
      "       0.93633867, 0.92502828]), 'split1_train_score': array([0.97014057, 0.97498788, 0.96125384, 0.97731459, 0.96742608,\n",
      "       0.9592826 , 0.9723057 , 0.96749071, 0.95708515, 0.96984973,\n",
      "       0.95369203, 0.94057198, 0.97847794, 0.93679108, 0.89193731,\n",
      "       0.95847471, 0.94322185, 0.91291   , 0.96933269, 0.9372435 ,\n",
      "       0.91714332, 0.94076587, 0.91888835, 0.91601228, 0.94028114,\n",
      "       0.90990467, 0.92308935]), 'split2_train_score': array([0.98203322, 0.9707232 , 0.95692497, 0.96859045, 0.96232146,\n",
      "       0.96170749, 0.96361404, 0.94936341, 0.95253021, 0.97088477,\n",
      "       0.94302979, 0.92257481, 0.94955729, 0.93181671, 0.91840626,\n",
      "       0.94025076, 0.91921411, 0.8577199 , 0.96177212, 0.93414335,\n",
      "       0.89724035, 0.93582369, 0.89071285, 0.89930847, 0.94845861,\n",
      "       0.91268662, 0.87665611]), 'mean_train_score': array([0.97921062, 0.97415877, 0.95673015, 0.97408339, 0.96442115,\n",
      "       0.95600838, 0.96950543, 0.95349869, 0.96005868, 0.97174588,\n",
      "       0.94866222, 0.92940246, 0.9626224 , 0.93697494, 0.90276385,\n",
      "       0.95091355, 0.92507223, 0.8877163 , 0.96299928, 0.93049029,\n",
      "       0.90143914, 0.94159599, 0.90885021, 0.90569398, 0.94657243,\n",
      "       0.91964332, 0.90825791]), 'std_train_score': array([0.00656413, 0.00253536, 0.00377563, 0.00390421, 0.00218004,\n",
      "       0.00642186, 0.00416753, 0.01016583, 0.00765531, 0.00199495,\n",
      "       0.00437365, 0.00796297, 0.01197055, 0.0042887 , 0.01132978,\n",
      "       0.00775643, 0.0130997 , 0.02278576, 0.00475015, 0.00746639,\n",
      "       0.01149819, 0.00508595, 0.01284945, 0.00736453, 0.00456592,\n",
      "       0.0118599 , 0.02235986])}\n"
     ]
    }
   ],
   "source": [
    "print(grid.cv_results_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#predictions\n",
    "predictions = grid.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(23209, 6)\n"
     ]
    }
   ],
   "source": [
    "#predictions\n",
    "predict = grid.predict_proba(X_test)\n",
    "print(predict.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Log_loss : 0.3504030505499869\n",
      "Hamming_loss : 15.158630990851249\n",
      "Accuracy : 21.612305571114653\n"
     ]
    }
   ],
   "source": [
    "#calculate score\n",
    "loss = log_loss(Y_test,predict)\n",
    "print(\"Log_loss : {}\".format(loss))\n",
    "predict = np.round(predict)\n",
    "loss = hamming_loss(Y_test,predict)\n",
    "print(\"Hamming_loss : {}\".format(loss*100))\n",
    "accuracy = accuracy_score(Y_test,predict)\n",
    "print(\"Accuracy : {}\".format(accuracy*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Free Form Visualisation \n",
    "Let us have a plot showing the **hamming-loss** and **log-loss** of different models, which we selected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import matplotlib.cm as cm\n",
    "import itertools"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAE5CAYAAACDPheMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3Xm4JVV97vHvyySRQaYWEWhwICAQ\naKXD8IDYihIgRIwxDHFAwbTmmiAOMeA1MphBoyFEMZBWCGgUJVdQEBC4IAIxIN3YzBgGGbpBaIQw\niF5oeO8fVRt2b/Y+p87ZZ5+q4ryf5znPqVpVu/aPw+r922vVqrVkm4iIiPGsVHcAERHRDkkYERFR\nSRJGRERUkoQRERGVJGFEREQlSRgREVFJEkZERFSShBEREZUkYURERCWr1B3AVNpggw28+eab1x1G\nRERrLFq06EHbs6qc+4JKGJtvvjkLFy6sO4yIiNaQdFfVc9MlFRERlSRhREREJUkYERFRSRJGRERU\nkoQRERGVJGFEREQlSRgREVFJEkZERFTygnpwb6bRMao7hGf5qKwNH/FClxZGRERUkoQRERGVJGFE\nREQlI0sYkjaV9ENJN0m6UdKHy/L1JF0k6dby97oDXn9wec6tkg4eVZwREVHNKFsYy4GP2d4a2Bn4\nkKStgSOAi21vAVxc7q9A0nrAUcBOwI7AUYMSS0RETI+RJQzb99m+ptx+DLgZ2BjYDzitPO004G19\nXv57wEW2H7L9MHARsNeoYo2IiPFNyz0MSZsDrwWuAja0fV956BfAhn1esjFwT9f+krIsIiJqMvKE\nIWlN4DvA4bYf7T5m28BQA/glzZe0UNLCZcuWDXOpiIgYw0gThqRVKZLFN2yfWRbfL2mj8vhGwAN9\nXroU2LRrf5Oy7HlsL7A91/bcWbMqrTIYERGTMMpRUgJOBm62fVzXobOBzqing4Hv9Xn5BcCektYt\nb3bvWZZFRERNRtnC2BV4N/AmSYvLn32AzwJvkXQr8OZyH0lzJX0VwPZDwGeAq8ufY8uyiIioycjm\nkrJ9BTBosqM9+py/EHh/1/4pwCmjiS4iIiYqT3pHREQlSRgREVFJEkZERFSShBEREZUkYURERCVJ\nGBERUUkSRkREVJKEERERlSRhREREJUkYERFRSRJGRERUkoQRERGVJGFEREQlSRgREVFJEkZERFSS\nhBEREZWMbAElSacA+wIP2N62LPs2sGV5yjrA/9ie0+e1dwKPAU8Dy23PHVWcERFRzcgSBnAqcALw\ntU6B7QM625L+EXhkjNe/0faDI4suIiImZJRLtF4mafN+xyQJ2B9406jePyIiplZd9zBeD9xv+9YB\nxw1cKGmRpPnTGFdERAwwyi6psRwEnD7G8d1sL5X0UuAiSbfYvqzfiWVCmQ8we/bsqY80IiKAGloY\nklYB3g58e9A5tpeWvx8AzgJ2HOPcBbbn2p47a9asqQ43IiJKdXRJvRm4xfaSfgclrSFprc42sCdw\nwzTGFxERfYwsYUg6HfgvYEtJSyQdWh46kJ7uKEkvl3ReubshcIWka4GfAOfa/sGo4oyIiGpGOUrq\noAHl7+1Tdi+wT7l9B7D9qOKKiIjJyZPeERFRSRJGRERUkoQRERGVJGFEREQlSRgREVFJEkZERFSS\nhBEREZUkYURERCVJGBERUUkSRkREVJKEERERlSRhREREJUkYERFRSRJGRERUkoQRERGVJGFEREQl\no1xx7xRJD0i6oavsaElLJS0uf/YZ8Nq9JP1M0m2SjhhVjBERUd0oWxinAnv1Kf8n23PKn/N6D0pa\nGfgysDewNXCQpK1HGGdERFQwyiVaL5O0+SReuiNwW7lUK5K+BewH3DR10UVETJ6kukNYge1peZ86\n7mH8uaTryi6rdfsc3xi4p2t/SVkWERE1mu6EcSLwKmAOcB/wj8NeUNJ8SQslLVy2bNmwl4uIiAGm\nNWHYvt/207afAb5C0f3Uaymwadf+JmXZoGsusD3X9txZs2ZNbcAREfGsaU0Ykjbq2v1D4IY+p10N\nbCHpFZJWAw4Ezp6O+CIiYrCR3fSWdDowD9hA0hLgKGCepDmAgTuBD5Tnvhz4qu19bC+X9OfABcDK\nwCm2bxxVnBERUc0oR0kd1Kf45AHn3gvs07V/HvC8IbcREVGfPOkdERGVVEoYknaVtEa5/S5Jx0na\nbLShRUREk1RtYZwIPCFpe+BjwO3A10YWVURENE7VhLHcxaOE+wEn2P4ysNbowoqIiKapetP7MUlH\nAu8Cdpe0ErDq6MKKiIimqZowDgD+BDjU9i8kzQY+P7qwImImufTS5szNNG/e9MzL1EaVWxjAP9t+\nWtJvA1sBp48urIiIaJqq9zAuA14kaWPgQuDdFNOXR0TEDFE1Ycj2E8DbgX+x/cfAtqMLKyIimqZy\nwpC0C/BO4NwJvjYiIl4Aqn7oHw4cCZxl+0ZJrwR+OLqwIiKiaSrd9Lb9I+BHktaUtGa5Gt5how0t\nIiKapOrUIL8j6afAjcBNkhZJ2ma0oUVERJNU7ZL6V+CjtjezPZtiepCvjC6siIhomqoJYw3bz96z\nsH0psMZIIoqIiEaq+uDeHZL+Gvh6uf8u4I7RhBQREU1UtYVxCDALOBP4DrAB8L6xXiDpFEkPSLqh\nq+zzkm6RdJ2ksyStM+C1d0q6XtJiSQsrxhgRESNUKWHYftj2YbZfZ3sH24cDnxrnZacCe/WUXQRs\na3s74L8phuoO8kbbc2zPrRJjRESM1jAP3+0/1kHblwEP9ZRdaHt5uXslsMkQ7x8REdNomIQx7PSS\nhwDnDzhm4MJy+O78Id8nIiKmwJg3vSWtN+gQQyQMSf8bWA58Y8Apu9leKumlwEWSbilbLP2uNR+Y\nDzB79uzJhhQREeMYb5TUIopv+/2Sw5OTeUNJ7wX2BfYoV/F7HttLy98PSDoL2JFixtx+5y4AFgDM\nnTs3E9lHRIzImAnD9ium8s0k7QV8AnhDOfttv3PWAFay/Vi5vSdw7FTGEREREzfhexiSjq543unA\nfwFbSloi6VDgBIq1wC8qh8yeVJ77cknnlS/dELhC0rXAT4Bzbf9gonFGRMTUqvrgXre3AkePd5Lt\ng/oUnzzg3HuBfcrtO4DtJxFXRESM0GRGSTVn8d2IiJg2k0kYO0x5FBER0XiVuqQkfbFnH+ARYKHt\n740groiIaJiqLYzVgTnAreXPdhRPaR8q6fgRxRYREQ1S9ab3dsCutp8GkHQicDmwG3D9iGKLiIgG\nqdrCWBdYs2t/DWC9MoH8vymPKiIiGqdqC+MfgMWSLqUYJbU78Hflg3X/d0SxRUREg1RKGLZPLh+s\n27Es+mT57ATAX44ksoiIaJSJDKtdCVgGPAy8WtLuowkpIiKaqOqw2s8BBwA3As+UxWbAhIAREfHC\nU/UextuALW3nBndExAxVtUvqDmDVUQYSERHNVrWF8QTFKKmL6RpGa/uwkUQVERGNUzVhnF3+RETE\nDFV1WO1pow4kIiKabbw1vc+wvb+k6ylGRa3A9nYjiywiIhplvBbGh8vf+07m4pJOKV/7gO1ty7L1\ngG8DmwN3AvvbfrjPaw8GPlXu/k1aORER9RpzlJTt+8rfd9m+i+Khvce6fsZzKrBXT9kRwMW2twAu\nLvdXUCaVo4CdKJ4uP0rSuhXeLyIiRqTSsFpJH5D0C+A6YFH5s3C819m+DHiop3g/oNNaOI3iGY9e\nvwdcZPuhsvVxEc9PPBERMY2qjpL6OLCt7Qen4D037LRcgF8AG/Y5Z2Pgnq79JWVZRETUpOqDe7dT\nPIsxpWybPjfTJ0LSfEkLJS1ctmzZFEUWERG9qrYwjgR+LOkqhn9w735JG9m+T9JGwAN9zlkKzOva\n3wS4tN/FbC8AFgDMnTt3qOQTERGDVW1h/CtwCXAlz93DWDTJ9zwbOLjcPhjotyb4BcCektYtb3bv\nWZZFRERNqrYwVrX90YleXNLpFC2FDSQtoRj59FngDEmHAncB+5fnzgU+aPv9th+S9Bng6vJSx9ru\nvXkeERHTqGrCOF/SfOAcVuySGvND3PZBAw7t0efchcD7u/ZPAU6pGF9ERIxY1YTR+eA/sqvMwCun\nNpyIiGiqqnNJvWLUgURERLNVXXFvZeD3KabzePY1to8bTVgREdE0VbukzgF+A1zPc0u0RkTEDFI1\nYWySmWkjIma2qs9hnC9pz5FGEhERjVa1hXElcJaklYCnAFHM7LH2yCKLiIhGqZowjgN2Aa4v53+K\niIgZpmqX1D3ADUkWEREzV9UWxh3ApZLOZ8UnvTOsNiJihqiaMH5e/qxW/kRExAxT9UnvY0YdSERE\nNFvVJ71nAZ8AtgFW75TbftOI4oqIiIapetP7G8AtwCuAY4A7eW7q8YiImAGqJoz1bZ8MPGX7R7YP\nAdK6iIiYQare9H6q/H2fpN8H7gXWG01IERHRRFVbGH8j6SXAx4CPA18FPjKZN5S0paTFXT+PSjq8\n55x5kh7pOufTk3mviIiYOlVHSX2/3HwEeOMwb2j7Z8AceHba9KXAWX1Ovdz2vsO8V0RETJ0xE4ak\nL1GsrNeX7cOGfP89gNtt3zXkdSIiYsTGa2Es7No+Bjhqit//QOD0Acd2kXQtxf2Sj9u+cYrfOyIi\nJmDMhGH7tM62pMO794claTXgray4TnjHNcBmth+XtA/wXWCLAdeZD8wHmD179lSFFxERPare9IYx\nuqYmaW/gGtv3P++N7EdtP15unwesKmmDvkHZC2zPtT131qxZUxxiRER0TCRhTLWDGNAdJellklRu\n70gR5y+nMbaIiOgx3k3vx3iuZfFiSY92DjHEAkqS1gDeAnygq+yDFBc9CXgH8GeSlgO/Bg7M1OoR\nEfUa7x7GWqN4U9u/AtbvKTupa/sE4IRRvHdERExOnV1SERHRIkkYERFRSRJGRERUkoQRERGVJGFE\nREQlSRgREVFJEkZERFSShBEREZUkYURERCVJGBERUUkSRkREVJKEERERlSRhREREJUkYERFRSRJG\nRERUkoQRERGV1JYwJN0p6XpJiyUt7HNckr4o6TZJ10l6XR1xRkREYcwV96bBG20/OODY3sAW5c9O\nwInl74iIqEGTu6T2A77mwpXAOpI2qjuoiIiZqs6EYeBCSYskze9zfGPgnq79JWVZRETUoM4uqd1s\nL5X0UuAiSbfYvmyiFymTzXyA2bNnT3WMERFRqq2FYXtp+fsB4Cxgx55TlgKbdu1vUpb1XmeB7bm2\n586aNWtU4UZEzHi1JAxJa0haq7MN7Anc0HPa2cB7ytFSOwOP2L5vmkONiIhSXV1SGwJnSerE8E3b\nP5D0QQDbJwHnAfsAtwFPAO+rKdaIiKCmhGH7DmD7PuUndW0b+NB0xhUREYM1eVhtREQ0SBJGRERU\nUveT3jGTFPesmsOuO4KIVkkLIyIiKknCiIiISpIwIiKikiSMiIioJAkjIiIqScKIiIhKMqw2Ygy6\n9NK6Q1iB582rO4SYwdLCiIiISpIwIiKiknRJlfIQckTE2JIwIl5gLtWldYewgnmeV3cIMUXSJRUR\nEZUkYURERCXTnjAkbSrph5JuknSjpA/3OWeepEckLS5/Pj3dcUZExIrquIexHPiY7WvKdb0XSbrI\n9k09511ue98a4ouIiD6mvYVh+z7b15TbjwE3AxtPdxwRETExtd7DkLQ58Frgqj6Hd5F0raTzJW0z\nrYFFRMTz1DasVtKawHeAw20/2nP4GmAz249L2gf4LrDFgOvMB+YDzJ49e4QRR0TMbLW0MCStSpEs\nvmH7zN7jth+1/Xi5fR6wqqQN+l3L9gLbc23PnTVr1kjjjoiYyeoYJSXgZOBm28cNOOdl5XlI2pEi\nzl9OX5QREdGrji6pXYF3A9dLWlyWfRKYDWD7JOAdwJ9JWg78GjjQzmQZERF1mvaEYfsKYMyZm2yf\nAJwwPRFFREQVedI7IiIqScKIiIhKkjAiIqKSJIyIiKgkCSMiIipJwoiIiEqSMCIiopIkjIiIqCQJ\nIyIiKknCiIiISpIwIiKikiSMiIioJAkjIiIqScKIiIhKkjAiIqKSJIyIiKikrjW995L0M0m3STqi\nz/EXSfp2efwqSZtPf5QREdGtjjW9Vwa+DOwNbA0cJGnrntMOBR62/Wrgn4DPTW+UERHRq44Wxo7A\nbbbvsP0k8C1gv55z9gNOK7f/D7CHpDGXdY2IiNGqI2FsDNzTtb+kLOt7ju3lwCPA+tMSXURE9LVK\n3QEMS9J8YH65+7ikn9UZD7AB8OCwF5nG9tTUxHv0tDYApyTm6fwjM1V/5ykIZAKm6O88fCAVTU28\n0/tXnqLPi6Fi3qzqiXUkjKXApl37m5Rl/c5ZImkV4CXAL/tdzPYCYMEI4pwUSQttz607jqraFi8k\n5unStpjbFi+0L+Y6uqSuBraQ9ApJqwEHAmf3nHM2cHC5/Q7gEtuexhgjIqLHtLcwbC+X9OfABcDK\nwCm2b5R0LLDQ9tnAycDXJd0GPESRVCIioka13MOwfR5wXk/Zp7u2fwP88XTHNUUa0z1WUdvihcQ8\nXdoWc9vihZbFrPT0REREFZkaJCIiKknCiIiISpIwIiKiktY/uBcTI2kz4Fe2H5S0M7AbcLvts2oO\nLWqWuhHjyU3vIUhaHTgAeBg4B/gE8HrgduAztqfgqdOpI+mvgfcCppjD683ApcBOwLW2D68tuAEk\nfRR4xPbJPeWHAmvZPr6eyPprW53oaFvdkHSF7d0kPUYR87OHANteu6bQxiXpt4G/pHjC+tkv7bbf\nVFtQFSVhDEHSGcBTwBrAusANFB8SuwFzbO9bY3jPI+kmYA7wYuBu4GW2nyifpl9se9taA+xD0iJg\nZ9tP9ZSvRvHcznb1RNZf2+pERxvrRltJuhY4CVgEPN0pt72otqAqSpfUcLa2vW35j2qJ7TeU5T8o\nK0XT/KacIfhJSbfbfgKefZjyyZpjG2SV3mQBYPvJhs5g3LY60dHGugE8u2TChqz4bf3u+iIa13Lb\nJ9YdxGQkYQznSXj2H9W9Pcee7nN+3daR9HaKZvva5Tbl/kvqC2tMK0na0Pb93YWSNqwroHG0rU50\ntLFuIOkvgKOA+4FnymIDjWp59jhH0v8CzgL+X6fQ9kP1hVRNuqSGIOkBiv5eUfRbf6tzCNjfdqM+\n1CT921jHbb9vumKpStJ7gMOAjwHXlMU7AJ8HTrB92qDX1qFtdaKjjXUDoJw+aCfbfScnbSJJP+9T\nbNuvnPZgJigJYwiSDh7reNM+zNpK0t7AEcC2FN8ebwQ+a/v8WgPrI3Viekn6IfCWct2cGLEkjBmk\n/LY+iG1/fdqCqUjS79q+uu44XujaWDcAJJ0MbAmcy4rdO8fVFtQAkt5k+5Ku7r4V2D5zumOaqNzD\nGELZjB+UcW370OmMp4LfHVD+VopVDpv4obBA0poUXTvftH1z3QGNpYV1oqONdQOKEV13A6uVP022\nO3AJ8Ad9jhlofMJIC2MIkv6oT/GmwEeAlW1vMs0hVVaOMHon8FfATcDf2r6u3qj6k7QlxRT3B1AM\nWT0d+JbtO+uMq58214mONtWNNpH0Ydv/LGk321fUHc9kJGFMEUmvBD5J8S3in4CTy2GKjVIO93wv\n8HHgSuDvbde9rG1lkranSB77A7+wvWvNIQ3UljrR0aa6Iel424dLOoc+LTrbb60hrDFJWmx7jqRr\nbL+u7ngmI11SQ5K0FfAp4LUUI3c+2NQbcJI+BHwYuBjYq4nf0MciaSXgpRRj7tcAHqg3ov7aVCc6\nWlg3Ol1kX6g1iom5WdKtwMsldbfYOk+nN3koMJAWxlAk/QfFEM9/BM6gZ5x908ZVS3qG4kN2Gf2n\nU2hkhZX0euAg4G3A9RT3M860/UitgfXRtjrR0da60U3S62xfM/6Z9ZH0MorVRp/XArJ91/RHNDFJ\nGEOQdCfP/ePq/O48fdy4cdXl5HIDNbHCSroHuIsiSZxhu5Gtio621YmONtaNXm3s6mlDkuuWhDED\nSVoD+LXtZ8qJ0LYCzu83BUfdJG3W+2ElaV3gf5zKO+Ukfc72X41X1kSSfmr7tXXHMRFtS3JZD2MK\nSLq4SlmDXAasLmlj4ELg3cCptUY02MHlPQEkvah8UOt24H5Jb643tMFaWCc63tKnbO9pj2JyjgGQ\n9PK6A5mAJs6HNlASxhAkrS5pfWADSetKWq/82Zxi7HpTqZxc7u3Av9j+Y2CbmmMa5ACgM1Kn8xT1\nLOANwN/VEtEY2lonJP2ZpOuBLSVd1/Xzc6AVQ2ptf7fcvLLWQCamVUkuo6SG8wHgcODlPDfPEcCj\nwAm1RFSNJO1CMda+8yDZyjXGM5Ynu7qefo/i+YunKUacNLH+trVOfBM4H/h7imlYOh5r6o36MbTm\nW3tPkptdZyxV5B7GFJD0F7a/VHccVUl6A8Vkfv9p+3Pl8wKH2z6s5tCeR9KVwPspZiP9GbCD7Z+X\nx26xvVWd8Q3Swjqx3ljH25Q0JN1tu/Efvt0k3WN707rjGE8SxhAGzQnT0Ya5YZpO0k7AaRTdUMfb\n/kxZvg/wbtsH1Rlfr7bWibLrqXdUV0fjRndJ+hL9p2ARcLAbvOJeP21JckkYQxhnSmjbPmTagpmA\n8sZxv6djG79EZNO1tU60TRtnBX4hJLkkjBlI0g5du6sDf0SxCtgnagppQiR93w1d6rTtJO3er9z2\nZdMdy2RJ+oLtj9cdR682JrleSRhTQNKn+5XbPna6Y5ksST+xvWPdcVTRhvH2ba0T5dxMHasDOwKL\n2tT6bEv3TremJrleTRxl0ka/6tpeHdgXaOw03D03OFeimMqisctw9vHTugOooFV1osP2ClNvS9oU\nOL6mcCarNaOkuuxPMeljo6WFMQKSXgRcYHte3bH003WDU8By4OfAsW2bclnSrrb/s+44qmh6nRik\nnOr8Rttb1x1LtzFGdQm4tg3TyHdryyiptDBG48VAYyus7VfUHUNVklam+Pa1MfAD2zdI2pdi2vDf\nopgRtg0aXSc6em7MrgTMYcXnSZpiEc996enVuCluYNwk14pWURLGFCifkO38I1uZYgho0/uqtwW2\npuguAcD21+qLaKCTKRYg+gnwRUn3AnOBI7oeemqcNtaJ0sKu7eXA6U1sxbXpS0+X1iW5XumSmgI9\nM30uB+5v8voHko4C5lEkjPMo5gq6wvY76oyrH0k3ANuVEyWuDvwCeJXtX9Yc2pjaVifaRtKYE/a1\naQbYNkkLYwhdTczHeg6tLanJT8e+A9ge+Knt90naEPj3mmMa5EnbzwDY/o2kO5qcLNpaJ3oW9Fnh\nEM1cD2MhcAPwYLnf/a3dQONGdb0QklwSxnAeBJZQfIOE51faRj0d26UztflySWtTLJzT1BtuW3V9\nmAl4Vbnf1A+yttaJZyji+yZwDvDresMZ10cpvvj8mmKtlLNsP15vSONqXZLrlYQxnC8CbwT+Ezid\nolunDX18CyWtA3yFol/1ceC/6g1poNfUHcAEtbJOuFhreiuKlQ2/CdxU/r6wiV1pto8Hji/nQTsQ\nuFjSXcDf2V5cb3QDtTHJrSD3MIZUDjucR/EPbUeK9SVO7EyQ13TltNtr227FFNYAkjYAftnUD+K2\n1wkASQcAXwY+Z/vzdcczFknbUCSNdwOfsH1GzSGNqSvJ7UexmmSTk9wKsh7GkFz4IfAJ4CTgfUAj\nF/aRtLKkNbv2d6aYUnkdSWvVF9lgknaWdKmkMyW9trwJfgPFAkp71R1fP22qE90kbSzpY5KuAN4F\nfAQ4seaw+pL0SkmflHQVxZoS1wKvaXqyALB9B/A9ii8SOwK/XW9E1aWFMQQVS53uR7HIzyzgTIp1\np++uNbABJH0BeMD2P5T7P6f48F0duKaJy3BKWkjxzMVLgAXA3ravLLtPTm/aFCFtqxMdkn4ErAWc\nAXwHWGFgQdNu1kt6hmJhp+9RrDWywgeZ7ePqiGssPS2Leyi6pc613fT7Rc9KwhiCpF8Bt1L8j7+V\n51faRk1lLemnwO92+qQ7czKVXSiX296t3gifT9Ji23PK7Zttv6brWOPmlGpbneiQdCfPxdo7zXkT\npzc/mv4zvwJg+5jpi6aaNia5XrnpPZz/oPifvmX5080U3y6bZKWeG5h/BcWnQXdXVcM807Xd+02s\nid922lYnALC9ed0xTITto+uOYRKO5bk629R/b2NKC2MGkXQzsKPtx3rKXwJc5QauXifpaYqJ/EQx\nFcgTnUPA6rZXrSu2FzpJR7fpg1nSNbbHfNYhhpOb3lNM0vfrjmEMXwG+LenZqZ/LJ5JPB75aW1Rj\nsL2y7bVtr2V7lXK7s9+KZNHwOjGWt9YdwAS1Yj6mbpIa/7Bet3RJTb2N6w5gENvHSXoCuKK8OQvF\nMxiftd3I0TAvEI2tE+No2wfwuXUHMAmt+hsnYUy9Rq/VYPsk4KTOMNre7qkYiUbXiW6SNrDdeRJ5\nhzFPrpmktwGvBq63fYHtT9Ud0yS0KsnlHsaItGWthix3OrUkzQJm2b6pp3xrYJntZfVENjZJfwCc\nQjGlydPA/rZ/XG9Ug0n6F2Ab4MfAHsA5tj9Tb1Tj601ydcczUbmHMYTyQbiDJH28nC4cSftK+jFw\nQs3hVdXW7pKm+hKwQZ/y9YF/nuZYJuJvgdfb3ohijfe/rzme8ewOvMn2kRRP1b+t3nDGVya5j1DU\nhc9I+uuaQ5qwdEkNp5VrNfRoTXdJS7za9mW9hbYvl9Tk+0TLbd8CYPuqpj753+VJ208D2H6ifJao\n6XYHtrf9tKQXA5cDjW8VdUvCGM5cWrhWQzfbh0B7utBaYKwP2iaP6nqppI8O2m/gQ2Vtm8UY2pnk\nVpCEMZy2rdXwQlnutMluk7SP7fO6CyXtDdxRU0xVfIUVk133fhNvdLZtFmNoZ5JbQW56D6Econpb\nZxd4VbnfyAog6VSe60LbCWhjF1qjSdqCYuTLjymmjofib7wLsK/t/64rtsmSdHg5nXijtWAW483G\nOm77rumKZbKSMIbQtgrQ1uVO20TSq4GXAVsA25bFNwL/Ddxn+/a6YpssSXfbnj3+mdOnnGn5s8BD\nFPcBvk4x2GAl4D22f1BjeJU1Pcn1SpfUEPolhIZXgFZ1obXU8cCRtv+tu1DS75TH/qCWqIbTxL72\nE3huFuNL6JnFGGhcwhgryUnUfmSNAAADi0lEQVRqRZJLwhhCCytA6/tQW2BD29f3Ftq+vlysqo2a\n+OVnFdsXAkg61vaVALZvafC95NYluV5JGMNpWwVo443CtllnjGO/NW1RTJCkx+ifGDqTPjZN22Yx\nhnYmuRUkYQynVRWghV1obbRQ0p/a/kp3oaT389xN8Max3fTnLnptL+lRyoRWblPur15fWGNqY5Jb\nQRLGcFpVAVrYhdZGhwNnSXonK46SWg34w9qieoGxvXLdMUxCG5PcCjJKaghtW6uhbcudtpmkN9I1\nSsr2JXXGEzEVkjBmkLYtdxoRzZLJB2eWVnWhRUSzpIUxg7StCy0imiUJIyIiKkmXVEREVJKEERER\nlSRhxAueJEv69679VSQtk/T9CV7nzvJBx6HOmcC1PlkxrvMkrVNuP17lNRGTkYQRM8GvgG0ldaa4\neAuwtMZ4qqqUMGzvY/t/Rh1MRBJGzBTnAb9fbh9EMdcXAJLWk/RdSddJulLSdmX5+pIulHSjpK/S\nNWurpHdJ+omkxZL+tVycaqCJXkvSZymeBl4s6Rvled+VtKi8xvyu1z+vtSJpI0mXla+/QdLrJ/uH\ni+hIwoiZ4lvAgeU6INsBV3UdOwb4aTlb7yeBr5XlRwFX2N4GOAuYDSDpNcABwK7lg5BPA+8c5/0n\ndC3bRwC/tj3Hdufah9jegWKqkcMkrT/G+/0JcEF5ze2BxePEFzGuzCUVM4Lt68rpxQ+iaG102w34\no/K8S8rWwNrA7sDby/JzJT1cnr8HsANwdTnJ5G8BD4wTwlRc6zBJnfmoNqVYpGnQeiZXA6dIWhX4\nru0kjBhaEkbMJGcDXwDmAWN9Ox+PgNNsHznwBOlDwJ+Wu/sMc63yevOANwO72H5C0qWMMWGd7csk\n7U7RDXeqpONsf23Q+RFVpEsqZpJTgGP6LHB0OWWXUvnB/KDtR4HLKLp2kLQ3sG55/sXAOyS9tDy2\nXu9yvba/XHYnzbF97ySv9VTZQoBiwsiHy2SxFbDzWP+h5TXuL6dZ/yrwuvH/PBFjSwsjZgzbS4Av\n9jl0NEX3zXUU06UcXJYfA5wu6Ubgx8Dd5XVukvQp4EJJKwFPAR8CxlrDfTLXWgBcJ+ka4BDgg5Ju\nBn4GXDnOf+484C8lPQU8DrxnnPMjxpWpQSIiopJ0SUVERCVJGBERUUkSRkREVJKEERERlSRhRERE\nJUkYERFRSRJGRERUkoQRERGV/H/LbTmVkDrdSwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1053d0da0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = ['BR-MultNB','BR-GausNB','BR-SVC','CC-MultNB','LP-MultNB','BP-MLL-ini','BP-MLL-fin']\n",
    "y = [3.27,20.74,4.26,3.56,3.17,13.96,15.158]\n",
    "colors = itertools.cycle(['b', 'g', 'r', 'c', 'm', 'y', 'k'])\n",
    "plt.ylabel('Hamming-Loss')\n",
    "plt.xlabel('Model-details')\n",
    "plt.xticks(rotation=90)\n",
    "for i in range(len(y)):\n",
    "    plt.bar(x[i], y[i], color=next(colors))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAE5CAYAAACDPheMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3Xu4HXV97/H3hwDiBQTMBhUIoFIB\nUUDSgEfU2CoEikCr5XIE8dacekRFWy14LAg8tlqtxQuKUSJaC6hVbNAo8IgUkYKEi1xFQ0BIagUJ\nclcIfM4fMxtWVvZldtaePTPZn9fzrGev+c2stb9sJuuz5jczv59sExERMZ71mi4gIiK6IYERERGV\nJDAiIqKSBEZERFSSwIiIiEoSGBERUUkCIyIiKqktMCRtI+lHkm6UdIOk94ywjSR9WtJSSddKemnP\nuqMk/bJ8HFVXnRERUY3qunFP0nOA59i+StLGwJXAwbZv7Nlmf+BdwP7AnsCnbO8paXNgCTAbcPna\nPWzfU0uxERExrvXremPbvwZ+XT6/X9JNwFbAjT2bHQR81UVqXSZp0zJo5gIX2F4JIOkCYB5w1li/\nc+bMmd5uu+0m+z8lImKddeWVV/7W9lCVbWsLjF6StgN2By7vW7UVcEfP8vKybbT2MW233XYsWbJk\nkFIjIqYVSb+qum3tJ70lPQP4FnCM7ftqeP/5kpZIWnLXXXdN9ttHRESp1sCQtAFFWPyb7W+PsMkK\nYJue5a3LttHa12B7ge3ZtmcPDVU6qoqIiLVQ51VSAk4HbrL9yVE2WwS8qbxaai/g3vLcx3nAPpI2\nk7QZsE/ZFhERDanzHMbLgSOB6yRdU7Z9EJgFYPs0YDHFFVJLgYeAt5TrVko6GbiifN1JwyfAIyKi\nGXVeJXUJoHG2MfDOUdYtBBbWUFpERKyF3OkdERGVJDAiIqKSBEZERFQyJTfudYHGPNsy9TLVekS0\nTY4wIiKikgRGRERUksCIiIhKEhgREVFJAiMiIipJYERERCUJjIiIqCSBERERlSQwIiKikgRGRERU\nksCIiIhKEhgREVFJAiMiIipJYERERCW1DW8uaSFwAHCn7V1GWP9+4I09dewEDJXzed8G3A88Bqyy\nPbuuOiMiopo6jzDOAOaNttL2x23vZns34DjgP22v7Nnk1eX6hEVERAvUFhi2LwZWjrth4XDgrLpq\niYiIwTV+DkPS0yiORL7V02zgfElXSprfTGUREdGrDVO0vg74SV931N62V0jaArhA0s/LI5Y1lIEy\nH2DWrFn1VxsRMU01foQBHEZfd5TtFeXPO4FzgDmjvdj2Atuzbc8eGhqqtdCIiOms0cCQ9EzgVcB/\n9LQ9XdLGw8+BfYDrm6kwIiKG1XlZ7VnAXGCmpOXACcAGALZPKzf7c+B82w/2vHRL4BxJw/WdafsH\nddUZERHV1BYYtg+vsM0ZFJff9rYtA3atp6qIiFhbbTiHERERHZDAiIiIShIYERFRSQIjIiIqSWBE\nREQlCYyIiKgkgREREZUkMCIiopIERkREVJLAiIiIShIYERFRSQIjIiIqSWBEREQlCYyIiKgkgRER\nEZUkMCIiopIERkREVJLAiIiISmoLDEkLJd0p6fpR1s+VdK+ka8rH8T3r5km6WdJSScfWVWNERFRX\n5xHGGcC8cbb5se3dysdJAJJmAKcC+wE7A4dL2rnGOiMiooLaAsP2xcDKtXjpHGCp7WW2HwHOBg6a\n1OIiImLCmj6H8TJJP5P0fUkvKtu2Au7o2WZ52RYREQ1av8HffRWwre0HJO0PfAfYYaJvImk+MB9g\n1qxZk1thTHu66KKmS1iN585tuoSYxho7wrB9n+0HyueLgQ0kzQRWANv0bLp12Tba+yywPdv27KGh\noVprjoiYzhoLDEnPlqTy+ZyylruBK4AdJG0vaUPgMGBRU3VGREShti4pSWcBc4GZkpYDJwAbANg+\nDXgD8A5Jq4CHgcNsG1gl6WjgPGAGsND2DXXVGbGuuUgXNV3CauZ6btMlxCSpLTBsHz7O+s8Cnx1l\n3WJgcR11RUTE2mn6KqmIiOiIBEZERFSSwIiIiEqavA8jBqQT1XQJT/AJbrqEiKhZjjAiIqKSBEZE\nRFSSwIiIiEoSGBERUUkCIyIiKklgREREJQmMiIioJIERERGVJDAiIqKSBEZERFSSwIiIiEoSGBER\nUUkCIyIiKklgREREJbUFhqSFku6UdP0o698o6VpJ10m6VNKuPetuK9uvkbSkrhojIqK6Oo8wzgDm\njbH+VuBVtl8MnAws6Fv/atu72Z5dU30RETEBtU2gZPtiSduNsf7SnsXLgK3rqiUiIgbXlnMYbwO+\n37Ns4HxJV0qa31BNERHRo/EpWiW9miIw9u5p3tv2CklbABdI+rnti0d5/XxgPsCsWbNqrzciYrpq\n9AhD0kuALwEH2b57uN32ivLnncA5wJzR3sP2Atuzbc8eGhqqu+SIiGmrscCQNAv4NnCk7V/0tD9d\n0sbDz4F9gBGvtIqIiKlTW5eUpLOAucBMScuBE4ANAGyfBhwPPAv4nCSAVeUVUVsC55Rt6wNn2v5B\nXXVGREQ1dV4ldfg4698OvH2E9mXArmu+IiIimtSWq6QiIqLlEhgREVFJpcCQ9E+SNpG0gaQfSrpL\n0hF1FxcREe1R9QhjH9v3AQcAtwEvAN5fV1EREdE+VQNj+OT4nwHftH1vTfVERERLVb1K6ruSfg48\nDLxD0hDw+/rKioiItql0hGH7WOB/AbNtPwo8CBxUZ2EREdEuVU96/yXwqO3HJH0I+Brw3Fori4iI\nVql6DuPvbd8vaW/gNcDpwOfrKysiItqmamA8Vv78M2CB7e8BG9ZTUkREtFHVwFgh6QvAocBiSU+Z\nwGsjImIdUPVD/xDgPGBf278DNif3YURETCtVr5J6CLgF2FfS0cAWts+vtbKIiGiVqldJvQf4N2CL\n8vE1Se+qs7CIiGiXqjfuvQ3Y0/aDAJI+BvwX8Jm6CouIiHapeg5DPHmlFOVzTX45ERHRVlWPML4M\nXC7pnHL5YGBhPSVFREQbVQoM25+UdBGwd9n0FttX11ZVRES0TuV7KWxfZfvT5eNqSbeP9xpJCyXd\nKen6UdZL0qclLZV0raSX9qw7StIvy8dRVeuMiIh6DHLzXZVzGGcA88ZYvx+wQ/mYTznciKTNgROA\nPYE5wAmSNhug1oiIGNAggeFxN7AvBlaOsclBwFdduAzYVNJzgH2BC2yvtH0PcAFjB09ERNRszHMY\nkt432irgGZPw+7cC7uhZXl62jdYeERENGe+k98ZjrPvUZBaytiTNp+jOYtasWQ1XExGx7hozMGyf\nWPPvXwFs07O8ddm2Apjb137RSG9gewGwAGD27NnjdpNFRMTamfA5DElXTeLvXwS8qbxaai/gXtu/\nphjocB9Jm5Unu/cp2yIioiFVb9zrVfkOb0lnURwpzJS0nOLKpw0AbJ8GLAb2B5YCDwFvKdetlHQy\ncEX5VifZHuvkeURE1GxtAuN7VTe0ffg46w28c5R1C8nd5BERrTHhLinbH6qjkIiIaLdKRxiS7mfN\n+y7uBZYAf2N72WQXFhER7VK1S+oUinshzqQ4h3EY8HzgKopuo7l1FBcREe1RtUvqQNtfsH2/7fvK\nS1n3tf11IEN2RERMA1UD4yFJh0har3wcAvy+XJd7HyIipoGqgfFG4EjgzvJxJHCEpKcCR9dUW0RE\ntEjV+TCWAa8bZfUlk1dORES0VaUjDElbSzqnnNviTknfkrR13cVFRER7VO2S+jLFMB7PLR/nlm0R\nETFNVA2MIdtftr2qfJwBDNVYV0REtEzVwLhb0hGSZpSPI4C76ywsIiLapWpgvBU4BPgf4NfAG4A3\n11RTRES0UKXAsP0r2wfaHrK9he2DgdfXXFtERLTIIHN6jzZ9a0RErIMGCYzK82JERET3DRIYGRIk\nImIaGfNO71GGNYfi6OKptVQUERGtNGZg2N54qgqJiIh2G6RLalyS5km6WdJSSceOsP5fJF1TPn4h\n6Xc96x7rWbeozjojImJ8azOndyWSZgCnAq+lmHzpCkmLbN84vI3t9/Zs/y5g9563eNj2bnXVFxER\nE1PnEcYcYKntZbYfAc4GDhpj+8OBs2qsJyIiBlBnYGwF3NGzvLxsW4OkbYHtgQt7mjeStETSZZIO\nrq/MiIioorYuqQk6DPh324/1tG1re4Wk5wEXSrrO9i39L5Q0H5gPMGvWrKmpNiJiGqrzCGMFsE3P\n8tZl20gOo687yvaK8ucy4CJWP7/Ru90C27Ntzx4aygC6ERF1qTMwrgB2kLS9pA0pQmGNq50k7Qhs\nBvxXT9tmkp5SPp8JvBy4sf+1ERExdWrrkrK9StLRwHnADGCh7RsknQQssT0cHocBZ9vuvUFwJ+AL\nkh6nCLWP9l5dFRERU6/Wcxi2FwOL+9qO71v+8AivuxR4cZ21RUTExNR6415ERKw7EhgREVFJAiMi\nIipJYERERCUJjIiIqCSBERERlSQwIiKikgRGRERUksCIiIhKEhgREVFJAiMiIipJYERERCUJjIiI\nqCSBERERlSQwIiKikgRGRERUksCIiIhKEhgREVFJrYEhaZ6kmyUtlXTsCOvfLOkuSdeUj7f3rDtK\n0i/Lx1F11hkREeOrbU5vSTOAU4HXAsuBKyQtsn1j36Zft31032s3B04AZgMGrixfe09d9UZExNjq\nPMKYAyy1vcz2I8DZwEEVX7svcIHtlWVIXADMq6nOiIiooM7A2Aq4o2d5ednW7/WSrpX075K2meBr\nIyJiijR90vtcYDvbL6E4ivjKRN9A0nxJSyQtueuuuya9wIiIKNQZGCuAbXqWty7bnmD7btt/KBe/\nBOxR9bU977HA9mzbs4eGhial8IiIWFOdgXEFsIOk7SVtCBwGLOrdQNJzehYPBG4qn58H7CNpM0mb\nAfuUbRER0ZDarpKyvUrS0RQf9DOAhbZvkHQSsMT2IuDdkg4EVgErgTeXr10p6WSK0AE4yfbKumqN\niIjx1RYYALYXA4v72o7veX4ccNwor10ILKyzvphiUtMVrM5uuoKITmn6pHdERHREAiMiIipJYERE\nRCUJjIiIqCSBERERlSQwIiKikgRGRERUksCIiIhKEhgREVFJAiMiIiqpdWiQiIgqLrqoPcPGzJ2b\nIWNGk8CIiJggtWxcNE/RuGjpkoqIiEoSGBERUUkCIyIiKklgREREJQmMiIioJIERERGV1BoYkuZJ\nulnSUknHjrD+fZJulHStpB9K2rZn3WOSrikfi+qsMyIixlfbfRiSZgCnAq8FlgNXSFpk+8aeza4G\nZtt+SNI7gH8CDi3XPWx7t7rqi4iIianzCGMOsNT2MtuPAGcDB/VuYPtHth8qFy8Dtq6xnoiIGECd\ngbEVcEfP8vKybTRvA77fs7yRpCWSLpN0cB0FRkREda0YGkTSEcBs4FU9zdvaXiHpecCFkq6zfcsI\nr50PzAeYNWvWlNQbETEd1XmEsQLYpmd567JtNZJeA/w/4EDbfxhut72i/LkMuAjYfaRfYnuB7dm2\nZw8NDU1e9RERsZo6A+MKYAdJ20vaEDgMWO1qJ0m7A1+gCIs7e9o3k/SU8vlM4OVA78nyiIiYYrV1\nSdleJelo4DxgBrDQ9g2STgKW2F4EfBx4BvDNcvTH220fCOwEfEHS4xSh9tG+q6siImKK1XoOw/Zi\nYHFf2/E9z18zyusuBV5cZ20RETExudM7IiIqSWBEREQlCYyIiKgkgREREZUkMCIiopIERkREVJLA\niIiIShIYERFRSQIjIiIqSWBEREQlCYyIiKgkgREREZUkMCIiopIERkREVJLAiIiIShIYERFRSQIj\nIiIqSWBEREQltQaGpHmSbpa0VNKxI6x/iqSvl+svl7Rdz7rjyvabJe1bZ50RETG+2gJD0gzgVGA/\nYGfgcEk79232NuAe2y8A/gX4WPnanYHDgBcB84DPle8XERENqfMIYw6w1PYy248AZwMH9W1zEPCV\n8vm/A38qSWX72bb/YPtWYGn5fhER0ZA6A2Mr4I6e5eVl24jb2F4F3As8q+JrIyJiCq3fdAGDkjQf\nmF8uPiDp5ibrAWYCvx30TaRJqKSayan3w1NXMJNU81T+kZmsv/MkFDIBk/R3HryQiian3qn9K0/S\n58VANW9bdcM6A2MFsE3P8tZl20jbLJe0PvBM4O6KrwXA9gJgwSTVPDBJS2zPbrqOqrpWL6TmqdK1\nmrtWL3Sv5jq7pK4AdpC0vaQNKU5iL+rbZhFwVPn8DcCFtl22H1ZeRbU9sAPw0xprjYiIcdR2hGF7\nlaSjgfOAGcBC2zdIOglYYnsRcDrwr5KWAispQoVyu28ANwKrgHfafqyuWiMiYny1nsOwvRhY3Nd2\nfM/z3wN/OcprPwJ8pM76atKa7rGKulYvpOap0rWau1YvdKxmFT1AERERY8vQIBERUUkCIyIiKklg\nREREJZ2/cS8mRtK2wIO2fytpL2Bv4Bbb5zRcWjQs+0aMJye9ByBpI+BQ4B7gXOADwCuAW4CTbU/C\nXaeTR9LfA28GTDG212uAi4A9gZ/ZPqax4kYh6X3AvbZP72t/G7Cx7VOaqWxkXdsnhnVt35B0ie29\nJd1PUfMTqwDb3qSh0sYl6Y+A91PcYf3El3bbf9JYURUlMAZQ3ivyKPB0YDPgeooPib2B3Wwf0GB5\na5B0I7Ab8DTgduDZth8q77K/xvYujRY4AklXAnvZfrSvfUOK+3le0kxlI+vaPjGsi/tGV0n6GXAa\ncCXwxP1ltq9srKiK0iU1mJ1t71L+o1pu+1Vl+w/KnaJtfl+OHPyIpFtsPwRP3GT5SMO1jWb9/rAA\nsP2IBhxApyZd2yeGdXHfAJ6YSmFLVv+2fntzFY1rle3PN13E2khgDOYReOIf1X/3rWvjnembSvoL\nisP2TcrnlMvPbK6sMa0naUvbv+ltlLRlUwWNo2v7xLAu7htIehdwAvAb4PGy2UCrjjz7nCvp/wLn\nAH8YbrS9srmSqkmX1AAk3UnR3yuKfuuzh1cBh9hu1YeapC+Ptd72W6aqlqokvQl4N/A3wFVl8x7A\nx4HP2v7KaK9tQtf2iWFd3DcAymGF9rR9d9O1VCXp1hGabft5U17MBCUwBiDpqLHWt+3DrKsk7Qcc\nC+xC8e3xBuCjtr/faGEjyD4xtST9CHhtOZ9O1CyBMY2U39ZHY9v/OmXFVCTpj21f0XQd67ou7hsA\nkk4HXgh8j9W7dz7ZWFGjkPQnti/s6e5bje1vT3VNE5VzGAMoD+NHS1zbfttU1lPBH4/SfiDFjIZt\n/FBYIOkZFF07Z9q+qemCxtLBfWJYF/cNKK7ouh3YsHy02SuBC4HXjbDOQOsDI0cYA5D0+hGatwHe\nC8ywvfUUl1RZeYXRG4G/oxhG/iO2r222qpFJeiHF0PeHUlyyehbFnO+3NVnXSLq8Twzr0r7RJZLe\nY/tTkva2fUnT9ayNBMYkkfQ84IMU3yL+BTi9vEyxVcrLPd8M/C1wGfCPtpue1rYySbtShMchwP/Y\nfnnDJY2qK/vEsC7tG5JOsX2MpHMZ4YjO9oENlDUmSdfY3k3SVbZf2nQ9ayNdUgOStCPwIWB3iit3\n/rqtJ+AkvRN4D/BDYF4bv6GPRdJ6wBYU19w/Hbiz2YpG1qV9YlgH943hLrJPNFrFxNwk6ZfAcyX1\nHrEN353e5kuBgRxhDETSNyku8fxn4Bv0XWfftuuqJT1O8SF7FyMPp9DKHVbSK4DDgYOB6yjOZ3zb\n9r2NFjaCru0Tw7q6b/SS9FLbV42/ZXMkPZtiFtI1joBs/2rqK5qYBMYAJN3Gk/+4hn8O333cuuuq\ny8HlRtXGHVbSHcCvKELiG7ZbeVQxrGv7xLAu7hv9utjV04WQ65XAmIYkPR142Pbj5UBoOwLfH2kI\njqZJ2rb/w0rSZsDvnJ130kn6mO2/G6+tjSRdbXv3puuYiK6FXObDmASSflilrUUuBjaStBVwPnAk\ncEajFY3uqPKcAJKeUt6odQvwG0mvaba00XVwnxj22hHa9pvyKtbOiQCSntt0IRPQxvHQRpXAGICk\njSQ9C5gpaTNJm5eP7SiuXW8rlYPL/QXwOdt/Cbyo4ZpGcygwfKXO8F3UQ8CrgH9opKIxdHWfkPQO\nSdcBL5R0bc/jVqATl9Ta/k759LJGC5mYToVcrpIazP8BjgGey5PjHAHcB3y2kYqqkaSXUVxrP3wj\n2YwG6xnLIz1dT/tS3H/xGMUVJ23cf7u6T5wJfB/4R4phWIbd39YT9WPozLf2vpCb1WQtVeQcxiSQ\n9C7bn2m6jqokvYpiML+f2P5Yeb/AMbbf3XBpa5B0GfB2itFIbwb2sH1rue7ntndssr7RdHCf2Hys\n9V0KDUm32279h28vSXfY3qbpOsaTwBjAaGPCDOvC2DBtJ2lP4CsU3VCn2D65bN8fONL24U3W16+r\n+0TZ9dR/Vdew1l3dJekzjDwEi4Cj3OIZ90bSlZBLYAxgnCGhbfutU1bMBJQnjke6O7b1U0S2XVf3\nia7p4qjA60LIJTCmIUl79CxuBLyeYhawDzRU0oRI+q5bOtVp10l65Ujtti+e6lrWlqRP2P7bpuvo\n18WQ65fAmASSjh+p3fZJU13L2pL0U9tzmq6jii5cb9/VfaIcm2nYRsAc4MouHX12pXunV1tDrl8b\nrzLpogd7nm8EHAC0dhjuvhOc61EMZdHaaThHcHXTBVTQqX1imO3Vht6WtA1wSkPlrK3OXCXV4xCK\nQR9bLUcYNZD0FOA823ObrmUkPSc4BawCbgVO6tqQy5JebvsnTddRRdv3idGUQ53fYHvnpmvpNcZV\nXQJ+1oVh5Ht15SqpHGHU42lAa3dY29s3XUNVkmZQfPvaCviB7eslHUAxbPhTKUaE7YJW7xPD+k7M\nrgfsxur3k7TFlTz5padf64a4gXFDrhNHRQmMSVDeITv8j2wGxSWgbe+r3gXYmaK7BADbX22uolGd\nTjEB0U+BT0v6b2A2cGzPTU+t08V9orSk5/kq4Kw2HsV16UtPj86FXL90SU2CvpE+VwG/afP8B5JO\nAOZSBMZiirGCLrH9hibrGomk64GXlAMlbgT8D/B823c3XNqYurZPdI2kMQfs69IIsF2SI4wB9Bxi\n3t+3ahNJbb479g3ArsDVtt8iaUvgaw3XNJpHbD8OYPv3kpa1OSy6uk/0Teiz2iraOR/GEuB64Lfl\ncu+3dgOtu6prXQi5BMZgfgssp/gGCWvutK26O7bH8NDmqyRtQjFxTltPuO3Y82Em4Pnlcls/yLq6\nTzxOUd+ZwLnAw82WM673UXzxeZhirpRzbD/QbEnj6lzI9UtgDObTwKuBnwBnUXTrdKGPb4mkTYEv\nUvSrPgD8V7MljWqnpguYoE7uEy7mmt6RYmbDM4Eby5/nt7ErzfYpwCnlOGiHAT+U9CvgH2xf02x1\no+piyK0m5zAGVF52OJfiH9ocivklPj88QF7blcNub2K7E0NYA0iaCdzd1g/iru8TAJIOBU4FPmb7\n403XMxZJL6IIjSOBD9j+RsMljakn5A6imE2yzSG3msyHMSAXfgR8ADgNeAvQyol9JM2Q9Iye5b0o\nhlTeVNLGzVU2Okl7SbpI0rcl7V6eBL+eYgKleU3XN5Iu7RO9JG0l6W8kXQIcAbwX+HzDZY1I0vMk\nfVDS5RRzSvwM2KntYQFgexnwHxRfJOYAf9RsRdXlCGMAKqY6PYhikp8h4NsU807f3mhho5D0CeBO\n2/9ULt9K8eG7EXBVG6fhlLSE4p6LZwILgP1sX1Z2n5zVtiFCurZPDJP0n8DGwDeAbwGrXVjQtpP1\nkh6nmNjpPyjmGlntg8z2J5uoayx9RxZ3UHRLfc92288XPSGBMQBJDwK/pPgf/0vW3GlbNZS1pKuB\nPx7ukx4ek6nsQvmx7b2brXBNkq6xvVv5/CbbO/Wsa92YUl3bJ4ZJuo0na+0f5ryNw5t/mJFHfgXA\n9olTV001XQy5fjnpPZhvUvxPf2H56GWKb5dtsl7fCcy/g+LToLerqmUe73ne/02sjd92urZPAGB7\nu6ZrmAjbH266hrVwEk/us2399zamHGFMI5JuAubYvr+v/ZnA5W7h7HWSHqMYyE8UQ4E8NLwK2Mj2\nBk3Vtq6T9OEufTBLusr2mPc6xGBy0nuSSfpu0zWM4YvA1yU9MfRzeUfyWcCXGqtqDLZn2N7E9sa2\n1y+fDy93Iixavk+M5cCmC5igTozH1EtS62/W65Uuqcm3VdMFjMb2JyU9BFxSnpyF4h6Mj9pu5dUw\n64jW7hPj6NoH8PeaLmAtdOpvnMCYfK2eq8H2acBpw5fR9ndPRS1avU/0kjTT9vCdyHuMuXHDJB0M\nvAC4zvZ5tj/UdE1roVMhl3MYNenKXA2Z7nRySRoChmzf2Ne+M3CX7buaqWxskl4HLKQY0uQx4BDb\nlzZb1egkfQ54EXAp8KfAubZPbraq8fWHXNP1TFTOYQygvBHucEl/Ww4XjqQDJF0KfLbh8qrqandJ\nW30GmDlC+7OAT01xLRPxEeAVtp9DMcf7PzZcz3heCfyJ7eMo7qo/uNlyxleG3Hsp9oWTJf19wyVN\nWLqkBtPJuRr6dKa7pCNeYPvi/kbbP5bU5vNEq2z/HMD25W2987/HI7YfA7D9UHkvUdu9EtjV9mOS\nngb8GGj9UVGvBMZgZtPBuRp62X4rdKcLrQPG+qBt81VdW0h632jLLbyprGujGEM3Q241CYzBdG2u\nhnVlutM2Wyppf9uLexsl7Qcsa6imKr7I6mHXu9zGE51dG8UYuhlyq8lJ7wGUl6guHV4Enl8ut3IH\nkHQGT3ah7Ql0sQut1STtQHHly6UUQ8dD8Td+GXCA7V80VdvaknRMOZx4q3VgFONtx1pv+1dTVcva\nSmAMoGs7QFenO+0SSS8Ang3sAOxSNt8A/AL4te1bmqptbUm63fas8becOuVIyx8FVlKcB/hXiosN\n1gPeZPsHDZZXWdtDrl+6pAYwUiC0fAfoVBdaR50CHGf7y72Nkl5crntdI1UNpo197Z/lyVGML6Rv\nFGOgdYExVshJ6kTIJTAG0MEdoPN9qB2wpe3r+httX1dOVtVFbfzys77t8wEknWT7MgDbP2/xueTO\nhVy/BMZgurYDdPFEYddsOsYjHbS5AAADTklEQVS6p05ZFRMk6X5GDobhQR/bpmujGEM3Q241CYzB\ndGoH6GAXWhctkfRXtr/Y2yjp7Tx5Erx1bLf9vot+u0q6jzLQyueUyxs1V9aYuhhyq0lgDKZTO0AH\nu9C66BjgHElvZPWrpDYE/ryxqtYxtmc0XcNa6GLIrSZXSQ2ga3M1dG260y6T9Gp6rpKyfWGT9URM\nhgTGNNK16U4jol0y+OD00qkutIholxxhTCNd60KLiHZJYERERCXpkoqIiEoSGBERUUkCI9Z5kizp\naz3L60u6S9J3J/g+t5U3Og60zQTe64MV61osadPy+QNVXhOxNhIYMR08COwiaXiIi9cCKxqsp6pK\ngWF7f9u/q7uYiARGTBeLgT8rnx9OMdYXAJI2l/QdSddKukzSS8r2Z0k6X9INkr5Ez6itko6Q9FNJ\n10j6Qjk51agm+l6SPkpxN/A1kv6t3O47kq4s32N+z+vXOFqR9BxJF5evv17SK9b2DxcxLIER08XZ\nwGHlPCAvAS7vWXcicHU5Wu8Hga+W7ScAl9h+EXAOMAtA0k7AocDLyxshHwPeOM7vn9B72T4WeNj2\nbraH3/uttvegGGrk3ZKeNcbv+9/AeeV77gpcM059EePKWFIxLdi+thxe/HCKo41eewOvL7e7sDwa\n2AR4JfAXZfv3JN1Tbv+nwB7AFeUgk08F7hynhMl4r3dLGh6PahuKSZpGm8/kCmChpA2A79hOYMTA\nEhgxnSwCPgHMBcb6dj4eAV+xfdyoG0jvBP6qXNx/kPcq328u8BrgZbYfknQRYwxYZ/tiSa+k6IY7\nQ9InbX91tO0jqkiXVEwnC4ETR5jg6MeUXUrlB/Nvbd8HXEzRtYOk/YDNyu1/CLxB0hblus37p+u1\nfWrZnbSb7f9ey/d6tDxCgGLAyHvKsNgR2Gus/9DyPX5TDrP+JeCl4/95IsaWI4yYNmwvBz49wqoP\nU3TfXEsxXMpRZfuJwFmSbgAuBW4v3+dGSR8Czpe0HvAo8E5grDnc1+a9FgDXSroKeCvw15JuAm4G\nLhvnP3cu8H5JjwIPAG8aZ/uIcWVokIiIqCRdUhERUUkCIyIiKklgREREJQmMiIioJIERERGVJDAi\nIqKSBEZERFSSwIiIiEr+P2MtWFYAG0YWAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x106d5f320>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = ['BR-MultNB','BR-GausNB','BR-SVC','CC-MultNB','LP-MultNB','BP-MLL-ini','BP-MLL-fin']\n",
    "y = [1.92,1.422,0.46,1.5,1.47,0.36,0.35]\n",
    "colors = itertools.cycle(['b', 'g', 'r', 'c', 'm', 'y', 'k'])\n",
    "plt.ylabel('Log-Loss')\n",
    "plt.xlabel('Model-details')\n",
    "plt.xticks(rotation=90)\n",
    "for i in range(len(y)):\n",
    "    plt.bar(x[i], y[i], color=next(colors))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- While showing among the best problem transformation method models, hamming-loss was considered (this is because for BP-MLL neural network we had to round the final results to get the hamming-loss because of the output being multivalued probabilities)\n",
    "- But while chosing among the best Adaptation Algorithm model, log loss was preferred."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}