SVM - PyTorch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Support Vector Machines\n",
    "\n",
    "In this notebook, we will build a Support Vector Machine (SVM) that will find the optimal hyperplane that maximizes the margin between two toy data classes using gradient descent. An SVM is supervised machine learning algorithm which can be used for both classification or regression problems. But it's usually used for classification. Given 2 or more labeled classes of data, it acts as a discriminative classifier, formally defined by an optimal hyperplane that seperates all the classes. New examples that are then mapped into that same space can then be categorized based on on which side of the gap they fall.\n",
    "\n",
    "Support vectors are the data points nearest to the hyperplane, the points of a data set that, if removed, would alter the position of the dividing hyperplane. Because of this, they can be considered the critical elements of a data set, they are what help us build our SVM.\n",
    "\n",
    "A hyperplane is a linear decision surface that splits the space into two parts; a hyperplane is a binary classifier. Geometry tells us that a hyperplane is a subspace of one dimension less than its ambient space. For instance, a hyperplane of an n-dimensional space is a flat subset with dimension n − 1. By its nature, it separates the space into two half spaces."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generating Toy Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.PathCollection at 0x1e6a163b390>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAD8CAYAAABXe05zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJztnX+MHOWZ57/P9JRNj8m5TXDucINjFkVGcQieYCWOfDotvhXOrQMZQRInR+4up7tDutPqAormNKyi2KBIWLKyoLs9aY/djS574VgDZkcGJ2dyMlFunTWszdhhfeDbEMDQIOHEHid4Grtn5r0/uqtdXf2+b71VXdVd1fP9SIhxT3X109XT3/ep59crSikQQggpDiODNoAQQkg8KNyEEFIwKNyEEFIwKNyEEFIwKNyEEFIwKNyEEFIwKNyEEFIwKNyEEFIwKNyEEFIwRrM46dVXX63WrVuXxakJIWQoOXbs2K+UUqtdjs1EuNetW4ejR49mcWpCCBlKRORN12MZKiGEkIJB4SaEkIJB4SaEkIJB4SaEkIJB4SaEkIJB4SaEkIKRSTkgGSzTMzXsOXgK78zWsaZSxuS29ZgYrw7aLEJISlC4h4zpmRruf/pl1BsLAIDabB33P/0yAFC8CRkSKNxDxp6Dp9qi7VNvLGDPwVNW4aaXTkhxoHAPGe/M1mM9DtBLJ6RoMDk5ZKyplGM9Dti9dEJI/lhSHvdSCAdMblvf4T0DQNkrYXLbeuNzknjpNpbCdSZkkCwZ4U4SDkhbgPohaP754rzOmkoZNY1I27x0Ewy7EJI9SyZUEjcc4AtQbbYOhcsCND1TS/T6Sc43PVPDlt2HcP3UAWzZfSjxa0cxuW09yl6p47EoL91EEcMu/brOhKTFkhHuuOGAtAWoXwuH7nn37j2O8QefMz53YryKh+68CdVKGQKgWinjoTtvSuQhpx12yZq0F2hC+oFTqERE3gDwWwALAOaVUpuyNCoL4oYD0hagNBcOm6DqngcA5+Ya1pDFxHhV+3jc8E6aYZd+YLrOu/afZGiH5JY4Me5blVK/ysySjImbtOtVgMKCVxnzcG6u4Xy+KKE3CaptYXER/uC5a7N1CADVetwlXp0kOTpITNdrtt7A9EyN4k1yyZIJlcQNB/QS99Xdfr//wTy8kjifz1bWZ7u9j1pYou4YgucGLou2T1S4KM2wSz+IKpMkJI+IUuGvpuYgkdcBnEPze/zflFKPao65B8A9ALB27dpb3nzTeRee3JK0CmTL7kNab71S9rBi+ajT+cLVGUBT6O+6pYrHX3gLC5rPrdo6Z/h54WMOT22NbXsQAfD67u3WY4rC9EwN9+49rv3dML1Pkn9E5JhrGNo1VLJFKfWOiHwEwI9F5FWl1E+DB7TE/FEA2LRpU/RqUABMcd8oTF7t+XoDx3fe5vzaQGdZ3603rsa+YzWtaANNz9t/3q79JzFb7wzNuNwxuMTwV5Y9bNl9aCjqtCfGq3jgmZOxwlhhWLdO+o2TcCul3mn9/z0R+SsAnwbwU/uzli5pJejCC8eW3YeMnjQAlEQ6npdEUEy2+3gjgguX5tuLwjDUaW//5DX4wZHTXY/femP0htusWyeDIFK4RWQFgBGl1G9bP98G4MHMLSswWSXoorzhsCee5I5BZ7ufoKxWypi7NN/lnQbj3kk9z1691l6e//yrZ2I9HsS1+odeOUkTF4/7HwL4K2l6c6MA/qdS6n9lalXBSdK96EKUN1w1ePRxRCPK9uunDmif53uaSTzPXr3WXp/fS+mny3PplZO0iRRupdQvAdzcB1uGCldvN46o2hKPJo8+iWjYbDctHiWRRHXnQPKa9ajn79p/0una9hLacnlur++PkDBLphwwj5jK+r41/bK2BTtYagdcjmnbSu7idGy6tH6byiRNCdM0vNYou2y12C4dkb2Ufro8t2jdpCT/LJkhU3lk1/6TWlF97MhpY9NL3Li1q2hMz9Qw+eQJNBZV+3Xv3XscR988i+9M3NQ+zhRK8Rt2wvTitSoA4w8+h/c/mO+wK3zHEBVC8jF5ub2EtlyeW7RuUpJ/KNwZYwqFTM/Uusr1fExNL74YxAmvuIrGrv0n2+IY5AdHTmPTR6/qOH9w8TB1WQLxvFZTCEhXphe+HlG160FMC1nUgmi75lHPdU1WM4FJXKFwZ4guvjz55Alj3bCNYNggTsw6SjR8sTAtIgCMsdiwLQqdFShRwhMUqpVlD1d4I87XJSjA/muYGmmCpDmq9uibZ/H8q2cihdbFK2cCk8SBwp0CJk9JF19uLKrYog1cFpy4iS6TaADAxgeeswq2T5xBWL5o27ozgW6hmq03umLFNsIC7F9vW8gkqZdruua2kFaYKK+cCUwSBwp3j9g8paTJJ29EOsIWQcFJkugKi4aund5GFhMUTUIlAkRNYTCFYKJq0JN6uab3ExXSigMTmCQOuRbuIsT8bJ7SyrLn5NGGufKKUYwt0880sSXytuw+5HSNTKNfdXgjEnuCYrAlvjLmQalmu7/LFEOTaPvjuUx/B/7fSr2xgJIIFpSKDNe4ermuyU8gudAygUnikCvhDgp1ZcyLrCYYtI22L7T/HpJwbq6BmW/rZ5rYEnHBawSYY6qu4lIpe9h1xwZtB2Bttg4R/fNm6432ghUMCwXtiyOGPg/v2Ig9B0/hvr3HsefgqY5Eb/CaLCjV9sptfyuuXu6tN67WtsTrSCq0RRuHSwZLboQ7/OVzqSboN7pb63Alhc+aSjmx91UyKKLOqwzjN55cnF80hgBsoln2Ssaa8K5kZIJRYv5nGKcSBAAqY54xrJE0Pmy6DuEF19T6nrSKRkdW3bZkOMmNcLvevvcj5qcL0QDAN5840SWWwUoKH/8LHJUsM6ETZJ1XaUIXngkKmUk0V4152Hn7BqNo695/Et4JTDF0uUZlrwSlYBRn099E1Hknt63H5FMn0FjofE/vfzDfsYmCLcZdbS3QaQht0mmUZOnhNI87Lps2bVJHjx6N9Zzrpw5oPdcwLhULvRA3ceeji6smPZd/juDiceHifKJ4uencgN67Cy5aZW8E9fnFRJ51lA3Bz9BW3eLbe9/e49q/D4HZcxY0wys2MTS9dtBG04zy4DFFyMeQfJPFPO7McYl59iPmFydxF0QXVzXN1PZrf8NxfKApNrXZeodQJfHaTfghhofuvKlrAQwvNHONxdRe10fQjBkHY+Uu2JJ3JmFXaN4lAea8yHnDghH0sl1q4VmDTfpJboRb9+XwSoIVy0a7KhLCpOnt9CKSSTocTZ2Hrk7uqjEPY8tGY9ltiv8mXbRsLCsJLgVCEQrA4y++hb0vvqXt1AzjNy2ZQjRjy0YwMV41Nt8sKGUVUZdqjqj4M2uwSb/JjXAnTc4k9XZMompK+rkSt8PRj2u6bBkWpuyVsPP2DQDcugaD1GbrXZvhppE/CMd8H3jmJC6FEs0LiwpxlgebwP/9exfar2u6fjYRtXnTrg4Ba7BJv8mNcAPJkjNR3o4p0WgSVVfRtlWTuNgVxuVL7nvXYSHZsvuQk81hwkOkTDvRu1IS6Yr59nK+OERVqdhmlAD6zlJXh4A12KTf5Eq4k2Dzdkxe7/LREaOo2jw3X6wrZQ+X5he6YsBxOhzDC0qUaPredTgMY0raueIPkQKa1RS94C96SZOySQneOZgqX2wiqnMYdNvEmRZe1mCTflP4edymL+SaStno9ZoqGN6ZrWvnKwNNsX54x0Y8smMjLs4vdon2qjGvo/7ZZpduDvf7H8zDK+nrtwXAXbd0TuQLPr9X/A0HXGLONla16p+ziJXb8GeLT4xX8d0v35x4tnaQOOGP4Jx0gX0+uo/L7HNCTBTe47Z5O/fFjPuuqZQjY+2mDXvHlo12fFFtdpmGT5lQAJ498W67GmWkxzh8mNl6w1hdoaNS9vDbi/NYCNl8fq6B8QefSxQeCc9niYNuUmCvyeq44Y84Yb40qlBYfri0KbzHbfN2TF+yVWOe1SubGK/i8NRWvL57Ow5PbXVK4IUft9mVJGkV3M0lTdH2iROPXbF8FMs0dweL0He8RlGS5KINZBNL7mVXnCji7Eqkw7RzEr32pUPhPW7A7O2YvF6/EiOOx+J7OCZ50YmHya4kczqyZNWYF6sFPW3be12I1n24bCyrdPFmbd5rFl5tr1UoLD8kQyHcJqK+fElvbcPE8cSmZ2q4cLG3JKALZW8E84uqq51bx/ZPXoOJ8SqOvnkWj7/wViYefZb87LWzeOn0+Y4NHYLYRC0qbJGFEPZahcLyQzLUwg2k8+WzJdtcdnrxSVJt4Sf84oYgRlwGW7d49sS7ePbEu6m01JswlU+mgUL3HJMwcTaDyNp77bUKheWHZOiF2xXb7bLpSy+AcW6K7nxJqi3OzTWw5YarOjxKFy5ccj82K8EObmJw642rO3aM6TdZbAaRlF7DMCw/JEtKuG0b99pul108nPD+iRcuzbfDFP75kpbI/ey1s7h781o8/+qZXMXGdZREsKiUUYxc51qniU3UBuW99nInyBGwJDfTAbNGF6bwZ0+bhh35099sz+1lCmAcSiL47pdvxsR4FeumDmT2OlE2RMW/BcDru7cbf3/3n/4NDr92NrENI60XWVRNe5aNCuqWYVhRoayoz5aQflHI6YBZY4tlRt0uJxkylDYLSuHevcexa/9JrOqxNd2nWilj7tK807l8MQNgXaRMdyH+ZMSXTp/vyeZ/vnltu0UfaI4DNvGGZgHR3XX5ize9V1IUnIVbREoAjgKoKaU+n51J2WATZ9cJcaYvcz+z+bP1BrwRw55hDqwa8/Dxaz6EI788195+bKTlwdq4wmuW/PvXYNf+k12x8ahRp2nEuB9rhVp88TZ9dlVNqMMUEtONuCUkz8RpwPkGgFeyMiRrbC3ovTZbuMRDbVI7gsvVIy700qzymw/mcfi1s+2Qh1LRog00k6R+k8fEeBXHd96GR3ZsNLZ56+5CXK32SmJcnBSa4u03m8T57HptfCEkLzgJt4hcC2A7gD/L1pzssH3Bk8yaiDq3NyJYNea1z3f35rXaGShAs+NQKRh/H4ePfWRF+33o9q4Mt6nHISxywQ5Tv2pm3dQB3HD/D2MlUStlr+Pa7/nizdjzpZuNxyt0zidx+eymZ2rWjZ0JKRKuoZJHAPwnAB8yHSAi9wC4BwDWrl3bu2Up49KMk3WWf9NHrzLOze61JK8kgq9+5jps+uhVbVHLopFGJ3Jx9sMMU/ZKXTvJ+9h2yAnPJ4nqevWrhHQkrSDhvBAyKCKFW0Q+D+A9pdQxEfld03FKqUcBPAo0q0pSs9ARly9Rlpuxupx7YryaeAPhKF576Pf7Ut2iE7m4ydlgfbdN7Gx7TcYRW5t9SeufuV0ZGSQuoZItAO4QkTcA/CWArSLyg0ytikmRhu6YQjamGHe1Um7Hkk2INKcW3rv3eKSAlnpIbJpELm6owRft8ACvMBPjVdy9eW1XfsBkh2lUqs2+pGV/acXLOd6VJCFSuJVS9yulrlVKrQPwFQCHlFJfy9yyGBQp6WSKye68fYM1Bn94aiu+tlkfgvI3GHbhQ8tHseWGq2DT77JXwtc2r+1YTCplzyhySUINJjGdnqlh/MHnsG7qANZNHcCzJ97F3ZvXOsWwTYu3yb5qYIxvWvbHWcSK5HCQfDEUddxFG7pjC6vYwj1+CVx4EFScfONsvYEX3zjX7nD0Cben7ztW61gML86bm1wmt62PveelTkynZ2qYfOpEx2Cs2XoDe198C3u+dLNVZG2LdxYt4ml0XHLKH0lKrHncSqmf5LGG21bqVyRsc8B9vjNxk3aXlzg0FlRXSWEwfPH8q2di3cFMjFdRKbuXM9rK9XTTDBuLKvLuyXTHUZut91w1pCONed1FczhIfhgKjztvQ3eyrDaYnqkZ91XsFV8wkgjKrjs2OCVGSyJG0bSdP0rMTO34fklk2olp/1wPPHOy3Xm6fDTeviSc8keSMhTCnaehO3GrDYIbAPjiE662CB6TJb5gJBGU4GdgstMrCfZ80RzysG0wESVmpoUs69niHwTmpMzWG86VJaa57JzyR1wYCuEGsi31i0OcuKWp/rk2W8fkkyfax8Up8Us69zooGEnvYIKfwfRMraMtfsWyErzSCO7be7wdd9btlh6OcQPNZqao167GaH1Pi6QxalPZ5qoxDztv19e0ExJkaIQ7L8QJM9jqixuLCrv2n8SK5aNOoh0cAhUnlCKAthnJty/pHUxYxO9/+uW2iAfvQp48erpjWuDHPrICv3r/Ujv8UCl7xgadIIMIlyWNUZs+9/CG04SYoHCnTJwwQ9QX3HX39XDc2HV3ewHw8I6NWrEIC++eg6dw397jiUTc5JlOPnkc4Ymsf//eBWy54So89u8+63x+317/tfoVLksao2ZSkvQKhTtl4nh+LpsGRx0jQHtOd5zzAs2QistGur12CJoEyTRGO+m87n6Hy5J6+UxKkl6JlwYnkcQpPdOVlAVZNebh1htXG38vAO7evFYbK3YtF4xqVEqjuWlYBSlpmWEapYRkaUOPOwNcPT/bbGuvJNh5+wajQAZ3xDGd1w8bjETsXJOkDC/Obb3JM81684l+kMTLz1MVFCkmFO4B43/xTbXfpnj1olKRcz7CyUGXXWt0v+v1tt4kVOHEpM+WG65yPndRyUsVFCkmFO6cEP4i+8OHTH5yEuEMNov4RN2ip1WtoROqifFq1x6USRKThCw1KNw5JMpD7kU443Z1Zn1bnweR5lxtUjSWzC7vRWLL7kPGqpCoGdYkHtzlneQF7vJecEyJPwE6NrW1eYr0It3ghD5SRCjcOcQlIWirrwbQc+31UoHNMKSIsI47h7jU+do8xSJtLDFohmUkMFlaULhziEtjh81TpBfpDpthSBFhqCSnRNX5RoVT2FLtBpthSBGhcBeUqPrqPG0skXfYDEOKBoW7oLh4isPiRbJChpBOWMdNco2uzjq4sTFFnAwLrOMmuSANT1lXIeO7GixzJEsVVpWQTPA95dpsHQqXRXZ6phbrPFGVMCxzJEsRCjfJhLRqyV0qYVjmSJYaFG6SCWnVkrtsCsEyR7LUoHCTTEirIzHYjAQ0E5NBWOZIliIUbpIJaXYkToxXcXhqK97YvR0P79gYe6swQoaNyKoSEbkCwE8BLG8d/5RSamfWhpFik1VHIptlCHErB7wIYKtS6n0R8QD8tYj8SCl1JGPbSMGhyBKSDZHCrZodOu+3/um1/ku/a4cQQogTTjFuESmJyHEA7wH4sVLqhWzNIoQQYsJJuJVSC0qpjQCuBfBpEflE+BgRuUdEjorI0TNnzqRtJyGEkBaxqkqUUrMAfgLgc5rfPaqU2qSU2rR69eqUzCOEEBImUrhFZLWIVFo/lwH8HoBXszaMEEKIHpeqkmsAfF9ESmgK/RNKqWezNYsQQogJl6qSnwMY74MthBBCHGDnJCGEFAwKNyGEFAwKNyGEFAwKNyGEFAwKNyGEFAwKNyGEFAwKNyGEFAwKNyGEFAwKNyGEFAwKNyGEFAwKNyGEFAwKNyGEFAwKNyGEFAwKNyGEFAwKNyGEFAwKNyGEFAwKNyGEFAwKNyGEFAwKNyGEFAwKNyGEFAwKNyGEFAwKNyGEFAwKNyGEFAwKNyGEFAwKNyGEFAwKNyGEFIxI4RaR60TkeRF5RUROisg3+mEYIYQQPaMOx8wD+KZS6iUR+RCAYyLyY6XU/83YNkJIH5ieqWHPwVN4Z7aONZUyJretx8R4ddBmEQuRwq2UehfAu62ffysirwCoAqBwE1JwpmdquP/pl1FvLAAAarN13P/0ywBA8c4xsWLcIrIOwDiAF7IwhhDSX/YcPNUWbZ96YwF7Dp4akEXEBWfhFpErAewDcK9S6jea398jIkdF5OiZM2fStJEQkhHvzNZjPU7ygZNwi4iHpmg/ppR6WneMUupRpdQmpdSm1atXp2kjISQj1lTKsR4n+cClqkQA/DmAV5RSf5S9SYQQG9MzNWzZfQjXTx3Alt2HMD1TS3yuyW3rUfZKHY+VvRImt63v1UySIaKUsh8g8o8B/B8ALwNYbD38h0qpH5qes2nTJnX06NHUjCSENAknEwFAACgA1YQVIawqyQcickwptcnlWJeqkr9G82+DEDJgdMlE3/WKqggxCbT/HykO7JwkpEBEJQ1NFSG+p16brUPhssj3EmYhg8OlAYcQkhPWVMqoRYi3L+5BD3tEBAuhsKgv8ml62wy79Ad63GSoSTORlwd0ycQwayrlLg87LNo+aZb90avvHxRuMrQMo5BMjFfx0J03odoq1wsnn/yKEF0sXEeaZX9s5ukfFG4ytAyrkEyMV3F4aive2L0dD+/YiGqlDEGzquShO2/CxHjVyZNOu+yPzTz9gzFuMrRkKSR5ieWaKkJMsfCSCBaVysRm02uymSd96HGToSWrrsAihGBMsfB/UB7Fwzs24vDUVkyMV9nMU1DocZOhZXLb+q5mlbhCovOsbSEYXwzz4I1f4Y102XlurtGu9QaAyadOoLHQTFzWZuuYfOoEgGSTAf3n5OG9DzuRnZNJYOckyQu9iKiuS7HslYxJPwHw8I6N2uf4sec0Mb03nd1hqpUy5i7N49xco+t3q8Y8zHz7tlRtJdHE6ZykcBNiYMvuQ8Y4sa68zq/0MNVZu7akuyw2pkXloTtvwp6DpyJrvf02eRNv7N5ufT5JnzjCzRg3IQZMScwFpYyxXFvi0yUW7ho/t4VrXJKv6btrpJ8wxk2IAVOVRDUQ6w57xVHeblS3YlQJo/+aJuH17YnyuG1Uyl7kMcG7gpVlD42FRVy4tNB+/q47NjC2nSH0uAmBvsNSVyXhlQQXLs7jvr3HAaCjQgNoJkS9EftMNptHbPqd73nXLKINoL2IeKVkc+G8EcGuOzZYjwnfFczWG23RRuvf9+49jm9Nv2w+CekJCjdZcoRF+lvTL2vDEwDaXYoCYMwbQWNBYbbesJcBRmimrRzR9LuSSGQnpB+umRivYsWyeDfTfgPPni/dHOkpu3ZlPnbkdK5KJIcJCjdZUuhiyI8dOW0t7zs8tRUP79iIemOx63zhTsw9B0+1y+t0RJUjmmqhTbNGAHR1TQLA+Xp3tYiJkghe3729487BhmsDkwIK36WaVyjcZElhm2cdJihQew6eMh5Xm623Pfio2LIv9CZPNDiLJCjItrizrvJkpUOc2se2KJhezxW2u2cDk5NkSRFHSIICZXuewFwCqCO84YGu/O/w1Nb28dMzNVy4NB/rfLbjw1Rb0wRd6911jU0m2O6eDRRusqRwrbgIhzRWlj3MGsIPSUrrgiGWqO7FqPBL8Hx+ZYvu+JIA4YfLXgm33ri6Q4ijdtIJd0iuLHuoNxZwcb4zlMR29+xgAw5Jlby0e5vsqs3WI5tPVo152Hl7Zznb+IPPabsMe0EAVMa8yO7F66cOOC8O1ZilgFtuuApv/LpufY5f/gjY29nz+tkXhVT3nCTElXA3X5TnNii7FOydg2PLRrvsnU1ZtAG79x8U8zh12S4LU5AjvzyHxQjnrTZbx72t8sfgY+HPNo29Kyn+bjA5SVIjr/Ov4yQkgcvJxmAC0RSrXTXmRe5Io8MPUbjgsutNEH9hcmGhNeI1CWl/tkWYupgXKNwkNfI6SD/J64dFw1Smt/P2DR1VIJWyB4lQzZIIPrV2JfYdMwtSsIokXGlSinoBNMXb5fiSSOyFIUian21eF/48QuEmqZHV/OteMb1+pWz3loOiYSrT88MDfq33xflFRKWNFpTCz147a6zK0HUv+q/x+u7tkaENtOxzOX7z76xqvzeXBSHMiEhqHnFeF/48whg3SY005l9ngckuXxxt80WComGK4QYTn67YpDeqe9El5h285rbjXzp9HtMztfbruZb5+SwolVoew2SnQnNSI+Pdl6HHTVLD5pXm1S7fk6063C3o5pkE47JpUK2UI3emiVoIK2Wv4xwuQ698lo9elgTfAY/yw9MKZ9hCNox3d8JyQJI5rvOl06gmSHoe23xr0+YEZa+EK7yR1MoE/dcDuj1fb0Rw5RWjmJ1rYE2ljF+9f7GrbtrnkR0bAXTWh9vwN4AIH++VBHu+eDOA6KmEQHPRSePz27X/pLFm3g8BDSOplgOKyPcAfB7Ae0qpT/RqHFlauJQIRh3jKsa9lCNGbbtlSpzFCSvYCI5C3bL7UNd5G4uqvUBEefcT41WMP/ick2gDAARd5X4A0FhQeOCZk5j59m3t62Dy4IPdo72WgZoWJIDxbh+XGPd/B/DHAP4iW1PIMBK1P2PUMQCcxdjltWzY6pDTCoWYWLH8cu14L+Lkh3zi3AXYbrrD59HlC3R143Gue5CoyYODTnTnhUjhVkr9VETWZW8KGUZcKgVsx8QR46yqEtKKq9q6GoOPJ90IoR+JYN2diUtiV3fX5J+nNls3bgcXJA+J7rzA5CTJFJcSQdsxccQ4q3LEtOqIJ7etN5bcBR9OUlddEsFdt1TboaW00FkbLE10SezqGmsmnzyByadOtEU/SrRLIrlIdOeF1IRbRO4RkaMicvTMmTNpnZYUkGBFxNyl+a4dYcKek6m5ZXLb+lhibDtPL6QRVxU0FwCTQCl12bOfGK/irluqzt2PQFP49h2rtTeFSAvf2qgqF9t11901NRaVcwy+7JXw3S9Hb/CwlEitjlsp9SiAR4FmVUla5yXFIpwgPDfXgFcSVMoeztcb2uRiVGLQVhsevgW/65Yqnn/1jPY8SStOet3DEUDb07QRDP88/+qZ2FMH640FPP7CW7Hna9vwR77a8gwT41UcffNs+7WD3j+QfOET6GeNJ2WY5qCwAYekita7WlBYsXwUx3feZnyeKTFoEnWge2JfbbaOfcdq2lvqXipOJretx+STJ9BYzNYfcYn7R2ET7Ud2bIzVKOSNSHtTZFueYXqmhn3Hau3XXlAKe198Cwd+/i5m5xoYcYhfh4lT9udabprHAWhJiQyViMjjAP4GwHoReVtE/k32ZpGikkWCMBxTBZpeuK5ywtQM0sscjInxKvZ86WZ4GWeEgjHhtPEbe9Z92D3ev2x0BBPj1cjP1BQKOTfX3JszyR2A6wAu18FUwzYHxaWq5Kv9MIQMB6awQpplXFElYzoze/1nAAAOhElEQVShSWNBGS2V0FiMX7ftMmbVD/98a/plPHbkdKLNGVzOffi1s87P83duj/pMXT14l8oRn+df7c6T6TxrkyB/84kTuG/v8fZxwzYHhVUlJFWyShAGifqy6RYJ08KxsuwZk25BXHc216GAyCqRu25p3q6nJdqrxryu9v7HX3gr0bmiPlPX2VSLSuGRHRudKmZqs/WOz8LkWZsWjQWlOo6rjOn34CxqXTiFm6RKP+aV2L5sflw2jE58vBHBhUvzHWJw397jWKcR8V48s0rZi5y+9/yrZ6wbEsflfL2BlWUPtdk6vvnECaybOpAoZLHxgefwwDMnUW8stO0Pf6aup13TCte4jqi9b+9xfGu6GYc2edYuEw3rjQVcNCy6riEZG7aKm6xgcpKkTho7odiwblZr+B7rkpxzl+a74uS+Bvl7P+7afxLn68kSbD6X5hespYD+66XJokJ73kcvVSbBmSELSrU97bifb9BDD/596GbA+Cg070A2ffQq48Lp2xR1NzTX0LfR60IycRhU0pMeNykctvnRjQVlTDiFk5xR25E1FhRm68kTbD5zjcXMW+b7Rb2xgHv3Hu/wLIObPuhYNeYZ77r8z9KEQnOxNd1l+d5/nE0mgvQa4x5U0pMeNykkE+NV3KcZjAR0fxlN7dZJvOg4CbZhJuhZfv7ma/CDI6eNx87ONXDv3uPYc/AUbr1xtbbO/oFnThrnq7wzW8fDOzYa6/mjPPiyV8Ly0RHtxMFeY9yDSnpSuEmuiNMkEVXtoBsR6odAoJJ50X6CrR913a6MeSO4OK+c349fIx01qzsKV88yGH4KCnxQ/G2m+7FxwL7LPGCv+89ik49+VFHp4DxukhuiZmLHOR6Iv5uLC77ohZt/wvhdf7o4ugurxjyMLRuN3LU9ODPb5f1GzRj3RgQQOLej+4GJXlTEn+NtOscjOzbmtnMy7t+sjVTncRPSL+KOZbV5YbqZ1i74IlIZ8/D+B/MdXnXQQ7PFx1eNeZj5drNL1JZ8s3FuroGdt2/omkdeGfOgFIzjA8LHzdYbHcJ/RaCLyOadunZY9pK09bFNGfR380mDLJLmrncCaUOPm+SG66cOaL0uAfD67u1dj9sELUkIINxmbfPQNj7wnHGXluCmCP55bDHcKESAuz+zFt+ZMCfxTOgaelw9wl5DKa5UW9c2Lc+1qNDjJoWkMuZpxU3XPKEbZuUTFV7QoYt32jYHvnBp3niu2XqjawjTnoOnEgu3UsAPjpzG62fexxu/rjt7dtMzNW1Dj+smB7YEW1pJ2rJXwq03rm7fbfnnrfbJcy0qLAckucGkA7rHozoZFaI3ufWJ2yS05+CpyBhwOHGXRpXB4dfORs7kCPLAMyeNi5eLPaYE26oxD4sW0TZ1RlYrZXxt89qO5qy7bqli37Fax1zupLXiSwl63CQ3nDeEHnSPuwiPwuWY9cqyhwuX5jsEN+mtuKsIB49bWfaMoZWkBBcHXZza5uG7VD1Mbluv3XD43FwDK5aV2rNMglQrZdx642pteEYnxrpcRNJtz5YSFG4SiyxnGscprXKJY8eJWadRhmizO2ZfiDO+5x3u3Fs+ar6ZFsCpDG5ivGrccf3CpQV4JelaCP3BT67hmWEb/tQvKNzEmazbe00JKtPsEVu1hv+8NGc1++dyiaGH7U4a346iJBJ79/m7N691/rxMd0EAsGLZKFYsH+26tq6NUcDg6qCLDoWbONPrLupRuJRWhStJlo+O4Hy9oS2TA9x2iHd5X2Fxj0rLhUMwWXRcCuI3EZW9EWt1SnBxKolY3+f5ekO7OUYcMY6zWA+CvO6aQ+EmzvTjttZWa6urJCl7JTxsaNBwjZ+6vK84Y139TQuCxBFYkzc/5o2g3lhs/05Zjh2R5qCpMPXGIrbsPuR05xFls8krjiPGg6qDdiHPu+ZQuIkzg76tjevxuy40Lu/LdXEyjZWtxqgtV63zhJt/lnsjXVPudOIdNS0vzp2HCZtXHFeMs54mmZSs7zB7geWAxJl+bJJgI67H77pDvMv7cl2crrxiVPul1r2GLV955RWjXTPNTd2afvVM8NhqhL26OSMui5PrjPXwJEZTPXy/51jHIc+JU3rcxJlB39aaPOMREVw/daDLHtdbdpf3FZUM9TGJq+41br1xtXGq3uxco90272NqQy+JaO2Ostf1zsMnzga+UeQhDBEVvx70HaYNCjeJxSBva03i6cdiw1/+OAtN1PsKnssmbrYvte41Dvz8XW3FSfg80zM1zBm6NU3vP8pe3Z2HSezjVOm4MOgwhMvCkefEKUMlpDC4bHsVDgG43LLHef3DU1utIY64X+qdt2+IDNP4IhMWeJ0dwffv26vb59F05xEMs4S3KgPgtKO6C4MOQ7hsgNCPbfiSQo+bFIqg13r91AHtMbXZOrbsPpRZOMd0C71qLP4ku4nxKo6+eRaPv/AWFpRCSQR33dLpmZuShrZ29rBnfNctVe0GBjoP2hQOSbPLcdBhCNeFI6+JU3rcpLCYvuQCpOIVmjAlM3fevsH4HFMibnqmhn3Hau1wx4JS2Hes1tNGxWVvpMsz3neshslt6zvuPEw7p5uuVZpe8qAT3a6J67xC4SaFxVSpYWq3BtKpZIh7C20TSJdbduMCZYjZ1OcXnfZBjLtfYppiN+gwxKAXjl5hqIQUFl3y0ZSI88MHaVUyxLmFtgmkixdrSpKZKkZMfTPh14rrQcdJ1rkkMQcZhhh0hVSvULhJoQl/+U3D/9dUygOrZLAJpEus1yQytvJAXddj2DOOG2d2Fbs8lPq5kNf4tQtOoRIR+ZyInBKRX4jIVNZGEZIU2y3woCoZbCEG11t2XXWM6blf/cx1TudMEi5wqdKJG4Ih8YkUbhEpAfivAP4ZgI8D+KqIfDxrwwhJgi12OqiElE0ge4n1mp77nYmbnM6ZVZx50KV+/WDQXZ+Re06KyGcB7FJKbWv9+34AUEo9ZHoO95wkeSTNHbmTvPag4qn9fm1TuCrNzstBktXfUdp7TlYBvBX499sAPpPEMEIGySATUoOKpw4i3pznjsM0GHTXJ+Am3Lqioy43XUTuAXAPAKxdu7ZHswjJhiInpJIwCJEpesVGFHkIBbkI99sArgv8+1oA74QPUko9CuBRoBkqScU6QkhPDEpkhnmBHHTXJ+BWVfK3AD4mIteLyDIAXwGwP1uzCCFpMOgOwUEn8bIgD807kcKtlJoH8AcADgJ4BcATSqmTWRtGCOmdQYpM3Jb6ojDork/AoaokCawqISQ/DKqiZdirS9Im7aoSQkhM8rTJ7KDizXlI4g0rHDJFSMoMa4ggLoOOrw8zFG5CUoYt303ykMQbVhgqISRl4oQI8hRSSZthr+ceJBRuQlLGtc63KFP0emGY67kHCUMlhKSMa4iAIRWSFHrchKSMa4iAVRckKRRuQjLAJUSQh9ZpUkwYKiFkQLDqgiSFHjchA4JVFyQpFG5CBgirLkgSGCohhJCCQeEmhJCCQeEmhJCCQeEmhJCCQeEmhJCCQeEmhJCCkckOOCJyBsCbqZ/4MlcD+FWG58+aIttfZNuBYttfZNuBYtvfD9s/qpRa7XJgJsKdNSJy1HWLnzxSZPuLbDtQbPuLbDtQbPvzZjtDJYQQUjAo3IQQUjCKKtyPDtqAHimy/UW2HSi2/UW2HSi2/bmyvZAxbkIIWcoU1eMmhJAlS66FW0Q+JyKnROQXIjKl+f1yEdnb+v0LIrKu/1bqcbD96yJyRkSOt/77t4OwU4eIfE9E3hORvzP8XkTkP7fe289F5FP9ttGGg/2/KyLnA9f+2/220YSIXCciz4vIKyJyUkS+oTkmt9ff0f5cXn8RuUJEXhSREy3bH9Ackw/NUUrl8j8AJQCvAfgdAMsAnADw8dAx/wHAn7R+/gqAvYO2O4btXwfwx4O21WD/PwHwKQB/Z/j97wP4EQABsBnAC4O2Oab9vwvg2UHbabDtGgCfav38IQD/T/O3k9vr72h/Lq9/63pe2frZA/ACgM2hY3KhOXn2uD8N4BdKqV8qpS4B+EsAXwgd8wUA32/9/BSAfyoi0kcbTbjYnluUUj8FcNZyyBcA/IVqcgRARUSu6Y910TjYn1uUUu8qpV5q/fxbAK8ACA/szu31d7Q/l7Su5/utf3qt/8JJwFxoTp6FuwrgrcC/30b3H0D7GKXUPIDzAD7cF+vsuNgOAHe1bnWfEpHr+mNaKri+vzzz2dYt8Y9EZMOgjdHRug0fR9PzC1KI62+xH8jp9ReRkogcB/AegB8rpYzXfpCak2fh1q1i4dXP5ZhB4GLXMwDWKaU+CeB/4/IqXgTyet1deQnN9uKbAfwXANMDtqcLEbkSwD4A9yqlfhP+teYpubr+Efbn9vorpRaUUhsBXAvg0yLyidAhubj2eRbutwEEvdBrAbxjOkZERgGsRD5ukSNtV0r9Wil1sfXPPwVwS59sSwOXzya3KKV+498SK6V+CMATkasHbFYbEfHQFL3HlFJPaw7J9fWPsj/v1x8AlFKzAH4C4HOhX+VCc/Is3H8L4GMicr2ILEMzEbA/dMx+AP+q9fMXARxSrazBgIm0PRSTvAPNWGBR2A/gX7aqGzYDOK+UenfQRrkiIv/Ij0uKyKfR/B78erBWNWnZ9ecAXlFK/ZHhsNxefxf783r9RWS1iFRaP5cB/B6AV0OH5UJzcrtZsFJqXkT+AMBBNKs0vqeUOikiDwI4qpTaj+YfyP8QkV+guep9ZXAWX8bR9v8oIncAmEfT9q8PzOAQIvI4mpn/q0XkbQA70UzUQCn1JwB+iGZlwy8AzAH414OxVI+D/V8E8O9FZB5AHcBXcrLgA8AWAP8CwMutWCsA/CGAtUAhrr+L/Xm9/tcA+L6IlNBcTJ5QSj2bR81h5yQhhBSMPIdKCCGEaKBwE0JIwaBwE0JIwaBwE0JIwaBwE0JIwaBwE0JIwaBwE0JIwaBwE0JIwfj/aTZjznuMI/YAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1e69f9f77b8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.datasets.samples_generator import make_blobs\n",
    "\n",
    "# Generate toy data that has two distinct classes and a huge gap between them\n",
    "X, Y = make_blobs(n_samples=500, centers=2, random_state=0, cluster_std=0.4)  # X - features, Y - labels\n",
    "# Plot the toy data\n",
    "plt.scatter(x=X[:, 0], y=X[:, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating our Support Vector Machine\n",
    "\n",
    "We will be using PyTorch to create our SVM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "from torch.autograd import Variable\n",
    "\n",
    "class SVM(nn.Module):\n",
    "    \"\"\"\n",
    "    Linear Support Vector Machine\n",
    "    -----------------------------\n",
    "    This SVM is a subclass of the PyTorch nn module that\n",
    "    implements the Linear  function. The  size  of  each \n",
    "    input sample is 2 and output sample  is 1.\n",
    "    \"\"\"\n",
    "    def __init__(self):\n",
    "        super().__init__()  # Call the init function of nn.Module\n",
    "        self.fully_connected = nn.Linear(2, 1)  # Implement the Linear function\n",
    "        \n",
    "    def forward(self, x):\n",
    "        fwd = self.fully_connected(x)  # Forward pass\n",
    "        return fwd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our Support Vector Machine (SVM) is a subclass of the `nn.Module` class and to initialize our SVM, we call the base class' `init` function. Our `Forward` function applies a linear transformation to the incoming data: *y = Ax + b*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Scaling\n",
    "\n",
    "Feature scaling is a method used to standardize the range of independent variables or features of data. In data processing, it is also known as data normalization and is generally performed during the data preprocessing step. Standardizing the features so that they are centered around 0 with a standard deviation of 1 is not only important if we are comparing measurements that have different units, but it is also a general requirement for many machine learning algorithms.\n",
    "\n",
    "But why feature scaling?\n",
    "\n",
    "\n",
    "> The true reason behind scaling features in SVM is the fact, that this classifier is not affine transformation invariant. In other words, if you multiply one feature by a 1000 than a solution given by SVM will be completely different. It has nearly nothing to do with the underlying optimization techniques (although they are affected by these scales problems, they should still converge to global optimum).\n",
    "\n",
    "> Consider an example: you have man and a woman, encoded by their sex and height (two features). Let us assume a very simple case with such data:\n",
    "\n",
    "> 0-man, 1-woman\n",
    "\n",
    "> 1 150\n",
    "\n",
    "> 1 160\n",
    "\n",
    "> 1 170\n",
    "\n",
    "> 0 180\n",
    "\n",
    "> 0 190\n",
    "\n",
    "> 0 200\n",
    "\n",
    "> And let us do something silly. Train it to predict the sex of the person, so we are trying to learn f(x,y)=x (ignoring second parameter).\n",
    "\n",
    "> It is easy to see, that for such data largest margin classifier will \"cut\" the plane horizontally somewhere around height \"175\", so once we get new sample \"0 178\" (a woman of 178cm height) we get the classification that she is a man.\n",
    "\n",
    "> However, if we scale down everything to [0,1] we get sth like\n",
    "\n",
    "> 0 0.0\n",
    "\n",
    "> 0 0.2\n",
    "\n",
    "> 0 0.4\n",
    "\n",
    "> 1 0.6\n",
    "\n",
    "> 1 0.8\n",
    "\n",
    "> 1 1.0\n",
    "\n",
    "> and now largest margin classifier \"cuts\" the plane nearly vertically (as expected) and so given new sample \"0 178\" which is also scaled to around \"0 0.56\" we get that it is a woman (correct!)\n",
    "\n",
    "Source: *scaling?, W. (2018). Why feature scaling?. [online] Stackoverflow.com. Available at: https://stackoverflow.com/questions/26225344/why-feature-scaling?utm_medium=organic&utm_source=google_rich_qa&utm_campaign=google_rich_qa [Accessed 6 Apr. 2018].*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.PathCollection at 0x1e6a2dfe940>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD8CAYAAABjAo9vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJztnX+MHOWZ579P9/SEbhtloO1TwpCZYbN7ZiEstmzlOCHtLU4Us3DJeskPdm/CWiKRk/FGghznOyNHAvZiYcnaEHS3LGddyEFmlIUcxJvEPjnZhb1IKGQzju0Qn/GJWzwOQ+5ixkyCmQG3x+/9UfO6q6vrrXrrR1d1V38/Ummmu6ur3q6xv/X093ne5xWlFAghhBSHUt4DIIQQki4UdkIIKRgUdkIIKRgUdkIIKRgUdkIIKRgUdkIIKRgUdkIIKRgUdkIIKRgUdkIIKRgDeZx01apVamxsLI9TE0JIz3Lo0KHXlVKrw/bLRdjHxsYwPT2dx6kJIaRnEZEZm/1oxRBCSMGgsBNCSMGgsBNCSMGgsBNCSMGgsBNCSMGgsBNCSMGgsBNCSMGgsJP+ZmoKGBsDSiXn59RU3iMiJDG5TFAipCuYmgK2bgUWFpzHMzPOYwAYH89vXIQkhBE76V927myKumZhwXneDaN60mMwYif9y6lT4c8zqic9CCN20jm6PdIdGQl/3jaq99Ltn50UGgo76Qw60p2ZAZRqRrpegYsrgGkI565dwOBg63ODg87zGpuo3m9sNp+dkE6hlMp8W79+vSIFZ3RUKUfWWrfR0eY+k5NK1Wqtr9dqzvNB2LxvctI5l4jz0++Yk5NKVSrtY6zXm/vbfI44n71T2Hxu0rMAmFYWGpuKUAM4CeBFAEdsTkxh7wNE/MVNpLlPXAEMe5/tDcN0HPf+fscaHHTE3ySeNp+9E8S9UZKeIQ9hX2W7P4W9D7AR7agCqKNRkxjr95n2qddbo1nTcfxuFHr/Usl8E4jy2TtB0OcmhcBW2Omxk86waxdQq7U+V6u1+tc2yUuN27c2od9n8r/n5lp9bxHzsdzHGR9vfp4LF9r38yZTbT57Jwj63PT3+wsb9Q/bALwC4KcADgHYGrY/I/Y+IczvjWIdhEXY7vfZROPuKD8sYrc5pvdbRh5ed9AYTd8W6Mn3FMjYirli+ec/A3AUwO/77LMVwDSA6ZGRkQwuAekJbIUlTIC9iVPvDSNoq9eDbxRh5weUKpfzF8fJyeAx+u1PT76nyFTYWw4I3A/g3wXtw4idRCaqb+2+YZTLwYLs3d9PnKN8CwhLrnYSv5uU/kZh+5n0mBnJdx22wp7YYxeRFSJyqf4dwEcA/DzpcQlpIapvPT4OnDzpeOJ+vrhmaSn++QGnjt7LuXOOr61UcA17lFp8230fftg/d6BU+6Qqkyevx8w6/N7FRv2DNgC/Bcd+OQrgGICdYe9hxE5iYYoik0Tben+bunh3NKxr3cMsGtM3C9M5JybaP4vfvpWK+VuBbR7AdF1M33CyqMMngSAvK8Zmo7CT1PCbZFQqtYrexIT/RKTBweASSne5o/f9lUp4+WUcUfXeKGo1s71iuhHZ2lamm4vtZyCZYyvsLHckvYXXkvjc54BGo3WfCxdarZDHHwc++1mgXm/uU68Djz3mWDZhbQPuuqv9HI2G87zJovHiLeE0nVOp1scLC85nCcNdcmmyrW65pfXaAcDevcDoqGPfjI42H9t8BtK92Kh/2hsjdhKLqNUu7s07Ocl2QpFNpYnXpgmKpvX+QQnduJs7ovbaUxMT9jNog6wpJlVzBbRiSM/jFREbS8J2c4utqW3AihXhx9EE+dVJSjH1TcnmPUHVLDaWkfea2Hj9LI/MFAo76W2SROe2m7chmRayet3fkw8S9qAkqlsco5RNeiPloChfJ15Nwhs3yesmz+ZmRCllL+z02IkZt5+9apWzdbK/uPt8W7a090FPm5kZYNs255x33OE8941vACtXtnvqfrg9+8svN++nlHOuO+4Ibongx969zQU9hobM49i7FzhwwNw7Pmh8boLaEcdpYUzywUb9094YsfcAYRFz2l/BOxWhj47aWSruz2Wzn66KCfPXk276G8TgYPtr7vbCSgVH5bZjZMTe1YBWDElElO6HcXHbH7ZWQVRRn5z0F8Wgza+Do9/rYfsFiXUan8/btdG2hNK0hd2s6bHnjq2w04oh/th8vU7yFXzbtqY1oWUijFIJKJftzzEz41g6585FG1vQTFX362H7mbD5rDZ4uzbu2mWedWq6bvr5crlp25hstvFx//JIrv3adVDYiT82Nct6n6jL1E1NAY8+Gl3garV2n9lvSr8b25YBfmiRrNfbl9DrFrZsaV7v8XHzNV1a8q9t37rV+amvU1j7AHerhpMnKepdCoWd+BM28Ub3aYmztumWLfGi1rNnWyfr1GrAE08Ak5NApRL9eGEo5Yj6ypXRo/6sWFpqvd6myUU6ItcRuo62gxKupHex8WvS3uix9wjeEkC/ySw2qxVFKR+M46Mr1dkEZi9sQcsCBnnpeS3jR2IBS49dnH2zZcOGDWp6ejrz85IOUCql5xnHZXQ0vIywXHaiW/2z04gA1WrnSzbd59Oe/9SUE3GfOuX8ffw+7+ioY6WMjflfO/066SpE5JBSakPYfrRiSDKy6B9Sr7fWjLsRsasNX1py9s1C1AHnZpe2qK9YYU6Cuv8ONi2LdeLbph1yJ9oLk85iE9anvdGKKRBJ6s/1TErvTz+bwe88nSiR7OZNxJld6tcB0lRyaFN7HtT/JUqJI8shOw5Yx04yI05PF7/GWEGC5neevIW2GzYt9qa/h19+I4rYRpmUxAlMHYfCTvLD1FTLtDBEWNRvEoZ+T5j6XZ+o1z6MKMlVJmI7jq2w02Mn6eM3keWxx4DXX/evf9650+xHu8sq3d7ttm3A/HwGH6YHcE8U87uW5845JZtxas9NORS/56PsSzqLjfqnvTFi7yP8+oJ7/VybHif95qfHjdjTjprpsXcVoBVDcse2ptpkqRRBzDuxoEaQcHbC546yuAYX4ugotsKemhUjImUROSwi30vrmKTHCbJYNPp1b8mdiCNJvYyIsyyfaTZoUnS7XqBpU509297+wFu+GJUobQTYcqArSNNjvwvA8RSPR3od2yZhZ860e/K9IurXXOPfeAtweqDrdgtpU687OQugtaWDXuu1Xmejrj4mFWEXkSsB3Argv6ZxPFIQbJNmIyPtkV6noty0OX4c2LjRX9zn5joz87RWAx5+2Pn9rrvaz9FoxE+WkkKQVsT+VQD/HkDMPqakkIQ1EgPMNoHfe93dFrsFpYCXX3ZWXoozLj2T1BT1a/xa5U5NtTZFc2PzbYmzRIuLjREftAH41wAeWf79DwB8z7DfVgDTAKZHRkYySDOQzAibuRhWFRPnuHknRb1JXqXir2eqP2vQvn7XI6iOPyxZGrWChUnRrgBZVcUAeBDAqwBOAvi/ABYATAa9h1UxBSKvErdumnmqRTRKFU+53H6NTEKtl8CL0roh7PpHqZ5hGWPXkJmwtxwsIGJ3bxT2ApHXNHI/salUoi+Dl8amp/RHudm468qD1k3Va6tGObZ3yTw/otS7s1VA12Ar7Jx5SpKR1cr1Xj8YaK+k+frXnRmu7ucmJsJ9/qQcOOD8jFJS6F596s47/b3yet35TOPj9pU17sSqzfltns/qb0zSw0b9094YsReILKK5MCsgzP+dnIy/8HTU6Htiwu49YQuVeK+haaKT/sxRvW/TOL0NxYLGyIg9c8CZpyQTsvBfg4TF5D1rX3pyUqmVK9MR8KC2wt5rEjTj1G2VBPny7htG0LjSvqZeorYKJh2Dwk6yo9MVE0F+cFDEWyopNTCQjqjXao7ARembYloOcGLC2cLaDbhFNu2o2dZjN/XB94vsScehsJPiECRqWfWT8do+7gjedDObnFRqxQr/G07Y+XTS1H2sNL8Z2d4oaMN0FbbCzuQp6X6Clm9L0hI2aMk9E+PjzfHoZfZmZpxp/d4JPuPjwKpV7ccwLVfnHpdOmrqP5U0WJ2kVYLMkHsDEaa9io/5pb4zYSWRMdk/cpflEmh68zfvj2iJRv1FkiY2Fxoi9q4BlxC7OvtmyYcMGNT09nfl5SUGZmgL+7M/CI2E3ExPAI480379zZ3BJoUjz+KWSI29B+2jGxqI1Acvh/2MgU1POtxF3P5pajY3FckJEDimlNoTtRyuG9D7j48ATT7S3qw1Ci7p+f1jjMbflE6UG3KZfjmblSrv9siRtC4hkAoWdFIPx8ebkJCC4qZZJwHft8r85VCqt3rOtP63H5RXGiQlgYKB1v4EB4NFHzWOOQtrNvdhjvfew8WvS3uixk44SVALpt7Cz22uu11vr3nU9vJekJZ6dKhFlX5dCA3rspG8xeeCAE303Gs3Hg4POvu7netlDNnn6o6NOtE16GnrspH8xeeDlcquAA8C5c+3PLSw4ydRehOWJBBR2UkRMHriuO7ehV4UwSmKXFBYKOykepkqOKMvt9aoQRknsksJCYSfFxK+Sw0/0Bgcd391NnkKYtKKF5YkEFHbST/iJ3mOPOdP3u0EI9WSgmRknoTszA3z6005bgigCz/LEvodVMYR0Ej2r9dQpx97ZtcsstEGzVHu5UoekBqtiCMkbvwjcr1mYJihh28uVOiRzKOyEdIqdO1t7rADBAh2WsO3VSh2SORR2QjpF1JrysL4yvVqpQzInsbCLyCUi8o8iclREjonIA2kMjJCeJ2pNuU7u+vWIZ8kiiUAaEfs7ADYqpa4HsBbAzSJyQwrHJaS3iVNTPj4OvP46MDnZHZU6pCcZCN8lmOXGNGeXH1aWty5rKk1IDmghtq2K8b6XQk5ikljYAUBEygAOAfhtAH+llPpxGsclpOehQJMcSCV5qpRaUkqtBXAlgA+KyAe8+4jIVhGZFpHp06dPp3FaQgghPqRaFaOUmgfwDwBu9nltr1Jqg1Jqw+rVq9M8LSGEEBdpVMWsFpGh5d+rAD4M4KWkxyWEEBKPNDz29wJ4fNlnLwF4Sin1vRSOSwghJAZpVMX8DMC6FMZCCCEkBTjzlBBCCgaFnRBCCgaFnRBCCgaFnRBCCgaFnRBCCgaFnRBCCgaFnRBCCgaFnRBCCgaFnRBCCgaFnRBCCgaFnRBCCgaFnRBCCgaFnRBCCkYqS+MRYN/hWew5eAKvzS/iiqEqtm9ag83rhvMeFiGkD6Gwp8C+w7O495kXsdhYAgDMzi/i3mdeBACKOyEkcyjsKbDn4ImLoq5ZbCxhz8ET2LxumNE8ISRTCivsWYrpa/OLxucZzRNCsqaQwh4kpgCsBN90Y/B7/oqhKmZ9xH2oVgmN5sPOl/Q68JsCIf2HKKUyP+mGDRvU9PR0x45/4+5nfYX2sloFbzcutAhttVLGg7dd1yay7huD3u/j64fx9KFZ3+ef/Mkv0FiKdi2Hl8UWgO/5vOOKgukzJDkmISRfROSQUmpD6H5FFParduxHlE81PFTF8zs2AnAE8Z6njmLJ57qURXyfBwABIp1TU62UcUmlhDcWGoHjiorp5jZUreDIfR+JdUxCSL7YCnviOnYReZ+IPCcix0XkmIjclfSYSbliqBppf+2R6yjXJN6m54F4og44toyfqLvHFQfTe+cXG9h3eDb2cQlJnakpYGwMKJWcn1NTeY+o50ljgtJ5APcopX4XwA0A/lxErknhuLHZvmkNqpVyy3PVShlD1Yrv/vpG4OeHuymLpDdIC6LeoGzfu+fgCQDOjezG3c/iqh37cePuZyn4JHumpoCtW4GZGUAp5+fWrRT3hCROniqlfgngl8u/vykixwEMA/hfSY8dF+0hexOHgL+XrV8LipBNHnun0OMKS4CaXt++aQ3ufvKI77FnA6p1pmfO4LmXTjPhSrJh505gYaH1uYUF5/nx8XzGVABS9dhFZAzADwF8QCn1G89rWwFsBYCRkZH1MzMzqZ03CkFCafKlNcNDVdx09WpMvnDK+nyVkmCgLFhsXLB+jwAYv2EEG0Yvb7sRaS9fj8XvRnNZrYL7PnotvvjUEfj9ecsieM+7L/H9rN5cAROupKOUSvD9RyoCXLD/P9MvZJ48FZGVAP4ngF1KqWeC9u108jQufpUkSSmXBEsXol/j4WUrJehGE5SwrVbKgZ8jSrI3SRKXkEDGxhz7xcvoKHDyZNaj6XoyS54un6wC4GkAU2Gi3s1sXjeMj68fTtVLjyPqgGMLBYk6ECzMi40l4+coi0RK9s7OL9KHJ3boRKgIMDDg/AxKiO7aBdRqrc/Vas7zJDaJPXYREQBfA3BcKfWV5EPKHm3PzM4vxi5bTJuhWgXzC41EY1lSyvfzBFX3mFDgrFkSgk6Eas98afkbo06IAo5vPjXleOinTgEjI8CWLcCBA83Hu3bRX09IGhH7jQDuALBRRI4sb7ekcFwjaVZzaPtFR8fdIOoA8EZCUQei19ZXK6W2aiIvetYsIW34JUI1OiHqVwXz+OOOmF+44NgvNqLOEslAemqC0r7Ds3jgu8fa6r7jJviCJiP1IwLgodvXXkwum66KAHhl960Zjoz0BKZEqJtyuRnJu4niqXu/GQCOfbN3b+Ej/Uw99izQkbXfZB7bKHLf4VmsfeD7GNuxH2M79uPuJ49Q1F2Ulj3553dsxCu7b72YwPWSpL6epEi3Ra0jI8Gvi/iLOuDYMLYElUgSAD0k7GGTh2bnFwPtmX2HZ/FvnzyC+UX/WZ5FwzQZK4glpXDvMy9evHamiV667p/kSDdO7PFLhGpEgqP5sJuCG9NNIMrNoeD0jBUT1v/F6yfrx5fVKlAKfSPomkpZIjcl03h757BDZBeStEzQm8BMK2Gpjzsz07RdRkf9x6qJaqP0cYlk4ZqABU0e6pZKlm4jLEgyvg/00LueJBN78vCoTWIMACtWAG+95fxerwMPPxw8DnrsxfHY/WwBwLEcKOr+KIXQKhc/hmqVUFuLPWZyxmRdmJ53+/Fbtvh71Hfd1RnPfmoKOHvW/LoWdQCYmwPuvDP43OPjjoiPjjo3stHRvhD1KPSMsG9eN4wHb7sOw0NVCBy74Ku3r8WR+z5iTPL1O5fVKrikEvwn9gp/pSw4+/Z5zC5XxejadS3e7vJQv9dJRkSZ2OP1400JzLm5Vs/+058GVq3yF9mgxK37tVWrHKGem7P/bOfOhSdCx8cd2yVKiWQf0TNWTBCdaAXQ65TEmWHaCJj5qhf6cHvob71z3jcfoX13kyXGtgM5YOuTB9kgNnhtjiArBGh/LQ7sFeNL4Tz2MLpx9miehPnrptp/U5Ja++5hr5MuxKa+PAx3YtJ0ozDVqCc9H9CalC2VmqJv48kXiMJ57GFsXjeM53dsxMndt+Kh29detGdK2bZQ7xqC/h8PD1WNE7pMNer6+bDXSU547Y9Vq5o2yeWXB7+3Xne2INylhKaywrREfXCw1VJyW0lAayQ/N+dYRtu2pXPuglAYYXej+5FXK2XE7MFVWLRlYipZDKtdZ217F+L10OfmnE175b/5jSOWJhYXgbVrg8/hTspGqTmPSr0OPPZYawQe1KpA8+ij+U/Q6iIKKexA+ISmfsRGgP2S1O7oPux1kgNhwtdoAJdeao7KFxaAv//74HPc4mr/FDQRKQnlMvD66+22is3EI6U489RF4Tx2nQQMa3kLBC9OXTTKIvjLT11/UYA58ahA2HjoIk6kHTeJOjgIfOYzzS6M2t4Jq3YRASoVp9LFBj1OdyLYNvnbBwlXW489cdvebsBvmbewBKpOHgLty+UVkQtKtYi637J4ACj8vYiNYCcRdcAR5r/+6+bjuTlH7Ov1YHFXCjh/3v487vYIgCPuu3bZVdp00iLqMQphxfjZLgpOpYYb/dhtH7itBaC5YHXRcq7u5Kbf9XI3UmOtegpk1aArbPIP0ExGlqNPVgvk3Dm7+vQ4UfTCAnDHHU5SVFtNevx+C8hwcY4WCiHspkWo9dqg2gt+6Pa1OLn71rbkoa6o+erta/Ged18CgTP7slKQkhoBWrx10/XSz4cJPwkhqwZd+jxh4tpYnpeQVtVKVijlfEvQ3zSWlhwB/8Y3gMnJaDNPu60TZocphBVj8tSjTJrx2hNvLDRQKQuqlVKkhaij0umae70wtvtGNlSr+LY/Hqo5HSFN+QnTDYF4CGorm2a9tU21COAI5Nat4bZJXGq15BOSbNHXMcpsU++EKq/VU0AKEbGnUYLnF6U2lhTe7qCoA46od+qPoL+lfHnzda3nNNxJlHJucKbvKaxVtySrtrJRjqdFzVvNktSeqdebfVuyYmYmWsTdh/3bCyHsaZTgBdk5nUQA/JsbRlAN6enix1C1YhThoHr1XxtaGP96sYE9B08YZ5ayVt2SqA260j6PiTNnWptn1evJhV3P+jx5Mlzc0/T4o1hbphtg1BtED1EIYQeaPvkrPh66DXlFowrAcy+dxvH/+IeRxL1SEtz/sWsxfsNIm7iHfVsJmj0adINjVYwlfnXeIo6QuP3dqL6vd/9bbmk/T6XivO6H90YwP29fhujHihVO1CsCDAyEV92k6fEvLDgzTm2uW9ANMO/FSTpEKsIuIo+JyK9E5OdpHC8P/OycrFKnWkyj2D57PunUpH9583UXWyjYflsJsq5Mos8OmhFwt5UFWhv3aH932zb/BOu2bf5ib1oEesuW1iTi178OPPGEf+fHW26x6/JoQ6XiJGXdic08sElMuydXeSmoJZPKBCUR+X0AZwE8oZT6QNj+nZiglAbe2u2brl6NqRdOWdsxlVJwN0UTYZ0TTfuHEVSLbnrNr1Nm3MXCCaI3zPJ2b9NdE3UDLC+mVYP8Oj9+7nOtvc/jIuJMUOpEItaLbWOxoKZhYfTQxKbMuzuKyBiA7/WysPvxpX0vWot7rVLCQkDUfVmtgrcbF4yiadN+uATgK7evDRXZJALNyUkpklZnxVOn4q+YBDjfBNwTjHoFEae8MWyCkt5Pi3mU5cN6aEk9CnuK7Ds8i7ufPJLoGO6ZrkGi6RbVSyolvN24cPGmUq2U8OBtv2clsuybnjNhEWOUFrdB7QBsRWlgIF27xN06t5PU68DKla1rqJr2W1yMXnbZY0vqdZ2wi8hWAFsBYGRkZP1MkunNOWBrk/gxnEPUy77pOeK3EIWbWs3xxh9/vHUfU5Spm3d5rQ/34hZhC274zdbsBbSX734ctU2BiSi93Du1+HdEuq4fu1Jqr1Jqg1Jqw+rVq7M6bWqY1lwNQ4BYVTombNcbZd/0HAmaOKRnST7ySPu6nZ//fHvSc2Cg2YbXja4fB+xmuabdTiArGo32x0lFvVx2Zq56O0maqpSymkmcIrRiIuD1nk3LyLmJOvs1zKbx+uZ6Zql3EhKToDli8tVt/HB3ZBiUoNQWjCk567VoetVj7wQTE86N1Y3pW5bp2xKQizefqRUjIt8E8AcAVgH4fwDuU0p9zbR/rwq7l7BkZxQhtRFikx0kAB7ySagyCZoTtmIb9zhA8yYRlJwdHW21DD784fC+6/1Ave5E6+6baKkUPQeRQzVNplaMUupPlVLvVUpVlFJXBol6kfDOeL2sVrk4GzTq7FebxltBk4f8GnQlnbRFYuI3QSlO98GglgF60k3Q5JuZGWcSz6pVjoi9/HK08xeVuTnnJpe0pj9syUFNDg3ICtEELE9069+khHVcBMzNzoLeT3JAR8hJk21BPdT1pBubXuVzc04L3F5fVMamiZlOTB84EFzDntU3l5wakBWmpUCvY5Ps3L5pDRt09Qq6f8qFC9E6EbrZtctczXLgQNNKcPcqN9Hrog44LRCC0AnlRx5xrvnkZGfHc+ZM+D45NSCjsHcJNh0qN68bjtUbhnQ5pq/q4+NmQZ6ZcaLwvKf0Z0nYZ5ybc2bXrlrlXMs44ul3IzXdXG2asGXV6dMDhb1LsO1QGac3DOliwkrpgjomFiEKT5u33nIEXl/LILyCbVrEw68M1TZnklWnTy9Kqcy39evXK0KIUmp0VClHhlq30VHn9clJpWq11tdE/N/TbZuIUgMDwa9/6EPtny+rsU1MONdZxPk5OWn+O01O2u/rfZ/389Vq9u/3AGBaWWgsk6eE5InNV/VqtdWnjRqpe2e0hvVRqdeBN9+M19JXH1uXWgLAXXf5Jz2VAn70o2ayU9fuv/FGZ8sIRZwo3FvLHsT4eLw8SVqJ9KjYqH/aGyN2Qpap1/0jynrdP9qLsunI0BttfuhDwe+bmDCPy2bzi2gnJoL3d5P2N5JSqf269ihgxE5Ij2O7pqkf5XJrcyt3hDg2FvzeAwfsKj5MzMwAd97pROpnzoS3+PV+awkq84yDO/pf7I+yYCZPCUlC0sknJgE9cyZ+5cTgIDA05FTN+I0p7LjaEknCuXPNJGZY7bk3kbhrl/MZ4lIqOXaLXwloQRfW8EJhJyQuaTSHMlVHXH65eYk7P3SFR73eFFO/MU1NhR93ZAR4+237cydBpOnF65vkHXcka/SllBOlm3z6TpUa5jDD1IiNX5P2Ro+dFIKwihYb/Hz0SkWpwcF43nbQmGw9+8nJdD3uoG3FimYeIEo+YXTUnAfQ1z+Nv0+Sv2OC6hcTsPTYKeyExMWU5BOJdhxvcjNu4lIkeEwmoXNv9bozpqyEXQtglM8cVArqFtOMxFYpldlNxFbYacUQEpe0Jp942w8EJS5HR5utZP3OGzQmGwviU59yfprO0QkWFuzXTx0cbFo37kXD9WQinTD2a7fgfj1tcpphaoLCTkhc0uri6CWsY+ObbzorCfmdN2hMNjecAwecnw8/bJ/AzHJ1pksvbX3s15PHnfsAnFYE+hokFXWTj57XDFMTNmF92hutGFIY4s5IDDtmmN9cr5vPaxqTzXHdNpL7OO5acO84Jiftas/L5eDPY+uxh9kpnbJFgqwdeuwUdkJC0aIaJsJRbyZhiVGT+E1OtrcHGBiwG6ffuP0e1+tOMjWKz+5HWrkPLzbtH9K+yXuwFXZaMYR0I9piCGsCFrXEcnzcfEy/0kNtOTz/fLvloh9H9ZGVar7X3d5Al2hOTDR9cxM2i5DYPm9LmI+eRqvmtLBR/7Q3RuyEWGJbBlgu20eKpsZiExPBr5ui1agRe5g1447G49gqnbLYrEiCAAANcElEQVRFsiyfNABaMYQUBPdX/DQ8aO8xvTeDKEItkrynTZBlYlPSaMonZJH76FT5pIFMhR3AzQBOAHgZwI6w/SnshMTEVnSTRJFRmnD5+ctBSdKoEbv32GHJ4E4LbQY+ehCZCTuAMoD/A+C3AAwCOArgmqD3UNgJiYltdJwkUWi6eXgF3ySiNmOs1RzrJ4kwd4E1EkgHbgK2wp5G8vSDAF5WSv2TUuocgL8B8EcpHJf0MPsOz+LG3c/iqh37cePuZ7Hv8GzeQyoG3kk5prVOkyQKTbXwn/+8/2SgsDGOjjYTooAz5oUFp2Z+yxa7Y/rRZZOCWkijj1ACxLkJJDiAyCcA3KyU+uzy4zsA/Aul1BdM79mwYYOanp5OdF7Svew7PIt7n3kRi43mGpXVSplL+HUCLSDu9r61WvIZlnrmZpqLQ6Q91rEx//a+o6NOVUqedGhsInJIKbUhbL80Ina/mqS2u4WIbBWRaRGZPn36dAqnJd3KnoMnWkQdABYbS9hz8EROIyowQdPqkx437dI9v/7ySdrodmrmbxrk/G0ijYU2XgXwPtfjKwG85t1JKbUXwF7AidhTOC/JiX2HZ3H/d45hfrEBALisVsF9H732YjT+2rz/Ygam50lC4i7bljVpi53+zO6l96rVeMdKG9NiIRm1GEgjYv8JgN8RkatEZBDAnwD4TgrHJV3IvsOz2P6toxdFHQDeWGhg+38/ii/texE37n62/evaMlcMdcl/OpIPnZo45F4VaW4uUy/bSM7fJhILu1LqPIAvADgI4DiAp5RSx5Iel3Qnew6eQONCu3Q3lhQmXziFWUNUXq2UsX3TmpbnmGDtMzohdmnbO2nRKYvMklRaCiilDiil/rlS6v1KqS4wuPqTLIQyjp0yPFRtS5zqBOvs/CIUgNn5Rdz7zIsU9zzp9ApAnRC7bq6MybHFABezLgjeShQtlAB8K1H2HZ7FnoMn8Nr8Iq4YqmL7pjVWFStXDFWNUbkfAuD5HRvbng9KsLJyJge8FSu6PA9IV5DSzgfk7GV3K2wCVhBsKlF0RD+2Yz+++OSRlmj5i08ewZhFpL990xpUSvb9txWA9997AF/a92LL80ywdhlhlkY3refpppsrY3KEwl4QwoTSbX0A7fWo+rEWea8QazavG8aeT16PWsX+n86Scvx39zFNiVQmWHMiyNLIebJNIDl72d0Khb0ghAmlX0RvQgGYeuHUxcjd691Pz5yB8p2+EMw3f/yLi7/7Rf6VkrQlWElGBFWsdGuCUtNN7XK7BAp7Qdi+aQ2qlfbp5Wfeegfr/uL7kXxxwBH3PQdP+CY5J184ZX2TcLOk1MWbxN1PHmmrrrkQ+YgkNYIsjW5OUBJfKOwFYfO6YTx423W4rNa6FuZi4wLeWGgY3hXMa/OLkSL9MEqCFjvIy9IFhQe+y0rZXAiyNLptPU8SCoW9QGxeN4zaYLqFTlEj/SDKgtCbRNybEEkBk6XBBGXPQWEvGGlWlaTd96FBr6U3YYKy52Ade5cQt67cy1Ct0tNR71C1aSWldU1ICvRKPxoCgMLeFUSdXBR0nLNvn+/IGLOgBOD+j10LIL1rQkg/QiumC0irza2pj0svMFSt4Cu3r70o2qZrcs9TR9l2gJAQKOxdgMkXn51ftBIxXUKYZqIza1a8q/XLo+maLCnFnjKEhEArpgsI6r+y/VtH8cB3j2F+oYErhqq46erVeO6l0xd955uuXo2nD82mVpKYF16rJeiasKcMIcEkXhovDlwarxW/peRsEaRfvZInZREsKRX6uQTAK7tvzWhUhHQHWS6NRxKyed0wPr4+XvQZJH7DQ1V8+oYRDA9VYzQAyIel5UAj7GbFnjKEmKEV0wXsOzyLpw91xjPeMHo5vrz5OgDA2I79HTlH1gjgu2gHSyMJcaCwdwFpTtt3U8QSQQEwfoMzlX3tA99vWaJPU8TPTUgU6LF3AVft2G+0HiplQWOpSC56MoaqFV8x92N4qOq7yAchvQo99h7C5BcPD1Wx5xPXX/TItWfez9iKOmDuc8O1VknRoRXTBWzftKatKkYv/rx53XCbnfDcS6d7umY9K8rSnjLmjFbSDySK2EXkkyJyTEQuiEjo1wPij265647M3Ys/eyPMm65ene+Ae4QlH5sxrVm+hHQzSSP2nwO4DcB/SWEsfY1fZA74R5idqqApGsM+FhfXWiX9QKKIXSl1XCnFUKeDmCLMXqlLzwttZXnhWqukH8gseSoiW0VkWkSmT58+ndVpex5TJNkLdTKduPnI8jY8VMVXb1+Lk7tvxcndt+Krt681Wllu/JYQNN0ECOlVQq0YEfk7AO/xeWmnUupvbU+klNoLYC/glDtaj7Cg2E6oCeqZEpWSAFk2f1QATu6+Fev+4vup9Yh/yNUB0o3JyvLbDwAnM5FCEyrsSqkPZzGQfiJKZcb2TWtw95NHUjlvHh199x2exXyKC3/EEWC/myjr20mRYR17DkSpzNi8brhlVaFe455vHU3NNvJLhoahb6Kz84tQaN5EWbtOikyiqhgR+WMA/wnAagD7ReSIUmpTKiMrMFErM+7/2LWxuz/mzVLErwmVsmBpScG7PGqlLFY+uDc6Xzh33ngTpf1CikoiYVdKfRvAt1MaS99g8s1NlRluX7hoE5NufP/lODm3eFGI33rnvO/s0hWDA6FC7GdxmWB5IykytGJyIE5lxuZ1w3h+x8ZIdkSlJKiUu7sw8qenfo3tm9bgld234vkdG/FrQ8sA0/NuojRTY3kjKTIU9hwIm2kahN9NwUTjgsLKdw1gqFrp2l4z3txCkjpz2yic5Y2k6LBXTE7Yluf5vQ8A7nnqqO+UeS9vLDRQrZRbygT3/+yXqZUfpsHs/CKu2rHfuNSfrRCbLK6hagUr3jXA8kbSNzBi70E2rxvGX37qeuvI3RsV3/fRa7vOotEVK08fmsXH1w+n9m2mWinj/o9di+d3bMRDt68FAHzxySPs6kgKDSP2HiVqQtVtU/hN0jElLdNkqFrBO+cvBPrgi40lPPfS6Vh15kGTj9jVkfQTXGijANgshh226ITNMSolwcpLBvDGQiPyItrVShkP3uYs0aeF1/T+TixUfePuZ31vgFyMg/QStgttMGIvAN7o3Su6Nh61X7R709Wr8dxLp329aXe9+LurFfzm7YZxZuuw5736p0ls392BCVns6kj6CUbsBSSPhZ1Ny/sFRd/7Ds9i+7eOouG5I1TKgj2fuD7VMTNiJ0WAEXsfE7XiJo0bQdRJV3qcD3z3WFuFTmNJpT4zNGiVKkKKBqti+py0eqnEbYdrahCWtkWSZO4AIb0GI/Y+x9SQ7IHvHosUxcdthxsn0o9L3LkDhPQaFPY+xxQZv7HQuGiR2JYGxhFOWiSEpA+tmD7HNjLu1ILPtEgISR9G7H2OX8RsolOlgbRICEkXCnufE2UWKjsiEtIbUNhJW8TsNwuVvjchvQOFnbTBBZ8J6W0o7MQX+t6E9C6siiGEkIKRSNhFZI+IvCQiPxORb4vIUFoDI4QQEo+kEfsPAHxAKfV7AP43gHuTD4kQQkgSEgm7Uur7Sqnzyw9fAHBl8iERQghJQpoe+50A/keKxyOEEBKD0KoYEfk7AO/xeWmnUupvl/fZCeA8gKmA42wFsHX54VkRSX9+ejasAvB63oPoEngtHHgdHHgdHDp5HUZtdkq80IaIbAHweQAfUkotJDpYDyAi0zaN7vsBXgsHXgcHXgeHbrgOierYReRmAP8BwL/qB1EnhJBeIKnH/p8BXArgByJyREQeTWFMhBBCEpAoYldK/XZaA+kh9uY9gC6C18KB18GB18Eh9+uQy2LWhBBCOgdbChBCSMGgsMdARD4pIsdE5IKI9F0VgIjcLCInRORlEdmR93jyQkQeE5FficjP8x5LXojI+0TkORE5vvx/4q68x5QXInKJiPyjiBxdvhYP5DUWCns8fg7gNgA/zHsgWSMiZQB/BeAPAVwD4E9F5Jp8R5Ub/w3AzXkPImfOA7hHKfW7AG4A8Od9/O/hHQAblVLXA1gL4GYRuSGPgVDYY6CUOq6U6tUJVkn5IICXlVL/pJQ6B+BvAPxRzmPKBaXUDwGcyXsceaKU+qVS6qfLv78J4DiAvuz3rBzOLj+sLG+5JDEp7CQqwwB+4Xr8Kvr0PzJpRUTGAKwD8ON8R5IfIlIWkSMAfgXgB0qpXK4FF9owYNNKoU8Rn+dYWtXniMhKAE8DuFsp9Zu8x5MXSqklAGuXW5h/W0Q+oJTKPAdDYTeglPpw3mPoUl4F8D7X4ysBvJbTWEgXICIVOKI+pZR6Ju/xdANKqXkR+Qc4OZjMhZ1WDInKTwD8johcJSKDAP4EwHdyHhPJCRERAF8DcFwp9ZW8x5MnIrJaLzYkIlUAHwbwUh5jobDHQET+WEReBfAvAewXkYN5jykrlvvvfwHAQTiJsqeUUsfyHVU+iMg3AfwIwBoReVVEPpP3mHLgRgB3ANi43FbkiIjckvegcuK9AJ4TkZ/BCYB+oJT6Xh4D4cxTQggpGIzYCSGkYFDYCSGkYFDYCSGkYFDYCSGkYFDYCSGkYFDYCSGkYFDYCSGkYFDYCSGkYPx/cJ4zHYpf0x0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1e6a1de6f98>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data = X  # Before feature scaling\n",
    "X = (X - X.mean())/X.std()  # Feature scaling\n",
    "Y[Y == 0] = -1  # Replace zeros with -1\n",
    "plt.scatter(x=X[:, 0], y=X[:, 1])  # After feature scaling\n",
    "plt.scatter(x=data[:, 0], y=data[:, 1], c='r')  # Before feature scaling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training\n",
    "\n",
    "Now let's go ahead and train our SVM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0, Loss: 4.120456695556641\n",
      "Epoch 1, Loss: 0.4514138102531433\n",
      "Epoch 2, Loss: 0.14271092414855957\n",
      "Epoch 3, Loss: 0.03346329927444458\n",
      "Epoch 4, Loss: 0.0\n",
      "Epoch 5, Loss: 0.0\n",
      "Epoch 6, Loss: 0.0\n",
      "Epoch 7, Loss: 0.0\n",
      "Epoch 8, Loss: 0.0\n",
      "Epoch 9, Loss: 0.0\n"
     ]
    }
   ],
   "source": [
    "learning_rate = 0.1  # Learning rate\n",
    "epoch = 10  # Number of epochs\n",
    "batch_size = 1  # Batch size\n",
    "\n",
    "X = torch.FloatTensor(X)  # Convert X and Y to FloatTensors\n",
    "Y = torch.FloatTensor(Y)\n",
    "N = len(Y)  # Number of samples, 500\n",
    "\n",
    "model = SVM()  # Our model\n",
    "optimizer = optim.SGD(model.parameters(), lr=learning_rate)  # Our optimizer\n",
    "model.train()  # Our model, SVM is a subclass of the nn.Module, so it inherits the train method\n",
    "for epoch in range(epoch):\n",
    "    perm = torch.randperm(N)  # Generate a set of random numbers of length: sample size\n",
    "    sum_loss = 0  # Loss for each epoch\n",
    "        \n",
    "    for i in range(0, N, batch_size):\n",
    "        x = X[perm[i:i + batch_size]]  # Pick random samples by iterating over random permutation\n",
    "        y = Y[perm[i:i + batch_size]]  # Pick the correlating class\n",
    "        \n",
    "        x = Variable(x)  # Convert features and classes to variables\n",
    "        y = Variable(y)\n",
    "\n",
    "        optimizer.zero_grad()  # Manually zero the gradient buffers of the optimizer\n",
    "        output = model(x)  # Compute the output by doing a forward pass\n",
    "        \n",
    "        loss = torch.mean(torch.clamp(1 - output * y, min=0))  # hinge loss\n",
    "        loss.backward()  # Backpropagation\n",
    "        optimizer.step()  # Optimize and adjust weights\n",
    "\n",
    "        sum_loss += loss[0].data.cpu().numpy()  # Add the loss\n",
    "        \n",
    "    print(\"Epoch {}, Loss: {}\".format(epoch, sum_loss[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Results\n",
    "\n",
    "![Imgur](https://i.imgur.com/oSLROan.png)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda root]",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}