nlp_text_preprocessing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text preprocessing and representation in NLP"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import pickle\n",
    "\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from gensim.models import Word2Vec\n",
    "\n",
    "import functions as f"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text preprocessing\n",
    "\n",
    "The goal of this part is to transform product reviews into lists of tokens. Since the dataset consists of over 1.4M reviews, for the purpose of this project it will be limited to 100k (to fasten computations).\n",
    "\n",
    "Text preprocessing includes:\n",
    "- converting text to lowercase\n",
    "- removing punctuation\n",
    "- tokenization\n",
    "- removing stop words\n",
    "- lemmatization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>review</th>\n",
       "      <th>sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>868012</th>\n",
       "      <td>Instead of the standard message board, this on...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>674370</th>\n",
       "      <td>my 2 yo loves washing dishes and putting them ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>900999</th>\n",
       "      <td>This was ordered for my twin grandsons 3rd bir...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>543468</th>\n",
       "      <td>Great! It provides hours of outdoors fun!</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>762935</th>\n",
       "      <td>Adorable Kyubey plush! It's bigger than I expe...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   review  sentiment\n",
       "868012  Instead of the standard message board, this on...          1\n",
       "674370  my 2 yo loves washing dishes and putting them ...          1\n",
       "900999  This was ordered for my twin grandsons 3rd bir...          1\n",
       "543468          Great! It provides hours of outdoors fun!          1\n",
       "762935  Adorable Kyubey plush! It's bigger than I expe...          1"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('data/reviews_toys_games.csv')\n",
    "df.sample(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Share of positive reviews in the whole dataset: 0.9147\n"
     ]
    }
   ],
   "source": [
    "print('Share of positive reviews in the whole dataset: %0.4f' % (sum(df['sentiment'])/len(df)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Share of positive reviews in the sample: 0.9148\n"
     ]
    }
   ],
   "source": [
    "df_sample = df.sample(100000, random_state=11).reset_index(drop=True)\n",
    "print('Share of positive reviews in the sample:', sum(df_sample['sentiment'])/len(df_sample))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "reviews = df_sample['review'].to_list()\n",
    "sentiment = df_sample['sentiment'].to_list()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "reviews_tokens = f.normalize_text(reviews)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "f.save_pickle(reviews_tokens, 'data/reviews_tokens_100k.pickle')\n",
    "f.save_pickle(sentiment, 'data/sentiment_100k.pickle')\n",
    "\n",
    "df_sample.to_csv('data/reviews_toys_games_100k.csv', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "del df, df_sample, reviews, reviews_tokens, sentiment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating word vectors\n",
    "\n",
    "There are several approaches aimed on transforming text into word vectors. In this script reviews will be presented as:\n",
    "- bag of words vectors\n",
    "- TF-IDF word vectors\n",
    "- co-occurrence matrix with SVD and NMF decomposition\n",
    "- Word2Vec embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "reviews_tokens = f.load_pickle('data/reviews_tokens_100k.pickle')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus_counter, n_tokens = f.distinct_corpus_words(reviews_tokens)\n",
    "word2ind, ind2word = f.build_dictionary(corpus_counter)\n",
    "reviews_ind = [f.text_token2ind(review, word2ind) for review in reviews_tokens]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Words in the dictionary:  45629\n",
      "Most common words:\n",
      " [('love', 42733), ('great', 26982), ('play', 20313), ('toy', 16773), ('old', 16037), ('like', 15440), ('buy', 14097), ('kid', 13786), ('game', 13429), ('fun', 13182)]\n"
     ]
    }
   ],
   "source": [
    "print('Words in the dictionary: ', n_tokens)\n",
    "print('Most common words:\\n', corpus_counter[:10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "f.save_pickle(reviews_ind, 'data/reviews_ind_100k.pickle')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bag of Words\n",
    "\n",
    "The bag of words model is the simplest representation of text used in NLP. It enables extracting features from text to use in machine learning algorithms. It consists of a vocabulary of known words and a count of the occurrence of words withing the document.\n",
    "\n",
    "In the bag of words model, each document is represented as a word-count vector and the size of this vector is equal to the number of elements in the vocabulary. Since the documents contain only limited number of words, vectors mostly consist of zeros, thus the matrix can be considered as sparse. To avoid memory errors, the output of this BOW implementation is a matrix represented in scipy sparse format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(100000, 6000)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "M_bow = f.build_bow(reviews_ind, n_tokens, 6000)\n",
    "M_bow.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "text of test review:  ['great', 'figure', 'right', 'price', 'love', 'amaze', 'figure', 'see']\n",
      "occurrences of words: \n",
      "figure: 2 \n",
      "love: 1\n"
     ]
    }
   ],
   "source": [
    "ind_test = 111\n",
    "print('text of test review: ', reviews_tokens[ind_test])\n",
    "print('occurrences of words:',\n",
    "      '\\nfigure:', M_bow[ind_test, word2ind['figure']],\n",
    "      '\\nlove:', M_bow[ind_test, word2ind['love']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Convert string to BOW vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "review_test = 'This game is amazing ^^, my son plays with it all the time!'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['game', 'amaze', 'son', 'play', 'time']\n",
      "[8, 171, 20, 2, 18]\n"
     ]
    }
   ],
   "source": [
    "review_tokens_test = f.normalize_single_text(review_test)\n",
    "print(review_tokens_test)\n",
    "\n",
    "review_ind_test = f.text_token2ind(review_tokens_test, word2ind)\n",
    "print(review_ind_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 0., 1., ..., 0., 0., 0.])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f.build_single_bow(review_ind_test, n_tokens, 6000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "f.save_pickle(M_bow, 'data/M_bow_100k.pickle')\n",
    "del M_bow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TF-IDF (Term Frequency – Inverse Document Frequency)\n",
    "\n",
    "TF-IDF is a statistical measure used to evaluate how important a word is to a document in a collection or corpus. The importance increases proportionally to the number of times a word appears in the document but is offset by the frequency of the word in the corpus. It is a weight often used in information retrieval and text mining.\n",
    "\n",
    "In order to optimize calculations, scikit-learn implementation was used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(100000, 6000)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vectorizer = TfidfVectorizer(max_features=6000)\n",
    "M_tfidf = vectorizer.fit_transform([' '.join(r) for r in reviews_tokens])\n",
    "M_tfidf.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "text of test review:  ['great', 'figure', 'right', 'price', 'love', 'amaze', 'figure', 'see']\n",
      "TFIDF of words: \n",
      "figure: 0.6571834159729407 \n",
      "love: 0.15079671030826042\n"
     ]
    }
   ],
   "source": [
    "ind_test = 111\n",
    "print('text of test review: ', reviews_tokens[ind_test])\n",
    "print('TFIDF of words:',\n",
    "      '\\nfigure:', M_tfidf[ind_test, vectorizer.vocabulary_['figure']],\n",
    "      '\\nlove:', M_tfidf[ind_test, vectorizer.vocabulary_['love']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Convert test string into TFIDF vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "matrix([[0., 0., 0., ..., 0., 0., 0.]])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vectorizer.transform([' '.join(review_tokens_test)]).todense()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "f.save_pickle(M_tfidf, 'data/M_tfidf_100k.pickle')\n",
    "f.save_pickle(vectorizer, 'data/tfidf_vectorizer_100k.pickle')\n",
    "\n",
    "del M_tfidf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Co-Occurrence Matrix\n",
    "\n",
    "Co-occurrence Matrix with SVD or NMF is one of the frequency-based approaches used to build word embeddings. The idea behind it is that similar words tend to occur together and will have a similar context. It is computed by counting how often two or more words occur together in a given context window.\n",
    "\n",
    "Co-occurrence matrix is not the word vector representation that is generally used. Instead, it is decomposed using techniques like SVD, NMF, PCA into factors and combination of these factors forms the word vector representation. Those decomposition methods are used to reduce a high-dimensional dataset into fewer dimensions while retaining important information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "M_co_occurrence = f.build_co_occurrence_matrix(reviews_ind, n_tokens, window_size=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Co-Occurrence Matrix with SVD (Singular Value Decomposition)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "svd_reduced_co_occurrence = f.matrix_reduce(M_co_occurrence, method='svd', n_dim=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(100000, 5)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "M_svd = np.stack([f.avg_cooc_embeddings(ind, svd_reduced_co_occurrence, word2ind) for ind in reviews_ind])\n",
    "M_svd.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1535.54419534  -88.85410665 -109.92193234  200.17761876  -67.75181056]\n",
      " [2649.78632729 -168.64916748 -184.34119014  522.55081926 -112.81073749]\n",
      " [5760.72399768  602.74718843 -429.02400627  807.77596857   74.03598068]\n",
      " ...\n",
      " [3241.88552377 -107.11166419 -308.16530576  931.15941756 -176.91970992]\n",
      " [1393.89739413 -832.52584809   16.83450896  -54.8309185  -696.31413259]\n",
      " [1413.60038863 -338.0630371    14.41326836 -189.74357591  238.11835686]]\n"
     ]
    }
   ],
   "source": [
    "print(M_svd)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### SVD word embeddings in vector space\n",
    "\n",
    "Although it seems that embeddings reflect some relations between words, eg \"son\" and \"daughter\" are close to each other, other word vectors don't follow the expected behaviour, eg \"game\" and \"toy\" seem to have nothing in common, \"good\" and \"bad\" are far from each other."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "viz_words = ['son', 'daughter', 'good', 'bad', 'great', 'game', 'toy', 'play']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAssAAAF1CAYAAAAeIKdDAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deZwV1Z3//9eHVdRGQBpopKGF+FUxi2IrII4SAwkiGJcQ0BGVxG0SNSSaRAMGHdcYydeZZCaOwVEEFROjX4X40wgJalxAROMScQ+LoOwIqDTI+f1xL20DXYDc3tDX8/HoR9+qc6rOqUvR/e5zT1VFSglJkiRJW2tU3x2QJEmSGirDsiRJkpTBsCxJkiRlMCxLkiRJGQzLkiRJUgbDsiRJkpTBsCxJVUTE5RExsb77UVsiYk1EdK3vfkjSrsKwLKnBiohLI+LBLda9nrFuWN32rmZERN+IWFBX7aWU9kwpvfVpt4uI/xMR90fEkohYHhEPR8T+Vcovj4j1EbE6//VaRPwmIkpq9ggkqW4ZliU1ZI8BfSKiMUBEdACaAj22WPeFfN0dFjm7/M/AiGhSR021Ah4A9gfaAzOB+7eoc3dKqQhoA5wIdACeNTBL2pXt8r8oJH2mPUMuHB+cXz4K+Cvw6hbr3kwpLQSIiCMi4pmIWJX/fsSmnUXE9Ii4OiKeAD4AukbEvhHxaH409BGgbVZnIuKViBhUZblJRCyNiB755V4R8WRErIyIv0dE3yp120TErRGxMCJWRMT/i4g9gP8P6JifHrEmIjpGRPOIuDFfd2H+dfP8fvpGxIKI+GlEvAvcGhFtI2JKvt3lEfF41h8CEZEi4gv517dFxH9FxJ/yxz8jIrpVt11KaWZK6ZaU0vKU0nrg/wL7R8Te1dRdn1J6GRgKLAEuyujLF/Lv/ar8+3j3Fv28MCLeypf9ctMxRUS3iPhLRCzLl90REa2qbFsaEffmR8GXRcRvqpR9J//vuCI/Ot6lur5J0iaGZUkNVkqpAphBLhCT//448Lct1j0GuUAK/An4T2Bv4FfAn7YIdMOBc4AiYC5wJ/AsuZB8JXDGNrp0F3BKleVvAEtTSrMjYp9821eRG1m9GPhjRBTn604AdgcOAtoB/zeltBY4FliYnx6xZz70jwJ6kfuD4CvA4cDoKu12yLfRJX8sFwELgGJyo74/A9I2jqOqU4ArgNbAG8DVO7jdUcC7KaVlWRVSSh+TG33+l4wqVwJ/zrfdCfj1FuUnAuVAD+CbwHfy6wO4FugIHAiUApcD5D9xmELu37YM2AeYlC87gdx7cxK59+pxcv+mkpTJsCypoXuUT4Lxv5ALOI9vse7R/OvjgNdTShNSShtSSncBc4DBVfZ3W0rp5ZTSBqAEOAy4LKW0LqX0GDB5G325Ezg+InbPL5+aXwdwGvBgSunBlNLGlNIjwCxgYH4awrHAeSmlFfmR10e32vsn/hX495TS4pTSEnJhdniV8o3AmHyfPwTW54+lS37fj6eUdjQs35sfNd4A3MEnI/aZIqIT8F/Aj3Zg/wvJBfvqrCcX+DumlD5KKf1ti/Jf5Eey5wE3kv9DJaX0RkrpkfzxLyH3R9HR+W0OJxeif5xSWrvFfs8Frk0pvZI/3muAgx1dlrQthmVJDd1jwJER0RooTim9DjwJHJFf90U+ma/ckdyIYlVzyY0ubjK/yuuOwIr8CG/V+tVKKb0BvAIMzgfm4/kkLHcBhuSnQqyMiJXAkeRCbCmwPKW0YgePecvjmJtft8mSlNJHVZZ/SW5U+M/5aQuX7GA7AO9Wef0BsOe2KudHyv8M/Hf+j5Ht2QdYnlH2E3KjxDMj4uWI+M4W5VX/rSrfg4hoFxGTIuKdiHgfmMgn02dKgbn5MLylLsB/VPn3WZ5vf59q6koSYFiW1PA9BexFbrrBEwAppffJjVieQ24Kw9v5ugvJBaKqOgPvVFmuOuK6CGidnztctf62bJqK8U3gH/kADblgNyGl1KrK1x4ppevyZW2qzqvN6M8mWx5H5/y6ardJKa1OKV2UUupKbhT9RxHxte0cx6eW/+Pkz8ADKaXtTtfIzzEeTO6TgK2klN5NKZ2dUupIbtT3vzfNp84rrfK66ntwLbn34MsppZbkRvUjXzYf6Jxx4eN84Nwt/o1apJSe3N6xSPr8MixLatDy0wxmkfvIv2ro+lt+XdW7YDwI/J+IODV/8d1QoDu5OazV7Xtuft9XRESziDiSzadsVGcS8HXg3/hkVBlyo5uDI+IbEdE4InbLX4zXKaW0iNyFfP8dEa0jomlEbJpG8h6wd0TsVWVfdwGjI6I4ItoCP8/vv1oRMSh/sVwA7wMf579qTES0BB4GnkgpbXPkOn98B5I7jg7kpklUV29IfkoHwApyAbhqv3+cf79KgR8Amy4ALALWACvzc8V/XGWbmeT+CLouIvbI/zv0yZfdBFwaEQfl298rIobsyPFL+vwyLEvaFTxK7qK4qnNaH8+vqwzL+YvNBpG74G0ZuY/5B6WUlm5j36cCPcl9JD8GuH1bHckH36eAI/gkvJFSmk9utPln5O4AMZ9ciNv0c3Y4uTm6c4DFwMj8dnPIhcq38tMDOpK7SHAW8ALwIjA7vy7LfsBUcgHyKXJTJKZv6zh2wonk5nePqHLnjjURUXUkfmhErAFWkrvN3DLg0E13KqnGYcCM/DYPAD+o8ikB5C4OfBZ4ntzFk7fk119B7qK/Vfn1927aIH9R4WBytxOcR+7Cx6H5svuAXwCT8tM3XiI3l1ySMsWOXwMiSVLdiIgE7Fdlmosk1QtHliVJkqQMhmVJkiQpg9MwJEmSpAyOLEuSJEkZDMuSJElShupu2t5gtG3bNpWVldV3NyRJkvQZ9uyzzy5NKRVXV9agw3JZWRmzZs2q725IkiTpMywi5maVOQ1DkiRJymBYliRJkjIYliVJkqQMhmVJkiQpg2FZkiRJymBYliRJkjIYliVJksT06dPp1KlTfXejwTEsS5IkSRkMy5IkSVIGw7IkSdLnSFlZGddeey3du3endevWjBgxgo8++miretdddx3dunWjqKiI7t27c9999wGwbt062rRpw4svvlhZd/HixbRo0YIlS5bU2XHUFcOyJEnS58wdd9zBww8/zJtvvslrr73GVVddtVWdbt268fjjj7Nq1SrGjBnDaaedxqJFi2jevDnDhg1j4sSJlXXvuusu+vXrR3FxcV0eRp0wLEuSJH3OnH/++ZSWltKmTRtGjRrFXXfdtVWdIUOG0LFjRxo1asTQoUPZb7/9mDlzJgBnnHEGd955Jxs3bgRgwoQJDB8+vE6Poa40qe8OSJIkqXZNnbOEcU/MZ/HqCpasqWAZLSvLunTpwsKFC7fa5vbbb+dXv/oV//znPwFYs2YNS5cuBaBnz57ssccePProo5SUlPDGG29w/PHH18mx1DXDsiRJ0mfY1DlLuGHq26zbkBsF/nhj4s7pL9Czb3/6HVDMvHnz6Nix42bbzJ07l7PPPptp06bRu3dvGjduzMEHH0xKqbLOGWecwcSJE+nQoQPf+ta32G233er0uOqKYVmSJOkzbNwT8yuD8iZzH/sD/zn5aHq068k111zD0KFDNytfu3YtEVE5B/nWW2/lpZde2qzO8OHD+fKXv0xRURETJkyo3YOoR85ZliRJ+gxbvLpiq3XtenydaWO/R9euXenatSujR4/erLx79+5cdNFF9O7dm/bt2/Piiy/Sp0+fzep06tSJHj16EBH8y7/8S60eQ31yZFmSJOkzrF1RM97bIjAXlR7I4SeezaTv9qhc17dvXxYsWFC5fPXVV3P11Vdvc9+dO3emV69eRETNdroBqZGR5YgYEBGvRsQbEXFJNeUHRMRTEbEuIi6uiTYlSZK0fWf1KaV5k80jX7PGjTirT2lB+/3nP//Jvffey3e/+92C9tPQFRyWI6Ix8F/AsUB34JSI6L5FteXAhcANhbYnSZKkHdfvgGIu7rcv7YuaEUDjRsGQHh3od8DO3xP5sssu44tf/CI//vGP2XfffWuusw1QVL2qcad2ENEbuDyl9I388qUAKaVrq6l7ObAmpbRDobm8vDzNmjWroP5JkiSpZpSVlTFu3Dj69etX312pURHxbEqpvLqympiGsQ8wv8rygvw6SZIkaZdWE2G5uhndOz1cHRHnRMSsiJj1WXy+uCRJ0q5o+PDhzJs3j8GDB7Pnnnty/fXX88ADD3DQQQfRqlUr+vbtyyuvvALAL3/5S04++eTNtr/gggsYOXJkfXS9IDURlhcAVWeIdwK2fgzMDkop3ZxSKk8plX8Wny8uSZK0K5owYQKdO3dm8uTJrFmzhhNOOIFTTjmFG2+8kSVLljBw4EAGDx5MRUUFp512Gg899BArV64EYMOGDdx999275COxayIsPwPsFxH7RkQzYBjwQA3sV5IkSQ3U3XffzXHHHUf//v1p2rQpF198MR9++CFPPvkkJSUlHHXUUfzhD38A4KGHHqJt27Yceuih9dzrT6/gsJxS2gCcDzwMvAL8PqX0ckScFxHnAUREh4hYAPwIGB0RCyKiZfZeJUmS1JAtXLiQLl26VC43atSI0tJS3nnnHeCTx2EDTJw4cZccVYYaeihJSulB4MEt1t1U5fW75KZnSJIkaRcydc4Sxj0xn8WrK1i6dj2z562kH9CxY0defPHFynopJebPn88+++Tu83DCCSfwb//2b7z00ktMmTKF66+/vp6OoDA+7lqSJEnVmjpnCTdMfZv3VleQgEZ7tGb8I88ydc4Svv3tb/OnP/2JadOmsX79esaOHUvz5s054ogjANhtt9341re+xamnnsrhhx9O586d6/dgdpJhWZIkSdUa98R81m3YWLnc5Wun8+ZDtzKwR1cmT57MxIkTueCCC2jbti2TJ09m8uTJNGvWrLL+GWecwYsvvrjLTsGAGpqGIUmSpM+exasrNltu+6WjaPulowjg4pG9ADjxxBMzt+/cuTMtWrTY6jZyuxJHliVJklStdkXNPtX6qjZu3MivfvUrhg0bRsuWu+59HRxZliRJUrXO6lPKDVPf3mwqRvMmjTirT+k2toK1a9fSvn17unTpwkMPPVTb3axVhmVJkiRVq98BuQfEbbobRruiZpzVp7RyfZY99tiDNWvW1EUXa51hWZIkSZn6HVC83XD8WeacZUmSJCmDYVmSJEnKYFiWJEmSMhiWJUmSpAyGZUmSJCmDYVmSJEnKYFiWJEmSMhiWJUmSpAyGZUmSJCmDYVmSJEnKYFiWJEmSMhiWJUmSpAyGZUmSJCmDYVmSJEnKYFiWJEmSMhiWJUmSpAyGZUmSJCmDYVmSJEnKYFiWJEmSMhiWJUmSpAyGZUmSJCmDYVmSJEnKYFiWJEmSMhiWJUmSpAyGZUmSJCmDYVmSJEnKYFiWJEmSMhiWJUmSpAyGZUmSJCmDYVmSJEnKYFiWJEmSMhiWJUmSpAyGZUmSJCmDYVmSJEnKYFiWJEmSMhiWJUmSpAyGZUmSJCmDYVmSJEnKYFiWJEmSMhiWJUmSpAw1EpYjYkBEvBoRb0TEJdWUR0T8Z778hYjoURPtSpIkSbWp4LAcEY2B/wKOBboDp0RE9y2qHQvsl/86B/htoe1KkiRJta0mRpYPB95IKb2VUqoAJgHf3KLON4HbU87TQKuIKKmBtiVJkqRaUxNheR9gfpXlBfl1n7YOABFxTkTMiohZS5YsqYHuSZIkSTunJsJyVLMu7USd3MqUbk4plaeUyouLiwvunCRJkrSzaiIsLwBKqyx3AhbuRB1JkiSpQamJsPwMsF9E7BsRzYBhwANb1HkAOD1/V4xewKqU0qIaaFuSJEmqNU0K3UFKaUNEnA88DDQG/jel9HJEnJcvvwl4EBgIvAF8AIwotF1JkiSpthUclgFSSg+SC8RV191U5XUCvl8TbUmSJEl1xSf4SZIkSRkMy5IkSVIGw7IkSZKUwbAsSZKkz43bbruNI488cofrG5YlSZK0y9iwYUOdtmdYliRJUr2bPXs2hxxyCEVFRQwZMoShQ4cyevRopk+fTqdOnfjFL35Bhw4dGDEidwfiKVOmcPDBB9OqVSuOOOIIXnjhhcp9XXfddXTr1o2ioiK6d+/OfffdB8Arr7zCeeedx1NPPcWee+5Jq1atttsvw7IkSZLqVUVFBSeeeCJnnnkmy5cv55RTTqkMuADvvvsuy5cvZ+7cudx8883Mnj2b73znO/zP//wPy5Yt49xzz+X4449n3bp1AHTr1o3HH3+cVatWMWbMGE477TQWLVrEgQceyE033UTv3r1Zs2YNK1eu3G7fDMuSJEmqV08//TQbNmzgwgsvpGnTppx00kkcfvjhleWNGjXiiiuuoHnz5rRo0YLf/e53nHvuufTs2ZPGjRtzxhln0Lx5c55++mkAhgwZQseOHWnUqBFDhw5lv/32Y+bMmTvVN8OyJEmS6sXUOUsYdsts/m3co3zUrDXTXl1aWVZaWlr5uri4mN12261yee7cuYwdO5ZWrVpVfs2fP5+FCxcCcPvtt1dO0WjVqhUvvfQSS5d+su9Po0ae4CdJkiR9GlPnLOGGqW+zbsNGmrXcmzUr3uOXj7wFQL8Dipk/fz7dunUDICI227a0tJRRo0YxatSorfY7d+5czj77bKZNm0bv3r1p3LgxBx98MLkHSm+9r+1xZFmSJEl1btwT81m3YSMALcu+RERj3vrr77n5sbe5//77tzlt4uyzz+amm25ixowZpJRYu3Ytf/rTn1i9ejVr164lIiguLgbg1ltv5aWXXqrctn379ixYsICKiood6qdhWZIkSXVu8epPwmqjJk05aMS1LJoxmXsv6MvEiRMZNGgQzZs3r3bb8vJyfve733H++efTunVrvvCFL3DbbbcB0L17dy666CJ69+5N+/btefHFF+nTp0/ltscccwwHHXQQHTp0oG3bttvtZ2wakm6IysvL06xZs+q7G5IkSaphw26ZzXurtx7dbV/UjEnf7UHPnj0577zzKm8VV5si4tmUUnl1ZY4sS5Ikqc6d1aeU5k0+iaIr35gNH6xgRM8Sxo8fzwsvvMCAAQPqsYc5XuAnSZKkOtfvgNyc4nFPzGfx6gqavL+Q5yeN4aSrP6Bbt27cc889lJSU1HMvnYYhSZKkzzmnYUiSJEk7wbAsSZIkZTAsS5IkSRkMy5IkSVIGw7IkSZKUwbAsSZIkZTAsS5IkSRkMy5IkSVIGw7IkSZKUwbAsSZIkZTAsS5IkSRkMy5IkSVIGw7IkSZKUwbAsSZIkZTAsS5IkSRkMy5IkSVIGw7IkSZKUwbAsSZIkZTAsS5IkSRkMy5IkSVIGw7IkSZKUwbAsSZIkZTAsS5IkSRkMy5IkSVIGw7IkSZKUwbAsSZIkZTAsS5IkSRkMy5IkSVIGw7IkSZKUwbAsSZIkZTAsS5IkSRkKCssR0SYiHomI1/PfW2fU+9+IWBwRLxXSniRJklSXCh1ZvgSYllLaD5iWX67ObcCAAtuSJEmS6lShYfmbwPj86/HACdVVSik9BiwvsC1JkiSpThUaltunlBYB5L+3K7RDEXFORMyKiFlLliwpdHeSJEnSTmuyvQoRMRXoUE3RqJrvDqSUbgZuBigvL0+10YYkSZK0I7YbllNK/bLKIuK9iChJKS2KiBJgcY32TpIkSapHhU7DeAA4I//6DOD+AvcnSZIkNRiFhuXrgP4R8TrQP79MRHSMiAc3VYqIu4CngP0jYkFEfLfAdiVJkqRat91pGNuSUloGfK2a9QuBgVWWTymkHUmSJKk++AQ/SZIkKYNhWZIkScpgWJYkSZIyGJYlSZKkDIZlSZIkKYNhWZIkScpgWJYkSZIyGJYlSZKkDIZlSZIkKYNhWZIkScpgWJYkSZIyGJYlSZKkDIZlSZIkKYNhWZIkScpgWJYkSZIyGJYlSZKkDIZlSZIkKYNhWZIkScpgWJYkSZIyGJYlSZKkDIZlSZIkKYNhWZIkScpgWJYkSZIyGJYlSZKkDIZlSZIkKYNhWZIkScpgWJYkSZIyGJYlSZKkDIZlSZIkKYNhWZIkScpgWJYkSZIyGJYlSZKkDIZlSZIkKYNhWZIkScpgWJYkSZIyGJYlSZKkDIZlSZIkKYNhWZIkScpgWJYkSZIyGJYlSZKkDIZlSZIkKYNhWZIkScpgWJYkSZIyGJYlSZKkDIZlSZIkKYNhWZIkScpQUFiOiDYR8UhEvJ7/3rqaOqUR8deIeCUiXo6IHxTSpiRJklRXCh1ZvgSYllLaD5iWX97SBuCilNKBQC/g+xHRvcB2JUmSpFpXaFj+JjA+/3o8cMKWFVJKi1JKs/OvVwOvAPsU2K4kSZJU6woNy+1TSosgF4qBdtuqHBFlwCHAjALblSRJkmpdk+1ViIipQIdqikZ9moYiYk/gj8DIlNL726h3DnAOQOfOnT9NE5IkSVKN2m5YTin1yyqLiPcioiSltCgiSoDFGfWakgvKd6SU7t1OezcDNwOUl5en7fVPkiRJqi2FTsN4ADgj//oM4P4tK0REALcAr6SUflVge5IkSVKdKTQsXwf0j4jXgf75ZSKiY0Q8mK/TBxgOHBMRz+e/BhbYriRJklTrtjsNY1tSSsuAr1WzfiEwMP/6b0AU0o4kSZJUH3yCnyRJkpTBsCxJkiRlMCxLkiRJGQzLkiTtwsrKypg6dWrB+7n88ss57bTTaqBH0meLYVmSJEnKYFiWJEmSMhiWJUnaxT3zzDN0796d1q1bM2LECD766CNWrFjBoEGDKC4upnXr1gwaNIgFCxZUbvP2229z9NFHU1RURP/+/Vm6dGk9HoHUcBmWJUnaxd1xxx08/PDDvPnmm7z22mtcddVVbNy4kREjRjB37lzmzZtHixYtOP/88yu3OfXUUzn00ENZunQpl112GePHj6/HI5Aarkgp1XcfMpWXl6dZs2bVdzckSWqwysrKuOSSSzjvvPMAePDBB7ngggt48803N6v3/PPP89WvfpUVK1Ywb948unbtyqpVq9hjjz2AXHhu1KgREydOrPNjkOpbRDybUiqvrqygJ/hJkqS6N3XOEsY9MZ/FqytYsqaCZbSsLOvSpQsLFy7kgw8+4Ic//CEPPfQQK1asAGD16tV8/PHHLFy4kNatW1cG5U3bzZ8/v86PRWronIYhSdIuZOqcJdww9W3eW11BAj7emLhz+gtMnbMEgHnz5tGxY0fGjh3Lq6++yowZM3j//fd57LHHAEgpUVJSwooVK1i7dm3lfufNm1cfhyM1eIZlSZJ2IeOemM+6DRs3Wzf3sT/wn5NnsXz5cq655hqGDh3K6tWradGiBa1atWL58uVcccUVlfW7dOlCeXk5Y8aMoaKigr/97W9Mnjy5rg9F2iUYliVJ2oUsXl2x1bp2Pb7OtLHfo2vXrnTt2pXRo0czcuRIPvzwQ9q2bUuvXr0YMGDAZtvceeedzJgxgzZt2nDFFVdw+umn19UhSLsUL/CTJGkXMuyW2bxXTWBuX9SMSd/tUQ89knZ927rAz5FlSZJ2IWf1KaV5k81/fTdv0oiz+pTWU4+kzzbvhiFJ0i6k3wHFAJV3w2hX1Iyz+pRWrpdUswzLkiTtYvodUGw4luqI0zAkSZKkDIZlSZIkKYNhWZIkScpgWJYkSZIyGJYlSZKkDIZlSZIkKYNhWZIkScpgWJYkSZIyGJYlSZKkDIZlSZIkKYNhWZIkScpgWJYkSZIyGJYlSZKkDIZlSZIkKYNhWZIkScpgWJYkSZIyGJYlSZKkDIZlSZIkKYNhWZIkScpgWJYkSZIyGJYlSZKkDIZlSZIkKYNhWZIkScpgWJYkSZIyGJYlSZKkDIZlSZIkKYNhWZIkScpgWJYkSZIyGJYlSZKkDIZlSZIkKUNBYTki2kTEIxHxev5762rq7BYRMyPi7xHxckRcUUibkiRJUl0pdGT5EmBaSmk/YFp+eUvrgGNSSl8BDgYGRESvAtuVJEmSal2hYfmbwPj86/HACVtWSDlr8otN81+pwHYlSZKkWldoWG6fUloEkP/errpKEdE4Ip4HFgOPpJRmFNiuJEmSVOuabK9CREwFOlRTNGpHG0kpfQwcHBGtgPsi4osppZcy2jsHOAegc+fOO9qEJEmSVOO2G5ZTSv2yyiLivYgoSSktiogSciPH29rXyoiYDgwAqg3LKaWbgZsBysvLna4hSZKkelPoNIwHgDPyr88A7t+yQkQU50eUiYgWQD9gToHtSpIkSbWu0LB8HdA/Il4H+ueXiYiOEfFgvk4J8NeIeAF4htyc5SkFtitJkiTVuu1Ow9iWlNIy4GvVrF8IDMy/fgE4pJB2JEmSpPrgE/wkSZKkDIZlSZIkKYNhWZIkScpgWJYkSZIyGJYlSZKkDIZlSZIkKYNhWZIkScpgWJYkSZIyGJYlSZKkDIZlSZIkKYNhWZIkScpgWJYkSZIyGJYlSZKkDIZlSZIkKYNhWZIkScpgWJYkSZIyGJYlSZKkDIZlSZIkKYNhWZIkScpgWJYkSZIyGJYlSZKkDIZlSZIkKYNhWWqgZs+ezSGHHEJRURFDhgxh6NChjB49mhUrVjBo0CCKi4tp3bo1gwYNYsGCBZXb9e3bl9GjR3PEEUew5557MnjwYJYtW8a//uu/0rJlSw477DD++c9/VtafM2cO/fv3p02bNuy///78/ve/r4ejlSSpYTIsSw1QRUUFJ554ImeeeSbLly/nlFNO4b777gNg48aNjBgxgrlz5zJv3jxatGjB+eefv9n2kyZNYsKECbzzzju8+eab9O7dmxEjRrB8+XIOPPBArrjiCgDWrl1L//79OfXUU1m8eDF33XUX3/ve93j55Zfr/JglSWqIDMtSA/T000+zYcMGLrzwQpo2bcpJJ53E4YcfDsDee+/NySefzO67705RURGjRo3i0Ucf3Wz7ESNG0K1bN/baay+OPfZYunXrRr9+/WjSpAlDhgzhueeeA2DKlCmUlZUxYsQImjRpQo8ePTj55JO555576vyYJUlqiJrUdwckfWLqnCWMe2I+Lz76KB81a820V5fS74BiAEpLSwH44IMP+OEPf8hDDz3EihUrAFi9ejUff/wxjRs3BqB9+/aV+2zRosVWy2vWrAFg7gxsTc0AABDGSURBVNy5zJgxg1atWlWWb9iwgeHDh9fugUqStIswLEsNxNQ5S7hh6tus27CRZi33Zs2K9/jlI28B0O+AYubPn0+3bt0YO3Ysr776KjNmzKBDhw48//zzHHLIIaSUPnWbpaWlHH300TzyyCM1fTiSJH0mOA1DaiDGPTGfdRs2AtCy7EtENOatv/6emx97m/vvv5+ZM2cCuVHkFi1a0KpVK5YvX145/3hnDBo0iNdee40JEyawfv161q9fzzPPPMMrr7xSI8ckSdKuzrAsNRCLV1dUvm7UpCkHjbiWRTMmc+8FfZk4cSKDBg2iefPmjBw5kg8//JC2bdvSq1cvBgwYsNNtFhUV8ec//5lJkybRsWNHOnTowE9/+lPWrVtXE4ckSdIuL3bmo9u6Ul5enmbNmlXf3ZDqxLBbZvNelcC8SfuiZkz6bg969uzJeeedx4gRI+qhd5IkfXZFxLMppfLqyhxZlhqIs/qU0rzJJ/8lV74xGz5YwYieJYwfP54XXnihoFFkSZL06XmBn9RAbLrrxbgn5rN4dQVN3l/I85PGcNLVH9CtWzfuueceSkpK6rmXkiR9vjgNQ5IkSZ9rTsOQJEmSdoJhWZIkScpgWJYkSZIyGJYlSZKkDIZlSZIkKYNhWQU588wzGT16dK22cfnll3PaaafVahuSJEnVMSyrRvziF79gn332oaioiP33359p06axbt06Ro4cSceOHenYsSMjR46sfIzy9OnT6dSpE2PHjqVdu3aUlJRw66237lTbZWVlTJ06tSYPR5IkCTAsqwYsW7aM3/zmNzzzzDOsXr2ahx9+mLKyMq6++mqefvppnn/+ef7+978zc+ZMrrrqqsrt3n33XVatWsU777zDLbfcwve//31WrFhRp31PKbFx48Y6bVOSJO06DMv6VJ577jl69OhBUVERQ4cO5aOPPiIi+PDDDznppJMoLi7mkEMO4Qc/+AHjx4/n5z//Oe3ateOwww5j8ODBTJgwAYDbbruNlBI///nPadq0KUuXLmXdunWUlZVx5ZVXbjVaXFFRwemnn05RUREHHXQQmx5WM3z4cObNm8fgwYPZc889uf766wF4+umnOeKII2jVqhVf+cpXmD59euW++vbty6hRo+jTpw+77747b731Vt29gZIkaZdiWNYOq6io4IQTTmD48OEsX76cIUOG8Mc//pE2bdpw9dVXs3LlStavX88xxxwDwIIFC+jSpUvl9u3bt2fhwoWVy82bN6dJkyb84x//4Hvf+x7FxcVMmjSpcrS5qgceeIBhw4axcuVKjj/+eM4//3wAJkyYQOfOnZk8eTJr1qzhJz/5Ce+88w7HHXcco0ePZvny5dxwww2cfPLJLFmypHJ/EyZM4Oabb2b16tWb9VGSJKkqw7J22NNPP8369esZOXIkTZs2pdUXj2avLt2ZOPMd/trkMH7z/x5n3rx5NGvWjE2PUZ87d27l9osXL6Zjx45b7feee+5h8ODB7LbbbjRt2pR///d/JyI2q3PkkUcycOBAGjduzPDhw/n73/+e2c+JEycycOBABg4cSKNGjejfvz/l5eU8+OCDlXXOPPNMDjroIJo0aULTpk0LfWskSdJnVJP67oAatqlzljDuifksXl3BR688QdHe7YkIps5Zwg1T36Zxy/asX7uKf8yYzpnjHqVi7rOsXLoYgI0bN3LllVdy2GGHsXHjRiZOnFjtXS0WLlxIaWkpTz31FAC77747e++992Z1OnToUPl6991356OPPmLDhg00abL1KTx37lz+8Ic/MHny5Mp169ev56tf/WrlcmlpaWFvjCRJ+lxwZFmZNgXi91ZXkIAPm+7F3HkLeOSVxYx7Yj7rNmzkoxXvkjZ+zKu/v5Z3Zkxh5cpVfP3rX2fKlCkAHHrooXz5y19m0aJFlJWVVd5mbvny5ZXtlJSUsGDBgsrlDz/8kGXLlu1wP7cchS4tLWX48OGsXLmy8mvt2rVccsklmdtIkiRVx7CsTJsC8SYty74EjRrz03+/nndXfsCSF6azet4/aFrUhvY9vkGb/Q+n9xVTuP322/ntb38LwI033siiRYv49re/TcuWLWncuDGzZs3iySef5KSTTgLgW9/6FpMnT+bOO+/kqKOOYsyYMZXTOHZE+/btN7tI77TTTmPy5Mk8/PDDfPzxx3z00UdMnz59s0AuSZK0IwoKyxHRJiIeiYjX899bb6Nu44h4LiKmFNKm6s7i1RWbLTdq0pSDRlzL648/wBOjvsHi56bS9st9Aeh09FA+Xr+OJy87ll69ejFgwIDNtr3yyit58803ad26NWPGjOHUU0+tLDvooIP49a9/zbBhwygpKaGoqIh27drRvHnzHernpZdeylVXXUWrVq244YYbKC0t5f777+eaa66huLiY0tJSfvnLX3qLOEmS9KnFpxnB22rjiOuB5Sml6yLiEqB1SumnGXV/BJQDLVNKg3Zk/+Xl5WnTLcJU94bdMpv3tgjMAO2LmnFWn1JumPr2ZiPPzZs04uJ++9LvgOKC2l2zZg2tWrXi9ddfZ9999y1oX5IkSdsTEc+mlMqrKyt0GsY3gfH51+OBEzI60Ak4DhhXYHuqQ2f1KaV5k81PkeZNGnFWn1L6HVDMxf32pX1RM4JcgC4kKE+ePJkPPviAtWvXcvHFF/OlL32JsrKywg9CkiSpAIXeDaN9SmkRQEppUUS0y6h3I/AToGh7O4yIc4BzADp37lxg9xq+yy+/nDfeeIOJEyfWd1e2sin4brobRrv8iPKm9f0OKC54FHmT+++/n+HDh5NSory8nEmTJnkRniRJqnfbDcsRMRXoUE3RqB1pICIGAYtTSs9GRN/t1U8p3QzcDLlpGDvShmpPTQbibRk3bhzjxvnBgyRJali2G5ZTSv2yyiLivYgoyY8qlwCLq6nWBzg+IgYCuwEtI2JiSmnrG+5KkiRJDUihc5YfAM7Ivz4DuH/LCimlS1NKnVJKZcAw4C+7SlCePXs2hxxyCEVFRQwZMoShQ4dW3if4d7/7HV/4whdo06YNxx9//GaPcX7yySc57LDD2GuvvTjssMN48sknK8vefvttjj76aIqKiujfvz9Lly6t8+OSJEnSjik0LF8H9I+I14H++WUiomNEPLjNLRu4iooKTjzxRM4880yWL1/OKaecwn333QfAX/7yFy699FJ+//vfs2jRIrp06cKwYcOA3MM2jjvuOC688EKWLVvGj370I4477rjKh2yceuqpHHrooSxdupTLLruM8ePHZ/ZBkiRJ9augW8fVtvq8ddxjjz3GKaecwoIFCyovNDvyyCPp27cvixYtYu+99+b6668Hcrc6a926Na+//jqPP/44v/71r5k5c2blvnr37s25557LMcccQ9euXVm1ahV77LEHkAvPjRo1apAX+EmSJH0e1Oat4z6zFi5cyD777LPZHRlKS0sry7p06VK5fs8992TvvffmnXfe2aoMoEuXLpVlrVu3rgzKm8okSZLUMBV667jPlKlzllTeJo2Fq3lr7nxSSpWBef78+XTr1o2OHTsyd+7cyu3Wrl3LsmXL2GeffbYqA5g3bx4DBgygpKSEFStWsHbt2srAPG/ePG+RJkmS1EA5spw3dc4Sbpj6Nu+triABH7c7gLXrExdcdh0bNmzg/vvvr5xaceqpp3Lrrbfy/PPPs27dOn72s5/Rs2dPysrKGDhwIK+99hp33nknGzZs4O677+Yf//gHgwYNokuXLpSXlzNmzBgqKir429/+xuTJk+v3wCVJkpTJsJw37on5mz26uVGTpnQ/81ruuP02WrVqxcSJExk0aBDNmzfna1/7GldeeSUnn3wyJSUlvPnmm0yaNAmAvffemylTpjB27NjKec1Tpkyhbdu2ANx5553MmDGDNm3acMUVV3D66afXy/FKkiRp+7zAL++YG5+muncigL+M7AVAz549Oe+88xgxYkSd9EmSJEm1zwv8dkC7omZbrVv5xmyKNr7Phg0bGD9+PC+88AIDBgyoh95JkiSpPhiW887qU0rzJpu/HRXL5vPYNf/KXnvtxdixY7nnnnsoKSmppx5KkiSprnk3jLx+BxQDVN4No11RM0aN+RH97rq2nnsmSZKk+mJYrqLfAcWVoVmSJElyGoYkSZKUwbAsSZIkZTAsS5IkSRkMy5IkSVIGw7IkSZKUwbAsSZIkZTAsS5IkSRkMy5IkSVIGw7IkSZKUwbAsSZIkZYiUUn33IVNELAHm1nc/tE1tgaX13Qk1OJ4X2pLnhKrjeaHq1Md50SWlVFxdQYMOy2r4ImJWSqm8vvuhhsXzQlvynFB1PC9UnYZ2XjgNQ5IkScpgWJYkSZIyGJZVqJvruwNqkDwvtCXPCVXH80LVaVDnhXOWJUmSpAyOLEuSJEkZDMuqVkQMiIhXI+KNiLikmvLWEXFfRLwQETMj4ov59aUR8deIeCUiXo6IH9R971Vbdva8qFLeOCKei4gpdddr1bZCzouIaBUR90TEnPzPjd5123vVhgLPiR/mf3+8FBF3RcRuddt71ZaI+N+IWBwRL2WUR0T8Z/68eSEielQp2+Y5VZsMy9pKRDQG/gs4FugOnBIR3beo9jPg+ZTSl4HTgf/Ir98AXJRSOhDoBXy/mm21CyrwvNjkB8Artd1X1Z0aOC/+A3gopXQA8BU8P3Z5hZwTEbEPcCFQnlL6ItAYGFZXfVetuw0YsI3yY4H98l/nAL+FHT6nao1hWdU5HHgjpfRWSqkCmAR8c4s63YFpACmlOUBZRLRPKS1KKc3Or19N7hffPnXXddWinT4vACKiE3AcMK7uuqw6sNPnRUS0BI4CbsmXVaSUVtZd11VLCvpZATQBWkREE2B3YGHddFu1LaX0GLB8G1W+Cdyecp4GWkVECTt2TtUaw7Kqsw8wv8ryArYOvH8HTgKIiMOBLkCnqhUiogw4BJhRS/1U3Sr0vLgR+AmwsXa7qTpWyHnRFVgC3JqfnjMuIvao/S6rlu30OZFSege4AZgHLAJWpZT+XOs9VkORde7syDlVawzLqk5Us27L26ZcB7SOiOeBC4DnyE3ByO0gYk/gj8DIlNL7tdVR1amdPi8iYhCwOKX0bC33UXWvkJ8XTYAewG9TSocAa4E6nYuoWlHIz4rW5EYM9wU6AntExGm12Vk1KFnnzo6cU7WmSV01pF3KAqC0ynIntvgYLB+AR0BuQj7wdv6LiGhKLijfkVK6ty46rDpRyHkxDDg+IgYCuwEtI2JiSslfgru+Qs6L3YEFKaVNnz7dg2H5s6CQc+IbwNsppSX5snuBI4CJtd9tNQBZ506zjPV1wpFlVecZYL+I2DcimpELOg9UrZC/gr1ZfvEs4LGU0vv5H3q3AK+klH5Vp71Wbdvp8yKldGlKqVNKqSy/3V8Myp8ZhZwX7wLzI2L/fNnXgH/UVcdVa3b6nCA3/aJXROye/33yNbzo8/PkAeD0/F0xepGbhrOIHTinapMjy9pKSmlDRJwPPEzuSuT/TSm9HBHn5ctvAg4Ebo+Ij8n9cvtufvM+wHDgxfzHawA/Syk9WKcHoRpX4Hmhz6gaOC8uAO7I/wJ8i/xoo3ZdhZwTKaUZEXEPMJvcVJ3naGBPc9POi4i7gL5A24hYAIwBmkLlefEgMBB4A/iA/M+DrHOqzvrtE/wkSZKk6jkNQ5IkScpgWJYkSZIyGJYlSZKkDIZlSZIkKYNhWZIkScpgWJYkSZIyGJYlSZKkDIZlSZIkKcP/DzRr9+iouu5oAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 864x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "M_svd_2d = f.matrix_reduce(M_svd, method='svd', n_dim=2)\n",
    "f.plot_vectors_2d(M_svd_2d, viz_words, word2ind)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Convert test string into SVD vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([3700.67999817, -820.60835517,   62.69732848, -657.55311825,\n",
       "       1061.47692555])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f.avg_cooc_embeddings(review_ind_test, svd_reduced_co_occurrence, word2ind)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "f.save_pickle(M_svd, 'data/M_svd_100k.pickle')\n",
    "del M_svd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Co-Occurrence Matrix with NMF (Non-Negative Matrix Factorization)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\klaud\\Anaconda3\\envs\\fastai-nlp\\lib\\site-packages\\sklearn\\decomposition\\_nmf.py:1077: ConvergenceWarning: Maximum number of iterations 200 reached. Increase it to improve convergence.\n",
      "  \" improve convergence.\" % max_iter, ConvergenceWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(100000, 5)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nmf_reduced_co_occurrence = f.matrix_reduce(M_co_occurrence, method='nmf', n_dim=5)\n",
    "M_nmf = np.stack([f.avg_cooc_embeddings(ind, nmf_reduced_co_occurrence, word2ind) for ind in reviews_ind])\n",
    "M_nmf.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 4.73204311  3.10003703  4.33827005  4.17181511  7.71958402]\n",
      " [ 8.04826308  5.81789377  7.47526516  5.45424254 14.58321622]\n",
      " [ 8.20413043 15.00659014 19.79551483 17.25447144 30.89723242]\n",
      " ...\n",
      " [10.16579462  8.22098302 10.5652474   1.76599422 19.62267803]\n",
      " [14.19922071  0.56729811  0.62328625  3.7909095   1.45206812]\n",
      " [ 2.90358797  0.75351253  1.38886852 10.85750284  7.32522433]]\n"
     ]
    }
   ],
   "source": [
    "print(M_nmf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### NMF word embeddings in vector space\n",
    "\n",
    "Based on the selected list of words, NMF word embeddings seem to perform similarly to SVD word vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAssAAAF1CAYAAAAeIKdDAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3de3xV1Z3//9eHq6ixgIT7HRkVp63SVFBstS3MUAXvDGBFpVXLdGi/dKy9QatUrbbVfm1/dX6Otd5AxSnVCmqlYse7IGgVb6B4SYjREgQVUAmR9f3jHGKI2YDkhER4PR+P8+Dsvddee+2zOPDOytp7R0oJSZIkSR/VoqkbIEmSJDVXhmVJkiQpg2FZkiRJymBYliRJkjIYliVJkqQMhmVJkiQpg2FZkmqJiPMjYmZTt6OxRMS6iOjf1O2QpE8Kw7KkZisifhQRd9VZ92LGunE7t3WFERFHRUT5zjpeSmnvlNLLH3e/iPiniLg9IiojYnVEzIuI/WttPz8iNkbE2vzrhYj4XUR0K+wZSNLOZViW1Jw9AAyLiJYAEdEVaA0MrrNuv3zZ7RY5n/h/AyOi1U46VHtgDrA/0AV4DLi9TplbUkpFQEfgBKAr8LiBWdIn2Sf+PwpJu7RF5MLxwfnlLwL/Cyyrs+6llFIFQEQcHhGLIuLt/J+Hb64sIu6LiIsi4mHgXaB/RPSLiPvzo6H3AJ2yGhMRz0fEqFrLrSJiVUQMzi8PjYhHIuKtiHgqIo6qVbZjRFwbERURsSYi/hwRewF/Abrnp0esi4juEdE2Ii7Pl63Iv2+br+eoiCiPiB9ExBvAtRHRKSLuyB93dUQ8mPWDQESkiNgv//66iLgiIu7Mn//CiBhQ334ppcdSSn9IKa1OKW0E/i+wf0TsW0/ZjSmlZ4GxQCVwTkZb9st/9m/nP8db6rTzOxHxcn7brzafU0QMiIi/RcSb+W03RkT7Wvv2iohb86Pgb0bE72pt+3q+H9fkR8f71Nc2SdrMsCyp2UopVQELyQVi8n8+CDxUZ90DkAukwJ3Ab4F9gV8Dd9YJdBOAs4EioBS4CXicXEi+ADh9K026GRhfa/lfgVUppSciokf+2BeSG1n9HvCniCjOl50B7AkcBHQG/m9KaT3wVaAiPz1i73zonwoMJfcDwWeBQ4FptY7bNX+MPvlzOQcoB4rJjfr+GEhbOY/axgPTgQ7AcuCi7dzvi8AbKaU3swqklD4gN/r8hYwiFwB/zR+7J/D/1dl+AlACDAaOA76eXx/AxUB34ECgF3A+QP43DneQ69u+QA9gVn7b8eQ+mxPJfVYPkutTScpkWJbU3N3Ph8H4C+QCzoN11t2ff38M8GJKaUZKqTqldDOwFBhdq77rUkrPppSqgW7A54GfpJQ2pJQeAOZupS03AcdGxJ755VPy6wBOBe5KKd2VUtqUUroHWAwcnZ+G8FVgUkppTX7k9f6P1P6hrwE/SymtTClVkguzE2pt3wScl2/ze8DG/Ln0ydf9YEppe8PyrflR42rgRj4csc8UET2BK4D/3I76K8gF+/psJBf4u6eU3k8pPVRn+y/yI9llwOXkf1BJKS1PKd2TP/9Kcj8UHZnf51ByIfrclNL6OvV+E7g4pfR8/nx/Dhzs6LKkrTEsS2ruHgCOiIgOQHFK6UXgEeDw/Lp/5sP5yt3JjSjWVkpudHGzFbXedwfW5Ed4a5evV0ppOfA8MDofmI/lw7DcBxiTnwrxVkS8BRxBLsT2AlanlNZs5znXPY/S/LrNKlNK79da/hW5UeG/5qct/HA7jwPwRq337wJ7b61wfqT8r8B/5X8Y2ZYewOqMbd8nN0r8WEQ8GxFfr7O9dl/VfAYR0TkiZkXEaxHxDjCTD6fP9AJK82G4rj7Ab2r1z+r88XvUU1aSAMOypObvUeBT5KYbPAyQUnqH3Ijl2eSmMLySL1tBLhDV1ht4rdZy7RHX14EO+bnDtctvzeapGMcBz+UDNOSC3YyUUvtar71SSpfkt3WsPa82oz2b1T2P3vl19e6TUlqbUjonpdSf3Cj6f0bEV7ZxHh9b/oeTvwJzUkrbnK6Rn2M8mtxvAj4ipfRGSumslFJ3cqO+/7V5PnVer1rva38GF5P7DD6TUtqH3Kh+5LetAHpnXPi4AvhmnT5ql1J6ZFvnImn3ZViW1KzlpxksJvcr/9qh66H8utp3wbgL+KeIOCV/8d1YYBC5Oaz11V2ar3t6RLSJiCPYcspGfWYB/wL8Ox+OKkNudHN0RPxrRLSMiD3yF+P1TCm9Tu5Cvv+KiA4R0ToiNk8j+Qewb0R8qlZdNwPTIqI4IjoBP83XX6+IGJW/WC6Ad4AP8q+CiYh9gHnAwymlrY5c58/vQHLn0ZXcNIn6yo3JT+kAWEMuANdu97n5z6sX8H+AzRcAFgHrgLfyc8XPrbXPY+R+CLokIvbK98Ow/LYrgR9FxEH5438qIsZsz/lL2n0ZliV9EtxP7qK42nNaH8yvqwnL+YvNRpG74O1Ncr/mH5VSWrWVuk8BhpD7lfx5wA1ba0g++D4KHM6H4Y2U0gpyo80/JncHiBXkQtzmf2cnkJujuxRYCUzJ77eUXKh8OT89oDu5iwQXA0uAp4En8uuyDATmkwuQj5KbInHf1s5jB5xAbn73xFp37lgXEbVH4sdGxDrgLXK3mXsT+NzmO5XU4/PAwvw+c4D/U+u3BJC7OPBx4ElyF0/+Ib9+OrmL/t7Or7918w75iwpHk7udYBm5Cx/H5rfdBvwCmJWfvvEMubnkkpQptv8aEEmSdo6ISMDAWtNcJKlJOLIsSZIkZTAsS5IkSRmchiFJkiRlcGRZkiRJylCQsBwRIyNiWUQsr+9m+BFxbkQ8mX89ExEf5B9LK0mSJDVbDZ6GEREtgReAEeRu0bMIGJ9Sei6j/GjguymlL2+r7k6dOqW+ffs2qH2SJEnS1jz++OOrUkrF9W2r7wlHH9ehwPKU0ssAETGL/JOtMsqPJ3dP0W3q27cvixcvLkATJUmSpPpFRGnWtkJMw+hB7ub7m5Xn19XXkD2BkcCfsiqLiLMjYnFELK6srCxA8yRJkqQdU4iwHPWsy5rbMZrco1JXZ1WWUroqpVSSUiopLq53NFySJEnaKQoRlsuBXrWWewJZjzYdx3ZOwZAkSZKaWiHC8iJgYET0i4g25ALxnLqFIuJTwJHA7QU4ZpM444wzmDZtWqMe4/zzz+fUU09t1GNIkiRp+zQ4LKeUqoHJwDzgeeB/UkrPRsSkiJhUq+gJwF9TSusbeszdWd++fZk/f35TN0OSJGm3UIi7YZBSugu4q866K+ssXwdcV4jjaceklEgp0aKFz6KRJEnaHqamrfj73//O4MGDKSoqYuzYsbz//vsArFmzhlGjRlFcXEyHDh0YNWoU5eXlNfvVHf2tO7XihhtuoE+fPuy7775ccMEFHylfVVXFaaedRlFREQcddFDN7fMmTJhAWVkZo0ePZu+99+aXv/wlAAsWLODwww+nffv2fPazn+W+++6rqeuoo45i6tSpDBs2jD333JOXX365UT4rSZKkXZFhOUNVVRXHH388EyZMYPXq1YwZM4Y//Sl3x7tNmzYxceJESktLKSsro127dkyePHm76n3uuef41re+xY033sjrr7/O22+/zWuvvbZFmTlz5jBu3Djeeustjj322Jq6Z8yYQe/evZk7dy7r1q3j+9//Pq+99hrHHHMM/fr145hjjuHSSy/lpJNOovZt92bMmMFVV13F2rVr6dOnT4E+IUmSpF2fYTnDggUL2LhxI1OmTKF169acfPLJfP7znwdg33335aSTTmLPPfekqKiIqVOncv/9929XvbNnz2b06NEcccQRtGnThp/97GdEbHn3vSOOOIKjjz6ali1bMmHCBJ566qnM+mbOnMnRRx/NwIEDiQhGjBhBSUkJd9314ayYM844g4MOOohWrVrRunXrHfg0JEmSdk8FmbO8K6qoqKBHjx5bBNnNo7Lvvvsu3/3ud7n77rtZs2YNAGvXruWDDz6gZcuW26y3V68P77S35557su+++25RpmvXrltsf//996murqZVq492V2lpKX/84x+B3Ij3HXfcwcaNG/nSl75UU6b28SRJkrT9DMu1zF9aydUPr2Dl2iqoWMvLpStIKdUE5rKyMgYMGMBll13GsmXLWLhwIV27duXJJ5/kkEMOIaXcs1j22msv3n333Zp6n376af73f/+XoqIievfuzbp162jTpg0XXnghV1xxBW+88QYnnngiRx11FAMHDqzZ75FHHuHf//3fARg6dCi//e1va9ryyiuvcMYZZ7BgwQI6derECSecwFtvvcXMmTM/cl51R64lSZK0fZyGkTd/aSWXzn+Ff6ytIgEfdD6Ad6vhP6b+nOrqam699VYee+wxIDeK3K5dO9q3b8/q1auZPn36FnUdfPDBzJo1i40bN/Loo4/y5z//mf79+7N69WrOOussysrKKC8vZ968eZx77rm0bNmSWbNm0adPH2bPng3A6tWrOeaYYzj99NMBmDJlCscccwwdO3bk5Zdf5pRTTuFzn/sczz77LO+99x7XXHMNKSXef/997rvvvi0uOJQkSdKOMSznXf3wCjZUb6pZbtGqNYPOuJibZ86gQ4cO3HLLLZx44olALri+9957dOrUiaFDhzJy5Mgt6rrgggt46aWX6NChA//5n/9Ju3bt2H///WndujVTpkxh4MCB3HbbbRx77LEMHjyYLl26UFRUxMUXX0x5eTnr1q3jzjvvZODAgTXHHDduHAcccADDhg3j/PPPrxlR3m+//fjLX/7C3nvvzezZs+nVqxe/+tWv2LRpE5IkSWqY2Dx1oDkqKSlJm2+b1ti+fPkC6vskAvjblKEfq67a0znef/5/Wf3oH1m65Ima7ePHj2fAgAE8/vjjjBgxgu9///u8+OKL9OvXj65du/KnP/2Jhx56iMWLF9fMR4ZcYP70pz/NV77yFUaPHr3FHS9+9KMfsWLFinqnYUiSJClbRDyeUiqpb5sjy3mdi9p8rPVZ6k7neK/1p3i1rJx7nl9ZU+app55i48aNdO7cmRtuuIFPf/rT9O3bl/Xr1/Pmm2/So0cPunfvTmlp6RZ1l5WV0aNHD7p168aaNWtYv379FtskSZJUWIblvDOH9aJtqy0/jratWnDmsI93J4m60zn26ftpiBb84IJLqa6u5vbbb2fZsmX85je/Yfbs2Tz33HOcd955VFVV8eMf/5ghQ4bQt29fjj76aF544QVuuukmqqurueWWW3juuecYNWoUffr0oaSkpGa/hx56iLlz5xbkc5AkSdKHDMt5ww8o5nvD+9GlqA0BdClqw/eG92P4AcUfq56Va6u2WG7RqjUHTbyYF+//M+3bt2fmzJkcd9xxTJ06lfXr1/Pb3/6Wc845h27duvHSSy8xa9YsIHcv5zvuuIPLLruMfffdl1/+8pfccccddOrUCYCbbrqJhQsX0rFjR6ZPn85pp51WkM9BkiRJH3LOcoGN+8MT/KNOYIZc+J71jcEADBkyhEmTJjFx4sSd3TxJkiTV4Zzlnai+6RzrX3mSE/+pDdXV1Vx//fUsWbLkI3fQkCRJUvPjQ0kKbPO0jc13w+hc1IZu+67n2yd/hYnr1jFgwABmz55Nt27dmrilkiRJ2hanYUiSJGm35jQMSZIkaQcYliVJkqQMhmVJkiQpg2FZkiRJymBYliRJkjIYliVJkqQMhmVJkiQpg2FZkiRJymBYliRJkjIYliVJkqQMhmXt9n7xi1/Qo0cPioqK2H///bn33nvZsGEDU6ZMoXv37nTv3p0pU6awYcMGAO677z569uzJZZddRufOnenWrRvXXnttE5+FJElqDIZl7daWLVvG7373OxYtWsTatWuZN28effv25aKLLmLBggU8+eSTPPXUUzz22GNceOGFNfu98cYbvP3227z22mv84Q9/4D/+4z9Ys2ZNE56JJElqDIZl7dZatmzJhg0beO6559i4cSN9+/ZlwIAB3Hjjjfz0pz+lc+fOFBcXc9555zFjxoya/Vq3bs1Pf/pTWrduzdFHH83ee+/NsmXLmvBMJElSYzAsa7e23377cfnll3P++efTuXNnxo0bR0VFBRUVFfTp06emXJ8+faioqKhZ3nfffWnVqlXN8p577sm6det2atslSVLjMyxrtzN/aSXj/vAEX758AeP+8ASdB4/goYceorS0lIjgBz/4Ad27d6e0tLRmn7KyMrp3796ErZYkSU3BsKzdyvyllVw6/xX+sbaKBLzy0ov86Io/cteScvbYYw/atWtHy5YtGT9+PBdeeCGVlZWsWrWKn/3sZ5x66qlN3XxJkrSTtdp2kW2LiJHAb4CWwNUppUvqKXMUcDnQGliVUjqyEMeWPo6rH17BhupNNcubqjeybM7vOO7351K0Z1sOP/xwrrrqKjp27Mg777zDZz7zGQDGjBnDtGnTmqrZkiSpiURKqWEVRLQEXgBGAOXAImB8Sum5WmXaA48AI1NKZRHROaW0clt1l5SUpMWLFzeofVJtX758AfX9jQ/gb1OG7uzmSJKkZiAiHk8pldS3rRDTMA4FlqeUXk4pVQGzgOPqlDkFuDWlVAawPUFZagydi9p8rPWSJGn3Voiw3ANYUWu5PL+utn8COkTEfRHxeEScVoDjSh/bmcN60bbVln/t27ZqwZnDejVRiyRJUnNWiDnLUc+6ur/pbgV8DvgK0A54NCIWpJRe+EhlEWcDZwP07t27AM2TPjT8gGIgN3d55doqOhe14cxhvWrWS5Ik1VaIsFwO1B6W6wlU1FNmVUppPbA+Ih4APkturvMWUkpXAVdBbs5yAdonbWH4AcWGY0mStF0KMQ1jETAwIvpFRBtgHDCnTpnbgS9ERKuI2BMYAjxfgGNLkiRJjabBI8sppeqImAzMI3fruGtSSs9GxKT89itTSs9HxN3AEmATudvLPdPQY0uSJEmNqcG3jmtM3jpOkiRJja2xbx0nSZIk7ZIMy5I+4oknnuCQQw6hqKiIMWPGMHbsWKZNm8aaNWsYNWoUxcXFdOjQgVGjRlFeXl6z31FHHcW0adM4/PDD2XvvvRk9ejRvvvkmX/va19hnn334/Oc/z6uvvlpTfunSpYwYMYKOHTuy//778z//8z9NcLaSJGUzLEvaQlVVFSeccAJnnHEGq1evZvz48dx2220AbNq0iYkTJ1JaWkpZWRnt2rVj8uTJW+w/a9YsZsyYwWuvvcZLL73EYYcdxsSJE1m9ejUHHngg06dPB2D9+vWMGDGCU045hZUrV3LzzTfzrW99i2effXann7MkSVkMy5K2sGDBAqqrq/nOd75D69atOfHEEzn00EMB2HfffTnppJPYc889KSoqYurUqdx///1b7D9x4kQGDBjApz71Kb761a8yYMAAhg8fTqtWrRgzZgx///vfAbjjjjvo27cvEydOpFWrVgwePJiTTjqJ2bNn7/RzliQpSyHusyxpF1JRUUGPHj2I+PB5Q7165W6l/u677/Ld736Xu+++mzVr1gCwdu1aPvjgA1q2bAlAly5davZr167dR5bXrVsHQGlpKQsXLqR9+/Y126urq5kwYULjnZwkSR+TYVkSAPOXVnL1wytY9vfVLHvxVe55fiUjDuwMwIoVKxgwYACXXXYZy5YtY+HChXTt2pUnn3ySQw45hB25q06vXr048sgjueeeewp9KpIkFYzTMCQxf2kll85/hX+srWKfvp9mEy34zk9/ybxnXuf222/nscceA3KjyO3ataN9+/asXr26Zv7xjhg1ahQvvPACM2bMYOPGjWzcuJFFixbx/PM+r0iS1HwYliVx9cMr2FC9CYAWrVpz0MSLWfHIHEaVDGDmzJmMGjWKtm3bMmXKFN577z06derE0KFDGTly5A4fs6ioiL/+9a/MmjWL7t2707VrV37wgx+wYcOGQp2WJEkN5kNJJPHlyxdQ378EAfxtylCGDBnCpEmTmDhx4s5umiRJjc6Hkkjaqs5FbbZYfmv5E2x450067dmC66+/niVLljRoFFmSpE8qL/CTxJnDenHp/FdqpmK8u7KM526Yxt+rN7B0vwHMnj2bbt26NXErJUna+QzLkhh+QDGQm7u8cm0Vh/zrv3HFz86pWS9J0u7KsCwJyAVmw7EkSVtyzrIkSZKUwbAsSZIkZTAsS5IkSRkMy5IkSVIGw7IkSZKUwbAsSZIkZTAsS5IkSRkMy5IkSVIGw7IkSZKUwbAsSZIkZTAsS5IkSRkMy5IkSVIGw7IkSZKUwbAsSZIkZTAsS5IkSRkMy5IkSVIGw7IkSZKUwbAsSZIkZShIWI6IkRGxLCKWR8QP69l+VES8HRFP5l8/LcRxJUmSpMbUqqEVRERL4ApgBFAOLIqIOSml5+oUfTClNKqhx5MkSZJ2lkKMLB8KLE8pvZxSqgJmAccVoF5JkiSpSRUiLPcAVtRaLs+vq+uwiHgqIv4SEQcV4LiSJElSo2rwNAwg6lmX6iw/AfRJKa2LiKOBPwMD660s4mzgbIDevXsXoHmSJEnSjinEyHI50KvWck+gonaBlNI7KaV1+fd3Aa0jolN9laWUrkoplaSUSoqLiwvQPEmSJGnHFCIsLwIGRkS/iGgDjAPm1C4QEV0jIvLvD80f980CHFuSJElqNA2ehpFSqo6IycA8oCVwTUrp2YiYlN9+JXAy8O8RUQ28B4xLKdWdqiFJkiQ1K9GcM2tJSUlavHhxUzdDkiRJu7CIeDylVFLfNp/gJ0mSJGUwLEuSJEkZDMuSJElSBsOyJEmSlMGwLEmSJGUwLEuSJEkZDMuSJElSBsOyJEmSlMGwLEmSJGUwLEuSJEkZDMuSJElSBsOyJEmSlMGwLEmSJGUwLEuSJEkZDMuSJElSBsOyJEmSlMGwLEmSJGUwLEuSJEkZDMuSJElSBsOyJEmSlMGwLEmSJGUwLEuSJEkZDMuSJElSBsOyJEmSlMGwLEmSJGUwLEuSJEkZDMuSJElSBsOyJEmSlMGwLEmSJGUwLEuSJEkZDMuSJElSBsOyJEmSlKEgYTkiRkbEsohYHhE/3Eq5z0fEBxFxciGOK0mSJDWmBofliGgJXAF8FRgEjI+IQRnlfgHMa+gxJUmSpJ2hECPLhwLLU0ovp5SqgFnAcfWU+zbwJ2BlAY4pSZIkNbpChOUewIpay+X5dTUiogdwAnDltiqLiLMjYnFELK6srCxA8yRJkqQdU4iwHPWsS3WWLwd+kFL6YFuVpZSuSimVpJRKiouLC9A8SZIkace0KkAd5UCvWss9gYo6ZUqAWREB0Ak4OiKqU0p/LsDxJUmSpEZRiLC8CBgYEf2A14BxwCm1C6SU+m1+HxHXAXcYlCVJktTcNTgsp5SqI2IyubtctASuSSk9GxGT8tu3OU9ZkiRJao4KMbJMSuku4K466+oNySmlMwpxTEmSJKmx+QQ/SZJE3759mT9/foPrOf/88zn11FML0CKpeTAsS5IkSRkMy5IkSVIGw7IkSQJg0aJFDBo0iA4dOjBx4kTef/991qxZw6hRoyguLqZDhw6MGjWK8vLymn1eeeUVjjzySIqKihgxYgSrVq1qwjOQCs+wLEmSALjxxhuZN28eL730Ei+88AIXXnghmzZtYuLEiZSWllJWVka7du2YPHlyzT6nnHIKn/vc51i1ahU/+clPuP7665vwDKTCi5TqPmyv+SgpKUmLFy9u6mZIkrTL69u3Lz/84Q+ZNGkSAHfddRff/va3eemll7Yo9+STT/KlL32JNWvWUFZWRv/+/Xn77bfZa6+9gFx4btGiBTNnztzp5yDtqIh4PKVUUt+2gtw6TpIkffLMX1rJ1Q+vYOXaKirXVfEm+9Rs69OnDxUVFbz77rt897vf5e6772bNmjUArF27lg8++ICKigo6dOhQE5Q377dixYqdfi5SY3EahiRJu6H5Syu5dP4r/GNtFQn4YFPipvuWMH9pJQBlZWV0796dyy67jGXLlrFw4ULeeecdHnjgAQBSSnTr1o01a9awfv36mnrLysqa4nSkRmNYliRpN3T1wyvYUL1pi3WlD/yR385dzOrVq/n5z3/O2LFjWbt2Le3ataN9+/asXr2a6dOn15Tv06cPJSUlnHfeeVRVVfHQQw8xd+7cnX0qUqMyLEuStBtaubbqI+s6D/4X7r3sW/Tv35/+/fszbdo0pkyZwnvvvUenTp0YOnQoI0eO3GKfm266iYULF9KxY0emT5/OaaedtrNOQdopvMBPkqTd0Lg/PME/6gnMXYraMOsbg5ugRVLT2doFfo4sS5K0GzpzWC/attoyBrRt1YIzh/VqohZJzZN3w5AkaTc0/IBigJq7YXQuasOZw3rVrJeUY1iWJGk3NfyAYsOxtA1Ow5AkSZIyGJYlSZKkDIZlSZIkKYNhWZIkScpgWJYkSZIyGJYlSZKkDIZlSZIkKYNhWZIkScpgWJYkSZIyGJYlSZKkDIZlSZIkKYNhWZIkScpgWJYkSZIyGJYlSZKkDIZlSZIkKYNhWZIkScpgWJYkSZIyFCQsR8TIiFgWEcsj4of1bD8uIpZExJMRsTgijijEcSVJkqTG1KqhFURES+AKYARQDiyKiDkppedqFbsXmJNSShHxGeB/gAMaemxJkiSpMRViZPlQYHlK6eWUUhUwCziudoGU0rqUUsov7gUkJEmSpGauEGG5B7Ci1nJ5ft0WIuKEiFgK3Al8PauyiDg7P1VjcWVlZQGaJ0mSJO2YQoTlqGfdR0aOU0q3pZQOAI4HLsiqLKV0VUqpJKVUUlxcXIDmSZIkSTumEGG5HOhVa7knUJFVOKX0ADAgIjoV4NiSJElSoylEWF4EDIyIfhHRBhgHzKldICL2i4jIvx8MtAHeLMCxJUmSpEbT4LthpJSqI2IyMA9oCVyTUno2Iiblt18JnAScFhEbgfeAsbUu+JMkSZKapWjOmbWkpCQtXry4qZshSZKkXVhEPJ5SKqlvm0/wkyRJkjIYliVJkqQMhmVJkiQpg2FZkiRJymBYliRJkjIYliVJkqQMhmVJkiQpg2FZkiRJymBYliRJkjIYliVJkqQMhmVJkiQpg2FZkiRJymBYliRJkjIYliVJkqQMhmVJkiQpgy8QYHEAABO5SURBVGFZkiRJymBYliRJkjIYliVJkqQMhmVJkiQpg2FZkiRJymBYliRJkjIYliVJkqQMhmVJkiQpg2FZkiRJymBYliRJkjIYliVJkqQMhmVJkiQpg2FZkiRJymBYliRJkjIYliVJkqQMhmVJkiQpQ0HCckSMjIhlEbE8In5Yz/avRcSS/OuRiPhsIY4rSZIkNaYGh+WIaAlcAXwVGASMj4hBdYq9AhyZUvoMcAFwVUOPK0mSJDW2QowsHwosTym9nFKqAmYBx9UukFJ6JKW0Jr+4AOhZgONKkiRJjaoQYbkHsKLWcnl+XZZvAH/J2hgRZ0fE4ohYXFlZWYDmSZIkSTumEGE56lmX6i0Y8SVyYfkHWZWllK5KKZWklEqKi4sL0DxJkiRpx7QqQB3lQK9ayz2BirqFIuIzwNXAV1NKbxbguJIkSVKjKsTI8iJgYET0i4g2wDhgTu0CEdEbuBWYkFJ6oQDHlCRJkhpdg0eWU0rVETEZmAe0BK5JKT0bEZPy268EfgrsC/xXRABUp5RKGnpsSZIkqTFFSvVOL24WSkpK0uLFi5u6GZIkSdqFRcTjWQO5PsFPkiRJymBYliRJkjIYliVJkqQMhmVJkiQpg2FZkiRJymBYliRJkjIYliVJkqQMhmVJkiQpg2FZkiRJymBYliRJkjIYliVJkqQMhmVJkiQpg2FZkiRJymBYliRJkjIYliVJkqQMhmVJkiQpg2FZkiRJymBYliRJkjIYliVJkqQMhmVJkiQpg2FZkiRJymBYliRJkjIYliVJklQwffv2Zf78+U3djIIxLEuSJEkZDMuSJEkqiAkTJlBWVsbo0aPZe++9+eUvf8mcOXM46KCDaN++PUcddRTPP/88AL/61a846aSTttj/29/+NlOmTGmKpmcyLEuSJKkgZsyYQe/evZk7dy7r1q3j+OOPZ/z48Vx++eVUVlZy9NFHM3r0aKqqqjj11FO5++67eeuttwCorq7mlltuYcKECU18FlsyLEuSJKlR3HLLLRxzzDGMGDGC1q1b873vfY/33nuPRx55hG7duvHFL36RP/7xjwDcfffddOrUic997nNN3OotGZYlSZLUKCoqKujTp0/NcosWLejVqxevvfYaAKeffjozZ84EYObMmc1uVBkMy5IkSWqg+UsrGfeHJ/jy5QtYtX4jT5TlplZ0796d0tLSmnIpJVasWEGPHj0AOP7441myZAnPPPMMd9xxB1/72teapP1bY1iWJEnSDpu/tJJL57/CP9ZWkYAWe3Xg+nseZ/7SSv7t3/6NO++8k3vvvZeNGzdy2WWX0bZtWw4//HAA9thjD04++WROOeUUDj30UHr37t20J1MPw7IkSZJ22NUPr2BD9aaa5T5fOY2X7r6Wowf3Z+7cucycOZNvf/vbdOrUiblz5zJ37lzatGlTU/7000/n6aefbpZTMABaFaKSiBgJ/AZoCVydUrqkzvYDgGuBwcDUlNKlhTiuJEmSmtbKtVVbLHf69Bfp9OkvEsD3pgwF4IQTTsjcv3fv3rRr1+4jt5FrLho8shwRLYErgK8Cg4DxETGoTrHVwHcAQ7IkSdIupHNRm4+1vrZNmzbx61//mnHjxrHPPvsUumkFUYhpGIcCy1NKL6eUqoBZwHG1C6SUVqaUFgEbC3A8SZIkNRNnDutF21ZbRsq2rVpw5rBeW91v/fr17LPPPtxzzz1Mnz69MZvYIIWYhtEDWFFruRwYUoB6JUmS1MwNP6AYyM1dXrm2is5FbThzWK+a9Vn22msv1q1btzOa2CCFCMtRz7q0w5VFnA2cDTTLKyIlSZK0peEHFG8zHH9SFWIaRjlQe5y9J1Cxo5WllK5KKZWklEqKi3fND12SJEmfDIUIy4uAgRHRLyLaAOOAOQWoV5IkSWpSDZ6GkVKqjojJwDxyt467JqX0bERMym+/MiK6AouBfYBNETEFGJRSeqehx5ckSZIaS0Hus5xSugu4q866K2u9f4Pc9AxJkiTpE8Mn+EmSJEkZDMuSJElSBsOyJEmSlMGwLEmSJGUwLEuSJEkZDMuSJElSBsOyJEmSlMGwLEmSJGUwLEuSJEkZDMuSJElSBsOyJEmSlMGwLEmSJGUwLEuSJEkZDMuSJElSBsOyJEmSlMGwLEmSJGUwLEuSJEkZDMuSJElSBsOyJEmSlMGwLEmSpN3GddddxxFHHLHd5Q3LkiRJ+sSorq7eqcczLEuSJKnJPfHEExxyyCEUFRUxZswYxo4dy7Rp07jvvvvo2bMnv/jFL+jatSsTJ04E4I477uDggw+mffv2HH744SxZsqSmrksuuYQBAwZQVFTEoEGDuO222wB4/vnnmTRpEo8++ih777037du332a7DMuSJElqUlVVVZxwwgmcccYZrF69mvHjx9cEXIA33niD1atXU1paylVXXcUTTzzB17/+df77v/+bN998k29+85sce+yxbNiwAYABAwbw4IMP8vbbb3Peeedx6qmn8vrrr3PggQdy5ZVXcthhh7Fu3TreeuutbbbNsCxJkqQmtWDBAqqrq/nOd75D69atOfHEEzn00ENrtrdo0YLp06fTtm1b2rVrx+9//3u++c1vMmTIEFq2bMnpp59O27ZtWbBgAQBjxoyhe/futGjRgrFjxzJw4EAee+yxHWqbYVmSJElNqqKigh49ehARNet69epV8764uJg99tijZrm0tJTLLruM9u3b17xWrFhBRUUFADfccEPNFI327dvzzDPPsGrVqh1qW6sdPCdJkiSpQeYvreTqh1ew7O+rWfbiq9zz/EpGHNgZgBUrVjBgwACALUI05IL01KlTmTp16kfqLC0t5ayzzuLee+/lsMMOo2XLlhx88MGklOqta1scWZYkSdJON39pJZfOf4V/rK1in76fZhMt+M5Pf8m8Z17n9ttv3+q0ibPOOosrr7yShQsXklJi/fr13Hnnnaxdu5b169cTERQXFwNw7bXX8swzz9Ts26VLF8rLy6mqqtqudhqWJUmStNNd/fAKNlRvAqBFq9YcNPFiVjwyh1ElA5g5cyajRo2ibdu29e5bUlLC73//eyZPnkyHDh3Yb7/9uO666wAYNGgQ55xzDocddhhdunTh6aefZtiwYTX7fvnLX+aggw6ia9eudOrUaZvtjM1D0s1RSUlJWrx4cVM3Q5IkSQX25csXUF8KDeBvU4YyZMgQJk2aVHOruMYUEY+nlErq2+bIsiRJkna6zkVttlh+a/kTbHjnTTrt2YLrr7+eJUuWMHLkyCZq3Ye8wE+SJEk73ZnDenHp/FdqpmK8u7KM526Yxt+rN7B0vwHMnj2bbt26NXErCzSyHBEjI2JZRCyPiB/Wsz0i4rf57UsiYnAhjitJkqRPpuEHFPO94f3oUtSGAA7513/jzseW8d6761myZAnHHHNMUzcRKMDIckS0BK4ARgDlwKKImJNSeq5Wsa8CA/OvIcD/n/9TkiRJu6nhBxQz/IDipm7GVhViZPlQYHlK6eWUUhUwCziuTpnjgBtSzgKgfUQ0/bi6JEmStBWFCMs9gBW1lsvz6z5uGUmSJKlZKURYru8xKHXvBLI9ZXIFI86OiMURsbiysrLBjZMkSZJ2VCHCcjnQq9ZyT6BiB8oAkFK6KqVUklIq2fzkFUmSJKkpFCIsLwIGRkS/iGgDjAPm1CkzBzgtf1eMocDbKaXXC3BsSZIkqdE0+G4YKaXqiJgMzANaAteklJ6NiEn57VcCdwFHA8uBd4HGfxSLJEmS1EAFeShJSukucoG49rora71PwH8U4liSJEnSzuLjriVJkqQMhmVJkiQpg2FZkiRJymBYliRJkjIYliVJkqQMhmVJkiQpg2FZkiRJymBYliRJkjIYliVJkqQMhmVJkiQpg2FZkiRJzcZ9991Hz549m7oZNQzLkiRJUgbDsiRJkpTBsCxJkqSdrm/fvlx88cUMGjSIDh06MHHiRN5///2PlLvkkksYMGAARUVFDBo0iNtuuw2ADRs20LFjR55++umasitXrqRdu3ZUVlYWrJ2GZUmSJDWJG2+8kXnz5vHSSy/xwgsvcOGFF36kzIABA3jwwQd5++23Oe+88zj11FN5/fXXadu2LePGjWPmzJk1ZW+++WaGDx9OcXFxwdpoWJYkSVKTmDx5Mr169aJjx45MnTqVm2+++SNlxowZQ/fu3WnRogVjx45l4MCBPPbYYwCcfvrp3HTTTWzatAmAGTNmMGHChIK2sVVBa5MkSZIyzF9aydUPr2Dl2ioq11XxJvvUbOvTpw8VFRUf2eeGG27g17/+Na+++ioA69atY9WqVQAMGTKEvfbai/vvv59u3bqxfPlyjj322IK22bAsSZKkRjd/aSWXzn+FDdW5UeAPNiVuum8JQ44awfADiikrK6N79+5b7FNaWspZZ53Fvffey2GHHUbLli05+OCDSSnVlDn99NOZOXMmXbt25eSTT2aPPfYoaLsNy5IkSWp0Vz+8oiYob1b6wB/57dwjGdx5CD//+c8ZO3bsFtvXr19PRNTMQb722mt55plntigzYcIEPvOZz1BUVMSMGTMK3m7nLEuSJKnRrVxb9ZF1nQf/C/de9i369+9P//79mTZt2hbbBw0axDnnnMNhhx1Gly5dePrppxk2bNgWZXr27MngwYOJCL7whS8UvN2OLEuSJKnRdS5qwz/qBOaiXgdy6AlnMesbg2vWHXXUUZSXl9csX3TRRVx00UVbrbt3794MHTqUiChso3FkWZIkSTvBmcN60bbVltGzTcsWnDmsV4PqffXVV7n11lv5xje+0aB6shiWJUmS1OiGH1DM94b3o0tRGwJo2SIYM7grww/Y8Xsi/+QnP+Gf//mfOffcc+nXr1/hGltL1L6asLkpKSlJixcvbupmSJIkaRcWEY+nlErq2+bIsiRJkpTBsCxJkiRlMCxLkiRJGQzLkiRJUgbDsiRJkpTBsCxJkiRlMCxLkiRJGQzLkiRJUoYGheWI6BgR90TEi/k/O2SUuyYiVkbEMw05niRJkrQzNXRk+YfAvSmlgcC9+eX6XAeMbOCxJEmSpJ2qoWH5OOD6/PvrgePrK5RSegBY3cBjSZIkSTtVQ8Nyl5TS6wD5Pzs3tEERcXZELI6IxZWVlQ2tTpIkSdphrbZVICLmA13r2TS18M2BlNJVwFUAJSUlqTGOIUmSJG2PbYbllNLwrG0R8Y+I6JZSej0iugErC9m4xx9/fFVElBayzp2oE7CqqRuhgrNfd032667Jft012a+7pqbu1z5ZG7YZlrdhDnA6cEn+z9sbWN8WUkrFhaxvZ4qIxSmlkqZuhwrLft012a+7Jvt112S/7pqac782dM7yJcCIiHgRGJFfJiK6R8RdmwtFxM3Ao8D+EVEeEd9o4HElSZKkRtegkeWU0pvAV+pZXwEcXWt5fEOOI0mSJDUFn+DXeK5q6gaoUdivuyb7dddkv+6a7NddU7Pt10jJG05IkiRJ9XFkWZIkScpgWN4BETEyIpZFxPKI+MgjviPi3Ih4Mv96JiI+iIiO+W2vRsTT+W2Ld37rlWU7+vVTETE3Ip6KiGcjYuL27qum08B+9fvaDG1Hn3aIiNsiYklEPBYR/7y9+6rpNLBf/a42UxFxTUSsjIhnMrZHRPw23+9LImJwrW3N4/uaUvL1MV5AS+AloD/QBngKGLSV8qOBv9VafhXo1NTn4evj9yvwY+AX+ffF5B7h3ubj/p3w9cno1/yy39dm9trOPv0VcF7+/QHAvdu7r69PXr/ml/2uNtMX8EVgMPBMxvajgb8AAQwFFm7v34md9XJk+eM7FFieUno5pVQFzAKO20r58cDNO6Vlaojt6dcEFEVEAHuTC1XV27mvmkZD+lXN0/b06SDgXoCU0lKgb0R02c591TQa0q9qxlJKD5D7dzXLccANKWcB0D7/oLtm8301LH98PYAVtZbL8+s+IiL2BEYCf6q1OgF/jYjHI+LsRmulPq7t6dffAQcCFcDTwP9JKW3azn3VNBrSr+D3tTnanj59CjgRICIOJfdkrp7bua+aRkP6FfyufpJl9X2z+b429Al+u6OoZ13WLUVGAw+nlGr/RDUspVQREZ2BeyJiaf6nLjWt7enXfwWeBL4MDCDXfw9u575qGjvcrymld/D72hxtT59eAvwmIp4k9wPQ38n9tsDvavPVkH4Fv6ufZFl932y+r44sf3zlQK9ayz3JjUjVZxx1pmCk3ANbSCmtBG4j92sGNb3t6deJwK35XxUtB14hN2/u4/yd0M7VkH71+9o8bbNPU0rvpJQmppQOBk4jNxf9le3ZV02mIf3qd/WTLavvm8331bD88S0CBkZEv4hoQy4Qz6lbKCI+BRwJ3F5r3V4RUbT5PfAvQL1Xh2qn255+LSP/xMr8PLn9gZe3c181jR3uV7+vzdY2+zQi2ue3AZwJPJD/TYHf1eZrh/vV7+on3hzgtPxdMYYCb6eUXqcZfV+dhvExpZSqI2IyMI/clZrXpJSejYhJ+e1X5oueAPw1pbS+1u5dgNty1xHRCrgppXT3zmu9smxnv14AXBcRT5P79dAPUkqrAOrbtynOQ1tqSL9GRH/8vjY729mnBwI3RMQHwHPAN7a2b1Och7bUkH7F/1ubtYi4GTgK6BQR5cB5QGuo6de7yN0RYznwLrnf9jWr76tP8JMkSZIyOA1DkiRJymBYliRJkjIYliVJkqQMhmVJkiQpg2FZkiRJymBYliRJkjIYliVJkqQMhmVJkiQpw/8DZODrH/Wrft0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "M_nmf_2d = f.matrix_reduce(M_nmf, method='svd', n_dim=2)\n",
    "f.plot_vectors_2d(M_nmf_2d, viz_words, word2ind)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Convert test string into NMF vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 3.39747279,  1.78241199,  3.35470569, 32.29559881, 21.61569199])"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f.avg_cooc_embeddings(review_ind_test, nmf_reduced_co_occurrence, word2ind)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "f.save_pickle(M_nmf, 'data/M_nmf_100k.pickle')\n",
    "del M_nmf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Word2Vec\n",
    "\n",
    "Word2Vec is a two-layer neural network used to learn word associations from a large corpus of text. Its output are word embeddings that represent words from text corpus. Word2Vec model represents words in dense low dimensional space in a way that similar words get similar word vectors, so they are mapped to nearby points."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "bigram_transformer = Phrases(common_texts)\n",
    "w2v_model = Word2Vec(min_count=5,\n",
    "                     window=3,\n",
    "                     size=100,\n",
    "                     workers=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(42810830, 50905110)"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w2v_model.build_vocab(reviews_tokens)\n",
    "w2v_model.train(reviews_tokens, total_examples=w2v_model.corpus_count, epochs=30)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(100000, 100)"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "M_word2vec = np.stack([f.avg_w2v_embeddings(review, w2v_model) for review in reviews_tokens])\n",
    "M_word2vec.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Word2Vec word embeddings in vector space\n",
    "\n",
    "Word2Vec word embeddings reflect the true relations between words, eg: \"good\" and \"great\" are the synonymes, close to the \"bad\" word (it's the adjective describing the same feature) but there is distance between them (since \"bad\" is the opposite word). \"Game\", \"toy\" and \"play\" are located in close neighbourhood in vector space, the same as \"son\" and \"daughter\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtwAAAF1CAYAAADbUj3SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3df5xVVb3/8ddHfokyOvwYfgyMg5CpaGY2KooV38KbP/BXyhVNMkrNb2FRWXaTwlJTb9H1e6+VmaYJJnZNU9SriaV5NX+MiigKhtoADsrIDx1AGUbW949zHIdxgIE5e2YYX8/HYx6cvffaa6191gN4nzXr7B0pJSRJkiRlY4f27oAkSZLUmRm4JUmSpAwZuCVJkqQMGbglSZKkDBm4JUmSpAwZuCVJkqQMGbglqZGIuCAiZrR3P7ISEasjYlh790OSPkgM3JI6rIj4t4i4q8m+f2xi3/i27V1hRMToiFjSVu2llHqllF7a2vMi4sMRcVtE1ETEioi4JyL2bHT8gohYHxG1+Z8XIuKKiBhU2CuQpO2PgVtSR/Y3YFREdAGIiIFAN+CAJvs+lC/bYpGz3f8bGBFd26ipYuB2YE9gAPAYcFuTMjellIqAPsAJwEDgCUO3pA+67f4/G0md2uPkAvb++e1PAn8FFjTZ92JKqRogIg6NiMcj4o38n4e+W1lE3B8RF0fEQ8BaYFhE7B4RD+RnZe8F+m2qMxHxfESMbbTdNSJej4gD8tsjI+LhiFgVEU9HxOhGZftExLURUR0RKyPiTxGxM/A/QGl+qcfqiCiNiB4RcXm+bHX+dY98PaMjYklEnBcRrwLXRkS/iLgj3+6KiHhwUx8mIiJFxIfyr6+LiF9ExJ356380IoY3d15K6bGU0jUppRUppfXAfwB7RkTfZsquTynNA04GaoBvb6IvH8q/92/k38ebmvTz6xHxUv7YT9+9pogYHhF/iYjl+WM3RERxo3PLIuKW/Gz88oi4otGxL+XHcWV+lr68ub5JUiEZuCV1WCmlOuBRcqGa/J8PAv/bZN/fIBdqgTuB/wT6Aj8H7mwSCicAZwFFQBXwe+AJckH7QuD0zXTpRuCURtufBV5PKT0ZEYPzbV9Ebob3XOCPEVGSLzsd2AnYB+gP/EdKaQ1wJFCdX+rRK//B4XxgJLkPFR8FDgKmNGp3YL6N8vy1fBtYApSQm33+PpA2cx2NnQL8COgNLAQubuF5nwReTSkt31SBlNI75GbBP7GJIhcCf863PQT4rybHTwAqgAOA44Av5fcHcAlQCuwNlAEXAOR/83EHubEdCgwGZuaPHU/uvfkcuffqQXJjKkmZMnBL6uge4L1w/QlyIenBJvseyL8+GvhHSml6Sqk+pXQjMB84plF916WU5qWU6oFBwIHAD1JK61JKfwNmbaYvvweOjYid8tun5vcBnAbclVK6K6W0IaV0L1AJHJVfUnEkcHZKaWV+BviB99X+ns8DP04pLUsp1ZALxBMaHd8ATM33+S1gff5ayvN1P5hSamngviU/e10P3MB7vznYpIgYAvwC+FYL6q8m9+GgOevJfWgoTSm9nVL63ybHL8vPqC8CLif/YSeltDCldG/++mvIfbD6VP6cg8gF8e+klNY0qfcrwCUppefz1/sTYH9nuSVlzcAtqaP7G3BYRPQGSlJK/wAeBg7N79uX99Zvl5Kb2Wysitws57sWN3pdCqzMzzQ3Lt+slNJC4HngmHzoPpb3Anc5MC6/rGNVRKwCDiMXhMuAFSmllS285qbXUZXf966alNLbjbZ/Sm52+s/5JRjfa2E7AK82er0W6LW5wvkZ+z8Dv8x/oNmSwcCKTRz7LrnZ6sciYl5EfKnJ8cZj1fAeRET/iJgZEa9ExJvADN5bClQGVOUDdVPlwP9rND4r8u0PbqasJBWMgVtSR/d3YFdySyceAkgpvUlu5vQscssxXs6XrSYXqhrbDXil0Xbjmd+lQO/8WurG5Tfn3WUlxwHP5UM45MLh9JRScaOfnVNKl+aP9Wm8zngT/XlX0+vYLb+v2XNSSrUppW+nlIaRm83/VkR8ZgvXsdXyH3D+DNyeUtri0pP8mutjyP1G4n1SSq+mlM5MKZWSm33+5bvry/PKGr1u/B5cQu492C+ltAu53y5E/thiYLdNfJl0MfCVJmPUM6X08JauRZJaw8AtqUPLL5moJLd8oXFw+9/8vsZ3J7kL+HBEnJr/QuPJwAhya3qbq7sqX/ePIqJ7RBzGxstPmjMT+Bfg//Le7DbkZlmPiYjPRkSXiNgx/wXHISmlpeS+HPnLiOgdEd0i4t0lMa8BfSNi10Z13QhMiYiSiOgH/DBff7MiYmz+C4gBvAm8k/8pmIjYBbgHeCiltNkZ9Pz17U3uOgaSW/LRXLlx+eUpACvJhejG/f5O/v0qA74BvPulyiJgNbAqv3b+O43OeYzcB6lLI2Ln/DiMyh+7Evi3iNgn3/6uETGuJdcvSa1h4Ja0PXiA3BcNG6/xfTC/ryFw57/AN5bclwiXk1uyMDal9Ppm6j4VOJjc8oKpwPWb60g+PP8dOJT3AiAppcXkZr2/T+7OHIvJBcF3/52dQG7N8nxgGTA5f958csH0pfxSh1JyX7ysBOYCzwBP5vdtyh7AbHIh9O/klnvcv7nr2AYnkFvvPrHRHVVWR0Tj3wicHBGrgVXkbiG4HPj4u3eQacaBwKP5c24HvtHotxWQ+8LlE8Accl9IvSa//0fkvkj5Rn7/Le+ekP+i5jHkbhW5iNyXSU/OH7sVuAyYmV+K8iy5tfWSlKlo+fdqJElqGxGRgD0aLdmRpO2WM9ySJElShgzckiRJUoZcUiJJkiRlyBluSZIkKUMGbkmSJClDzT0YoMPo169fGjp0aHt3Q5IkSZ3YE0888XpKqSSr+jt04B46dCiVlZXt3Q1JkiR1YhFRlWX9LimRJEmSMmTgliRJkjJk4JYkSZIyZOCWJEmSMmTgliRJkjJk4JYkSZIyZOBugaFDhzJ79uxW13PBBRdw2mmnFaBH2dYpSZKkwjFwq1nXXXcdhx12WHt3Q5Ikabtn4O6k6uvr27sLkiRJwsDdYo8//jgjRoygd+/eTJw4kbfffpuVK1cyduxYSkpK6N27N2PHjmXJkiUN58yaNYtevXoREQwYMIAbbriBp59+GoDf/OY3fOhDH6JPnz4ce+yxVFdXN5z38MMPc+CBB7Lrrrty4IEH8vDDD2+2zjlz5nD//fczZMgQLrvsMgYOHMjEiRMBuOOOO9h///0pLi7m0EMPZe7cuQ11XXrppQwfPpyioiJGjBjBrbfeCsDzzz/P2Wefzd///nd69epFcXFxpu+tJElSZ2bgbqEbbriBe+65hxdffJEXXniBiy66iA0bNjBx4kSqqqpYtGgRPXv2ZNKkSQDU1dVx0kknUVFRQW1tLZMmTWLhwoUA/OUvf+Hf/u3f+MMf/sDSpUspLy9n/PjxAKxYsYKjjz6ar3/96yxfvpxvfetbHH300Sxfvvx9dZ5zzjkNdQK8+uqrrFixgqqqKq666iqefPJJvvSlL/HrX/+a5cuX85WvfIVjjz2WdevWATB8+HAefPBB3njjDaZOncppp53G0qVL2Xvvvbnyyis55JBDWL16NatWrWrjd1uSJKkTSSl12J+Pf/zjqSMoLy9Pv/rVrxq277zzzjRs2LD3lXvqqadScXFxSimlP/zhDwlItbW1Dcf79euX9t133/SlL30pfec732nYX1tbm7p27ZpefvnldP3116cDDzxwo3pHjhyZrr322k3Wuc8++6S//vWvqVu3bumtt95qOHb22WenKVOmbFTXhz/84XT//fc3e50f/ehH05/+9KeUUkrXXnttGjVq1BbfG0mSpO0dUJkyzLQFmeGOiN9GxLKIeHYTxyMi/jMiFkbE3Ig4oBDtZmn2/BrGX/Mkn778EWpW17GcXRqOlZeXU11dzdq1a/nKV75CeXk5u+yyC5/85CdZtWoV//qbx/nub+8ldujCI0veajivpKQEgOrqasrLyxv29+rVi759+/LKK6+87xjAjr0H8vPbNl/nu6933HHHhu2qqiqmTZtGcXFxw8/ixYsblq9cf/31DctNiouLefbZZ3n99dcL9A5KkiQJCrek5DrgiM0cPxLYI/9zFvCrArWbidnza/jZ7Jd5rbaOBLyzIfH7++cye34NAIsWLaK0tJRp06axYMECHn30Ud58803+/Xd/AuC1N9exU0kZacM7XHLHvIbzampyf5aWllJVVdXQ3po1a1i+fDmDBw9+37HZ82uYM/9F6nr2Yaf+u72vzsYBOSI2uo6ysjLOP/98Vq1a1fCzdu1aTjnlFKqqqjjzzDO54oorWL58OatWrWLfffcl9yHv/XVJkiRp2xQkcKeU/gas2EyR44Dr87P2jwDFETGoEG1n4eqHFrOufsNG+6r+9t/856xKVqxYwU9+8hNOPvlkamtr6dmzJ8XFxaxYsYKLLrywoXy//UYTXbrx5K/P5cq/vMAll1zSEI5PPfVUrr32WubMmcO6dev4/ve/z8EHH8zQoUM56qijeOGFF/j9739PfX09P/yPa6hd+jJ9R4yi30c+1VDnr/J1vhvim3PmmWdy5ZVX8uijj5JSYs2aNdx5553U1tayZs0aIqJhhvzaa6/l2Wff+wXFgAEDWLJkCXV1dYV8ayVJkj5w2upLk4OBxY22l+T3vU9EnBURlRFRubkwmaVlte8Pmf0P+Bfum/ZVhg0bxrBhw5gyZQqTJ0/mrbfeol+/fowcOZKdP3RQQ/kdunZjxBcu5M1F8/jj1w7j8ssvZ9iwYeywww585jOf4cILL+TEE09k0KBBvPjii8ycOROAvn37cscddzBt2jT69u3LM3ddx0fO/BndexVvVOctjers0qVLs9dRUVHBb37zGyZNmkTv3r350Ic+xHXXXQfAiBEj+Pa3v80hhxzCgAEDeOaZZxg1alTDuZ/+9KfZZ599GDhwIP369SvguytJkvTBEu8uIWh1RRFDgTtSSvs2c+xO4JKU0v/mt+8DvptSemJzdVZUVKTKysqC9G9rjL/mSV5rJnQPKOrOzC9vevn5ls47+OCDOfvssxtu2VeIvmxLnZIkSXpPRDyRUqrIqv62muFeApQ12h4CVG+ibLs7Y1QZPbpu/Nb06LoDZ4wq28QZzZ+3auGTsHYlEw8exO9+9zvmzp3LEUdsbql729QpSZKkttO1jdq5HZgUETOBg4E3UkpL26jtrTZmr9y65qsfWsyy2jr6F3XnjFFlDftbel7XN6uZM3Mqn7t4LcOHD+fmm29m0KCtW7qeRZ2SJElqOwVZUhIRNwKjgX7Aa8BUoBtASunKyN3y4gpydzJZC0xMKW1xrUh7LSmRJEnSB0fWS0oKMsOdUjplC8cT8LVCtCVJkiRtT3y0uyRJkpQhA7ckSZKUIQO3JEmSlCEDtyRJkpQhA7ckSZKUIQO3JEmSlCEDtyRJkpQhA7ckSZKUIQO3JEmSlCEDtyRJkpQhA7ckSZKUIQO3JEmSlCEDtyRJkpQhA7ckSZKUIQO3JEmSlCEDtyRJkpQhA7ckSZKUIQO3JEmSlCEDtyRJkpQhA7ckSZKUIQO3JEmSlCEDtyRJkpQhA7ckSZKUIQO3JEmSlCEDtyRJkpQhA7ckSZKUIQO3JEmSlCEDtyRJkpQhA7ckSZKUIQO3JEmSlCEDtyRJkpQhA7ckSZKUIQO3JEmSlCEDtyRJkpQhA7ckSZKUIQO3JEmSlCEDtyRJkpQhA7ckSZKUIQO3JEmSlCEDtyRJkpQhA7ckSZKUIQO3JEmSlKGCBO6IOCIiFkTEwoj4XjPHd42IWRHxdETMi4iJhWhXkiRJ6uhaHbgjogvwC+BIYARwSkSMaFLsa8BzKaWPAqOBaRHRvbVtS5IkSR1dIWa4DwIWppReSinVATOB45qUSUBRRATQC1gB1BegbUmSJKlDK0TgHgwsbrS9JL+vsSuAvYFq4BngGymlDc1VFhFnRURlRFTW1NQUoHuSJElS+ylE4I5m9qUm258F5gClwP7AFRGxS3OVpZSuSilVpJQqSkpKCtA9SZIkqf0UInAvAcoabQ8hN5Pd2ETglpSzEHgZ2KsAbUuSJEkdWiEC9+PAHhGxe/6LkOOB25uUWQR8BiAiBgB7Ai8VoG1JkiSpQ+va2gpSSvURMQm4B+gC/DalNC8izs4fvxK4ELguIp4htwTlvJTS661tW5IkSeroWh24AVJKdwF3Ndl3ZaPX1cC/FKItSZIkaXvikyYlSZKkDBm4JUmSpAwZuCVJkqQMGbglSZKkDBm4JUmSpAwZuCVJkqQMGbglSZKkDBm4JUmSpAwZuCVJkqQMGbglSZKkDBm4JUmSpAwZuCVJkqQMGbglSZKkDBm4JUmSpAwZuCVJkqQMGbglSZKkDBm4JUmSpAwZuCVJkqQMGbglSZKkDBm4JUmSpAwZuCVJkqQMGbglSZKkDBm4JUmSpAwZuCVJkqQMGbglSZKkDBm4JUmSpAwZuCVJkqQMGbglSZKkDBm4JUmSpAwZuCVJkqQMGbglSZKkDBm4JUmSpAwZuCVJkqQMGbglSZKkDBm4JUmSpAwZuCVJkqQMGbglSZKkDBm4JUmSpAwZuCVJkqQMGbglSZKkDBm4JUmSpAwZuCVJkqQMFSRwR8QREbEgIhZGxPc2UWZ0RMyJiHkR8UAh2pUkSZI6uq6trSAiugC/AA4HlgCPR8TtKaXnGpUpBn4JHJFSWhQR/VvbriRJkrQ9KMQM90HAwpTSSymlOmAmcFyTMqcCt6SUFgGklJYVoF1JkiSpwytE4B4MLG60vSS/r7EPA70j4v6IeCIivrCpyiLirIiojIjKmpqaAnRPkiRJaj+FCNzRzL7UZLsr8HHgaOCzwA8i4sPNVZZSuiqlVJFSqigpKSlA9yRJkqT20+o13ORmtMsabQ8Bqpsp83pKaQ2wJiL+BnwUeKEA7UuSJEkdViFmuB8H9oiI3SOiOzAeuL1JmduAT0RE14jYCTgYeL4AbUuSJEkdWqtnuFNK9RExCbgH6AL8NqU0LyLOzh+/MqX0fETcDcwFNgBXp5SebW3bkiRJUkcXKTVdbt1xVFRUpMrKyvbuhiRJkjqxiHgipVSRVf0+aVKSJEnKkIFbkiRJypCBW5IkScqQgVuSJEnKkIFbkiRJypCBW5IkScqQgVuSJEnKkIFbkiRJypCBW5IkaRs8+eSTfOxjH6OoqIhx48Zx8sknM2XKFFauXMnYsWMpKSmhd+/ejB07liVLljScN3r0aKZMmcKhhx5Kr169OOaYY1i+fDmf//zn2WWXXTjwwAP55z//2VB+/vz5HH744fTp04c999yTP/zhD+1wtWoNA7ckSdJWqqur44QTTuCLX/wiK1as4JRTTuHWW28FYMOGDUycOJGqqioWLVpEz549mTRp0kbnz5w5k+nTp/PKK6/w4osvcsghhzBx4kRWrFjB3nvvzY9+9CMA1qxZw+GHH86pp57KsmXLuPHGG/nqV7/KvHnz2vyate0M3JIkSVvpkUceob6+nq9//et069aNz33ucxx00EEA9O3blxNPPJGddtqJoqIizj//fB544IGNzp84cSLDhw9n11135cgjj2T48OGMGTOGrl27Mm7cOJ566ikA7rjjDoYOHcrEiRPp2rUrBxxwACeeeCI333xzm1+ztl3X9u6AJEnS9qa6uprBgwcTEQ37ysrKAFi7di3f/OY3ufvuu1m5ciUAtbW1vPPOO3Tp0gWAAQMGNJzXs2fP922vXr0agKqqKh599FGKi4sbjtfX1zNhwoTsLk4FZ+CWJElqgdnza7j6ocUsq62D6lpeqlpMSqkhdC9evJjhw4czbdo0FixYwKOPPsrAgQOZM2cOH/vYx0gpbXWbZWVlfOpTn+Lee+8t9OWoDbmkRJIkaQtmz6/hZ7Nf5rXaOhLwTv+9WLM+cc4PLqW+vp7bbruNxx57DMjNZvfs2ZPi4mJWrFjRsB57W4wdO5YXXniB6dOns379etavX8/jjz/O888/X6ArU1swcEuSJG3B1Q8tZl39hobtHbp2Y8QXL+GG66+juLiYGTNmMHbsWHr06MHkyZN566236NevHyNHjuSII47Y5naLior485//zMyZMyktLWXgwIGcd955rFu3rhCXpTYS2/LrjbZSUVGRKisr27sbkiTpA+7Tlz9Cc4kpgL9MHgnAwQcfzNlnn83EiRPbtG9qvYh4IqVUkVX9znBLkiRtQf+i7u/bt2rhkxRteJP6+np+97vfMXfu3FbNZqvzMnBLkiRtwRmjyujRdePYVLd8MX/7yefZddddmTZtGjfffDODBg1qpx6qI/MuJZIkSVswZq8SgIa7lPQv6s75U7/FmBsvaeeeaXtg4JYkSWqBMXuVNARvaWu4pESSJEnKkIFbkiRJypCBW5IkScqQgVuSJEnKkIFbkiRJypCBW5IkScqQgVuSJEnKkIFbkiRJypCBW5IkScqQgVuSJEnKkIFbkiRJypCBW5IkScqQgVuSJEnKkIFbkiRJypCBW5IkScqQgVuSJEnKkIFbkiRJypCBW5IkScqQgVuSJEnKkIFbkiRJypCBW5IkScqQgVuSJEnKUEECd0QcERELImJhRHxvM+UOjIh3IuKkQrQrSZIkdXStDtwR0QX4BXAkMAI4JSJGbKLcZcA9rW1TkiRJ2l4UYob7IGBhSumllFIdMBM4rply5wB/BJYVoE1JkiRpu1CIwD0YWNxoe0l+X4OIGAycAFy5pcoi4qyIqIyIypqamgJ0T5IkSWo/hQjc0cy+1GT7cuC8lNI7W6ospXRVSqkipVRRUlJSgO5JkiRJ7adrAepYApQ12h4CVDcpUwHMjAiAfsBREVGfUvpTAdqXJEmSOqxCBO7HgT0iYnfgFWA8cGrjAiml3d99HRHXAXcYtiVJkvRB0OrAnVKqj4hJ5O4+0gX4bUppXkScnT++xXXbkiRJUmdViBluUkp3AXc12dds0E4pfbEQbUqSJEnbA580KUmSJGXIwC1JkvQBNnToUGbPnt3e3ejUDNySJElShgzckiRJH1ATJkxg0aJFHHPMMfTq1Yt///d/5/bbb2efffahuLiY0aNH8/zzzwPw05/+lBNPPHGj88855xwmT57cHl3frhi4JUmSPqCmT5/ObrvtxqxZs1i9ejXHH388p5xyCpdffjk1NTUcddRRHHPMMdTV1XHaaadx9913s2rVKgDq6+u56aabmDBhQjtfRcdn4JYkSRIAN910E0cffTSHH3443bp149xzz+Wtt97i4YcfZtCgQXzyk5/kv//7vwG4++676devHx//+Mfbudcdn4FbkiTpA2b2/BrGX/Mkn778EWpW1/HkotysdXV1NeXl5Q3ldthhB8rKynjllVcAOP3005kxYwYAM2bMcHa7hQzckiRJHyCz59fws9kv81ptHQnYkOC/n3yV2fNrKC0tpaqqqqFsSonFixczePBgAI4//njmzp3Ls88+yx133MHnP//5drqK7YuBW5Ik6QPk6ocWs65+Q8N2t6I+vLlsCVc/tJh//dd/5c477+S+++5j/fr1TJs2jR49enDooYcCsOOOO3LSSSdx6qmnctBBB7Hbbru112VsVwzckiRJHyDLaus22i7/zBeouvc6/jjpU8yaNYsZM2Zwzjnn0K9fP2bNmsWsWbPo3r17Q/nTTz+dZ555xuUkW6Egj3aXJEnS9qF/UXdeaxS6+33kk/T7yCcZUNSdc798AAAnnHDCJs/fbbfd6Nmz5/tuEahNc4ZbkiTpA+SMUWX06LpxBOzRdQfOGFW2xXM3bNjAz3/+c8aPH88uu+ySVRc7HWe4JUmSPkDG7FUC5NZyL6uto39Rd84YVdawf1PWrFnDgAEDKC8v5+67726LrnYaBm5JkqQPmDF7lWwxYDe18847s3r16ox61Lm5pESSJEnKkIFbkiRJypCBW5IkScqQgVuSJEnKkIFbkiRJypCBW5IkScqQgVuSJEnKkIFbkiRJypCBW5IkScqQgVuSJEnKkIFbkiRJypCBW5IkScqQgVuSJEmZuf/++xkyZEh7d6NdGbglSZKkDBm4JUmSpAwZuCVJktRqQ4cO5ZJLLmHEiBH07t2biRMn8vbbb7+v3KWXXsrw4cMpKipixIgR3HrrrQCsW7eOPn368MwzzzSUXbZsGT179qSmpqbNriMLBm5JkiQVxA033MA999zDiy++yAsvvMBFF130vjLDhw/nwQcf5I033mDq1KmcdtppLF26lB49ejB+/HhmzJjRUPbGG29kzJgxlJSUtOVlFJyBW5IkSQUxadIkysrK6NOnD+effz433njj+8qMGzeO0tJSdthhB04++WT22GMPHnvsMQBOP/10fv/737NhwwYApk+fzoQJE9r0GrJg4JYkSVJBlJWVNbwuLy+nurr6fWWuv/569t9/f4qLiykuLubZZ5/l9ddfB+Dggw9m55135oEHHmD+/PksXLiQY489ts36n5Wu7d0BSZIkbZ9mz6/h6ocWs6y2jprVdfz58ec4+uijAVi0aBGlpaUbla+qquLMM8/kvvvu45BDDqFLly7sv//+pJQaypx++unMmDGDgQMHctJJJ7Hjjju26TVlwcAtSZKkrTZ7fg0/m/0y6+pzyz/e2ZD47VW/ZuTof+Gz+5Xxk5/8hJNPPnmjc9asWUNENKzJvvbaa3n22Wc3KjNhwgT2228/ioqKmD59ettcTMZcUiJJkqStdvVDixvC9rtKDjics089gWHDhjFs2DCmTJmy0fERI0bw7W9/m0MOOYQBAwbwzDPPMGrUqI3KDBkyhAMOOICI4BOf+ETm19EWnOGWJEnSVltWW/e+fUVlezN0zOn8ZfLIhn2jR49myZIlDdsXX3wxF1988Wbr3m233Rg5ciQRUbgOtyMDtyRJkrZa/6LuvNZM6O5f1L1V9f7zn//klltu4amnnmpVPR2JS0okSZK01c4YVUaPrhtHye5dduCMUWWbOGPLfvCDH7Dvvvvyne98h9133721XewwovG3QjuaioqKVFlZ2d7dkCRJUjMa36Wkf1F3zhhVxpi9tr+H1ETEEymliqzqd0mJJEmStsmYvUq2y9PgbSQAABA4SURBVIDd1lxSIkmSJGWoIIE7Io6IiAURsTAivtfM8c9HxNz8z8MR8dFCtCtJkiR1dK0O3BHRBfgFcCQwAjglIkY0KfYy8KmU0n7AhcBVrW1XkiRJ2h4UYob7IGBhSumllFIdMBM4rnGBlNLDKaWV+c1HgCEFaFeSJEnq8AoRuAcDixttL8nv25QvA/+zqYMRcVZEVEZEZU1NTQG6J0mSJLWfQgTu5h4B1Oy9BiPi/5AL3OdtqrKU0lUppYqUUkVJid96lSRJ0vatELcFXAI0vsP5EKC6aaGI2A+4GjgypbS8AO1KkiRJHV4hZrgfB/aIiN0jojswHri9cYGI2A24BZiQUnqhAG1KkiRJ24VWz3CnlOojYhJwD9AF+G1KaV5EnJ0/fiXwQ6Av8MuIAKjP8mk+kiRJUkfho90lSZL0gZb1o9190qQkSZKUIQO3JEmSlCEDtyRJkpQhA7ckSZKUIQO3JEmSlCEDtyRJkpQhA7ckSZKUIQO3JEmSlCEDtyRJkpQhA7ckSZKUIQO3JEmSlCEDtyRJkpQhA7ckSZKUIQO3JEmSlCEDtyRJkpQhA7ckSZKUIQO3JEmSlCEDtyRJkpQhA7ckSZKUIQO3JEmSlCEDtyRJkpQhA7ckSZKUIQO3JEmSlCEDtyRJkpQhA7ckSZKUIQO3JEmSlCEDtyRJkpQhA7ckSZKUIQO3JEmSlCEDtyRJkpQhA7ckSZKUIQO3JEmSlCEDtyRJkpQhA7ckSZKUIQO3JEmSlCEDtyRJkpQhA7ckSZKUIQO3JEmSlCEDtyRJkpQhA7ckSZKUIQO3JEmSlCEDtyRJkpShggTuiDgiIhZExMKI+F4zxyMi/jN/fG5EHFCIdiVJkqSOrtWBOyK6AL8AjgRGAKdExIgmxY4E9sj/nAX8qrXtSpIkSduDQsxwHwQsTCm9lFKqA2YCxzUpcxxwfcp5BCiOiEEFaFuSJEnq0AoRuAcDixttL8nv29oyAETEWRFRGRGVNTU1BeieJEmS1H4KEbijmX1pG8rkdqZ0VUqpIqVUUVJS0urOSZIkSe2pEIF7CVDWaHsIUL0NZSRJkqROpxCB+3Fgj4jYPSK6A+OB25uUuR34Qv5uJSOBN1JKSwvQtiRJktShdW1tBSml+oiYBNwDdAF+m1KaFxFn549fCdwFHAUsBNYCE1vbriRJkrQ9aHXgBkgp3UUuVDfed2Wj1wn4WiHakiRJkrYnPmlSkiRJypCBW5IkScqQgVuSJEnKkIFbkiRJypCBW5IkScqQgVuSJEnKkIFbkiRJypCBW5IkScqQgVuSJEnt7rLLLmPw4MEUFRWx5557ct9997Fu3TomT55MaWkppaWlTJ48mXXr1gFw//33M2TIEKZNm0b//v0ZNGgQ1157bTtfRfMM3JIkSWpXCxYs4IorruDxxx+ntraWe+65h6FDh3LxxRfzyCOPMGfOHJ5++mkee+wxLrrooobzXn31Vd544w1eeeUVrrnmGr72ta+xcuXKdryS5hm4JUmS1K66dOnCunXreO6551i/fj1Dhw5l+PDh3HDDDfzwhz+kf//+lJSUMHXqVKZPn95wXrdu3fjhD39It27dOOqoo+jVqxcLFixoxytpXtf27oAkSZI+eGbPr+HqhxazrLaO/kXdOeO7P+aCCy5g3rx5fPazn+XnP/851dXVlJeXN5xTXl5OdXV1w3bfvn3p2vW9OLvTTjuxevXqNr2OlnCGW5IkSW1q9vwafjb7ZV6rrSMBr9XW8eSOB3DB1bdSVVVFRHDeeedRWlpKVVVVw3mLFi2itLS0/Tq+jQzckiRJalNXP7SYdfUbGrbXLqvi1ecf49f3v8iOO+5Iz5496dKlC6eccgoXXXQRNTU1vP766/z4xz/mtNNOa8eebxuXlEiSJKlNLaut22h7Q/16Xrrjlzx7TRUDv9+DQw89lKuuuoo+ffrw5ptvst9++wEwbtw4pkyZ0h5dbpVIKbV3HzapoqIiVVZWtnc3JEmSVEDjr3mS15qEboABRd2Z+eUD2rw/EfFESqkiq/pdUiJJkqQ2dcaoMnp03TiG9ui6A2eMKmunHmXLJSWSJElqU2P2KgHY+C4lo8oa9nc2Bm5JkiS1uTF7lXTagN2US0okSZKkDBm4JUmSpAwZuCVJkqQMGbglSZKkDBm4JUmS1K6++MUvZv5AmwsuuKDdnlJp4JYkSdIHXkT8MyLGZFG3gVuSJElqhcjZZK42cEuSJKlNPfXUUxxwwAEUFRVx8skn8/bbbwOwcuVKxo4dS0lJCb1792bs2LEsWbKk4byhQ4cye/bshu2my0Suv/56ysvL6du3LxdeeOH7ytfV1fGFL3yBoqIi9tlnHyorK989tDuwGzArIlZHxHcBImJkRDwcEasi4umIGP3uCRFxf0RcHBEPAWuBYZu6XgO3JEmS2kxdXR3HH388EyZMYMWKFYwbN44//vGPAGzYsIGJEydSVVXFokWL6NmzJ5MmTWpRvc899xxf/epXueGGG1i6dClvvPEGr7zyykZlbr/9dsaPH8+qVas49thjG9f9MrAIOCal1Cul9O8RMRi4E7gI6AOcC/wxIho/rWcCcBZQBFRtqm8GbkmSJLWZRx55hPXr1zN58mS6devGSSedxIEHHghA3759OfHEE9lpp50oKiri/PPP54EHHmhRvTfffDPHHHMMhx12GN27d+fHP/4xEbFRmcMOO4yjjjqKLl26MGHCBJ5++unNVXkacFdK6a6U0oaU0r1AJXBUozLXpZTmpZTqU0rrN1WRj3aXJElSm6murmbw4MEbheHy8nIA1q5dyze/+U3uvvtuVq5cCUBtbS3vvPMOXbp02WK9ZWVlDds77bQTffv23ajMwIEDNzr+9ttvU19fv6kqy4FxEXFMo33dgL822l682U7lGbglSZKUqdnza7j6ocUsq62D6lpeqlpMSqkhdC9atIjhw4czbdo0FixYwKOPPsrAgQOZM2cOH/vYx0gpAbDzzjuzdu3ahnpfffXVhteDBg1iwYIFDdtvvfUWy5cv35pupibbi4HpKaUzt+KcZrmkRJIkSZmZPb+Gn81+mddq60jAO/33Ym09fO38n1BfX88tt9zCY489BuRms3v27ElxcTErVqzgRz/60UZ17b///sycOZP169dTWVnJzTff3HDspJNOYtasWTz88MPU1dUxderUhqDeQq+x8RcfZwDHRMRnI6JLROwYEaMjYsjWvgcGbkmSJGXm6ocWs65+Q8P2Dl27MeKLl3DjjOn07t2bm266ic997nMATJ48mbfeeot+/foxcuRIjjjiiI3quvDCC3nxxRfp3bs3U6dO5dRTT204ts8++/Bf//VfjB8/nkGDBlFUVET//v3p0aNHS7t6CTAlf0eSc1NKi4HjgO8DNeRmvL/DNuTn2Mrk36YqKipSo9u1SJIkaTvz6csfaXbdRQB/mTwys3ZXr15NcXEx//jHP9h99903WzYinkgpVWTVF2e4JUmSlJn+Rd23an9rzJo1i7Vr17JmzRrOPfdcPvKRjzB06NCCt7O1DNySJEnKzBmjyujRdePI2aPrDpwxqmwTZ2y72267jdLSUkpLS/nHP/7BzJkz33drwPbgkhJJkiRlqvFdSvoXdeeMUWWM2atkyye2kayXlHhbQEmSJGVqzF4lHSpgtzWXlEiSJEkZMnBLkiRJGTJwS5IkSRkycEuSJEkZalXgjog+EXFvRPwj/2fvZsqURcRfI+L5iJgXEd9oTZuSJEnS9qS1M9zfA+5LKe0B3Jffbqoe+HZKaW9gJPC1iBjRynYlSZKk7UJrA/dxwO/yr38HHN+0QEppaUrpyfzrWuB5YHAr25UkSZK2C60N3ANSSkshF6yB/psrHBFDgY8Bj26mzFkRURkRlTU1Na3sniRJktS+tvjgm4iYDQxs5tD5W9NQRPQC/ghMTim9ualyKaWrgKsg96TJrWlDkiRJ6mi2GLhTSmM2dSwiXouIQSmlpRExCFi2iXLdyIXtG1JKt2xzbyVJkqTtTKS07ZPIEfFTYHlK6dKI+B7QJ6X03SZlgtz67hUppclbWX8NULXNHewY+gGvt3cnVHCOa+fkuHY+jmnn5Lh2Tu05ruUppcyePd/awN0X+AOwG7AIGJdSWhERpcDVKaWjIuIw4EHgGWBD/tTvp5Tual3Xtw8RUZlSqmjvfqiwHNfOyXHtfBzTzslx7Zw687hucUnJ5qSUlgOfaWZ/NXBU/vX/AtGadiRJkqTtlU+alCRJkjJk4M7eVe3dAWXCce2cHNfOxzHtnBzXzqnTjmur1nBLkiRJ2jxnuCVJkqQMGbgLJCKOiIgFEbEwf4vEpscjIv4zf3xuRBzQHv3U1mnBuH4+P55zI+LhiPhoe/RTLbelMW1U7sCIeCciTmrL/mnbtGRcI2J0RMyJiHkR8UBb91FbrwX/Bu8aEbMi4un8uE5sj36q5SLitxGxLCKe3cTxTpmXDNwFEBFdgF8ARwIjgFMiYkSTYkcCe+R/zgJ+1aad1FZr4bi+DHwqpbQfcCGdeP1ZZ9DCMX233GXAPW3bQ22LloxrRBQDvwSOTSntA4xr845qq7Tw7+vXgOdSSh8FRgPTIqJ7m3ZUW+s64IjNHO+UecnAXRgHAQtTSi+llOqAmcBxTcocB1yfch4BivNP51THtcVxTSk9nFJamd98BBjSxn3U1mnJ31WAc8g9HbfZp+eqw2nJuJ4K3JJSWgSQUnJsO76WjGsCivIP2esFrADq27ab2hoppb+RG6dN6ZR5ycBdGIOBxY22l+T3bW0ZdSxbO2ZfBv4n0x6ptbY4phExGDgBuLIN+6XWacnf1Q8DvSPi/oh4IiK+0Ga907ZqybheAewNVJN7wN43Ukob0PasU+alVj34Rg2ae7BP09u/tKSMOpYWj1lE/B9ygfuwTHuk1mrJmF4OnJdSeic3aabtQEvGtSvwcXIPa+sJ/D0iHkkpvZB157TNWjKunwXmAJ8GhgP3RsSDKaU3s+6cMtMp85KBuzCWAGWNtoeQ+7S9tWXUsbRozCJiP+Bq4Mj801fVcbVkTCuAmfmw3Q84KiLqU0p/apsuahu09N/g11NKa4A1EfE34KOAgbvjasm4TgQuTbl7HC+MiJeBvYDH2qaLykCnzEsuKSmMx4E9ImL3/Jc1xgO3NylzO/CF/LdvRwJvpJSWtnVHtVW2OK4RsRtwCzDBmbLtwhbHNKW0e0ppaEppKHAz8FXDdofXkn+DbwM+ERFdI2In4GDg+Tbup7ZOS8Z1EbnfWhARA4A9gZfatJcqtE6Zl5zhLoCUUn1ETCJ3R4MuwG9TSvMi4uz88SuBu4CjgIXAWnKfytWBtXBcfwj0BX6ZnxGtTylVtFeftXktHFNtZ1oyriml5yPibmAusAG4OqXU7G3J1DG08O/rhcB1EfEMuaUI56WUXm+3TmuLIuJGcneU6RcRS4CpQDfo3HnJJ01KkiRJGXJJiSRJkpQhA7ckSZKUIQO3JEmSlCEDtyRJkpQhA7ckSZKUIQO3JEmSlCEDtyRJkpQhA7ckSZKUof8PP65Xa86HgsgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "M_w2v_words = w2v_model.wv[viz_words]\n",
    "M_w2v_words_2d = f.matrix_reduce(M_w2v_words, 'svd', n_dim=2)\n",
    "w2v_word2ind = {e:i for i, e in enumerate(viz_words)}\n",
    "f.plot_vectors_2d(M_w2v_words_2d, viz_words, w2v_word2ind)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Convert test string into Word2Vec vector\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.31646168, -0.56161016,  1.0844812 , -0.18825021,  1.3932753 ,\n",
       "        0.14343838,  1.0295175 ,  0.9405424 , -0.41131052, -0.04139958,\n",
       "       -0.13122456, -0.2395148 ,  0.12359241, -0.17586541,  0.2841231 ,\n",
       "       -0.662642  , -0.4899149 , -0.82126445, -0.4849453 , -0.01437436,\n",
       "        0.5155698 ,  0.00157979, -0.12170897,  0.12294563, -0.56744236,\n",
       "        0.09137762,  0.13993731, -0.50038534, -0.6875189 ,  0.20462283,\n",
       "       -0.2606817 , -0.20736933, -0.14295428,  0.79791254, -0.55176574,\n",
       "        0.64750224, -0.573746  ,  0.03562661, -0.3149734 , -0.36117828,\n",
       "       -0.5166572 ,  1.468988  ,  0.35668358, -0.93026483,  0.07668543,\n",
       "        0.16054472, -0.542458  , -0.80053425,  0.49514374, -0.70084465,\n",
       "       -0.85109246,  0.36010742,  0.28695327, -0.6446176 , -0.26563647,\n",
       "       -0.32876414,  0.233358  ,  0.03792597, -1.2165335 , -0.46383953,\n",
       "       -0.761924  , -1.0288746 , -0.37649328, -0.4844675 ,  1.208699  ,\n",
       "       -0.18409283,  0.23793969,  0.4282012 ,  0.42860514, -0.00332098,\n",
       "       -0.11296084, -0.24559149,  0.19468245,  0.4535489 , -0.7435466 ,\n",
       "        0.44443297,  0.6071012 , -0.34038827,  1.0002236 , -0.276553  ,\n",
       "        0.90246135,  0.05409582,  0.74569976, -0.33316666, -0.3515421 ,\n",
       "        0.6093632 ,  0.17268491, -1.0197427 , -0.12005872,  0.8795897 ,\n",
       "       -0.2181303 , -0.6902615 ,  0.7760292 , -0.04611551, -0.04813138,\n",
       "        0.09687453, -0.25432467,  0.6559578 , -0.50627196, -0.353421  ],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f.avg_w2v_embeddings(review_tokens_test, w2v_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "f.save_pickle(M_word2vec, 'data/M_word2vec_100k.pickle')\n",
    "del M_word2vec"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "fastai-nlp",
   "language": "python",
   "name": "fastai-nlp"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}