Recommendations_with_IBM_submit.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Recommendations with IBM (Submit Version)\n",
    "\n",
    "In this notebook, you will be putting your recommendation skills to use on real data from the IBM Watson Studio platform. \n",
    "\n",
    "\n",
    "You may either submit your notebook through the workspace here, or you may work from your local machine and submit through the next page.  Either way assure that your code passes the project [RUBRIC](https://review.udacity.com/#!/rubrics/2322/view).  **Please save regularly.**\n",
    "\n",
    "By following the table of contents, you will build out a number of different methods for making recommendations that can be used for different situations. \n",
    "\n",
    "\n",
    "## Table of Contents\n",
    "\n",
    "I. [Exploratory Data Analysis](#Exploratory-Data-Analysis)<br>\n",
    "II. [Rank Based Recommendations](#Rank)<br>\n",
    "III. [User-User Based Collaborative Filtering](#User-User)<br>\n",
    "IV. [Content Based Recommendations (EXTRA - NOT REQUIRED)](#Content-Recs)<br>\n",
    "V. [Matrix Factorization](#Matrix-Fact)<br>\n",
    "VI. [Extras & Concluding](#conclusions)\n",
    "\n",
    "At the end of the notebook, you will find directions for how to submit your work.  Let's get started by importing the necessary libraries and reading in the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>article_id</th>\n",
       "      <th>title</th>\n",
       "      <th>email</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1430.0</td>\n",
       "      <td>using pixiedust for fast, flexible, and easier...</td>\n",
       "      <td>ef5f11f77ba020cd36e1105a00ab868bbdbf7fe7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1314.0</td>\n",
       "      <td>healthcare python streaming application demo</td>\n",
       "      <td>083cbdfa93c8444beaa4c5f5e0f5f9198e4f9e0b</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1429.0</td>\n",
       "      <td>use deep learning for image classification</td>\n",
       "      <td>b96a4f2e92d8572034b1e9b28f9ac673765cd074</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1338.0</td>\n",
       "      <td>ml optimization using cognitive assistant</td>\n",
       "      <td>06485706b34a5c9bf2a0ecdac41daf7e7654ceb7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1276.0</td>\n",
       "      <td>deploy your python model as a restful api</td>\n",
       "      <td>f01220c46fc92c6e6b161b1849de11faacd7ccb2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   article_id                                              title  \\\n",
       "0      1430.0  using pixiedust for fast, flexible, and easier...   \n",
       "1      1314.0       healthcare python streaming application demo   \n",
       "2      1429.0         use deep learning for image classification   \n",
       "3      1338.0          ml optimization using cognitive assistant   \n",
       "4      1276.0          deploy your python model as a restful api   \n",
       "\n",
       "                                      email  \n",
       "0  ef5f11f77ba020cd36e1105a00ab868bbdbf7fe7  \n",
       "1  083cbdfa93c8444beaa4c5f5e0f5f9198e4f9e0b  \n",
       "2  b96a4f2e92d8572034b1e9b28f9ac673765cd074  \n",
       "3  06485706b34a5c9bf2a0ecdac41daf7e7654ceb7  \n",
       "4  f01220c46fc92c6e6b161b1849de11faacd7ccb2  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import random\n",
    "from collections import Counter\n",
    "from pprint import pp\n",
    "import time\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import project_tests as t\n",
    "import pickle\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "df = pd.read_csv('data/user-item-interactions.csv')\n",
    "df_content = pd.read_csv('data/articles_community.csv')\n",
    "del df['Unnamed: 0']\n",
    "del df_content['Unnamed: 0']\n",
    "\n",
    "# Show df to get an idea of the data\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a class=\"anchor\" id=\"Exploratory-Data-Analysis\">Part I : Exploratory Data Analysis</a>\n",
    "\n",
    "Use the dictionary and cells below to provide some insight into the descriptive statistics of the data.\n",
    "\n",
    "`1.` What is the distribution of how many articles a user interacts with in the dataset?  Provide a visual and descriptive statistics to assist with giving a look at the number of times each user interacts with an article.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "median_val = df.groupby('email').size().sort_values(ascending=True).median()\n",
    "max_views_by_user = df.groupby('email').size().sort_values(ascending=False).head(1).values[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`2.` Explore and remove duplicate articles from the **df_content** dataframe.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>doc_body</th>\n",
       "      <th>doc_description</th>\n",
       "      <th>doc_full_name</th>\n",
       "      <th>doc_status</th>\n",
       "      <th>article_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>50</th>\n",
       "      <td>Follow Sign in / Sign up Home About Insight Da...</td>\n",
       "      <td>Community Detection at Scale</td>\n",
       "      <td>Graph-based machine learning</td>\n",
       "      <td>Live</td>\n",
       "      <td>50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>365</th>\n",
       "      <td>Follow Sign in / Sign up Home About Insight Da...</td>\n",
       "      <td>During the seven-week Insight Data Engineering...</td>\n",
       "      <td>Graph-based machine learning</td>\n",
       "      <td>Live</td>\n",
       "      <td>50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>221</th>\n",
       "      <td>* United States\\r\\n\\r\\nIBM® * Site map\\r\\n\\r\\n...</td>\n",
       "      <td>When used to make sense of huge amounts of con...</td>\n",
       "      <td>How smart catalogs can turn the big data flood...</td>\n",
       "      <td>Live</td>\n",
       "      <td>221</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>692</th>\n",
       "      <td>Homepage Follow Sign in / Sign up Homepage * H...</td>\n",
       "      <td>One of the earliest documented catalogs was co...</td>\n",
       "      <td>How smart catalogs can turn the big data flood...</td>\n",
       "      <td>Live</td>\n",
       "      <td>221</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>232</th>\n",
       "      <td>Homepage Follow Sign in Get started Homepage *...</td>\n",
       "      <td>If you are like most data scientists, you are ...</td>\n",
       "      <td>Self-service data preparation with IBM Data Re...</td>\n",
       "      <td>Live</td>\n",
       "      <td>232</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>971</th>\n",
       "      <td>Homepage Follow Sign in Get started * Home\\r\\n...</td>\n",
       "      <td>If you are like most data scientists, you are ...</td>\n",
       "      <td>Self-service data preparation with IBM Data Re...</td>\n",
       "      <td>Live</td>\n",
       "      <td>232</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>399</th>\n",
       "      <td>Homepage Follow Sign in Get started * Home\\r\\n...</td>\n",
       "      <td>Today’s world of data science leverages data f...</td>\n",
       "      <td>Using Apache Spark as a parallel processing fr...</td>\n",
       "      <td>Live</td>\n",
       "      <td>398</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>761</th>\n",
       "      <td>Homepage Follow Sign in Get started Homepage *...</td>\n",
       "      <td>Today’s world of data science leverages data f...</td>\n",
       "      <td>Using Apache Spark as a parallel processing fr...</td>\n",
       "      <td>Live</td>\n",
       "      <td>398</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>578</th>\n",
       "      <td>This video shows you how to construct queries ...</td>\n",
       "      <td>This video shows you how to construct queries ...</td>\n",
       "      <td>Use the Primary Index</td>\n",
       "      <td>Live</td>\n",
       "      <td>577</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>970</th>\n",
       "      <td>This video shows you how to construct queries ...</td>\n",
       "      <td>This video shows you how to construct queries ...</td>\n",
       "      <td>Use the Primary Index</td>\n",
       "      <td>Live</td>\n",
       "      <td>577</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              doc_body  \\\n",
       "50   Follow Sign in / Sign up Home About Insight Da...   \n",
       "365  Follow Sign in / Sign up Home About Insight Da...   \n",
       "221  * United States\\r\\n\\r\\nIBM® * Site map\\r\\n\\r\\n...   \n",
       "692  Homepage Follow Sign in / Sign up Homepage * H...   \n",
       "232  Homepage Follow Sign in Get started Homepage *...   \n",
       "971  Homepage Follow Sign in Get started * Home\\r\\n...   \n",
       "399  Homepage Follow Sign in Get started * Home\\r\\n...   \n",
       "761  Homepage Follow Sign in Get started Homepage *...   \n",
       "578  This video shows you how to construct queries ...   \n",
       "970  This video shows you how to construct queries ...   \n",
       "\n",
       "                                       doc_description  \\\n",
       "50                        Community Detection at Scale   \n",
       "365  During the seven-week Insight Data Engineering...   \n",
       "221  When used to make sense of huge amounts of con...   \n",
       "692  One of the earliest documented catalogs was co...   \n",
       "232  If you are like most data scientists, you are ...   \n",
       "971  If you are like most data scientists, you are ...   \n",
       "399  Today’s world of data science leverages data f...   \n",
       "761  Today’s world of data science leverages data f...   \n",
       "578  This video shows you how to construct queries ...   \n",
       "970  This video shows you how to construct queries ...   \n",
       "\n",
       "                                         doc_full_name doc_status  article_id  \n",
       "50                        Graph-based machine learning       Live          50  \n",
       "365                       Graph-based machine learning       Live          50  \n",
       "221  How smart catalogs can turn the big data flood...       Live         221  \n",
       "692  How smart catalogs can turn the big data flood...       Live         221  \n",
       "232  Self-service data preparation with IBM Data Re...       Live         232  \n",
       "971  Self-service data preparation with IBM Data Re...       Live         232  \n",
       "399  Using Apache Spark as a parallel processing fr...       Live         398  \n",
       "761  Using Apache Spark as a parallel processing fr...       Live         398  \n",
       "578                              Use the Primary Index       Live         577  \n",
       "970                              Use the Primary Index       Live         577  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find and explore duplicate articles\n",
    "\n",
    "dups = df_content.groupby('article_id').size().sort_values(ascending=False) # duplicates of article id\n",
    "dups_idx = dups[dups > 1].index # duplicates index\n",
    "\n",
    "df_content[df_content['article_id'].isin(dups_idx)].sort_values(by='article_id') # retrieve duplicated content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove any rows that have the same article_id - only keep the first\n",
    "\n",
    "df_content = df_content.drop_duplicates('article_id', keep='first')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`3.` Use the cells below to find:\n",
    "\n",
    "**a.** The number of unique articles that have an interaction with a user.  \n",
    "**b.** The number of unique articles in the dataset (whether they have any interactions or not).<br>\n",
    "**c.** The number of unique users in the dataset. (excluding null values) <br>\n",
    "**d.** The number of user-article interactions in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "unique_articles = df.article_id.nunique() # The number of unique articles that have at least one interaction\n",
    "total_articles = df_content.article_id.nunique() # The number of unique articles on the IBM platform\n",
    "unique_users = df.email.nunique() # The number of unique users\n",
    "user_article_interactions = df.shape[0] # The number of user-article interactions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`4.` Use the cells below to find the most viewed **article_id**, as well as how often it was viewed.  After talking to the company leaders, the `email_mapper` function was deemed a reasonable way to map users to ids.  There were a small number of null values, and it was found that all of these null values likely belonged to a single user (which is how they are stored using the function below)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "most_viewed_ds = df.groupby('article_id').size().sort_values(ascending=False).head(1)\n",
    "\n",
    "# The most viewed article in the dataset as a string with one value following the decimal \n",
    "most_viewed_article_id = str(most_viewed_ds.index[0])\n",
    "\n",
    "# The most viewed article in the dataset was viewed how many times?\n",
    "max_views = most_viewed_ds.values[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>article_id</th>\n",
       "      <th>title</th>\n",
       "      <th>user_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1430.0</td>\n",
       "      <td>using pixiedust for fast, flexible, and easier...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1314.0</td>\n",
       "      <td>healthcare python streaming application demo</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1429.0</td>\n",
       "      <td>use deep learning for image classification</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1338.0</td>\n",
       "      <td>ml optimization using cognitive assistant</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1276.0</td>\n",
       "      <td>deploy your python model as a restful api</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   article_id                                              title  user_id\n",
       "0      1430.0  using pixiedust for fast, flexible, and easier...        1\n",
       "1      1314.0       healthcare python streaming application demo        2\n",
       "2      1429.0         use deep learning for image classification        3\n",
       "3      1338.0          ml optimization using cognitive assistant        4\n",
       "4      1276.0          deploy your python model as a restful api        5"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## No need to change the code here - this will be helpful for later parts of the notebook\n",
    "# Run this cell to map the user email to a user_id column and remove the email column\n",
    "\n",
    "def email_mapper():\n",
    "    coded_dict = dict()\n",
    "    cter = 1\n",
    "    email_encoded = []\n",
    "    \n",
    "    for val in df['email']:\n",
    "        if val not in coded_dict:\n",
    "            coded_dict[val] = cter\n",
    "            cter+=1\n",
    "        \n",
    "        email_encoded.append(coded_dict[val])\n",
    "    return email_encoded\n",
    "\n",
    "email_encoded = email_mapper()\n",
    "del df['email']\n",
    "df['user_id'] = email_encoded\n",
    "\n",
    "# show header\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "It looks like you have everything right here! Nice job!\n"
     ]
    }
   ],
   "source": [
    "## If you stored all your results in the variable names above, \n",
    "## you shouldn't need to change anything in this cell\n",
    "\n",
    "sol_1_dict = {\n",
    "    '`50% of individuals have _____ or fewer interactions.`': median_val,\n",
    "    '`The total number of user-article interactions in the dataset is ______.`': user_article_interactions,\n",
    "    '`The maximum number of user-article interactions by any 1 user is ______.`': max_views_by_user,\n",
    "    '`The most viewed article in the dataset was viewed _____ times.`': max_views,\n",
    "    '`The article_id of the most viewed article is ______.`': most_viewed_article_id,\n",
    "    '`The number of unique articles that have at least 1 rating ______.`': unique_articles,\n",
    "    '`The number of unique users in the dataset is ______`': unique_users,\n",
    "    '`The number of unique articles on the IBM platform`': total_articles\n",
    "}\n",
    "\n",
    "# Test your dictionary against the solution\n",
    "t.sol_1_test(sol_1_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a class=\"anchor\" id=\"Rank\">Part II: Rank-Based Recommendations</a>\n",
    "\n",
    "Unlike in the earlier lessons, we don't actually have ratings for whether a user liked an article or not.  We only know that a user has interacted with an article.  In these cases, the popularity of an article can really only be based on how often an article was interacted with.\n",
    "\n",
    "`1.` Fill in the function below to return the **n** top articles ordered with most interactions as the top. Test your function using the tests below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_top_articles(n, df=df):\n",
    "    '''\n",
    "    INPUT:\n",
    "    n - (int) the number of top articles to return\n",
    "    df - (pandas dataframe) df as defined at the top of the notebook \n",
    "    \n",
    "    OUTPUT:\n",
    "    top_articles - (list) A list of the top 'n' article titles \n",
    "    \n",
    "    '''\n",
    "    # Your code here\n",
    "    article_ids_ds = df.groupby('article_id').size().sort_values(ascending=False).head(n)\n",
    "    ids = article_ids_ds.index\n",
    "    \n",
    "    top_articles = []\n",
    "    for i in ids:\n",
    "        top_articles.append(df[df['article_id'] == i].head(1).title.values[0])\n",
    "    \n",
    "    \n",
    "    \n",
    "    return top_articles # Return the top article titles from df (not df_content)\n",
    "\n",
    "\n",
    "def get_top_article_ids(n, df=df):\n",
    "    '''\n",
    "    INPUT:\n",
    "    n - (int) the number of top articles to return\n",
    "    df - (pandas dataframe) df as defined at the top of the notebook \n",
    "    \n",
    "    OUTPUT:\n",
    "    top_articles_ids - (list) A list of the top 'n' article titles \n",
    "    \n",
    "    '''\n",
    "    # Your code here\n",
    "    article_ids_ds = df.groupby('article_id').size().sort_values(ascending=False).head(n)\n",
    "    ids = article_ids_ds.index\n",
    "    top_articles_ids = list(ids)\n",
    " \n",
    "    return top_articles_ids # Return the top article ids"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['use deep learning for image classification', 'insights from new york car accident reports', 'visualize car data with brunel', 'use xgboost, scikit-learn & ibm watson machine learning apis', 'predicting churn with the spss random tree algorithm', 'healthcare python streaming application demo', 'finding optimal locations of new store using decision optimization', 'apache spark lab, part 1: basic concepts', 'analyze energy consumption in buildings', 'gosales transactions for logistic regression model']\n",
      "[1429.0, 1330.0, 1431.0, 1427.0, 1364.0, 1314.0, 1293.0, 1170.0, 1162.0, 1304.0]\n"
     ]
    }
   ],
   "source": [
    "print(get_top_articles(10))\n",
    "print(get_top_article_ids(10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Your top_5 looks like the solution list! Nice job.\n",
      "Your top_10 looks like the solution list! Nice job.\n",
      "Your top_20 looks like the solution list! Nice job.\n"
     ]
    }
   ],
   "source": [
    "# Test your function by returning the top 5, 10, and 20 articles\n",
    "top_5 = get_top_articles(5)\n",
    "top_10 = get_top_articles(10)\n",
    "top_20 = get_top_articles(20)\n",
    "\n",
    "# Test each of your three lists from above\n",
    "t.sol_2_test(get_top_articles)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a class=\"anchor\" id=\"User-User\">Part III: User-User Based Collaborative Filtering</a>\n",
    "\n",
    "\n",
    "`1.` Use the function below to reformat the **df** dataframe to be shaped with users as the rows and articles as the columns.  \n",
    "\n",
    "* Each **user** should only appear in each **row** once.\n",
    "\n",
    "\n",
    "* Each **article** should only show up in one **column**.  \n",
    "\n",
    "\n",
    "* **If a user has interacted with an article, then place a 1 where the user-row meets for that article-column**.  It does not matter how many times a user has interacted with the article, all entries where a user has interacted with an article should be a 1.  \n",
    "\n",
    "\n",
    "* **If a user has not interacted with an item, then place a zero where the user-row meets for that article-column**. \n",
    "\n",
    "Use the tests to make sure the basic structure of your matrix matches what is expected by the solution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create the user-article matrix with 1's and 0's\n",
    "\n",
    "def create_user_item_matrix(df):\n",
    "    '''\n",
    "    INPUT:\n",
    "    df - pandas dataframe with article_id, title, user_id columns\n",
    "    \n",
    "    OUTPUT:\n",
    "    user_item - user item matrix \n",
    "    \n",
    "    Description:\n",
    "    Return a matrix with user ids as rows and article ids on the columns with 1 values where a user interacted with \n",
    "    an article and a 0 otherwise\n",
    "    '''\n",
    "    \n",
    "    df_user_item = df.copy()\n",
    "    \n",
    "    # Fill in the function here\n",
    "    df_user_item['title'] = 1\n",
    "    df_user_item = df_user_item[['user_id', 'article_id', 'title']]\n",
    "    \n",
    "    user_item = pd.pivot_table(df_user_item, index='user_id', columns='article_id', values='title', fill_value=0)\n",
    "    \n",
    "    return user_item # return the user_item matrix \n",
    "\n",
    "user_item = create_user_item_matrix(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>article_id</th>\n",
       "      <th>0.0</th>\n",
       "      <th>2.0</th>\n",
       "      <th>4.0</th>\n",
       "      <th>8.0</th>\n",
       "      <th>9.0</th>\n",
       "      <th>12.0</th>\n",
       "      <th>14.0</th>\n",
       "      <th>15.0</th>\n",
       "      <th>16.0</th>\n",
       "      <th>18.0</th>\n",
       "      <th>...</th>\n",
       "      <th>1434.0</th>\n",
       "      <th>1435.0</th>\n",
       "      <th>1436.0</th>\n",
       "      <th>1437.0</th>\n",
       "      <th>1439.0</th>\n",
       "      <th>1440.0</th>\n",
       "      <th>1441.0</th>\n",
       "      <th>1442.0</th>\n",
       "      <th>1443.0</th>\n",
       "      <th>1444.0</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>user_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5145</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5146</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5147</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5148</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5149</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5149 rows × 714 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "article_id  0.0     2.0     4.0     8.0     9.0     12.0    14.0    15.0    \\\n",
       "user_id                                                                      \n",
       "1                0       0       0       0       0       0       0       0   \n",
       "2                0       0       0       0       0       0       0       0   \n",
       "3                0       0       0       0       0       1       0       0   \n",
       "4                0       0       0       0       0       0       0       0   \n",
       "5                0       0       0       0       0       0       0       0   \n",
       "...            ...     ...     ...     ...     ...     ...     ...     ...   \n",
       "5145             0       0       0       0       0       0       0       0   \n",
       "5146             0       0       0       0       0       0       0       0   \n",
       "5147             0       0       0       0       0       0       0       0   \n",
       "5148             0       0       0       0       0       0       0       0   \n",
       "5149             0       0       0       0       0       0       0       0   \n",
       "\n",
       "article_id  16.0    18.0    ...  1434.0  1435.0  1436.0  1437.0  1439.0  \\\n",
       "user_id                     ...                                           \n",
       "1                0       0  ...       0       0       1       0       1   \n",
       "2                0       0  ...       0       0       0       0       0   \n",
       "3                0       0  ...       0       0       1       0       0   \n",
       "4                0       0  ...       0       0       0       0       0   \n",
       "5                0       0  ...       0       0       0       0       0   \n",
       "...            ...     ...  ...     ...     ...     ...     ...     ...   \n",
       "5145             0       0  ...       0       0       0       0       0   \n",
       "5146             0       0  ...       0       0       0       0       0   \n",
       "5147             0       0  ...       0       0       0       0       0   \n",
       "5148             0       0  ...       0       0       0       0       0   \n",
       "5149             1       0  ...       0       0       0       0       0   \n",
       "\n",
       "article_id  1440.0  1441.0  1442.0  1443.0  1444.0  \n",
       "user_id                                             \n",
       "1                0       0       0       0       0  \n",
       "2                0       0       0       0       0  \n",
       "3                0       0       0       0       0  \n",
       "4                0       0       0       0       0  \n",
       "5                0       0       0       0       0  \n",
       "...            ...     ...     ...     ...     ...  \n",
       "5145             0       0       0       0       0  \n",
       "5146             0       0       0       0       0  \n",
       "5147             0       0       0       0       0  \n",
       "5148             0       0       0       0       0  \n",
       "5149             0       0       0       0       0  \n",
       "\n",
       "[5149 rows x 714 columns]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "user_item"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "You have passed our quick tests!  Please proceed!\n"
     ]
    }
   ],
   "source": [
    "## Tests: You should just need to run this cell.  Don't change the code.\n",
    "assert user_item.shape[0] == 5149, \"Oops!  The number of users in the user-article matrix doesn't look right.\"\n",
    "assert user_item.shape[1] == 714, \"Oops!  The number of articles in the user-article matrix doesn't look right.\"\n",
    "assert user_item.sum(axis=1)[1] == 36, \"Oops!  The number of articles seen by user 1 doesn't look right.\"\n",
    "print(\"You have passed our quick tests!  Please proceed!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`2.` Complete the function below which should take a user_id and provide an ordered list of the most similar users to that user (from most similar to least similar).  The returned result should not contain the provided user_id, as we know that each user is similar to him/herself. Because the results for each user here are binary, it (perhaps) makes sense to compute similarity as the dot product of two users. \n",
    "\n",
    "Use the tests to test your function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def find_similar_users(user_id, user_item=user_item):\n",
    "    '''\n",
    "    INPUT:\n",
    "    user_id - (int) a user_id\n",
    "    user_item - (pandas dataframe) matrix of users by articles: \n",
    "                1's when a user has interacted with an article, 0 otherwise\n",
    "    \n",
    "    OUTPUT:\n",
    "    similar_users - (list) an ordered list where the closest users (largest dot product users)\n",
    "                    are listed first\n",
    "    \n",
    "    Description:\n",
    "    Computes the similarity of every pair of users based on the dot product\n",
    "    Returns an ordered\n",
    "    \n",
    "    '''\n",
    "    \n",
    "    user_ids_all = user_item.index\n",
    "    \n",
    "    # a dictionary holds [user id : similarity] value pairs\n",
    "    dot_product_dict = {}\n",
    "\n",
    "    # compute similarity of each user to the provided user\n",
    "    for i in user_ids_all:\n",
    "        dot_product = np.dot(user_item.loc[user_id:user_id,:], user_item.loc[i:i,:].transpose())\n",
    "        dot_product_dict[i] = dot_product[0][0]\n",
    "        \n",
    "    # sort by highest (most similar), also drop against self    \n",
    "    ds = pd.Series(dot_product_dict.values(), index=dot_product_dict.keys()).drop(index=user_id).sort_values(ascending=False)\n",
    "    \n",
    "    # extract only index as list of ids. \n",
    "    most_similar_users = list(ds.index)\n",
    "       \n",
    "    return most_similar_users # return a list of the users in order from most to least similar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The 10 most similar users to user 1 are: [3933, 23, 3782, 203, 4459, 3870, 131, 46, 4201, 395]\n",
      "The 5 most similar users to user 3933 are: [1, 23, 3782, 4459, 203]\n",
      "The 3 most similar users to user 46 are: [4201, 23, 3782]\n"
     ]
    }
   ],
   "source": [
    "# Do a spot check of your function\n",
    "print(\"The 10 most similar users to user 1 are: {}\".format(find_similar_users(1)[:10]))\n",
    "print(\"The 5 most similar users to user 3933 are: {}\".format(find_similar_users(3933)[:5]))\n",
    "print(\"The 3 most similar users to user 46 are: {}\".format(find_similar_users(46)[:3]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`3.` Now that you have a function that provides the most similar users to each user, you will want to use these users to find articles you can recommend.  Complete the functions below to return the articles you would recommend to each user. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_article_names(article_ids, df=df):\n",
    "    '''\n",
    "    INPUT:\n",
    "    article_ids - (list) a list of article ids\n",
    "    df - (pandas dataframe) df as defined at the top of the notebook\n",
    "    \n",
    "    OUTPUT:\n",
    "    article_names - (list) a list of article names associated with the list of article ids \n",
    "                    (this is identified by the title column)\n",
    "    '''\n",
    "    # Your code here\n",
    "    \n",
    "    article_names = []\n",
    "    \n",
    "    for i in article_ids:\n",
    "        \n",
    "        # print(i)\n",
    "\n",
    "        name = df[df.article_id == i].title.head(1).values[0]\n",
    "\n",
    "        article_names.append(name)\n",
    "    \n",
    "    return article_names # Return the article names associated with list of article ids\n",
    "\n",
    "\n",
    "\n",
    "def get_user_articles(user_id, user_item=user_item):\n",
    "    '''\n",
    "    INPUT:\n",
    "    user_id - (int) a user id\n",
    "    user_item - (pandas dataframe) matrix of users by articles: \n",
    "                1's when a user has interacted with an article, 0 otherwise\n",
    "    \n",
    "    OUTPUT:\n",
    "    article_ids - (list) a list of the article ids seen by the user\n",
    "    article_names - (list) a list of article names associated with the list of article ids \n",
    "                    (this is identified by the doc_full_name column in df_content)\n",
    "    \n",
    "    Description:\n",
    "    Provides a list of the article_ids and article titles that have been seen by a user\n",
    "    '''\n",
    "    # Your code here\n",
    "    \n",
    "    \n",
    "    article_ids = list(user_item.loc[user_id][user_item.loc[user_id] > 0].index)\n",
    "    article_names = get_article_names(article_ids)\n",
    "    \n",
    "    return article_ids, article_names # return the ids and names\n",
    "\n",
    "\n",
    "\n",
    "def user_user_recs(user_id, m=10):\n",
    "    '''\n",
    "    INPUT:\n",
    "    user_id - (int) a user id\n",
    "    m - (int) the number of recommendations you want for the user\n",
    "    \n",
    "    OUTPUT:\n",
    "    recs - (list) a list of recommendations for the user\n",
    "    \n",
    "    Description:\n",
    "    Loops through the users based on closeness to the input user_id\n",
    "    For each user - finds articles the user hasn't seen before and provides them as recs\n",
    "    Does this until m recommendations are found\n",
    "    \n",
    "    Notes:\n",
    "    Users who are the same closeness are chosen arbitrarily as the 'next' user\n",
    "    \n",
    "    For the user where the number of recommended articles starts below m \n",
    "    and ends exceeding m, the last items are chosen arbitrarily\n",
    "    \n",
    "    '''\n",
    "    # Your code here\n",
    "    \n",
    "    \n",
    "    # get all similar user ids for the targeted user\n",
    "    similar_uids = find_similar_users(user_id)\n",
    "\n",
    "\n",
    "    # get all article ids of the targeted user\n",
    "    article_ids_target_user = get_user_articles(user_id)[0]\n",
    "    # print(f\"[article_ids_target_user]:\\n {article_ids_target_user} \\n\")\n",
    "\n",
    "    \n",
    "    # a list contain unseen articles to recommend\n",
    "    recs = []\n",
    "\n",
    "    for uid in similar_uids:\n",
    "\n",
    "        # print(f\"\\n\\n\\n[number of recs]: {len(recs)}\\n\")\n",
    "        if len(recs) == m:\n",
    "            # print(f\"Number of recs reaches threadhold. Enough. Stop\")\n",
    "            break\n",
    "\n",
    "\n",
    "        #print(f\"[similar user id]: {uid}\")\n",
    "\n",
    "        # get this uid's article ids, and arbitrarily shuffle\n",
    "        article_ids_similar_user = get_user_articles(uid)[0]\n",
    "\n",
    "        # compute the differences of articles seen between the this user and targeted user\n",
    "        # subtraction's order matters\n",
    "        set_diff = list(set(article_ids_similar_user) - set(article_ids_target_user))\n",
    "        # print(f\"[set_diff]:\\n {set_diff} \\n\")\n",
    "        \n",
    "        # make a shuffle for arbitraily chocies from the diff set\n",
    "        random.shuffle(set_diff)\n",
    "\n",
    "        # add the differences of article ids to recs [], append only unique (no duplicate)\n",
    "        for i in set_diff:\n",
    "            if i not in recs and len(recs) < m:\n",
    "                recs.append(i)\n",
    "                # print(f\"[id] {i} appended\")\n",
    "\n",
    "    return recs # return your recommendations for this user_id "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['pixiedust 1.0 is here! – ibm watson data lab',\n",
       " 'a dynamic duo – inside machine learning – medium',\n",
       " 'using deep learning with keras to predict customer churn',\n",
       " 'deep learning with data science experience',\n",
       " 'analyze accident reports on amazon emr spark',\n",
       " 'deep forest: towards an alternative to deep neural networks',\n",
       " 'analyze open data sets with pandas dataframes',\n",
       " 'brunel interactive visualizations in jupyter notebooks',\n",
       " 'maximize oil company profits',\n",
       " 'optimizing a marketing campaign: moving from predictions to actions']"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check Results\n",
    "get_article_names(user_user_recs(1, 10)) # Return 10 recommendations for user 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "float64\n",
      "int64\n"
     ]
    }
   ],
   "source": [
    "# According to dataset, df and df_content, article id are all actually numeric. \n",
    "\n",
    "print(df.article_id.dtype)\n",
    "print(df_content.article_id.dtype)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "If this is all you see, you passed all of our tests!  Nice job!\n"
     ]
    }
   ],
   "source": [
    "# Test your functions here. \n",
    "# I use numeric id instead of string id for article id, since they are actually all numeric.\n",
    "# It makes sense to make it consistent numeric over this notebook\n",
    "\n",
    "assert set(get_article_names([1024.0, 1176.0, 1305.0, 1314.0, 1422.0, 1427.0])) == set(['using deep learning to reconstruct high-resolution audio', 'build a python app on the streaming analytics service', 'gosales transactions for naive bayes model', 'healthcare python streaming application demo', 'use r dataframes & ibm watson natural language understanding', 'use xgboost, scikit-learn & ibm watson machine learning apis']), \"Oops! Your the get_article_names function doesn't work quite how we expect.\"\n",
    "assert set(get_article_names([1320.0, 232.0, 844.0])) == set(['housing (2015): united states demographic measures','self-service data preparation with ibm data refinery','use the cloudant-spark connector in python notebook']), \"Oops! Your the get_article_names function doesn't work quite how we expect.\"\n",
    "assert set(get_user_articles(20)[0]) == set([1320.0, 232.0, 844.0])\n",
    "assert set(get_user_articles(20)[1]) == set(['housing (2015): united states demographic measures', 'self-service data preparation with ibm data refinery','use the cloudant-spark connector in python notebook'])\n",
    "assert set(get_user_articles(2)[0]) == set([1024.0, 1176.0, 1305.0, 1314.0, 1422.0, 1427.0])\n",
    "assert set(get_user_articles(2)[1]) == set(['using deep learning to reconstruct high-resolution audio', 'build a python app on the streaming analytics service', 'gosales transactions for naive bayes model', 'healthcare python streaming application demo', 'use r dataframes & ibm watson natural language understanding', 'use xgboost, scikit-learn & ibm watson machine learning apis'])\n",
    "print(\"If this is all you see, you passed all of our tests!  Nice job!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`4.` Now we are going to improve the consistency of the **user_user_recs** function from above.  \n",
    "\n",
    "* Instead of arbitrarily choosing when we obtain users who are all the same closeness to a given user - choose the users that have the most total article interactions before choosing those with fewer article interactions.\n",
    "\n",
    "\n",
    "* Instead of arbitrarily choosing articles from the user where the number of recommended articles starts below m and ends exceeding m, choose articles with the articles with the most total interactions before choosing those with fewer total interactions. This ranking should be  what would be obtained from the **top_articles** function you wrote earlier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_top_sorted_users(user_id, df=df, user_item=user_item):\n",
    "    '''\n",
    "    INPUT:\n",
    "    user_id - (int)\n",
    "    df - (pandas dataframe) df as defined at the top of the notebook \n",
    "    user_item - (pandas dataframe) matrix of users by articles: \n",
    "            1's when a user has interacted with an article, 0 otherwise\n",
    "    \n",
    "            \n",
    "    OUTPUT:\n",
    "    neighbors_df - (pandas dataframe) a dataframe with:\n",
    "                    neighbor_id - is a neighbor user_id\n",
    "                    similarity - measure of the similarity of each user to the provided user_id\n",
    "                    num_interactions - the number of articles viewed by the user - if a u\n",
    "                    \n",
    "    Other Details - sort the neighbors_df by the similarity and then by number of interactions where \n",
    "                    highest of each is higher in the dataframe\n",
    "     \n",
    "    '''\n",
    "    # Your code here\n",
    "    \n",
    "    # get all neighbors ids\n",
    "    nbh_ids = find_similar_users(user_id)\n",
    "    \n",
    "    \n",
    "    # assemble a data matrix\n",
    "    data_matrix = np.array([\n",
    "    [\n",
    "        x, # neighbor id\n",
    "        np.dot(user_item.loc[user_id:user_id,:], user_item.loc[x:x,:].transpose())[0][0], # similarity score\n",
    "        df[df.user_id == x].shape[0] # number of content interaction\n",
    "    ] for x in nbh_ids])\n",
    "    \n",
    "    # make a dataframe\n",
    "    neighbors_df = pd.DataFrame(data=data_matrix, \n",
    "                                columns=['neighbor_id', 'similarity', 'num_interactions'], \n",
    "                                index=data_matrix[:,0]).sort_values(by=['similarity', 'num_interactions'],\n",
    "                                                                    ascending=[False, False])\n",
    "    \n",
    "    \n",
    "    \n",
    "    return neighbors_df # Return the dataframe specified in the doc_string\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "def user_user_recs_part2(user_id, m=10):\n",
    "    '''\n",
    "    INPUT:\n",
    "    user_id - (int) a user id\n",
    "    m - (int) the number of recommendations you want for the user\n",
    "    \n",
    "    OUTPUT:\n",
    "    recs - (list) a list of recommendations for the user by article id\n",
    "    rec_names - (list) a list of recommendations for the user by article title\n",
    "    \n",
    "    Description:\n",
    "    Loops through the users based on closeness to the input user_id\n",
    "    For each user - finds articles the user hasn't seen before and provides them as recs\n",
    "    Does this until m recommendations are found\n",
    "    \n",
    "    Notes:\n",
    "    * Choose the users that have the most total article interactions \n",
    "    before choosing those with fewer article interactions.\n",
    "\n",
    "    * Choose articles with the articles with the most total interactions \n",
    "    before choosing those with fewer total interactions. \n",
    "   \n",
    "    '''\n",
    "    # Your code here\n",
    "    \n",
    "    \n",
    "    # get all similar user ids for the targeted user\n",
    "    # fetch with the 'neighbors_df'\n",
    "    similar_uids = list(get_top_sorted_users(user_id).index)\n",
    "    #print(f\"[similar_uids]: \\n{similar_uids}\")\n",
    "\n",
    "\n",
    "    # get all article ids of the targeted user\n",
    "    article_ids_target_user = get_user_articles(user_id)[0]\n",
    "    # print(f\"[article_ids_target_user]:\\n {article_ids_target_user} \\n\")\n",
    "\n",
    "    \n",
    "    # a list contain unseen articles to recommend\n",
    "    recs = []\n",
    "\n",
    "    for uid in similar_uids:\n",
    "\n",
    "        # print(f\"\\n\\n\\n[number of recs]: {len(recs)}\\n\")\n",
    "        if len(recs) == m:\n",
    "            #print(f\"Number of recs reaches threadhold {m}. Enough. Stop\")\n",
    "            break\n",
    "\n",
    "\n",
    "        #print(f\"[similar user id]: {uid}\")\n",
    "\n",
    "        # get this uid's article ids, and arbitrarily shuffle\n",
    "        article_ids_similar_user = get_user_articles(uid)[0]\n",
    "\n",
    "        # compute the differences of articles seen between the this user and targeted user\n",
    "        # subtraction's order matters\n",
    "        set_diff = list(set(article_ids_similar_user) - set(article_ids_target_user))\n",
    "        #print(f\"[set_diff before sort]:\\n {set_diff} \\n\")\n",
    "        \n",
    "        # Sort the set. Determine with highest total interactions metric \n",
    "        set_diff = list(df[df.article_id.isin(set_diff)]['article_id'].value_counts().index)\n",
    "        #print(f\"[set_diff after sort]:\\n {set_diff} \\n\")\n",
    "\n",
    "        # add the differences of article ids to recs [], append only unique (no duplicate)\n",
    "        for i in set_diff:\n",
    "            if i not in recs and len(recs) < m:\n",
    "                recs.append(i)\n",
    "                #print(f\"[id] {i} appended\")\n",
    "    \n",
    "    \n",
    "    \n",
    "    \n",
    "    rec_names = get_article_names(recs)\n",
    "    \n",
    "    return recs, rec_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The top 10 recommendations for user 20 are the following article ids:\n",
      "[1330.0, 1427.0, 1364.0, 1170.0, 1162.0, 1304.0, 1351.0, 1160.0, 1354.0, 1368.0]\n",
      "\n",
      "The top 10 recommendations for user 20 are the following article names:\n",
      "['insights from new york car accident reports', 'use xgboost, scikit-learn & ibm watson machine learning apis', 'predicting churn with the spss random tree algorithm', 'apache spark lab, part 1: basic concepts', 'analyze energy consumption in buildings', 'gosales transactions for logistic regression model', 'model bike sharing data with spss', 'analyze accident reports on amazon emr spark', 'movie recommender system with spark machine learning', 'putting a human face on machine learning']\n"
     ]
    }
   ],
   "source": [
    "# Quick spot check - don't change this code - just use it to test your functions\n",
    "rec_ids, rec_names = user_user_recs_part2(20, 10)\n",
    "print(\"The top 10 recommendations for user 20 are the following article ids:\")\n",
    "print(rec_ids)\n",
    "print()\n",
    "print(\"The top 10 recommendations for user 20 are the following article names:\")\n",
    "print(rec_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`5.` Use your functions from above to correctly fill in the solutions to the dictionary below.  Then test your dictionary against the solution.  Provide the code you need to answer each following the comments below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>neighbor_id</th>\n",
       "      <th>similarity</th>\n",
       "      <th>num_interactions</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3933</th>\n",
       "      <td>3933</td>\n",
       "      <td>35</td>\n",
       "      <td>45</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      neighbor_id  similarity  num_interactions\n",
       "3933         3933          35                45"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_top_sorted_users(1).head(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>neighbor_id</th>\n",
       "      <th>similarity</th>\n",
       "      <th>num_interactions</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3870</th>\n",
       "      <td>3870</td>\n",
       "      <td>74</td>\n",
       "      <td>144</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3782</th>\n",
       "      <td>3782</td>\n",
       "      <td>39</td>\n",
       "      <td>363</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>23</td>\n",
       "      <td>38</td>\n",
       "      <td>364</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>203</th>\n",
       "      <td>203</td>\n",
       "      <td>33</td>\n",
       "      <td>160</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4459</th>\n",
       "      <td>4459</td>\n",
       "      <td>33</td>\n",
       "      <td>158</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98</th>\n",
       "      <td>98</td>\n",
       "      <td>29</td>\n",
       "      <td>170</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3764</th>\n",
       "      <td>3764</td>\n",
       "      <td>29</td>\n",
       "      <td>169</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>49</td>\n",
       "      <td>29</td>\n",
       "      <td>147</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3697</th>\n",
       "      <td>3697</td>\n",
       "      <td>29</td>\n",
       "      <td>145</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>242</th>\n",
       "      <td>242</td>\n",
       "      <td>25</td>\n",
       "      <td>148</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      neighbor_id  similarity  num_interactions\n",
       "3870         3870          74               144\n",
       "3782         3782          39               363\n",
       "23             23          38               364\n",
       "203           203          33               160\n",
       "4459         4459          33               158\n",
       "98             98          29               170\n",
       "3764         3764          29               169\n",
       "49             49          29               147\n",
       "3697         3697          29               145\n",
       "242           242          25               148"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_top_sorted_users(131).head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Tests with a dictionary of results\n",
    "\n",
    "user1_most_sim = get_top_sorted_users(1).head(1).index.values[0] # Find the user that is most similar to user 1 \n",
    "user131_10th_sim = get_top_sorted_users(131).head(10).index.values[9] # Find the 10th most similar user to user 131"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This all looks good!  Nice job!\n"
     ]
    }
   ],
   "source": [
    "## Dictionary Test Here\n",
    "sol_5_dict = {\n",
    "    'The user that is most similar to user 1.': user1_most_sim, \n",
    "    'The user that is the 10th most similar to user 131': user131_10th_sim,\n",
    "}\n",
    "\n",
    "t.sol_5_test(sol_5_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`6.` If we were given a new user, which of the above functions would you be able to use to make recommendations?  Explain.  Can you think of a better way we might make recommendations?  Use the cell below to explain a better method for new users."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Answer** \n",
    "\n",
    "**If we were given a new user, we would use rank based functions. get_top_article_ids()**\n",
    "\n",
    "Because we have limited information about any new user, we would try pulling most-interacted (viewed) content.\n",
    "\n",
    "There can be some better way, for exmaple, also pull content that similar to top articles. Combination of rank based / knowledge based with content based approach.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1429.0,\n",
       " 1330.0,\n",
       " 1431.0,\n",
       " 1427.0,\n",
       " 1364.0,\n",
       " 1314.0,\n",
       " 1293.0,\n",
       " 1170.0,\n",
       " 1162.0,\n",
       " 1304.0]"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_user = 0.0\n",
    "\n",
    "# What would your recommendations be for this new user '0.0'?  As a new user, they have no observed articles.\n",
    "# Provide a list of the top 10 article ids you would give to \n",
    "\n",
    "\n",
    "new_user_recs = get_top_article_ids(10, df) # Your recommendations here\n",
    "new_user_recs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "That's right!  Nice job!\n"
     ]
    }
   ],
   "source": [
    "assert set(new_user_recs) == set([1314.0, 1429.0, 1293.0, 1427.0, 1162.0, 1364.0, 1304.0, 1170.0, 1431.0,\n",
    "                                  1330.0]), \"Oops!  It makes sense that in this case we would want to recommend the most popular articles, because we don't know anything about these users.\"\n",
    "\n",
    "print(\"That's right!  Nice job!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "### <a class=\"anchor\" id=\"Content-Recs\">Part IV: Content Based Recommendations (EXTRA - NOT REQUIRED)</a>\n",
    "\n",
    "Another method we might use to make recommendations is to perform a ranking of the highest ranked articles associated with some term.  You might consider content to be the **doc_body**, **doc_description**, or **doc_full_name**.  There isn't one way to create a content based recommendation, especially considering that each of these columns hold content related information.  \n",
    "\n",
    "`1.` Use the function body below to create a content based recommender.  Since there isn't one right answer for this recommendation tactic, no test functions are provided.  Feel free to change the function inputs if you decide you want to try a method that requires more input values.  The input values are currently set with one idea in mind that you may use to make content based recommendations.  One additional idea is that you might want to choose the most popular recommendations that meet your 'content criteria', but again, there is a lot of flexibility in how you might make these recommendations.\n",
    "\n",
    "### This part is NOT REQUIRED to pass this project.  However, you may choose to take this on as an extra way to show off your skills."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to /Users/apple/nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n",
      "[nltk_data] Downloading package stopwords to /Users/apple/nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n",
      "[nltk_data] Downloading package wordnet to /Users/apple/nltk_data...\n",
      "[nltk_data]   Package wordnet is already up-to-date!\n",
      "[nltk_data] Downloading package averaged_perceptron_tagger to\n",
      "[nltk_data]     /Users/apple/nltk_data...\n",
      "[nltk_data]   Package averaged_perceptron_tagger is already up-to-\n",
      "[nltk_data]       date!\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "import nltk\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.stem.wordnet import WordNetLemmatizer\n",
    "from nltk.tokenize import word_tokenize\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "\n",
    "\n",
    "from nltk.util import bigrams\n",
    "from nltk.util import ngrams\n",
    "from nltk.lm.preprocessing import flatten\n",
    "\n",
    "nltk.download('punkt')\n",
    "nltk.download('stopwords')\n",
    "nltk.download('wordnet')\n",
    "nltk.download('averaged_perceptron_tagger')\n",
    "\n",
    "# Constants and Reusable objects for tokenization\n",
    "url_regex = 'http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\\(\\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+'\n",
    "stop_words = stopwords.words(\"english\")\n",
    "lemmatizer = WordNetLemmatizer()\n",
    "\n",
    "# Soft copy a article content dataframe for NLP processing.\n",
    "df_nlp = df_content.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Tokenization helper\n",
    "\n",
    "def tokenize(text):\n",
    "    '''\n",
    "    private tokenizer to transform each text.\n",
    "    As a NLP helper function including following tasks:\n",
    "    - Replace URLs\n",
    "    - Normalize text\n",
    "    - Remove punctuation\n",
    "    - Tokenize words\n",
    "    - Remove stop words\n",
    "    - Legmmatize words\n",
    "    :param text: A message text.\n",
    "    :return: cleaned tokens extracted from original message text.\n",
    "    '''\n",
    "\n",
    "    # print(f\"original text: \\n {text}\")\n",
    "\n",
    "    # replace urls\n",
    "    detected_urls = re.findall(url_regex, text)\n",
    "    for url in detected_urls:\n",
    "        text = text.replace(url, \"urlplaceholder\")\n",
    "\n",
    "    # normalize case and remove punctuation\n",
    "    text = re.sub(r\"[^a-zA-Z0-9]\", \" \", text.lower())\n",
    "\n",
    "    # tokenize text\n",
    "    tokens = word_tokenize(text)\n",
    "\n",
    "    # lemmatize and remove stop words\n",
    "    tokens = [lemmatizer.lemmatize(word).strip() for word in tokens if word not in stop_words]\n",
    "\n",
    "    # in case after normalize/lemmatize, if there is no words, make a dummy element. otherwise follwing transformation\n",
    "    # may breaks\n",
    "    if len(tokens) < 1:\n",
    "        tokens = ['none']\n",
    "\n",
    "    # print(f\"tokens: \\n {tokens} \\n\\n\")\n",
    "    return tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Do some wrangling, cleaning to the df_nlp dataset.\n",
    "\n",
    "\n",
    "# Found index and article id mismatched. \n",
    "# Update index with article to make it consistent and eaiser to process with.\n",
    "df_nlp.index = df_nlp.article_id\n",
    "\n",
    "\n",
    "# Clean empty / missing page body and desc content. Update the empty (NaN) with 'empty' placeholder\n",
    "df_nlp.loc[df_nlp[df_nlp.doc_body.isnull()].index, 'doc_body'] = 'empty'\n",
    "df_nlp.loc[df_nlp[df_nlp.doc_description.isnull()].index, 'doc_description'] = 'empty'\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Internal private similarity function of body, title, desc respectly\n",
    "# These internal functions will be called by similarity overall function (a weighed sum of all similarity).\n",
    "\n",
    "\n",
    "\n",
    "def _compute_article_body_similarity(article_id_1, article_id_2, data=df_nlp):\n",
    "    # Compute the consine similarity based on tfidf of body content\n",
    "    # This is a private helper function that is used by overall similarity function.\n",
    "    \n",
    "\n",
    "    doc_a = ' '.join(tokenize(data.loc[article_id_1].doc_body))\n",
    "    doc_b = ' '.join(tokenize(data.loc[article_id_2].doc_body))\n",
    "    \n",
    "    \n",
    "\n",
    "    # combine to a list of documents\n",
    "    documents = [doc_a, doc_b]\n",
    "\n",
    "    \n",
    "    # instanciate a scikitlearn tfidf vecorizer\n",
    "    vectorizer = TfidfVectorizer()\n",
    "\n",
    "    # fit transform to get a sparse matrix\n",
    "    matrix = vectorizer.fit_transform(documents)\n",
    "\n",
    "\n",
    "    # # UnComment out belows to examinate details (term features, tfidf values)\n",
    "    # ====================================\n",
    "    # term_features =  vectorizer.get_feature_names_out()\n",
    "\n",
    "    # # convert to readable array \n",
    "    # matrix_array = matrix.toarray()\n",
    "\n",
    "    # # assemble to a dataframe for explor \n",
    "    # tfidf_dataframe = pd.DataFrame(data=matrix_array, columns=term_features)\n",
    "    # ====================================\n",
    "\n",
    "\n",
    "    return cosine_similarity(matrix[:1], matrix[1:])[0][0]\n",
    "\n",
    "\n",
    "\n",
    "def _compute_article_title_similarity(article_id_1, article_id_2, data=df_nlp):\n",
    "    # Compute the consine similarity based on tfidf of title (doc_full_name)\n",
    "    # think of title tag for seo pagerank\n",
    "    \n",
    "    # This is a private helper function that is used by overall similarity function.\n",
    "    \n",
    "\n",
    "    doc_a = ' '.join(tokenize(data.loc[article_id_1].doc_full_name))\n",
    "    doc_b = ' '.join(tokenize(data.loc[article_id_2].doc_full_name))\n",
    "    \n",
    "    \n",
    "\n",
    "    # combine to a list of documents\n",
    "    documents = [doc_a, doc_b]\n",
    "\n",
    "    \n",
    "    # instanciate a scikitlearn tfidf vecorizer\n",
    "    vectorizer = TfidfVectorizer()\n",
    "\n",
    "    # fit transform to get a sparse matrix\n",
    "    matrix = vectorizer.fit_transform(documents)\n",
    "\n",
    "\n",
    "    # # UnComment out belows to examinate details (term features, tfidf values)\n",
    "    # ====================================\n",
    "    # term_features =  vectorizer.get_feature_names_out()\n",
    "\n",
    "    # # convert to readable array \n",
    "    # matrix_array = matrix.toarray()\n",
    "\n",
    "    # # assemble to a dataframe for explor \n",
    "    # tfidf_dataframe = pd.DataFrame(data=matrix_array, columns=term_features)\n",
    "    # ====================================\n",
    "\n",
    "\n",
    "    return cosine_similarity(matrix[:1], matrix[1:])[0][0]\n",
    "\n",
    "\n",
    "\n",
    "def _compute_article_desc_similarity(article_id_1, article_id_2, data=df_nlp):\n",
    "    # Compute the consine similarity based on tfidf of desc content (doc_description)\n",
    "    # think of desc tag for seo pagerank\n",
    "    \n",
    "    # This is a private helper function that is used by overall similarity function.\n",
    "    \n",
    "\n",
    "    doc_a = ' '.join(tokenize(data.loc[article_id_1].doc_description))\n",
    "    doc_b = ' '.join(tokenize(data.loc[article_id_2].doc_description))\n",
    "    \n",
    "    \n",
    "\n",
    "    # combine to a list of documents\n",
    "    documents = [doc_a, doc_b]\n",
    "\n",
    "    \n",
    "    # instanciate a scikitlearn tfidf vecorizer\n",
    "    vectorizer = TfidfVectorizer()\n",
    "\n",
    "    # fit transform to get a sparse matrix\n",
    "    matrix = vectorizer.fit_transform(documents)\n",
    "\n",
    "\n",
    "    # # UnComment out belows to examing details (term features, tfidf values)\n",
    "    # ====================================\n",
    "    # term_features =  vectorizer.get_feature_names_out()\n",
    "\n",
    "    # # convert to readable array \n",
    "    # matrix_array = matrix.toarray()\n",
    "\n",
    "    # # assemble to a dataframe for explor \n",
    "    # tfidf_dataframe = pd.DataFrame(data=matrix_array, columns=term_features)\n",
    "    # ====================================\n",
    "\n",
    "\n",
    "    return cosine_similarity(matrix[:1], matrix[1:])[0][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_article_similarity(article_id_1, article_id_2):\n",
    "    \"\"\"\n",
    "    Cosine similarty of overall content details, \n",
    "    in consideration of: similarity_title, similarity_body, similarity_desc\n",
    "    \"\"\"\n",
    "    \n",
    "    # calculate similary for body, title, desc, then combine a single value.\n",
    "    # think about how google weight title,desc,body. Title tag is very heavy. SEO-wised\n",
    "    # so having 3 consine similarity values, then do a normailized one. \n",
    "    # what's the formular? think of a course, assignments weight x, final exam weight y,\n",
    "    # then what's total grade.\n",
    "    # https://www.indeed.com/career-advice/career-development/how-to-calculate-weighted-average\n",
    "    \n",
    "    similarity_title = _compute_article_title_similarity(article_id_1,article_id_2)\n",
    "    similarity_body = _compute_article_body_similarity(article_id_1,article_id_2)\n",
    "    similarity_desc = _compute_article_desc_similarity(article_id_1,article_id_2)\n",
    "    \n",
    "    # a weighted sum caluculation for final score\n",
    "    # I give title 0.5 weight, body 0.4 weight, desc 0.1 weight. (adjust if necessary)\n",
    "    \n",
    "    overall = similarity_title * 0.5 + similarity_body * 0.4 + similarity_desc * 0.1\n",
    "    \n",
    "    return overall"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.07122043497708203"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Spot Check:\n",
    "\n",
    "compute_article_similarity(55, 50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Find similar articles for a given article.\n",
    "\n",
    "This only works on the df_content/df_nlp dataset. \n",
    "\n",
    "Use full details of an article. (title, desc, body), this info only available in the df_content/df_nlp dataset\n",
    "Calculate in real-time. It might take some seconds. \n",
    "(Because it loops to all articles against the targeted article, use cosine similarity, not dot product.)\n",
    "For articles not in the df_content/df_nlp dataset, another \n",
    "\n",
    "method is suitable lookup_similar_title_articles(), which checks only on the title. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Find similar articles for a given article based on content similarity.(overall of title+desc+body)\n",
    "# This only works on the df_content/df_nlp dataset. \n",
    "# Use full details of an article. (title, desc, body), this info only available in the df_content/df_nlp dataset\n",
    "# Calculate in real-time. It might take some seconds. \n",
    "# (since it loops to all articles against the targeted article, use cosine similarity, not dot product.)\n",
    "# For articles not in the df_content/df_nlp dataset, another \n",
    "# method is suitable lookup_similar_title_articles(), which checks only on the title. \n",
    "\n",
    "\n",
    "def find_similar_content_articles(article_id, data=df_nlp):\n",
    "    \n",
    "    article_ids_all = data.index\n",
    "    similarity_dict = {}\n",
    "    \n",
    "    for i in article_ids_all:\n",
    "        # print(f\"\\n[i]: {i}\")\n",
    "        \n",
    "        if i == article_id:\n",
    "            continue\n",
    "\n",
    "        similarity_score = compute_article_similarity(article_id,i)\n",
    "        similarity_dict[i] = similarity_score\n",
    "        # print(f\"[similarity_score]: {similarity_score}\")\n",
    "    \n",
    "    \n",
    "    similarity_ds = pd.Series(data=similarity_dict.values(), \n",
    "                           index=similarity_dict.keys()).sort_values(ascending=False).index\n",
    "    \n",
    "    return list(similarity_ds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.33220330079396565 seconds\n",
      "[389, 993, 949, 592, 714, 117, 678, 942, 231, 925, 15, 463, 353, 835, 907, 284, 977, 600, 595, 997]\n"
     ]
    }
   ],
   "source": [
    "# Spot Check: \n",
    "# Articles that relevant to article id 420, based on content details similarity (title, desc, body)\n",
    "# Calculate in real time, it might take some seconds\n",
    "# (Because it loops to all articles against the targeted article, use language consine similarity not dot product.)\n",
    "start = time.time()\n",
    "relevant_content_articles_for_420th = find_similar_content_articles(420)\n",
    "end = time.time()\n",
    "\n",
    "print(f\"{(end - start) / 60} seconds\")\n",
    "print(relevant_content_articles_for_420th[:20])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>doc_full_name</th>\n",
       "      <th>doc_description</th>\n",
       "      <th>doc_body</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>article_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>389</th>\n",
       "      <td>Apache Spark™ 2.0: Impressive Improvements to ...</td>\n",
       "      <td>What a difference a version number makes! With...</td>\n",
       "      <td>* Home\\r\\n * Community\\r\\n * Projects\\r\\n * Bl...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>993</th>\n",
       "      <td>Configuring the Apache Spark SQL Context</td>\n",
       "      <td>The Apache Spark website documents the propert...</td>\n",
       "      <td>* Home\\r\\n * Community\\r\\n * Projects\\r\\n * Bl...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>949</th>\n",
       "      <td>Apache Spark SQL Analyzer Resolves Order-by Co...</td>\n",
       "      <td>The Apache Spark SQL component has several sub...</td>\n",
       "      <td>* Home\\r\\n * Community\\r\\n * Projects\\r\\n * Bl...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>592</th>\n",
       "      <td>Apache Spark Analytics</td>\n",
       "      <td>Combine Apache® Spark™ with other cloud servic...</td>\n",
       "      <td>APACHE SPARK ANALYTICSCombine Apache® Spark™ w...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>714</th>\n",
       "      <td>A Survey of Books about Apache Spark™</td>\n",
       "      <td>From the big crop of books about Apache Spark™...</td>\n",
       "      <td>* Home\\r\\n * Community\\r\\n * Projects\\r\\n * Bl...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>117</th>\n",
       "      <td>Apache Spark™ 2.0: Migrating Applications</td>\n",
       "      <td>This post provides a brief summary of sample c...</td>\n",
       "      <td>* Home\\r\\n * Community\\r\\n * Projects\\r\\n * Bl...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>678</th>\n",
       "      <td>Spark SQL - Rapid Performance Evolution</td>\n",
       "      <td>Spark SQL Version 1.6 runs queries faster! Tha...</td>\n",
       "      <td>{ spark .tc } * Community\\r\\n * Projects\\r\\n *...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>942</th>\n",
       "      <td>Interview with Sean Li, New Apache Spark™ Comm...</td>\n",
       "      <td>Sean looks back on his first encounter with Sp...</td>\n",
       "      <td>* Home\\r\\n * Community\\r\\n * Projects\\r\\n * Bl...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>231</th>\n",
       "      <td>Speed your SQL Queries with Spark SQL</td>\n",
       "      <td>Get faster queries and write less code too. Le...</td>\n",
       "      <td>Skip to main content IBM developerWorks / Deve...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>925</th>\n",
       "      <td>Build SQL queries with Apache Spark in DSX</td>\n",
       "      <td>This video shows you how to use the Spark SQL ...</td>\n",
       "      <td>Skip navigation Sign in SearchLoading...\\r\\n\\r...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Apache Spark™ 2.0: Extend Structured Streaming...</td>\n",
       "      <td>Early methods to integrate machine learning us...</td>\n",
       "      <td>* Home\\r\\n * Community\\r\\n * Projects\\r\\n * Bl...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>463</th>\n",
       "      <td>What is Spark?</td>\n",
       "      <td>Learn Why and How To Use Spark for large amoun...</td>\n",
       "      <td>Skip navigation Upload Sign in SearchLoading.....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>353</th>\n",
       "      <td>sparklyr — R interface for Apache Spark</td>\n",
       "      <td>We’re excited today to announce sparklyr, a ne...</td>\n",
       "      <td>RStudio Blog * Home\\r\\n\\r\\n * Subscribe to fee...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>835</th>\n",
       "      <td>Build SQL Queries in a Scala notebook using Ap...</td>\n",
       "      <td>How to build SQL Queries in a Scala notebook u...</td>\n",
       "      <td>Skip to main content IBM developerWorks / Deve...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>907</th>\n",
       "      <td>Build Spark SQL Queries</td>\n",
       "      <td>How to build SQL Queries in a Scala notebook u...</td>\n",
       "      <td>Skip to main content IBM developerWorks / Deve...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>284</th>\n",
       "      <td>Apache Spark 2.0: Machine Learning. Under the ...</td>\n",
       "      <td>Now that the dust has settled on Apache Spark ...</td>\n",
       "      <td>* Home\\r\\n * Community\\r\\n * Projects\\r\\n * Bl...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>977</th>\n",
       "      <td>Apache Spark as the New Engine of Genomics</td>\n",
       "      <td>A handful of talks at the recent Spark Summit ...</td>\n",
       "      <td>* Home\\r\\n * Community\\r\\n * Projects\\r\\n * Bl...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>600</th>\n",
       "      <td>Access IBM Analytics for Apache Spark from RSt...</td>\n",
       "      <td>In this post I will show you how to use the IB...</td>\n",
       "      <td>Homepage Follow Sign in / Sign up Homepage * H...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>595</th>\n",
       "      <td>Load dashDB Data with Apache Spark</td>\n",
       "      <td>Learn how to create a connection to dashDB dat...</td>\n",
       "      <td>Skip to main content IBM developerWorks / Deve...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>997</th>\n",
       "      <td>Apache Spark: Upgrade and speed-up your analytics</td>\n",
       "      <td>One of the best things about Apache Spark is t...</td>\n",
       "      <td>Skip to main content IBM developerWorks / Deve...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                doc_full_name  \\\n",
       "article_id                                                      \n",
       "389         Apache Spark™ 2.0: Impressive Improvements to ...   \n",
       "993                  Configuring the Apache Spark SQL Context   \n",
       "949         Apache Spark SQL Analyzer Resolves Order-by Co...   \n",
       "592                                    Apache Spark Analytics   \n",
       "714                     A Survey of Books about Apache Spark™   \n",
       "117                 Apache Spark™ 2.0: Migrating Applications   \n",
       "678                   Spark SQL - Rapid Performance Evolution   \n",
       "942         Interview with Sean Li, New Apache Spark™ Comm...   \n",
       "231                     Speed your SQL Queries with Spark SQL   \n",
       "925                Build SQL queries with Apache Spark in DSX   \n",
       "15          Apache Spark™ 2.0: Extend Structured Streaming...   \n",
       "463                                            What is Spark?   \n",
       "353                   sparklyr — R interface for Apache Spark   \n",
       "835         Build SQL Queries in a Scala notebook using Ap...   \n",
       "907                                   Build Spark SQL Queries   \n",
       "284         Apache Spark 2.0: Machine Learning. Under the ...   \n",
       "977                Apache Spark as the New Engine of Genomics   \n",
       "600         Access IBM Analytics for Apache Spark from RSt...   \n",
       "595                        Load dashDB Data with Apache Spark   \n",
       "997         Apache Spark: Upgrade and speed-up your analytics   \n",
       "\n",
       "                                              doc_description  \\\n",
       "article_id                                                      \n",
       "389         What a difference a version number makes! With...   \n",
       "993         The Apache Spark website documents the propert...   \n",
       "949         The Apache Spark SQL component has several sub...   \n",
       "592         Combine Apache® Spark™ with other cloud servic...   \n",
       "714         From the big crop of books about Apache Spark™...   \n",
       "117         This post provides a brief summary of sample c...   \n",
       "678         Spark SQL Version 1.6 runs queries faster! Tha...   \n",
       "942         Sean looks back on his first encounter with Sp...   \n",
       "231         Get faster queries and write less code too. Le...   \n",
       "925         This video shows you how to use the Spark SQL ...   \n",
       "15          Early methods to integrate machine learning us...   \n",
       "463         Learn Why and How To Use Spark for large amoun...   \n",
       "353         We’re excited today to announce sparklyr, a ne...   \n",
       "835         How to build SQL Queries in a Scala notebook u...   \n",
       "907         How to build SQL Queries in a Scala notebook u...   \n",
       "284         Now that the dust has settled on Apache Spark ...   \n",
       "977         A handful of talks at the recent Spark Summit ...   \n",
       "600         In this post I will show you how to use the IB...   \n",
       "595         Learn how to create a connection to dashDB dat...   \n",
       "997         One of the best things about Apache Spark is t...   \n",
       "\n",
       "                                                     doc_body  \n",
       "article_id                                                     \n",
       "389         * Home\\r\\n * Community\\r\\n * Projects\\r\\n * Bl...  \n",
       "993         * Home\\r\\n * Community\\r\\n * Projects\\r\\n * Bl...  \n",
       "949         * Home\\r\\n * Community\\r\\n * Projects\\r\\n * Bl...  \n",
       "592         APACHE SPARK ANALYTICSCombine Apache® Spark™ w...  \n",
       "714         * Home\\r\\n * Community\\r\\n * Projects\\r\\n * Bl...  \n",
       "117         * Home\\r\\n * Community\\r\\n * Projects\\r\\n * Bl...  \n",
       "678         { spark .tc } * Community\\r\\n * Projects\\r\\n *...  \n",
       "942         * Home\\r\\n * Community\\r\\n * Projects\\r\\n * Bl...  \n",
       "231         Skip to main content IBM developerWorks / Deve...  \n",
       "925         Skip navigation Sign in SearchLoading...\\r\\n\\r...  \n",
       "15          * Home\\r\\n * Community\\r\\n * Projects\\r\\n * Bl...  \n",
       "463         Skip navigation Upload Sign in SearchLoading.....  \n",
       "353         RStudio Blog * Home\\r\\n\\r\\n * Subscribe to fee...  \n",
       "835         Skip to main content IBM developerWorks / Deve...  \n",
       "907         Skip to main content IBM developerWorks / Deve...  \n",
       "284         * Home\\r\\n * Community\\r\\n * Projects\\r\\n * Bl...  \n",
       "977         * Home\\r\\n * Community\\r\\n * Projects\\r\\n * Bl...  \n",
       "600         Homepage Follow Sign in / Sign up Homepage * H...  \n",
       "595         Skip to main content IBM developerWorks / Deve...  \n",
       "997         Skip to main content IBM developerWorks / Deve...  "
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Spot Check:\n",
    "# Checking relavancy. They are ordered by relavancy. \n",
    "\n",
    "df_nlp.iloc[relevant_content_articles_for_420th].head(20)[['doc_full_name','doc_description','doc_body']]\n",
    "\n",
    "# Looks like related to 'Apache Spark'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Thoughs\n",
    "\n",
    "The above find_similar_content_articles() work well. However, it is calculating on the fly, could take time to load. \n",
    "\n",
    "Such real-time calculating is too expensive in terms of user experience. (Waited xx seconds for similar articles)\n",
    "\n",
    "To improve the experience, we can pre-calculate or cache the cosine similarity scores.\n",
    "\n",
    "So make an article_article_similary data frame for lookup. (the content-content recs). \n",
    "\n",
    "(people who view X article also might be interested in Y article based on the relevancy of article_article NLP cosine similarity.)\n",
    "\n",
    "#### Approach\n",
    "\n",
    "Make a dataframe, store article-article-similarity. (Based on the df_nlp dataframe, the cleaned df_content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>article_id</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>1041</th>\n",
       "      <th>1042</th>\n",
       "      <th>1043</th>\n",
       "      <th>1044</th>\n",
       "      <th>1045</th>\n",
       "      <th>1046</th>\n",
       "      <th>1047</th>\n",
       "      <th>1048</th>\n",
       "      <th>1049</th>\n",
       "      <th>1050</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>article_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 1051 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "article_id  0     1     2     3     4     5     6     7     8     9     ...  \\\n",
       "article_id                                                              ...   \n",
       "0            0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  ...   \n",
       "1            0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  ...   \n",
       "2            0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  ...   \n",
       "3            0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  ...   \n",
       "4            0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  ...   \n",
       "\n",
       "article_id  1041  1042  1043  1044  1045  1046  1047  1048  1049  1050  \n",
       "article_id                                                              \n",
       "0            0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  \n",
       "1            0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  \n",
       "2            0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  \n",
       "3            0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  \n",
       "4            0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  \n",
       "\n",
       "[5 rows x 1051 columns]"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Make a article_article dummy dataframe based on shape of df_nlp\n",
    "\n",
    "article_article = pd.DataFrame(\n",
    "    data=np.zeros((len(df_nlp.index), len(df_nlp.index))),\n",
    "    index=df_nlp.index,\n",
    "    columns=df_nlp.index\n",
    ")\n",
    "\n",
    "article_article.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compute article_article cosine similarity for each article, and update to the article_article dataframe\n",
    "\n",
    "# ATTENTION: THE ITERATION TAKES HOURS to finish. 1050 x 1050 instances. \n",
    "\n",
    "def compute_and_update_similarity():\n",
    "    \n",
    "    start = time.time()\n",
    "\n",
    "\n",
    "    # Number of articles\n",
    "    len_articles = article_article.shape[0]\n",
    "\n",
    "    # List of index of articles\n",
    "    idx_articles = list(article_article.index)\n",
    "\n",
    "\n",
    "    # Loop thru each article: for each row, loop each columns. \n",
    "    for i in idx_articles:\n",
    "        print(f\"Processing row {i} of {len_articles-1}.\")\n",
    "\n",
    "        for j in idx_articles:\n",
    "            article_article.loc[i, j] = compute_article_similarity(i, j)\n",
    "\n",
    "\n",
    "    end = time.time()\n",
    "    print(f\"{(end - start) / 60} seconds\")\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Uncomment the below cell ONLY if want to run the compute and update again. (Warning: took hours)\n",
    "\n",
    "Basically you don't need to do so, as I have already done it and exported to pkl.\n",
    "(**It took 8 hours**)\n",
    "\n",
    "Ready to use. You can download from this link:\n",
    "\n",
    "https://www.dropbox.com/s/oogukiggii75ot3/article_article_similarity_df.pkl\n",
    "\n",
    "Just load it and use it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check info above. Uncomment only if necessary\n",
    "\n",
    "# compute_and_update_similarity()\n",
    "# article_article.to_pickle('article_article_similarity_df.pkl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>article_id</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>1041</th>\n",
       "      <th>1042</th>\n",
       "      <th>1043</th>\n",
       "      <th>1044</th>\n",
       "      <th>1045</th>\n",
       "      <th>1046</th>\n",
       "      <th>1047</th>\n",
       "      <th>1048</th>\n",
       "      <th>1049</th>\n",
       "      <th>1050</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>article_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.032471</td>\n",
       "      <td>0.072460</td>\n",
       "      <td>0.020439</td>\n",
       "      <td>0.259333</td>\n",
       "      <td>0.033410</td>\n",
       "      <td>0.016156</td>\n",
       "      <td>0.051355</td>\n",
       "      <td>0.046419</td>\n",
       "      <td>0.047824</td>\n",
       "      <td>...</td>\n",
       "      <td>0.063201</td>\n",
       "      <td>0.051671</td>\n",
       "      <td>0.143709</td>\n",
       "      <td>0.020163</td>\n",
       "      <td>0.010646</td>\n",
       "      <td>0.003492</td>\n",
       "      <td>0.004095</td>\n",
       "      <td>0.028436</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.078722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.032471</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.186967</td>\n",
       "      <td>0.028671</td>\n",
       "      <td>0.093232</td>\n",
       "      <td>0.098205</td>\n",
       "      <td>0.035012</td>\n",
       "      <td>0.155317</td>\n",
       "      <td>0.206362</td>\n",
       "      <td>0.103742</td>\n",
       "      <td>...</td>\n",
       "      <td>0.104562</td>\n",
       "      <td>0.152361</td>\n",
       "      <td>0.093032</td>\n",
       "      <td>0.030101</td>\n",
       "      <td>0.082490</td>\n",
       "      <td>0.014128</td>\n",
       "      <td>0.019350</td>\n",
       "      <td>0.056505</td>\n",
       "      <td>0.010927</td>\n",
       "      <td>0.090637</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.072460</td>\n",
       "      <td>0.186967</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.053018</td>\n",
       "      <td>0.172458</td>\n",
       "      <td>0.106455</td>\n",
       "      <td>0.041646</td>\n",
       "      <td>0.204603</td>\n",
       "      <td>0.339824</td>\n",
       "      <td>0.153497</td>\n",
       "      <td>...</td>\n",
       "      <td>0.149979</td>\n",
       "      <td>0.208527</td>\n",
       "      <td>0.139453</td>\n",
       "      <td>0.036274</td>\n",
       "      <td>0.009098</td>\n",
       "      <td>0.009416</td>\n",
       "      <td>0.013930</td>\n",
       "      <td>0.104403</td>\n",
       "      <td>0.018914</td>\n",
       "      <td>0.160396</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.020439</td>\n",
       "      <td>0.028671</td>\n",
       "      <td>0.053018</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.027609</td>\n",
       "      <td>0.037732</td>\n",
       "      <td>0.084109</td>\n",
       "      <td>0.048761</td>\n",
       "      <td>0.031276</td>\n",
       "      <td>0.049991</td>\n",
       "      <td>...</td>\n",
       "      <td>0.027299</td>\n",
       "      <td>0.026927</td>\n",
       "      <td>0.033499</td>\n",
       "      <td>0.010938</td>\n",
       "      <td>0.036107</td>\n",
       "      <td>0.012851</td>\n",
       "      <td>0.011922</td>\n",
       "      <td>0.010652</td>\n",
       "      <td>0.009425</td>\n",
       "      <td>0.036531</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.259333</td>\n",
       "      <td>0.093232</td>\n",
       "      <td>0.172458</td>\n",
       "      <td>0.027609</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.095990</td>\n",
       "      <td>0.026990</td>\n",
       "      <td>0.154261</td>\n",
       "      <td>0.135846</td>\n",
       "      <td>0.152328</td>\n",
       "      <td>...</td>\n",
       "      <td>0.122135</td>\n",
       "      <td>0.141956</td>\n",
       "      <td>0.137278</td>\n",
       "      <td>0.024670</td>\n",
       "      <td>0.007071</td>\n",
       "      <td>0.006955</td>\n",
       "      <td>0.003037</td>\n",
       "      <td>0.038724</td>\n",
       "      <td>0.072108</td>\n",
       "      <td>0.138237</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 1051 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "article_id      0         1         2         3         4         5     \\\n",
       "article_id                                                               \n",
       "0           1.000000  0.032471  0.072460  0.020439  0.259333  0.033410   \n",
       "1           0.032471  1.000000  0.186967  0.028671  0.093232  0.098205   \n",
       "2           0.072460  0.186967  1.000000  0.053018  0.172458  0.106455   \n",
       "3           0.020439  0.028671  0.053018  1.000000  0.027609  0.037732   \n",
       "4           0.259333  0.093232  0.172458  0.027609  1.000000  0.095990   \n",
       "\n",
       "article_id      6         7         8         9     ...      1041      1042  \\\n",
       "article_id                                          ...                       \n",
       "0           0.016156  0.051355  0.046419  0.047824  ...  0.063201  0.051671   \n",
       "1           0.035012  0.155317  0.206362  0.103742  ...  0.104562  0.152361   \n",
       "2           0.041646  0.204603  0.339824  0.153497  ...  0.149979  0.208527   \n",
       "3           0.084109  0.048761  0.031276  0.049991  ...  0.027299  0.026927   \n",
       "4           0.026990  0.154261  0.135846  0.152328  ...  0.122135  0.141956   \n",
       "\n",
       "article_id      1043      1044      1045      1046      1047      1048  \\\n",
       "article_id                                                               \n",
       "0           0.143709  0.020163  0.010646  0.003492  0.004095  0.028436   \n",
       "1           0.093032  0.030101  0.082490  0.014128  0.019350  0.056505   \n",
       "2           0.139453  0.036274  0.009098  0.009416  0.013930  0.104403   \n",
       "3           0.033499  0.010938  0.036107  0.012851  0.011922  0.010652   \n",
       "4           0.137278  0.024670  0.007071  0.006955  0.003037  0.038724   \n",
       "\n",
       "article_id      1049      1050  \n",
       "article_id                      \n",
       "0           0.000000  0.078722  \n",
       "1           0.010927  0.090637  \n",
       "2           0.018914  0.160396  \n",
       "3           0.009425  0.036531  \n",
       "4           0.072108  0.138237  \n",
       "\n",
       "[5 rows x 1051 columns]"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load from pre-calculated pkl. \n",
    "\n",
    "article_article = pd.read_pickle('article_article_similarity_df.pkl')\n",
    "\n",
    "\n",
    "# All calculated and cached cosine similarity score for every article. Ready to use.\n",
    "article_article.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Instead of finding in real time calculation with find_similar_content_articles(),\n",
    "# this is another approach that uses a lookup from pre-calculated\n",
    "# article_article df. It saves time for user.\n",
    "\n",
    "def lookup_similar_content_articles(article_id, data=article_article, n=20):\n",
    "    \"\"\"\n",
    "    NOTE THAT: the cosine similarity score is based on content: title + desc + body, \n",
    "    therefore only ids are in df_content are available to work with this method. \n",
    "    Otherwise, use lookup_similar_title_articles()\n",
    "    \n",
    "    input: n - number of top similar to return\n",
    "    \n",
    "    \"\"\"\n",
    "    \n",
    "    \n",
    "\n",
    "    if article_id in article_article.index.values:\n",
    "\n",
    "        ids = list(\n",
    "            data.loc[article_id][data.loc[article_id].index != article_id].sort_values(ascending=False).head(n).index)\n",
    "        names = list(df_nlp.loc[ids].doc_full_name.values)\n",
    "\n",
    "    else:\n",
    "        # print(f\"Article {article_id} is not in df_content.\")\n",
    "        # print(f\"We are unable to compute overall content similarity for it.\")\n",
    "        # print(f\"Alternatively, you might try lookup_similar_title_articles() which is based on title relevancy.\")\n",
    "        ids = []\n",
    "        names = []\n",
    "\n",
    "    return ids, names\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[800, 1035, 313, 805, 444, 721, 260, 967, 122, 96, 384, 809, 124, 723, 234, 892, 54, 253, 812, 861, 412, 732, 871, 74, 221, 89, 479, 616, 500, 567]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['Machine Learning for the Enterprise',\n",
       " 'Machine Learning for the Enterprise.',\n",
       " 'What is machine learning?',\n",
       " 'Machine Learning for everyone',\n",
       " 'Declarative Machine Learning',\n",
       " 'The power of machine learning in Spark',\n",
       " 'The Machine Learning Database',\n",
       " 'ML Algorithm != Learning Machine',\n",
       " 'Watson Machine Learning for Developers',\n",
       " 'Improving quality of life with Spark-empowered machine learning',\n",
       " 'Continuous Learning on Watson',\n",
       " 'Use the Machine Learning Library',\n",
       " 'Python Machine Learning: Scikit-Learn Tutorial',\n",
       " '10 Essential Algorithms For Machine Learning Engineers',\n",
       " '3 Scenarios for Machine Learning on Multicloud',\n",
       " 'Breaking the 80/20 rule: How data catalogs transform data scientists’ productivity',\n",
       " '8 ways to turn data into value with Apache Spark machine learning',\n",
       " 'Lifelong (machine) learning: how automation can help your models get smarter over time',\n",
       " 'Machine Learning Exercises In Python, Part 1',\n",
       " 'Cleaning the swamp: Turn your data lake into a source of crystal-clear insight',\n",
       " 'Adoption of machine learning to software failure prediction',\n",
       " 'Rapidly build Machine Learning flows with DSX',\n",
       " 'Overfitting in Machine Learning: What It Is and How to Prevent It',\n",
       " 'The 3 Kinds of Context: Machine Learning and the Art of the Frame',\n",
       " 'How smart catalogs can turn the big data flood into an ocean of opportunity',\n",
       " 'Top 20 R Machine Learning and Data Science packages',\n",
       " 'Drowning in data sources: How data cataloging could fix your findability problems',\n",
       " 'Three reasons machine learning models go out of sync',\n",
       " 'The Difference Between AI, Machine Learning, and Deep Learning?',\n",
       " 'You could be looking at it all wrong']"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Spot Check:\n",
    "# Lookup most relevant articles for 455th article\n",
    "\n",
    "relevant_for_455th_article = lookup_similar_content_articles(455, n=30)\n",
    "\n",
    "print(relevant_for_455th_article[0])\n",
    "relevant_for_455th_article[1]\n",
    "\n",
    "# Revealing that they are about on 'machine learning'."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Side Note on Datasets Inconsistency: \n",
    "\n",
    "- df_content (page content of articles)\n",
    "- df interaction between users and articles)\n",
    "\n",
    "Since that original df_content only contains articles from [0 - 1050].\n",
    "\n",
    "However, df (the interaction) dataset shows more unique article ids than those beyond 1050. e,g 11xx, 12xx, 13xx, 14xx.\n",
    "\n",
    "So these two datasets' articles are not up to date.\n",
    "\n",
    "The given df_content is 'late' (not catch up), which means that:\n",
    "\n",
    "**articles in the df (interaction) dataset might not be found in the df_content (content info) dataset.**\n",
    "\n",
    "Therefore, due to this inconsistency, we will not see content details for some articles mentioned in df (interaction set).\n",
    "\n",
    "The inconsistency is a limitation of given datasets. \n",
    "\n",
    "---\n",
    "\n",
    "Due to the limitation of inconsistency mentioned above,\n",
    "\n",
    "Let's create a unified article data frame that **stores every unique article's \\[ID\\] and \\[TITLE\\]**.\n",
    "\n",
    "This dataset serves the purpose of lookup the article's title. (Think of an index of all articles).\n",
    "\n",
    "Because of the inconsistency, the bottomline: **not every article has body/desc info, but all articles DO have titles**.\n",
    "\n",
    "The idea is that: **for those articles that cannot do NLP processing on \\[TITLE + DESC + BODY\\] content, we at least can process NLP on \\[TITLE\\] for them**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>article_id</th>\n",
       "      <th>title</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>article_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>Detect Malfunctioning IoT Sensors with Streami...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Communicating data science: A guide to present...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>This Week in Data Science (April 18, 2017)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>DataLayer Conference: Boost the performance of...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>Analyze NY Restaurant data using Spark in DSX</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1440</th>\n",
       "      <td>1440</td>\n",
       "      <td>world marriage data</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1441</th>\n",
       "      <td>1441</td>\n",
       "      <td>world tourism data by the world tourism organi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1442</th>\n",
       "      <td>1442</td>\n",
       "      <td>worldwide county and region - national account...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1443</th>\n",
       "      <td>1443</td>\n",
       "      <td>worldwide electricity demand and production 19...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1444</th>\n",
       "      <td>1444</td>\n",
       "      <td>worldwide fuel oil consumption by household (i...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1328 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           article_id                                              title\n",
       "article_id                                                              \n",
       "0                   0  Detect Malfunctioning IoT Sensors with Streami...\n",
       "1                   1  Communicating data science: A guide to present...\n",
       "2                   2         This Week in Data Science (April 18, 2017)\n",
       "3                   3  DataLayer Conference: Boost the performance of...\n",
       "4                   4      Analyze NY Restaurant data using Spark in DSX\n",
       "...               ...                                                ...\n",
       "1440             1440                                world marriage data\n",
       "1441             1441  world tourism data by the world tourism organi...\n",
       "1442             1442  worldwide county and region - national account...\n",
       "1443             1443  worldwide electricity demand and production 19...\n",
       "1444             1444  worldwide fuel oil consumption by household (i...\n",
       "\n",
       "[1328 rows x 2 columns]"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Make a all titles dataframe. Store every unique article.\n",
    "# Schema: article id, article title\n",
    "\n",
    "def make_titles_df(df=df, df_content=df_content):\n",
    "    \"\"\"\n",
    "    Generate an article index dataframe, from orginal df and df_content\n",
    "    Only store article id and title for every unique article.\n",
    "    \"\"\"\n",
    "    \n",
    "    \n",
    "    unique_ids_in_df = sorted(list(df.article_id.astype('int64').value_counts().index))\n",
    "    unique_ids_in_df_content = sorted(list(df_content.article_id.astype('int64').value_counts().index))\n",
    "    \n",
    "    # ids that are not in df_content, but appear in df\n",
    "    ids_not_in_df_content = list(set(unique_ids_in_df) - set(unique_ids_in_df_content))\n",
    "    # print(f\"How many?: {len(ids_not_in_df_content)}\")\n",
    "    \n",
    "    # Subset the diff articles dataframe\n",
    "    ids_not_in_df_content_float = [float(x) for x in ids_not_in_df_content]\n",
    "    df_diff = df[df.article_id.isin(ids_not_in_df_content_float)][['article_id', 'title']].drop_duplicates()\n",
    "    df_diff.article_id = df_diff.article_id.astype('int64')\n",
    "    df_diff = df_diff.sort_values(by='article_id', ascending=True)\n",
    "    \n",
    "    # Subset from df_content\n",
    "    df_content_titles_subset = df_content[['article_id', 'doc_full_name']]\n",
    "    df_content_titles_subset.columns = [['article_id', 'title']]\n",
    "    df_content_titles_subset.index = df_content_titles_subset['article_id'].values.flatten()\n",
    "    \n",
    "    # convert to numpy array shape and concatenate\n",
    "    a = df_diff.to_numpy()\n",
    "    b = df_content_titles_subset.to_numpy()\n",
    "    all_titles_np = np.concatenate((a, b))\n",
    "    \n",
    "    # make a dataframe\n",
    "    titles_df = pd.DataFrame(all_titles_np)\n",
    "    titles_df.columns = ['article_id', 'title']\n",
    "    titles_df.index = titles_df.article_id\n",
    "    titles_df = titles_df.sort_index()\n",
    "    \n",
    "    return titles_df\n",
    "    \n",
    "    \n",
    "    \n",
    "# Get titles df\n",
    "titles_df = make_titles_df()\n",
    "\n",
    "# Spot check titles df\n",
    "titles_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Also Do Similarty Finding Based on Title Only \n",
    "\n",
    "As we can see, we do have articles that have no content details avaiable.\n",
    "\n",
    "For these items, we use title to find similarity. \n",
    "\n",
    "We can do real-time compuation across all, or like above, we precalculate and cache to title-title dataframe for lookup."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Find similar articles for a given article based on title only similarity.\n",
    "# This only works on all articles in df or df_content.\n",
    "# Use only the TITLE of an article. this info available in both df and df_content\n",
    "# Calculate in real-time. It might take some seconds. \n",
    "# (since it loops to all articles against the targeted article, use cosine similarity, not dot product.)\n",
    "# For articles also in the df_content/df_nlp dataset, another \n",
    "# method is suitable lookup_similar_content_articles(), which checks based on more content information. \n",
    "\n",
    "\n",
    "def find_similar_title_articles(article_id, data=titles_df):\n",
    "    \n",
    "    \"\"\"\n",
    "    Note that it is based on page title only.\n",
    "    According orginal datasets, every article has title, not not all article have body/desc info.\n",
    "    \n",
    "    \"\"\"\n",
    "    \n",
    "    if article_id not in titles_df.index:\n",
    "        print(f\"Article {article_id} is not found.\")\n",
    "        print(f\"Please double check the id or try another.\")\n",
    "        result = []\n",
    "        \n",
    "    else:\n",
    "        \n",
    "        # similarities holder\n",
    "        similarity_dict = {}\n",
    "\n",
    "        # loop thru article_id aginst every article\n",
    "\n",
    "        for i in data.article_id.to_list():\n",
    "            # print(f\"\\n[i]: {i}\")\n",
    "\n",
    "            if i == article_id:\n",
    "                # if against self, skip to next loop\n",
    "                continue\n",
    "\n",
    "            # Tokenize self and another. \n",
    "            # Rejoin as documents.\n",
    "            doc_a = ' '.join(tokenize(data.loc[article_id].title))\n",
    "            doc_b = ' '.join(tokenize(data.loc[i].title))\n",
    "\n",
    "\n",
    "            # combine to a list of documents\n",
    "            documents = [doc_a, doc_b]\n",
    "\n",
    "\n",
    "            # instanciate a scikitlearn tfidf vecorizer\n",
    "            vectorizer = TfidfVectorizer()\n",
    "\n",
    "            # fit transform to get a sparse matrix\n",
    "            matrix = vectorizer.fit_transform(documents)\n",
    "\n",
    "\n",
    "            # # Uncomment belows to explore details (term features, tfidf values)\n",
    "            # ====================================\n",
    "            # term_features =  vectorizer.get_feature_names_out()\n",
    "\n",
    "            # # convert to readable array \n",
    "            # matrix_array = matrix.toarray()\n",
    "\n",
    "            # # assemble to a dataframe for explor \n",
    "            # tfidf_dataframe = pd.DataFrame(data=matrix_array, columns=term_features)\n",
    "            # ====================================\n",
    "\n",
    "            similarity_score = cosine_similarity(matrix[:1], matrix[1:])[0][0]\n",
    "            similarity_dict[i] = similarity_score\n",
    "            # print(f\"[similarity_score]: {similarity_score}\")\n",
    "\n",
    "\n",
    "        similarity_ds = pd.Series(data=similarity_dict.values(), \n",
    "                               index=similarity_dict.keys()).sort_values(ascending=False).index\n",
    "\n",
    "        result = list(similarity_ds)\n",
    "        \n",
    "        # Reminder check. \n",
    "        if article_id in df_content.article_id.index:\n",
    "            # print(f\"We also have content details (desc, body) information for this article: {article_id}.\")\n",
    "            # print(f\"Alternatively, you might want to try find_similar_content_articles() for it.\")\n",
    "            # print(f\"It might give you better result, because it checks on title, desc, body\")\n",
    "            pass\n",
    "    \n",
    "    return result\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Machine Learning', 'Deep Learning']\n",
      "['Apache Spark', 'Cloud Data']\n",
      "['Data Science', 'Ibm Data']\n",
      "['Ibm Watson', 'Watson Data']\n"
     ]
    }
   ],
   "source": [
    "# Test Semantic Topics Extracting from given an article id\n",
    "\n",
    "print(extract_topics_by_title_bigrams(500))\n",
    "\n",
    "print(extract_topics_by_title_bigrams(55))\n",
    "\n",
    "print(extract_topics_by_title_bigrams(1400))\n",
    "\n",
    "print(extract_topics_by_title_bigrams(937))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>article_id</th>\n",
       "      <th>title</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>article_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1420</th>\n",
       "      <td>1420</td>\n",
       "      <td>use apache systemml and spark for machine lear...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>161</th>\n",
       "      <td>161</td>\n",
       "      <td>Use the Machine Learning Library in Spark</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>893</th>\n",
       "      <td>893</td>\n",
       "      <td>Use the Machine Learning Library in IBM Analyt...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1172</th>\n",
       "      <td>1172</td>\n",
       "      <td>apache spark lab, part 3: machine learning</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>284</th>\n",
       "      <td>284</td>\n",
       "      <td>Apache Spark 2.0: Machine Learning. Under the ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>112</th>\n",
       "      <td>112</td>\n",
       "      <td>Building Custom Machine Learning Algorithms Wi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>809</th>\n",
       "      <td>809</td>\n",
       "      <td>Use the Machine Learning Library</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>721</th>\n",
       "      <td>721</td>\n",
       "      <td>The power of machine learning in Spark</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>313</th>\n",
       "      <td>313</td>\n",
       "      <td>What is machine learning?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>375</th>\n",
       "      <td>375</td>\n",
       "      <td>Apache SystemML</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54</th>\n",
       "      <td>54</td>\n",
       "      <td>8 ways to turn data into value with Apache Spa...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>762</th>\n",
       "      <td>762</td>\n",
       "      <td>From Machine Learning to Learning Machine (Din...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1354</th>\n",
       "      <td>1354</td>\n",
       "      <td>movie recommender system with spark machine le...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>260</th>\n",
       "      <td>260</td>\n",
       "      <td>The Machine Learning Database</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>805</th>\n",
       "      <td>805</td>\n",
       "      <td>Machine Learning for everyone</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1049</th>\n",
       "      <td>1049</td>\n",
       "      <td>Use dashDB with Spark</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1035</th>\n",
       "      <td>1035</td>\n",
       "      <td>Machine Learning for the Enterprise.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>444</th>\n",
       "      <td>444</td>\n",
       "      <td>Declarative Machine Learning</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>800</th>\n",
       "      <td>800</td>\n",
       "      <td>Machine Learning for the Enterprise</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>592</th>\n",
       "      <td>592</td>\n",
       "      <td>Apache Spark Analytics</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           article_id                                              title\n",
       "article_id                                                              \n",
       "1420             1420  use apache systemml and spark for machine lear...\n",
       "161               161          Use the Machine Learning Library in Spark\n",
       "893               893  Use the Machine Learning Library in IBM Analyt...\n",
       "1172             1172         apache spark lab, part 3: machine learning\n",
       "284               284  Apache Spark 2.0: Machine Learning. Under the ...\n",
       "112               112  Building Custom Machine Learning Algorithms Wi...\n",
       "809               809                   Use the Machine Learning Library\n",
       "721               721             The power of machine learning in Spark\n",
       "313               313                          What is machine learning?\n",
       "375               375                                    Apache SystemML\n",
       "54                 54  8 ways to turn data into value with Apache Spa...\n",
       "762               762  From Machine Learning to Learning Machine (Din...\n",
       "1354             1354  movie recommender system with spark machine le...\n",
       "260               260                      The Machine Learning Database\n",
       "805               805                      Machine Learning for everyone\n",
       "1049             1049                              Use dashDB with Spark\n",
       "1035             1035               Machine Learning for the Enterprise.\n",
       "444               444                       Declarative Machine Learning\n",
       "800               800                Machine Learning for the Enterprise\n",
       "592               592                             Apache Spark Analytics"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Spot Check:\n",
    "\n",
    "# Relevant articles for 1420, based on title NLP cosine similarity calculation.\n",
    "\n",
    "relevant_title_articles_for_1420th = find_similar_title_articles(1420)\n",
    "\n",
    "titles_df.loc[[1420] + relevant_title_articles_for_1420th].head(20)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "As we can see, the relevancy of title calculation works fine, it is calculated in real-time. \n",
    "\n",
    "It loops thru a targeted article against every article. \n",
    "\n",
    "Agian, make a precalcualted title-title dataframe to store all title-title cosine similarity score,\n",
    "so that we can cache and lookup directly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>article_id</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>1434</th>\n",
       "      <th>1435</th>\n",
       "      <th>1436</th>\n",
       "      <th>1437</th>\n",
       "      <th>1439</th>\n",
       "      <th>1440</th>\n",
       "      <th>1441</th>\n",
       "      <th>1442</th>\n",
       "      <th>1443</th>\n",
       "      <th>1444</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>article_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 1328 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "article_id  0     1     2     3     4     5     6     7     8     9     ...  \\\n",
       "article_id                                                              ...   \n",
       "0            0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  ...   \n",
       "1            0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  ...   \n",
       "2            0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  ...   \n",
       "3            0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  ...   \n",
       "4            0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  ...   \n",
       "\n",
       "article_id  1434  1435  1436  1437  1439  1440  1441  1442  1443  1444  \n",
       "article_id                                                              \n",
       "0            0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  \n",
       "1            0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  \n",
       "2            0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  \n",
       "3            0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  \n",
       "4            0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0   0.0  \n",
       "\n",
       "[5 rows x 1328 columns]"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Make a title-title dummy dataframe based on titles_df\n",
    "title_title = pd.DataFrame(\n",
    "    data=np.zeros((titles_df.shape[0], titles_df.shape[0])),\n",
    "    index=titles_df.index,\n",
    "    columns=titles_df.index\n",
    ")\n",
    "\n",
    "title_title.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compute title-title cosine similarity for each article, and update to the title-title dataframe\n",
    "\n",
    "# ATTENTION: THE ITERATION TAKES HOURS to finish. \n",
    "\n",
    "\n",
    "def _compute_title_title_similarity(article_id_1, article_id_2, data=titles_df):\n",
    "    # Compute the consine similarity based on tfidf of title (doc_full_name)\n",
    "    # Based on the titles df (union all titles in df and df_content)\n",
    "    \n",
    "\n",
    "    doc_a = ' '.join(tokenize(data.loc[article_id_1].title))\n",
    "    doc_b = ' '.join(tokenize(data.loc[article_id_2].title))\n",
    "    \n",
    "    \n",
    "\n",
    "    # combine to a list of documents\n",
    "    documents = [doc_a, doc_b]\n",
    "\n",
    "    \n",
    "    # instanciate a scikitlearn tfidf vecorizer\n",
    "    vectorizer = TfidfVectorizer()\n",
    "\n",
    "    # fit transform to get a sparse matrix\n",
    "    matrix = vectorizer.fit_transform(documents)\n",
    "\n",
    "\n",
    "    # # UnComment out belows to examing details (term features, tfidf values)\n",
    "    # ====================================\n",
    "    # term_features =  vectorizer.get_feature_names_out()\n",
    "\n",
    "    # # convert to readable array \n",
    "    # matrix_array = matrix.toarray()\n",
    "\n",
    "    # # assemble to a dataframe for explor \n",
    "    # tfidf_dataframe = pd.DataFrame(data=matrix_array, columns=term_features)\n",
    "    # ====================================\n",
    "\n",
    "\n",
    "    return cosine_similarity(matrix[:1], matrix[1:])[0][0]\n",
    "\n",
    "\n",
    "\n",
    "def compute_and_update_title_title_similarity():\n",
    "    \n",
    "    \"\"\"\n",
    "    Update every cell of title_title dataframe with title_title cosine similarity score.\n",
    "    \"\"\"\n",
    "    \n",
    "    start = time.time()\n",
    "\n",
    "\n",
    "    # Number of articles in title_title\n",
    "    len_articles = title_title.shape[0]\n",
    "\n",
    "\n",
    "    # List of index of all articles\n",
    "    idx_articles = list(title_title.index)\n",
    "\n",
    "    # Last idx for tracking in print\n",
    "    idx_last = title_title.tail(1).index[0]\n",
    "\n",
    "    # Loop thru each article: for each row, loop each columns. \n",
    "    for i in idx_articles:\n",
    "        print(f\"Processing row {i} of {idx_last}.\")\n",
    "\n",
    "        for j in idx_articles:\n",
    "            title_title.loc[i, j] = _compute_title_title_similarity(i, j)\n",
    "\n",
    "\n",
    "    end = time.time()\n",
    "    print(f\"{(end - start) / 60} seconds\")\n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Uncomment the below cell ONLY if want to run the compute and update again. (Warning: took time)\n",
    "\n",
    "Basically you don't need to do so, as I have already done it and exported to pkl. (It took 1 hour)\n",
    "\n",
    "Ready to use. You can download from this link:\n",
    "\n",
    "https://www.dropbox.com/s/j74bmaese35i2pz/title_title_similarity_df.pkl\n",
    "\n",
    "Just load it and use it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check info above. Uncomment only if necessary\n",
    "\n",
    "# compute_and_update_title_title_similarity()\n",
    "# title_title.to_pickle('title_title_similarity_df.pkl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>article_id</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>1434</th>\n",
       "      <th>1435</th>\n",
       "      <th>1436</th>\n",
       "      <th>1437</th>\n",
       "      <th>1439</th>\n",
       "      <th>1440</th>\n",
       "      <th>1441</th>\n",
       "      <th>1442</th>\n",
       "      <th>1443</th>\n",
       "      <th>1444</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>article_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.201993</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.08458</td>\n",
       "      <td>0.115216</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.136276</td>\n",
       "      <td>0.260556</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.136276</td>\n",
       "      <td>0.069973</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.201993</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.08458</td>\n",
       "      <td>0.115216</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.136276</td>\n",
       "      <td>0.450176</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.136276</td>\n",
       "      <td>0.069973</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.084580</td>\n",
       "      <td>0.084580</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>0.105992</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.125367</td>\n",
       "      <td>0.105992</td>\n",
       "      <td>0.130494</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.125367</td>\n",
       "      <td>0.064371</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 1328 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "article_id  0         1         2     3        4         5     6         7     \\\n",
       "article_id                                                                      \n",
       "0            1.0  0.000000  0.000000   0.0  0.00000  0.000000   0.0  0.000000   \n",
       "1            0.0  1.000000  0.201993   0.0  0.08458  0.115216   0.0  0.136276   \n",
       "2            0.0  0.201993  1.000000   0.0  0.08458  0.115216   0.0  0.136276   \n",
       "3            0.0  0.000000  0.000000   1.0  0.00000  0.000000   0.0  0.000000   \n",
       "4            0.0  0.084580  0.084580   0.0  1.00000  0.105992   0.0  0.125367   \n",
       "\n",
       "article_id      8         9     ...  1434  1435  1436  1437  1439      1440  \\\n",
       "article_id                      ...                                           \n",
       "0           0.000000  0.000000  ...   0.0   0.0   0.0   0.0   0.0  0.000000   \n",
       "1           0.260556  0.000000  ...   0.0   0.0   0.0   0.0   0.0  0.136276   \n",
       "2           0.450176  0.000000  ...   0.0   0.0   0.0   0.0   0.0  0.136276   \n",
       "3           0.000000  0.000000  ...   0.0   0.0   0.0   0.0   0.0  0.000000   \n",
       "4           0.105992  0.130494  ...   0.0   0.0   0.0   0.0   0.0  0.125367   \n",
       "\n",
       "article_id      1441  1442  1443  1444  \n",
       "article_id                              \n",
       "0           0.000000   0.0   0.0   0.0  \n",
       "1           0.069973   0.0   0.0   0.0  \n",
       "2           0.069973   0.0   0.0   0.0  \n",
       "3           0.000000   0.0   0.0   0.0  \n",
       "4           0.064371   0.0   0.0   0.0  \n",
       "\n",
       "[5 rows x 1328 columns]"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load from pre-calculated pkl. \n",
    "\n",
    "title_title = pd.read_pickle('title_title_similarity_df.pkl')\n",
    "\n",
    "\n",
    "# All calculated and cached cosine similarity score for every article. Ready to use.\n",
    "title_title.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Instead of finding in real time calculation with find_similar_title_articles(),\n",
    "# this is another approach that uses a lookup from pre-calculated\n",
    "# title_title df. It saves time for user.\n",
    "\n",
    "def lookup_similar_title_articles(article_id, data=title_title, n=20):\n",
    "    \"\"\"\n",
    "    NOTE THAT: the cosine similarity score is based on title only.\n",
    "    \n",
    "    input: n - number of top similar to return\n",
    "    \n",
    "    \n",
    "    \"\"\"\n",
    "    \n",
    "    if article_id in title_title.index.values: \n",
    "    \n",
    "        ids = list(data.loc[article_id][data.loc[article_id].index != article_id].sort_values(ascending=False).head(20).index)\n",
    "        names = list(titles_df.loc[ids].title.values)\n",
    "        \n",
    "    else:\n",
    "        print(f\"[article_id] {article_id} not found in datasets.\")\n",
    "        ids = []\n",
    "        names = []\n",
    "        \n",
    "    \n",
    "    return ids, names\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([122,\n",
       "  313,\n",
       "  1427,\n",
       "  762,\n",
       "  254,\n",
       "  1390,\n",
       "  655,\n",
       "  805,\n",
       "  444,\n",
       "  800,\n",
       "  260,\n",
       "  1035,\n",
       "  384,\n",
       "  1001,\n",
       "  40,\n",
       "  500,\n",
       "  893,\n",
       "  1297,\n",
       "  390,\n",
       "  1185],\n",
       " ['Watson Machine Learning for Developers',\n",
       "  'What is machine learning?',\n",
       "  'use xgboost, scikit-learn & ibm watson machine learning apis',\n",
       "  'From Machine Learning to Learning Machine (Dinesh Nirmal)',\n",
       "  'Apple, IBM add machine learning to partnership with Watson-Core ML coupling',\n",
       "  'style transfer experiments with watson machine learning',\n",
       "  'Create a project for Watson Machine Learning in DSX',\n",
       "  'Machine Learning for everyone',\n",
       "  'Declarative Machine Learning',\n",
       "  'Machine Learning for the Enterprise',\n",
       "  'The Machine Learning Database',\n",
       "  'Machine Learning for the Enterprise.',\n",
       "  'Continuous Learning on Watson',\n",
       "  'Get started in Bluemix',\n",
       "  'Ensemble Learning to Improve Machine Learning Results',\n",
       "  'The Difference Between AI, Machine Learning, and Deep Learning?',\n",
       "  'Use the Machine Learning Library in IBM Analytics for Apache Spark',\n",
       "  'from local spark mllib model to cloud with watson machine learning',\n",
       "  'Introducing IBM Watson Studio ',\n",
       "  'classify tumors with machine learning'])"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Spot Check\n",
    "\n",
    "lookup_similar_title_articles(437)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Internal Utility Helper Functions\n",
    "\n",
    "\n",
    "\n",
    "def _union_list(list_a, list_b):\n",
    "    \"\"\"\n",
    "    Combine two lists. \n",
    "    Use Case:\n",
    "    List that from title similarity, list that from content similarity. \n",
    "    Intersection plus each list's unique items.\n",
    "    Return the blended union result.\n",
    "    \n",
    "    \n",
    "    INPUT: two lists \n",
    "    OUTPUT: a union list. (unique of each list plus intersection)\n",
    "    \n",
    "    \"\"\"\n",
    "    \n",
    "    \n",
    "    \n",
    "    \n",
    "    intersect = np.intersect1d(list_a, list_b)\n",
    "    unique_in_a = set(list_a) - set(intersect)\n",
    "    unique_in_b = set(list_b) - set(intersect)\n",
    "    \n",
    "    union = list(set.union(set.union(unique_in_a, unique_in_b), intersect))\n",
    "    \n",
    "    # shuffle the list\n",
    "    random.shuffle(union)\n",
    "\n",
    "    \n",
    "    return union\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "def _get_article_titles(ids, data=titles_df):\n",
    "    \"\"\"\n",
    "    With given article ids (list), return titles as list. \n",
    "    Assume that the ids given are valid. \n",
    "    This is a helper function, only serves for output title name, which means  \n",
    "    No Sanity Check Here: the outside is responsible for passing valid ids into here. \n",
    "    \n",
    "    \n",
    "    INPUT: list of article ids\n",
    "    OUTPUT: list of article titles\n",
    "    \n",
    "    \"\"\"\n",
    "    # Input: a list of article ids\n",
    "    # data: the all article title dataframe\n",
    "    \n",
    "    titles = data.loc[ids].title.apply(lambda x: x.title()).values\n",
    "    result = list(titles)\n",
    "\n",
    "    return result\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "\n",
    "### Putting All Content-Content together to Make Recs\n",
    "\n",
    "#### Strategy Explain: \n",
    "\n",
    "1. Get relevant articles based on TITLE similarity. (**Guaranteed**. )\n",
    "2. Get relevant articles based on CONTENT (title + desc + body weighted sum) similarity. (**Not Guaranteed**. Only if an article has details available. )\n",
    "3. Combine them to a union list. \n",
    "\n",
    "Then, look further for an extra layer, to fetch relevant content of its relevant content. \n",
    "Think of 'relevant of relevant.' \n",
    "\n",
    "4. Add these extra on the union list. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_content_recs(article_id, data=titles_df):\n",
    "    '''\n",
    "    Content Based Recommendations. 'Content-Content'.\n",
    "\n",
    "    Strategy:\n",
    "    1. Get relevant articles based on TITLE similarity. (Guaranteed. )\n",
    "    2. Get relevant articles based on CONTENT (title + desc + body weighted sum) similarity.\n",
    "    (Not Guaranteed. Only if an article has details available. )\n",
    "    3. Combine them to a union list.\n",
    "    Then, look further for an extra layer, to fetch relevant content of its relevant content. Think of 'relevant of relevant.'\n",
    "    4. Add these extra on the union list.\n",
    "\n",
    "    INPUT: article id\n",
    "    OUTPUT: tuple of recommendation ids and titles\n",
    "    '''\n",
    "\n",
    "    recs_based_on_title = lookup_similar_title_articles(article_id)[0]\n",
    "    recs_based_on_content = lookup_similar_content_articles(article_id)[0]\n",
    "\n",
    "    union_recs = _union_list(recs_based_on_title, recs_based_on_content)\n",
    "    \n",
    "    # a list holder for second layer recs items\n",
    "    extra_recs = []\n",
    "\n",
    "    # If article_id not in df_content, find one from the union_recs list that is in df_content or article_article,\n",
    "    # so that it can use content similarity score to load extra relevant content.\n",
    "    if article_id not in article_article.index.values:\n",
    "        # print(f\"{article_id} not in article_article, pulling additional relevant content.\")\n",
    "\n",
    "        # loop thru first layer recs\n",
    "        for i in union_recs:\n",
    "            if i in article_article.index.values:\n",
    "                # find similar base on content similarity score, and slice first 5 instances\n",
    "                extra = lookup_similar_content_articles(i)[0][:5]\n",
    "                extra_recs.append(extra)\n",
    "\n",
    "    extra_recs = np.array(extra_recs).flatten()\n",
    "    extra_recs = list(np.unique(extra_recs))\n",
    "\n",
    "    for e in extra_recs:\n",
    "        if e not in union_recs:\n",
    "            union_recs.append(e)\n",
    "\n",
    "    titles = _get_article_titles(union_recs)\n",
    "\n",
    "    return union_recs, titles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([805,\n",
       "  34,\n",
       "  1035,\n",
       "  800,\n",
       "  1390,\n",
       "  1422,\n",
       "  260,\n",
       "  437,\n",
       "  1420,\n",
       "  893,\n",
       "  124,\n",
       "  655,\n",
       "  762,\n",
       "  313,\n",
       "  444,\n",
       "  809,\n",
       "  122,\n",
       "  254,\n",
       "  161,\n",
       "  384,\n",
       "  96,\n",
       "  112,\n",
       "  250,\n",
       "  455,\n",
       "  495,\n",
       "  511,\n",
       "  555,\n",
       "  592,\n",
       "  595,\n",
       "  600,\n",
       "  686,\n",
       "  721,\n",
       "  723,\n",
       "  751,\n",
       "  793,\n",
       "  806,\n",
       "  812,\n",
       "  907,\n",
       "  967,\n",
       "  972,\n",
       "  1006],\n",
       " ['Machine Learning For Everyone',\n",
       "  'Top 10 Machine Learning Use Cases: Part 1',\n",
       "  'Machine Learning For The Enterprise.',\n",
       "  'Machine Learning For The Enterprise',\n",
       "  'Style Transfer Experiments With Watson Machine Learning',\n",
       "  'Use R Dataframes & Ibm Watson Natural Language Understanding',\n",
       "  'The Machine Learning Database',\n",
       "  'Ibm Watson Machine Learning: Get Started',\n",
       "  'Use Apache Systemml And Spark For Machine Learning',\n",
       "  'Use The Machine Learning Library In Ibm Analytics For Apache Spark',\n",
       "  'Python Machine Learning: Scikit-Learn Tutorial',\n",
       "  'Create A Project For Watson Machine Learning In Dsx',\n",
       "  'From Machine Learning To Learning Machine (Dinesh Nirmal)',\n",
       "  'What Is Machine Learning?',\n",
       "  'Declarative Machine Learning',\n",
       "  'Use The Machine Learning Library',\n",
       "  'Watson Machine Learning For Developers',\n",
       "  'Apple, Ibm Add Machine Learning To Partnership With Watson-Core Ml Coupling',\n",
       "  'Use The Machine Learning Library In Spark',\n",
       "  'Continuous Learning On Watson',\n",
       "  'Improving Quality Of Life With Spark-Empowered Machine Learning',\n",
       "  'Building Custom Machine Learning Algorithms With Apache Systemml',\n",
       "  'Building Your First Machine Learning System ',\n",
       "  'Make Machine Learning A Reality For Your Enterprise',\n",
       "  'Top 10 Machine Learning Algorithms For Beginners',\n",
       "  'Put A Human Face On Machine Learning With Wml & Dsx',\n",
       "  'Build A Naive-Bayes Model With Wml & Dsx',\n",
       "  'Apache Spark Analytics',\n",
       "  'Load Dashdb Data With Apache Spark',\n",
       "  'Access Ibm Analytics For Apache Spark From Rstudio',\n",
       "  'Score A Predictive Model Built With Ibm Spss Modeler, Wml & Dsx',\n",
       "  'The Power Of Machine Learning In Spark',\n",
       "  '10 Essential Algorithms For Machine Learning Engineers',\n",
       "  'Build A Predictive Analytic Model',\n",
       "  '10 Powerful Features On Watson Data Platform, No Coding Necessary',\n",
       "  'Collaborate On Projects In Dsx',\n",
       "  'Machine Learning Exercises In Python, Part 1',\n",
       "  'Build Spark Sql Queries',\n",
       "  'Ml Algorithm != Learning Machine',\n",
       "  'Create A Project In Dsx',\n",
       "  'Essentials Of Machine Learning Algorithms (With Python And R Codes)'])"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "make_content_recs(1427)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extracting Semantic Topic(s) from Article"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "# helper function: get n-grams (two grams here)\n",
    "\n",
    "# Why 2-grams, not 3-grams or everygram? Because thru experiment, I found \n",
    "# 2 grams generalize the meaning well, \n",
    "# 3 grams often off the track, every-gram went too verbose and miss out meaning\n",
    "# It is case by case, however for these articles, 2 grams works well. \n",
    "\n",
    "\n",
    "def get_content_bigrams_by_article_id(article_id, data=df_nlp):\n",
    "    \"\"\"\n",
    "    Input an article id, return the bigrams results of its content\n",
    "    Then Reorder grams By most frequent.\n",
    "    \n",
    "    Note that the content is combination string of title, desc, body.\n",
    "    So assume that the article id is IN df_content, if the artcile is not in, we will\n",
    "    not be able to get its decs, body content.\n",
    "    \n",
    "    \n",
    "    \"\"\"\n",
    "    \n",
    "    if article_id not in data.index:\n",
    "        print(f\"{article_id} is not found in article content details dataset.\")\n",
    "        print(f\"You can try ngrams from title. \\nAlternative method: get_title_bigrams_by_article_id()\")\n",
    "        return []\n",
    "    \n",
    "    else:\n",
    "        title = data.loc[article_id].doc_full_name\n",
    "        desc = data.loc[article_id].doc_description\n",
    "        body = data.loc[article_id].doc_body\n",
    "\n",
    "        # a list of tokens\n",
    "        tokened_text = tokenize(title) + tokenize(desc) + tokenize(body)\n",
    "\n",
    "        # ngrams 2\n",
    "        grams = [f\"{x[0]} {x[1]}\" for x in list(bigrams(tokened_text))]\n",
    "\n",
    "        # sort with frequency descending\n",
    "        sorted_grams = [x[0] for x in Counter(grams).most_common()]\n",
    "\n",
    "        return sorted_grams\n",
    "    \n",
    "\n",
    "    \n",
    "\n",
    "    \n",
    "def get_title_bigrams_by_article_id(article_id, data=titles_df):\n",
    "    \"\"\"\n",
    "    Input an article id, return the bigrams results of its title\n",
    "    Then Reorder grams By most frequent.\n",
    "    \n",
    "    Note that it is based on page title only.\n",
    "    According orginal datasets, every article has title, not not all article have body/desc info.\n",
    "    \n",
    "    \n",
    "    \"\"\"\n",
    "    if article_id not in data.index:\n",
    "        print(f\"{article_id} is not found in article titles dataset.\")\n",
    "        print(f\"We don't have any information for this article. \\nPlease check article id again.\")\n",
    "        return []\n",
    "    \n",
    "    else:\n",
    "    \n",
    "        title = data.loc[article_id].title\n",
    "\n",
    "        # a list of tokens\n",
    "        tokened_text = tokenize(title)\n",
    "\n",
    "        # ngrams 2\n",
    "        grams = [f\"{x[0]} {x[1]}\" for x in list(bigrams(tokened_text))]\n",
    "\n",
    "        # sort with frequency descending\n",
    "        sorted_grams = [x[0] for x in Counter(grams).most_common()]\n",
    "        \n",
    "        # Reminder check\n",
    "        if article_id in df_content.article_id.index:\n",
    "            # print(f\"We also have content details, e,g desc, body information for this article: {article_id}.\")\n",
    "            # print(f\"Alternatively, you might want to try get_content_bigrams_by_article_id() for it.\")\n",
    "            pass\n",
    "        \n",
    "        \n",
    "        return sorted_grams\n",
    "    \n",
    "\n",
    "    \n",
    "def extract_topics_by_title_bigrams(article_id, m=2):\n",
    "    \"\"\"\n",
    "    Extract semantic topic with a given article ID.\n",
    "    It takes all similar articles into consideration, then output highly possible topics. \n",
    "    \n",
    "    Algorithm Explain:\n",
    "    1. For this targeted article, we pull out its most similar articles.\n",
    "    2. Combine them to a list of documents.\n",
    "    3. Batch process NLP bigrams on each doc.\n",
    "    4. Find out most frequent terms amount all bigrams. \n",
    "    5. Slice first m (e.g 2) terms. \n",
    "\n",
    "    INPUT:\n",
    "    m - threshold of top m to slice from the frequent term counter\n",
    "\n",
    "    OUTPUT:\n",
    "    topics - Possible topics.\n",
    "    \"\"\"\n",
    "\n",
    "    # Get relevant articles' ids. (also append self)\n",
    "    ids = [article_id] + make_content_recs(article_id)[0]\n",
    "\n",
    "    # Get ngrams for each title, and assemble to documents list.\n",
    "    documents = [get_title_bigrams_by_article_id(i) for i in ids]\n",
    "\n",
    "    # Find most common bigrams of all its relevant articles.\n",
    "    # These bigrams will highly possible be the topics.\n",
    "    documents_flatten = [item for sublist in documents for item in sublist]\n",
    "\n",
    "    # most counted terms. Select first m instances.\n",
    "    topics_counter = Counter(documents_flatten).most_common()[:m]\n",
    "\n",
    "    # Convert to list.\n",
    "    topics = [x[0].title() for x in topics_counter]\n",
    "\n",
    "    return topics\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Machine Learning', 'Deep Learning']\n",
      "['Apache Spark', 'Cloud Data']\n",
      "['Data Science', 'Ibm Data']\n",
      "['Ibm Watson', 'Watson Data']\n",
      "['Apache Spark', 'Jupyter Notebook']\n"
     ]
    }
   ],
   "source": [
    "# Spot Check:\n",
    "# Test Sementic Topics Extracting from given an article id\n",
    "\n",
    "print(extract_topics_by_title_bigrams(500))\n",
    "print(extract_topics_by_title_bigrams(55))\n",
    "print(extract_topics_by_title_bigrams(1400))\n",
    "print(extract_topics_by_title_bigrams(937))\n",
    "print(extract_topics_by_title_bigrams(1314))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "### Get Insight of Top Semantic Topics within the Raw Dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Internal Helpler Private Function.\n",
    "# Get most interacted (popular) articles.\n",
    "\n",
    "def _get_most_popular_articles(data=df, n=20):\n",
    "    # input: n - number of top viewed to return\n",
    "    \n",
    "    ids = data.groupby('article_id').size().sort_values(ascending=False).head(n).index\n",
    "    \n",
    "    return list(ids.values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show Top Sementic Topics (NLP mining)\n",
    "\n",
    "def show_top_topics():\n",
    "    \"\"\"\n",
    "    A Business intelligence (BI) tool that helps executives, managers and workers make informed business decisions.\n",
    "\n",
    "    What are the highly possible top semantic topics amount the data.\n",
    "\n",
    "    No output return. Just print information to console.\n",
    "    \"\"\"\n",
    "\n",
    "\n",
    "\n",
    "    # Checked the top interacted content\n",
    "    most_popular_list = _get_most_popular_articles()\n",
    "    print(f\"💡 Most popular(interacted) contents:\\n\")\n",
    "    print(most_popular_list)\n",
    "    print('\\n' * 3)\n",
    "\n",
    "    # A list to hold each hot/popular content's topics\n",
    "    popular_topics_overall = []\n",
    "\n",
    "    print(f\"💡 What are their topics? (Based on NLP mining)\\n\")\n",
    "\n",
    "    # Loop thru each popular article, extract its topic and combine them.\n",
    "    for article_id in most_popular_list:\n",
    "        article_id = int(article_id)\n",
    "        topics = extract_topics_by_title_bigrams(article_id)\n",
    "        print(f\"[{article_id}] is about: {topics}\")\n",
    "        popular_topics_overall.append(topics)\n",
    "\n",
    "    # Sort with most frequent to represent most popular topics\n",
    "    print('\\n' * 3)\n",
    "    print(f\"💡 Based on the most viewed articles (top 20 instances), the top 10 popular topics are:\\n\")\n",
    "    most_popular_topics = [x[0] for x in Counter(np.array(popular_topics_overall).flatten()).most_common()[0:10]]\n",
    "\n",
    "    # Show top topics\n",
    "    for k, v in enumerate(most_popular_topics):\n",
    "        print(k + 1, v)\n",
    "\n",
    "    print('\\n' * 3)\n",
    "    print(f\"💡 Thoughts:\\n\")\n",
    "    print(\n",
    "        f\"This is valuable because, based on such demand, we can feed more related content in the future. \\n\"\n",
    "        f\"Also, to serve as a knowledge-based recommendation for brand new users.\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "💡 Most popular(interacted) contents:\n",
      "\n",
      "[1429.0, 1330.0, 1431.0, 1427.0, 1364.0, 1314.0, 1293.0, 1170.0, 1162.0, 1304.0, 1436.0, 1271.0, 1398.0, 43.0, 1351.0, 1393.0, 1185.0, 1160.0, 1354.0, 1368.0]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "💡 What are their topics? (Based on NLP mining)\n",
      "\n",
      "[1429] is about: ['Deep Learning', 'Machine Learning']\n",
      "[1330] is about: ['Cloudant Query', 'Apache Spark']\n",
      "[1431] is about: ['Data Science', 'Watson Data']\n",
      "[1427] is about: ['Machine Learning', 'Watson Machine']\n",
      "[1364] is about: ['Machine Learning', 'Data Science']\n",
      "[1314] is about: ['Apache Spark', 'Jupyter Notebook']\n",
      "[1293] is about: ['Data Science', 'Cloudant Query']\n",
      "[1170] is about: ['Apache Spark', 'Spark Sql']\n",
      "[1162] is about: ['Machine Learning', 'Jupyter Notebook']\n",
      "[1304] is about: ['Machine Learning', 'Logistic Regression']\n",
      "[1436] is about: ['Data Science', 'Ibm Watson']\n",
      "[1271] is about: ['Making Compose', 'Compose Customer']\n",
      "[1398] is about: ['Country Statistic', 'Country Population']\n",
      "[43] is about: ['Deep Learning', 'Data Science']\n",
      "[1351] is about: ['Data Science', 'Watson Data']\n",
      "[1393] is about: ['Data Science', 'Web Pick']\n",
      "[1185] is about: ['Machine Learning', 'Deep Learning']\n",
      "[1160] is about: ['Apache Spark', 'Spark 2']\n",
      "[1354] is about: ['Machine Learning', 'Apache Spark']\n",
      "[1368] is about: ['Machine Learning', 'Wml Dsx']\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "💡 Based on the most viewed articles (top 20 instances), the top 10 popular topics are:\n",
      "\n",
      "1 Machine Learning\n",
      "2 Data Science\n",
      "3 Apache Spark\n",
      "4 Deep Learning\n",
      "5 Cloudant Query\n",
      "6 Watson Data\n",
      "7 Jupyter Notebook\n",
      "8 Watson Machine\n",
      "9 Spark Sql\n",
      "10 Logistic Regression\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "💡 Thoughts:\n",
      "\n",
      "This is valuable because, based on such demand, we can feed more related content in the future. \n",
      "Also, to serve as a knowledge-based recommendation for brand new users.\n"
     ]
    }
   ],
   "source": [
    "# BI Spot Check\n",
    "\n",
    "show_top_topics()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`2.` Now that you have put together your content-based recommendation system, use the cell below to write a summary explaining how your content based recommender works.  Do you see any possible improvements that could be made to your function?  Is there anything novel about your content based recommender?\n",
    "\n",
    "### This part is NOT REQUIRED to pass this project.  However, you may choose to take this on as an extra way to show off your skills."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A: As above shown, the content-based recommendation system with the blending of multiple approaches:\n",
    "- NLP TFIDF cosine similarity computed on the content (Weighted Sum of title, desc, body)\n",
    "- Due to dataset inconsistency, to overcome it, I also give a 'fallback' for those articles that don't have content details, compute cosine similarity on the title.\n",
    "- Each similarity score allows the system to compute in real-time or by lookup from precalculate / cache scores.\n",
    "- The make recs is a blending article title similarity and content similarity strategy.\n",
    "- In addition, the system extracts highly possible semantic topics for any given article by mining from its neighbors.\n",
    "- Finally, it has a Business Intelligence tool that shows the insight of top semantic topics. It can guide investors/managers/executives to feed quality information and understand trends, demands."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "`3.` Use your content-recommendation system to make recommendations for the below scenarios based on the comments.  Again no tests are provided here, because there isn't one right answer that could be used to find these content based recommendations.\n",
    "\n",
    "### This part is NOT REQUIRED to pass this project.  However, you may choose to take this on as an extra way to show off your skills."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "### A: Recs for Brand New Users\n",
    "\n",
    "#### Strategy: Content-based + Knowleadge-based Blending \n",
    "\n",
    "**Approach:**\n",
    "1. Pull the most popular content, ranked based. (First layer - most popular items)\n",
    "2. Pull similar contents of each popular content (Second layer - relevant of popular)\n",
    "3. Blend them. Make sure the first layer's content shows first (ranked order), followed by the second layer (random shuffle)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cold Start recs\n",
    "\n",
    "def make_recs_for_brand_new_user(threshold_second_layer=5):\n",
    "    \"\"\"\n",
    "    Make recs for brand new user.\n",
    "    \n",
    "    Strategy: Content-based + Knowledge-based Blending \n",
    "\n",
    "    **Approach:**\n",
    "    1. Pull the most popular content, ranked based. (First layer - most popular items)\n",
    "    2. Pull similar contents of each popular content (Second layer - relevant of popular)\n",
    "    3. Blend them. Make sure the first layer's content shows first (ranked order), \n",
    "                    followed by the second layer (random shuffle).\n",
    "\n",
    "    INPUT:\n",
    "    threshold_second_layer: How many recs items for each 2nd layer fetch.\n",
    "\n",
    "    OUTPUT: a tuple of ids and titles\n",
    "    \"\"\"\n",
    "\n",
    "    # Get list of most popular contents\n",
    "    popular_ids = _get_most_popular_articles(n=10)\n",
    "    popular_ids = [int(x) for x in popular_ids]\n",
    "\n",
    "    # List of recs of similar content for popular content. (the 'All 2nd-layer recs')\n",
    "    recs_for_similar = []\n",
    "\n",
    "    # We will make sure it has no duplication\n",
    "    recs_for_similar_no_duplicate = []\n",
    "\n",
    "    for i in popular_ids:\n",
    "        # print(i)\n",
    "\n",
    "        # fetch relevant content for this popular article\n",
    "        recs_items = make_content_recs(i)[0]\n",
    "\n",
    "        # random select some from similar\n",
    "        recs_items = random.sample(recs_items, threshold_second_layer)\n",
    "\n",
    "        # append to recs_for_similar list\n",
    "        recs_for_similar.append(recs_items)\n",
    "\n",
    "    # Flatten the recs_for_similar (All 2nd layer recs), and shuffle it\n",
    "    recs_for_similar = list(np.array(recs_for_similar).flatten())\n",
    "    random.shuffle(recs_for_similar)\n",
    "\n",
    "    # Make sure recs_for_similar has no duplicated content\n",
    "    for r in recs_for_similar:\n",
    "        if (r not in recs_for_similar_no_duplicate) and (r not in popular_ids):\n",
    "            recs_for_similar_no_duplicate.append(r)\n",
    "\n",
    "    # print(popular_ids)\n",
    "    # print(recs_for_similar_no_duplicate)\n",
    "\n",
    "    # Combine popular_ids (ranked-based / knowledge based) + recs_for_similar (content-based)\n",
    "    # Order Matters: ranked based come first, follow by content-based. (since it is cold start)\n",
    "    all_recs_ids = popular_ids + recs_for_similar_no_duplicate\n",
    "    all_recs_titles = _get_article_titles(all_recs_ids)\n",
    "\n",
    "    return all_recs_ids, all_recs_titles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "💡 Welcome, @[new user]! Here are some content you might be interested: \n",
      "\n",
      "1 Use Deep Learning For Image Classification\n",
      "2 Insights From New York Car Accident Reports\n",
      "3 Visualize Car Data With Brunel\n",
      "4 Use Xgboost, Scikit-Learn & Ibm Watson Machine Learning Apis\n",
      "5 Predicting Churn With The Spss Random Tree Algorithm\n",
      "6 Healthcare Python Streaming Application Demo\n",
      "7 Finding Optimal Locations Of New Store Using Decision Optimization\n",
      "8 Apache Spark Lab, Part 1: Basic Concepts\n",
      "9 Analyze Energy Consumption In Buildings\n",
      "10 Gosales Transactions For Logistic Regression Model\n",
      "11 Store Tweets Using Bluemix, Node-Red, Cloudant, And Dashdb\n",
      "12 Compose'S 2016 — All About The Database\n",
      "13 I'D Rather Predict Basketball Games Than Elections: Elastic Nba Rankings\n",
      "14 May 2016: Scripts Of The Week\n",
      "15 Continuous Learning On Watson\n",
      "16 Ibm Watson Machine Learning: Get Started\n",
      "17 Modeling Energy Usage In New York City\n",
      "18 Create A Business Intelligence And Analytics Service In Ruby With The Dashdb            Service\n",
      "19 Data Tidying In Data Science Experience\n",
      "20 Reading & Writing\n",
      "21 Biginsights On Cloud For Data Scientists\n",
      "22 Watson Machine Learning For Developers\n",
      "23 Add Data Assets To A Catalog Using Ibm Data Catalog\n",
      "24 Create A Project For Watson Machine Learning In Dsx\n",
      "25 Sharing Non-Public Data In Jupyter Notebooks – Ibm Watson Data Lab – Medium\n",
      "26 New Shiny Cheat Sheet And Video Tutorial\n",
      "27 Apache Spark As The New Engine Of Genomics\n",
      "28 A Guide To Convolution Arithmetic For Deep Learning\n",
      "29 Find The User In Data Science \n",
      "30 Apache Spark Lab, Part 2: Querying Data\n",
      "31 Getting Started With Apache Mahout\n",
      "32 Dt: An R Interface To The Datatables Library\n",
      "33 Awesome Deep Learning Papers\n",
      "34 Apple, Ibm Add Machine Learning To Partnership With Watson-Core Ml Coupling\n",
      "35 Using Notebooks With Pixiedust For Fast, Flexible, And Easier Data Analysis And Experimentation\n",
      "36 Jupyter Notebooks With Scala, Python, Or R Kernels\n",
      "37 Leverage Scikit-Learn Models With Core Ml\n",
      "38 Load Data From The Cloud Into Dashdb\n",
      "39 Load Dashdb Data With Apache Spark\n",
      "40 Analyze Open Data Sets With Pandas Dataframes\n",
      "41 What’S New In Data Refinery?\n",
      "42 The Machine Learning Database\n",
      "43 Analyze Open Data Sets Using Pandas In A Python Notebook\n",
      "44 Higher-Order Logistic Regression For Large Datasets\n",
      "45 Use The Machine Learning Library In Ibm Analytics For Apache Spark\n",
      "46 Practical Tutorial On Random Forest And Parameter Tuning In R\n",
      "47 Machine Learning For Everyone\n",
      "48 Move A Toy Car With Your Mind\n",
      "49 Data Model With Streaming Analytics And Python\n",
      "50 Load Geospatial Data Into Dashdb To Analyze In Esri Arcgis\n",
      "51 A Survey Of Books About Apache Spark™\n",
      "52 Government Consumption Expenditure\n",
      "53 Load Cloudant Data In Apache Spark Using A Python Notebook\n",
      "54 Build A Naive-Bayes Model With Wml & Dsx\n",
      "55 What Is Spark?\n",
      "56 Data Visualization Playbook: Revisiting The Basics\n",
      "57 Seven Databases In Seven Days – Day 5: Etcd\n"
     ]
    }
   ],
   "source": [
    "# make recommendations for a brand new user\n",
    "\n",
    "recs_for_new_user = make_recs_for_brand_new_user()[1]\n",
    "\n",
    "print(f\"💡 Welcome, @[new user]! Here are some content you might be interested: \\n\")\n",
    "\n",
    "for k, v in enumerate(recs_for_new_user):\n",
    "    print(k+1, v)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "💡 \n",
      "Since you checked article 1427:\n",
      " (Use Xgboost, Scikit-Learn & Ibm Watson Machine Learning Apis), \n",
      "you might also be interested in these:\n",
      "\n",
      "1 Declarative Machine Learning\n",
      "2 What Is Machine Learning?\n",
      "3 The Machine Learning Database\n",
      "4 Style Transfer Experiments With Watson Machine Learning\n",
      "5 Create A Project For Watson Machine Learning In Dsx\n",
      "6 From Machine Learning To Learning Machine (Dinesh Nirmal)\n",
      "7 Top 10 Machine Learning Use Cases: Part 1\n",
      "8 Watson Machine Learning For Developers\n",
      "9 Use R Dataframes & Ibm Watson Natural Language Understanding\n",
      "10 Machine Learning For The Enterprise\n",
      "11 Continuous Learning On Watson\n",
      "12 Use The Machine Learning Library In Spark\n",
      "13 Use The Machine Learning Library\n",
      "14 Use The Machine Learning Library In Ibm Analytics For Apache Spark\n",
      "15 Machine Learning For Everyone\n",
      "16 Use Apache Systemml And Spark For Machine Learning\n",
      "17 Machine Learning For The Enterprise.\n",
      "18 Apple, Ibm Add Machine Learning To Partnership With Watson-Core Ml Coupling\n",
      "19 Python Machine Learning: Scikit-Learn Tutorial\n",
      "20 Ibm Watson Machine Learning: Get Started\n",
      "21 Improving Quality Of Life With Spark-Empowered Machine Learning\n",
      "22 Building Custom Machine Learning Algorithms With Apache Systemml\n",
      "23 Building Your First Machine Learning System \n",
      "24 Make Machine Learning A Reality For Your Enterprise\n",
      "25 Top 10 Machine Learning Algorithms For Beginners\n",
      "26 Put A Human Face On Machine Learning With Wml & Dsx\n",
      "27 Build A Naive-Bayes Model With Wml & Dsx\n",
      "28 Apache Spark Analytics\n",
      "29 Load Dashdb Data With Apache Spark\n",
      "30 Access Ibm Analytics For Apache Spark From Rstudio\n",
      "31 Score A Predictive Model Built With Ibm Spss Modeler, Wml & Dsx\n",
      "32 The Power Of Machine Learning In Spark\n",
      "33 10 Essential Algorithms For Machine Learning Engineers\n",
      "34 Build A Predictive Analytic Model\n",
      "35 10 Powerful Features On Watson Data Platform, No Coding Necessary\n",
      "36 Collaborate On Projects In Dsx\n",
      "37 Machine Learning Exercises In Python, Part 1\n",
      "38 Build Spark Sql Queries\n",
      "39 Ml Algorithm != Learning Machine\n",
      "40 Create A Project In Dsx\n",
      "41 Essentials Of Machine Learning Algorithms (With Python And R Codes)\n"
     ]
    }
   ],
   "source": [
    "# make a recommendations for a user who only has interacted with article id '1427.0'\n",
    "\n",
    "print(\n",
    "    f\"💡 \\nSince you checked article 1427:\\n \"\n",
    "    f\"({_get_article_titles([1427])[0]}), \\n\"\n",
    "    f\"you might also be interested in these:\\n\")\n",
    "\n",
    "recs_for_user_1427th_titles = make_content_recs(1427)[1]\n",
    "\n",
    "for k, v in enumerate(recs_for_user_1427th_titles):\n",
    "    print(k + 1, v)\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a class=\"anchor\" id=\"Matrix-Fact\">Part V: Matrix Factorization</a>\n",
    "\n",
    "In this part of the notebook, you will build use matrix factorization to make article recommendations to the users on the IBM Watson Studio platform.\n",
    "\n",
    "`1.` You should have already created a **user_item** matrix above in **question 1** of **Part III** above.  This first question here will just require that you run the cells to get things set up for the rest of **Part V** of the notebook. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>article_id</th>\n",
       "      <th>0.0</th>\n",
       "      <th>2.0</th>\n",
       "      <th>4.0</th>\n",
       "      <th>8.0</th>\n",
       "      <th>9.0</th>\n",
       "      <th>12.0</th>\n",
       "      <th>14.0</th>\n",
       "      <th>15.0</th>\n",
       "      <th>16.0</th>\n",
       "      <th>18.0</th>\n",
       "      <th>...</th>\n",
       "      <th>1434.0</th>\n",
       "      <th>1435.0</th>\n",
       "      <th>1436.0</th>\n",
       "      <th>1437.0</th>\n",
       "      <th>1439.0</th>\n",
       "      <th>1440.0</th>\n",
       "      <th>1441.0</th>\n",
       "      <th>1442.0</th>\n",
       "      <th>1443.0</th>\n",
       "      <th>1444.0</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>user_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 714 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "article_id  0.0     2.0     4.0     8.0     9.0     12.0    14.0    15.0    \\\n",
       "user_id                                                                      \n",
       "1                0       0       0       0       0       0       0       0   \n",
       "2                0       0       0       0       0       0       0       0   \n",
       "3                0       0       0       0       0       1       0       0   \n",
       "4                0       0       0       0       0       0       0       0   \n",
       "5                0       0       0       0       0       0       0       0   \n",
       "\n",
       "article_id  16.0    18.0    ...  1434.0  1435.0  1436.0  1437.0  1439.0  \\\n",
       "user_id                     ...                                           \n",
       "1                0       0  ...       0       0       1       0       1   \n",
       "2                0       0  ...       0       0       0       0       0   \n",
       "3                0       0  ...       0       0       1       0       0   \n",
       "4                0       0  ...       0       0       0       0       0   \n",
       "5                0       0  ...       0       0       0       0       0   \n",
       "\n",
       "article_id  1440.0  1441.0  1442.0  1443.0  1444.0  \n",
       "user_id                                             \n",
       "1                0       0       0       0       0  \n",
       "2                0       0       0       0       0  \n",
       "3                0       0       0       0       0  \n",
       "4                0       0       0       0       0  \n",
       "5                0       0       0       0       0  \n",
       "\n",
       "[5 rows x 714 columns]"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load the matrix here\n",
    "# The matrix is generated in Part 3 above, use it directly. \n",
    "\n",
    "# quick look at the matrix\n",
    "user_item.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`2.` In this situation, you can use Singular Value Decomposition from [numpy](https://docs.scipy.org/doc/numpy-1.14.0/reference/generated/numpy.linalg.svd.html) on the user-item matrix.  Use the cell to perform SVD, and explain why this is different than in the lesson."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((5149, 5149), (714,), (714, 714))"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Perform SVD on the User-Item Matrix Here\n",
    "\n",
    "u, s, vt = np.linalg.svd(user_item) # use the built in to get the three matrices\n",
    "\n",
    "u.shape, s.shape, vt.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Provide your response here.**\n",
    "\n",
    "#### A: \n",
    "\n",
    "As the values of user_item matrix are binary. 0, 1. Default value is 0 indicates no interaction. \n",
    "\n",
    "And no missing values. So SVD is suitable. If there are missing values, we can approach with FunkSVD."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`3.` Now for the tricky part, how do we choose the number of latent features to use?  Running the below cell, you can see that as the number of latent features increases, we obtain a lower error rate on making predictions for the 1 and 0 values in the user-item matrix.  Run the cell below to get an idea of how the accuracy improves as we increase the number of latent features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAAw1ElEQVR4nO3deXxcZd3//9c7W9MtTfe9pC2l0AJdKGUVqghCgVYFlIIC/pDKreByqwi33tyI+nNXVEAEVJB9x4KsYgFFhLZ0X0kXmnRNl3TP/vn+ca6UaUiaSclkZjKf5+Mxj5xtzvmcM5PzOee65lyXzAznnHOZKyvZATjnnEsuTwTOOZfhPBE451yG80TgnHMZzhOBc85lOE8EzjmX4TwROHcQktZI+niStt1X0uuSdkn6ZTJicJnBE0GSSHpV0nZJHZIdS7qQVCTJJD3XYPr9km5KUliJNB3YAhSY2TcbzpR0j6QftnSlkm6SdH9rBBjW96qkLx5kfv3ntjvmNf9DbrN+nTkfZj0u4okgCSQVAR8BDJjSxttuD/84J0g6OdlBtMQhHvfDgCXWfp76LDSzLuE1JpmBKOLnv8APRHJcBvwHuAe4PHaGpMGSnpRUJmmrpFtj5l0laWkoKlgiaXyYbpIOj1lu/5WipEmSSiV9R9JG4M+Sukt6NmxjexgeFPP+HpL+LGl9mP90mL5I0vkxy+VK2iJpXMMdDHGeFzOeE7Y3XlJ+uIrfKqlc0ixJfVtw/H4G/KixGZKukPSvBtP2H59wbG6X9Hy4Mn1DUj9Jt4R9XdbI/hwfjvf2cFzyY9Z9nqR5YT/+LenYmHlrwnFfAOxpLBlIOjns/47w9+T6OIm+G9eFOFtUPCXpN5JKJO2UNEfSR8L0s4H/AT4be2UuqZukP0raIGmdpB9Kyo49ppJ+EY7BaknnhHk/IrqouTWs79bGI2oyziMlvSxpm6Tlkj4TM+9cSXPDPpTowLu+18Pf8rDdk9TgTkcN7hoU3bn8SNIbwF5gWDPbnxw+913hmHyrJfuWVszMX238AoqBLwPHAdVA3zA9G5gP/BroDOQDp4Z5FwHrgOMBAYcDh4V5Bhwes/57gB+G4UlADfBToAPQEegJXAB0AroCjwFPx7z/b8AjQHcgFzg9TL8OeCRmuanAwib28UbggZjxc4GlYfhLwDNh+9nhOBTEcdyKwr52Dcfi42H6/cBNYfgK4F8N3rf/+IRjsyVsMx/4B7CaKDlnAz8EZsa8dw2wCBgM9ADeiDm244DNwAnhvZeH5TvEvHdeeG/HRvanB7Ad+DyQA0wL4z0bfo5NHI8m5wOfC59zDvBNYCOQH+bdBNzfYPmngD8Qfe/6AG8DX4o5ptXAVWE//wtYDyjMfxX4YhyfW06D6Z2BEuALIc5x4bMZFfPdPYbogvVYYBPwyabW2XC/Gi4T4lwLjA7b69bM9jcAHwnD3YHxyT53JOrldwRtTNKpRLf8j5rZHGAlcEmYPREYAHzbzPaYWYWZ1V/dfhH4mZnNskixmb0X52brgP8zs0oz22dmW83sCTPba2a7iK6uTw/x9QfOAa42s+1mVm1mr4X13A9MllQQxj8P3NfENh8EpkjqFMYvAR4Kw9VEJ6nDzazWzOaY2c449wVgX4i5xeXjwVNhmxVEJ8AKM/uLmdUSJcCGdwS3mlmJmW0L250Wpk8H/mBmb4X9uBeoBE6Mee9vw3v3NRLHucC7ZnafmdWY2UPAMuD8RpZtETO7P3zONWb2S6KLgJGNLRvuxiYDXw/fu81EFyMXxyz2npndFY7RvUB/oCV3cQBbwp1Tebi6Pg9YY2Z/DnHOBZ4guujBzF41s4VmVmdmC4i+P6e3cJsN3WNmi82sBjj7YNsn+p6OklQQ/hfe+ZDbTlmeCNre5cBLZrYljD/I+8VDg4n+4Woaed9goqRxKMrCSQ8ASZ0k/UHSe5J2Et1mF4aigMHANjPb3nAlZrae6Ir4AkmFRAnjgcY2aGbFwFLg/JAMphDtK0TJ40Xg4VD89DNJuS3cp7uBvrFFVS2wKWZ4XyPjXRosXxIz/B5RsoYooX8z5uRWTnT8BjTx3oYGhPXFeg8YeNDo4yDpW6F4bkeIqxvQq4nFDyO689sQsx9/ILozqLexfsDM9obBhsepOb3MrDC8fhG2e0KD43cp0C/swwmSZioqUtwBXH2QfYhX7Odx0O0T3TVPBt6T9Jqkkz7ktlNWe6g4TBuSOgKfAbIVlddDdKVWKGkM0Zd0iKScRpJBCTC8iVXvJSpmqdcPKI0Zb1jZ+E2iq8MTzGyjpLHAXKIipxKgh6RCMytvZFv3Et2d5ABvmtm6pvaX6ApuGtEFx5KQHDCzauD7wPcVVZw/BywH/niQdR3AzKokfR/4AbA4ZtYeYo6FpH4N33sIBscMDyEqFoHoWP3IzBqtr6gP9SDz1hOdjGINAV5ocYQxQn3AdcAZwGIzq5O0nejzbSymEqI7mV5NXIQ051Ars0uA18zszCbmPwjcCpxjZhWSbuH9RNDYNg/47Hn/hN5UrAfdvpnNAqaGi5RrgEc58LvQbvgdQdv6JFALjALGhtdRwD+JyqjfJiqX/ImkzooqVU8J770b+Jak4xQ5XFL9SWQecImk7FAZ2Nztc1eiK99yST2A/6ufYWYbgOeB2xVVKudKOi3mvU8D44GvAX9pZjsPA2cRlSnX3w0g6aOSjgl3IDuJbsHrmllXY+4jKuc/O2bafGC0pLGKKnVvOoT1NvQVSYPCsfouUfERwF3A1eHKVeEzO1dS1zjX+xxwhKRLFFWmf5bou/FsC2LLDt+T+lce0edbA5QBOZJuBApi3rMJKFL41Uz4zF8CfimpQFKWpOGS4i2G2QQMa0HM9Z4l2v/Ph+9ZrqTjJR0V5nclujutkDSR94tQCftW12C784DTJA2R1A244VC3LylP0qWSuoULl50c2nc0LXgiaFuXA382s7VmtrH+RXTVcynRFdv5RBXBa4mu6j8LYGaPEZVPPwjsIjoh9wjr/Vp4X3lYz9PNxHELUaXxFqJfLzW8Av080cl5GVFl6NfrZ4Sy7ieAocCTB9tIOMG8CZzM+ydPiK7UHif651oKvEaoa5B0h6Q7mom/fv21RJXSPWKmrQBuBv4OvAv8q/F3t8iDRCfKVUTFcz8M25pNVIF6K1ElbzFRxWpczGwrUTn5N4GtRFfx58UUG8bjeqKkXv/6B1Gx2wvACqKipgoOLBJ5LPzdKqm+3PsyIA9YEvblcaJ6gHj8BrhQ0S+Kfhtv4KF+6iyiuoj1RMVP9T9qgOgHFTdL2kX0OT8a8969RP8Pb4RinRPN7GWi79kCYA7NJNQ4tv95YE0oPr2a6H+rXaqv9XcubuEK8wgz+1yyY3HOfXheR+BaJBSPXEl0teScawe8aMjFTdJVREUMz5vZ680t75xLD1405JxzGc7vCJxzLsOlXR1Br169rKioKNlhOOdcWpkzZ84WM+vd2Ly0SwRFRUXMnj072WE451xakdRkkzReNOSccxnOE4FzzmU4TwTOOZfhPBE451yG80TgnHMZLmGJQNKfJG2WtKiJ+ZL0W0nFkhYodLvonHOubSXyjuAeDmweuKFzgBHhNR34fQJjcc4514SEPUdgZq+HTkeaMhX4i0VtXPxHUqGk/qHpYueci1tNbR3VtUZVbR3VtXXU1Bo1dfV/jdq6aLy2zqiuPXC8ps6obWS5mrr6Pn2j3myiv1GTPPXTsGhKbZ1RZ1Ef8PXDdWbU1U8nWk+9/YMtbOLnjKP6MmZw4Yc+Xg0l84GygRzYRnppmPaBRCBpOtFdA0OGDGmT4Jxzh6a6to7dFTXsrox5VdSwt6qW6to6qmrqqAx/68frhytr6thXVUtFTS0V1bXsq66joro25lVHZU0t1bVGdU3d/hN/XRo3mSY1v0y9PgX57S4RxM3M7gTuBJgwYUIaf+TOpb66OmNXRQ3b9laxbU8l5Xur2VlRzc59NezcFzNc8f7wropqdlfWsKuihsqalnfkJUFedhYdcrLomJdNfm42+TnZ5Odlk5+TRY/OedF4bhYdcrLJy8kiNzuL3ByRlx2Gs7PIzdb+edlZIidLZGfpA+M5WVnkZB84np0lcrK1f7ksRcP18QmFv/VBR9OyBFkSWVnvD2dnRctmSeFVv58tOOu3oWQmgnUc2P/noDDNOdeK9lXVsnVPJdv3VLNtbxXb91SxbU8V2/dGf7c1GN++t5rag1xid8zNpqBjDgX5uRR0zKVXlzyKenWma34OXTvk0LlDDl065NAljHfJj8Y75mWTl51FXk706pCdvf9EnpPtP2BMpmQmghnANZIeBk4Adnj9gHPxq6mto2x3JevLK9iwYx8bd1TsH96wo4LNOyvYtreKiurGr9CzBN075dGjcx7dO+cxrFcXJhTl0aNTNN6jcy7dO+XRvVMe3TrmRif6/Fzycvyk3d4kLBFIegiYBPSSVErUQXougJndQdRx92Sifl73Al9IVCzOpRMzY+e+GjbtqmDTzgo27awMfw8c37yr8gNX7h1zs+lfmM+Abh0ZNrwnvbp0CCf76KTes0ve/pN/QX4uWVmpWVTh2lYifzU0rZn5BnwlUdt3LtXV1Nbx3ra9vLtpN8Wbd/Hu5t2s2LSb1Vt2N3oVX5CfQ9+CfPp1y2dY754M6NZx/0m/X7fob0HHnJQth3apKy0qi51LZ7V1xntb97Bi0y6Wb9zNu5t3Ubx5N6vK9lBV+/4Jf2BhR0b07cJJw3oyoDCfPgX59CvIp29BB/p0zadjXnYS98K1Z54InGslZsa68n37T/jR310Ul+2mKvySRoLB3TtxRN8uTBrZhxF9ujCibxeG9+5C5w7+7+iSw795zh2iLbsrmbe2nLkl25m7tpwFpTvYXVmzf37/bvkc0bcrp47oxRF9uzKyb1cO79PFr+xdyvFE4FwcqmrqWLJhJ3PXRif9uSXbKdm2D4CcLHFU/wI+OW4AR/UvYGTfrozo25VuHXOTHLVz8fFE4FwD9UU8c9eWM6+knLlrt7No/c79xTv9u+Uzbkghl51YxNghhRw9oJtf5bu05onAZby9VTUsKN0RXemv3c7cknLKdlUC0CEni2MHdePykw5j/JDujB1SSP9uHZMcsXOtyxOByzhmxrKNu3h1eRmvLt/MnPe2UxN+j1/UsxOnHt6LcUMKGTe4O0f270quP/Xq2jlPBC4j7Kqo5o3iLeHkX8bGnRUAHNmvK1d+ZCgnDO3B2MHd6dE5L8mROtf2PBG4dmtl2W7+vmQTM5dvZvaa6Kq/a4ccTh3Ri0kje3P6EX3o1y0/2WE6l3SeCFy7UVdnzCst56XFm3h5yUZWlu0Boqv+L35kGJNG9ua4w7p7UY9zDXgicGmtsqaWf6/cykuLN/H3pZso21VJTpY4cVhPLjupiI+P6svAQq/cde5gPBG4tGNm/HvlVh56ey0zl21mT1UtnfOymTSyD2eN7sukkX38N/zOtYAnApc2duyt5vF3SnngP++xasseunfKZcrYgZw1ui8nD+9Jhxz/Lb9zh8ITgUt5C0rLue/N93hmwXoqqusYP6SQX392DOcc3Z/8XD/5O/dheSJwKWlfVS3PzF/P/W+9x4LSHXTKy+ZT4wbxuROHMHpAt2SH51y74onApZSNOyq49801PPjWWnbsq2ZEny7cPHU0nxw3kIJ8L/d3LhE8EbiUsGjdDv74r9U8M389dWZ8YnQ/Lj+5iBOG9vCOVpxLME8ELmnq6oxXlm3m7n+u4q3V2+icl81lJxXxhVOKGNyjU7LDcy5jeCJwbW5vVQ1PzCnlT2+sYfWWPQzols93Jx/FZycO9uIf55LAE4FrM7V1xiOzSvjFS8vZtqeKMYML+d20cZxzdD9y/Glf55LGE4FrE2+t2sr3n1nCkg07mTi0B9d9YiTHHdbdy/+dSwEJTQSSzgZ+A2QDd5vZTxrMPwz4E9Ab2AZ8zsxKExmTa1ul2/fy4+eX8bcFGxjQLZ9bLxnHucf09wTgXApJWCKQlA3cBpwJlAKzJM0wsyUxi/0C+IuZ3SvpY8CPgc8nKibXdvZV1fL711byh9dWIsE3Pn4E008b5j15OZeCEnlHMBEoNrNVAJIeBqYCsYlgFPDfYXgm8HQC43FtwMx4ZsEGfvLcUtbvqOC8Y/tzw+SjvOE351JYIhPBQKAkZrwUOKHBMvOBTxMVH30K6Cqpp5ltjV1I0nRgOsCQIUMSFrD7cFaV7eb6Jxfy9uptjOpfwC0Xj2Pi0B7JDss514xkVxZ/C7hV0hXA68A6oLbhQmZ2J3AnwIQJE6wtA3TNq6mt4+5/reZXL68gPyeL//9Tx/DZ4weTneX1AM6lg0QmgnXA4JjxQWHafma2nuiOAEldgAvMrDyBMblWtmzjTq57fAELSnfwidF9+cHUo+lT4L1+OZdOEpkIZgEjJA0lSgAXA5fELiCpF7DNzOqAG4h+QeTSQFVNHbfNLOb2V4spyM/ltkvGM/mYfv5rIOfSUMISgZnVSLoGeJHo56N/MrPFkm4GZpvZDGAS8GNJRlQ09JVExeNaz4LScq57fAHLNu7ik2MHcOP5o73Td+fSmMzSq8h9woQJNnv27GSHkZEqqmv59d9XcNfrq+jTNZ8ffepozjiqb7LDcs7FQdIcM5vQ2LxkVxa7NDG/pJxvPDqPVWV7mDZxMDdMPsrbBXKunfBE4A6qujaqC/jdP4rp27UDD3zxBE45vFeyw3LOtSJPBK5Jq8p2841H5zO/pJxPjxvI/00Z7Z3CO9cOeSJwH2Bm3P/WWn70tyXk52Zz2yXjOffY/skOyzmXIJ4I3AE27azguscX8NqKMk47ojc/v/BY+vpzAc61a54I3H7PLdzA/zy1kIrqWn4wdTSfO/Ewfy7AuQzgicBRVVPH955eyKOzSxkzqBu/+uxYhvfukuywnHNtxBNBhttVUc3V98/hjeKtXPuxw/nqGSPI9d7CnMsonggy2KadFVzx51m8u2kXv7hoDBceNyjZITnnksATQYYq3ryLy/80i+17q/jjFcdz+hG9kx2Scy5JPBFkoNlrtnHlvbPJzc7ikeknccygbskOyTmXRJ4IMswLizbytYfnMqCwI/d+YSJDenZKdkjOuSTzRJBB7ntzDTfOWMyYQYX86YrjvcVQ5xzgiSAjmBk/f3E5t7+6ko8f1YffTRvvncg75/bzRNDOmRn/89RCHnq7hGkTB/ODqUeT4z8Pdc7F8ETQzt39z9U89HYJV58+nO+cPdKfFHbOfYBfGrZjM5dv5sfPL2XyMf247hOeBJxzjfNE0E6tLNvNVx+ay8h+BfziojFkZXkScM41zhNBO7RjXzVX3TubvOws7rrsODrleQmgc65pfoZoZ2rrjGsfmsvabXt58KoTGdTdnxNwzh2cJ4J25ifPL+X1FWX8+NPHMHFoj2SH45xLAwktGpJ0tqTlkoolXd/I/CGSZkqaK2mBpMmJjKe9e2JOKXf9czWXn3QY0yYOSXY4zrk0kbBEICkbuA04BxgFTJM0qsFi3wMeNbNxwMXA7YmKp717Z+12bnhyIScP78n3zmt4mJ1zrmmJvCOYCBSb2SozqwIeBqY2WMaAgjDcDVifwHjarY07KvjSfXPo1y2f2y4Z7/0JOOdaJJFnjIFAScx4aZgW6ybgc5JKgeeAaxtbkaTpkmZLml1WVpaIWNNWRXUtX7pvNnsra7jrsgl09/aDnHMtlOxLx2nAPWY2CJgM3CfpAzGZ2Z1mNsHMJvTu7e3mx/ruU4tYsG4Ht1w8jpH9uiY7HOdcGkpkIlgHDI4ZHxSmxboSeBTAzN4E8oFeCYypXXlqbilPvFPKVz82gjNH9U12OM65NJXIRDALGCFpqKQ8osrgGQ2WWQucASDpKKJE4GU/cXhv6x6+99QiJhb14KtnjEh2OM65NJawRGBmNcA1wIvAUqJfBy2WdLOkKWGxbwJXSZoPPARcYWaWqJjai+raOr768Dyys8SvLx5Ltjcf4Zz7EBL6QJmZPUdUCRw77caY4SXAKYmMoT361csrmF9Szu8vHc/Awo7JDsc5l+aSXVnsWujfxVu447WVTJs4mHOO6Z/scJxz7YAngjSybU8VX39kHsN6deZ//aEx51wr8baG0oSZcd3j8ynfW809X5joLYo651qN3xGkib+8+R5/X7qZ6885klEDCpp/g3POxckTQRpYumEnP3puKR8d2ZsvnFKU7HCcc+2MJ4IUt6+qlq8+NJeC/Fx+ftEY727SOdfqvKA5xf3wb0t4d/Nu/vL/TaRXlw7JDsc51w75HUEKe3HxRh54ay3TTxvGaUd4G0vOucTwRJCitu+p4n+eXMjoAQV866yRyQ7HOdeOedFQivrBs0vYsa+a+794Ank5nq+dc4njZ5gUNHP5Zp6cu44vTxrOUf39p6LOucTyRJBidlfW8N0nF3J4ny585WOHJzsc51wG8KKhFPPzF5axYWcFj199Eh1yspMdjnMuAzR7RyDp/MZ6DXOtb9aabfzlP+9x+UlFHHdYj2SH45zLEPGc4D8LvCvpZ5KOTHRAmaqiupbvPLGAAd068u1P+K+EnHNtp9lEYGafA8YBK4F7JL0ZOpP3DnJb0e/+8S6ryvbw408fQ+cOXmLnnGs7cRX5mNlO4HHgYaA/8CngHUnXJjC2jLF4/Q7+8NoqLhg/yB8cc861uXjqCKZIegp4FcgFJprZOcAYoq4m3YdQU1vHd55YQGGnXP73vKOSHY5zLgPFUwZxAfBrM3s9dqKZ7ZV0ZWLCyhx3/2s1i9bt5PZLx1PYKS/Z4TjnMlA8ieAmYEP9iKSOQF8zW2NmryQqsEywessefv3yCj4xui/nHN0v2eE45zJUPHUEjwF1MeO1YVqzJJ0tabmkYknXNzL/15LmhdcKSeVxRd0O1NUZ1z+xgLycLG6eerQ3L+2cS5p47ghyzKyqfsTMqiQ1W4YhKRu4DTgTKAVmSZphZkti1vWNmOWvJfp1UkZ4eFYJb63exk8vOIa+BfnJDsc5l8HiuSMokzSlfkTSVGBLHO+bCBSb2aqQSB4Gph5k+WnAQ3GsN+1V1tTym1dWcHxRdz4zYXCyw3HOZbh47giuBh6QdCsgoAS4LI73DQzL1isFTmhsQUmHAUOBfzQxfzowHWDIkCFxbDq1PTFnHZt2VvLLi8Z6kZBzLumaTQRmthI4UVKXML47AXFcDDxuZrVNxHAncCfAhAkTLAHbbzM1tXXc8dpKxgzqximH90x2OM45F1+jc5LOBUYD+fVXsGZ2czNvWwfElnsMCtMaczHwlXhiSXfPLtjA2m17+d65x/ndgHMuJcTzQNkdRO0NXUtUNHQRcFgc654FjJA0NFQuXwzMaGT9RwLdgTdbEHdaqqszbn+1mCP6duHjR/VNdjjOOQfEV1l8spldBmw3s+8DJwFHNPcmM6sBrgFeBJYCj5rZYkk3x1Y+EyWIh80srYt84vHy0k2s2LSbL086nKwsvxtwzqWGeIqGKsLfvZIGAFuJ2htqlpk9BzzXYNqNDcZvimdd6c7MuH1mMUN6dOK8Y+M6fM451ybiuSN4RlIh8HPgHWAN8GACY2qX3ijeyvzSHVx9+nBysr17B+dc6jjoHUHokOYVMysHnpD0LJBvZjvaIrj25NaZ79K3oAMXHDcw2aE459wBDnppamZ1RE8H149XehJouTnvbeM/q7Zx1UeGefeTzrmUE08ZxSuSLpD/1vGQ3T5zJd075TJtYvo/DOeca3/iSQRfImpkrlLSTkm7JO1McFztxpL1O3ll2Wa+cMpQ73nMOZeS4nmy2Luk/BBuf7WYLh1yuPykomSH4pxzjWo2EUg6rbHpDTuqcR+0qmw3f1u4gS+dNpxunXKTHY5zzjUqnrKKb8cM5xO1KjoH+FhCImpH7nhtJXnZWVx56tBkh+Kcc02Kp2jo/NhxSYOBWxIVUHuxrnwfT76zjktPGELvrh2SHY5zzjXpUJ5sKgW8l/Vm3PX6KgCmnz48yZE459zBxVNH8Dugvh2gLGAs0RPGrglbdlfy0Ntr+dS4gQws7JjscJxz7qDiqSOYHTNcAzxkZm8kKJ524Z431lBVW8fVk/xuwDmX+uJJBI8DFfWdxkjKltTJzPYmNrT0VFNbx6OzS/joyD4M790l2eE451yz4nqyGIgt3+gI/D0x4aS/f767hc27KvnMhEHJDsU55+ISTyLIj+2eMgx3SlxI6e2xOSX06JzHx470jmecc+khnkSwR9L4+hFJxwH7EhdS+tq2p4qXl2xi6tgB5OV4U9POufQQTx3B14HHJK0n6qqyH1HXla6Bv85bR3WtcdFxg5tf2DnnUkQ8D5TNCv0KjwyTlptZdWLDSk+PzS7l6IEFjBpQkOxQnHMubvF0Xv8VoLOZLTKzRUAXSV9OfGjpZfH6HSzZsNPvBpxzaSeeguyrQg9lAJjZduCqhEWUph6bXUpedhZTxw5IdijOOdci8SSC7NhOaSRlA3mJCyn9VNbU8td56zhzdF8KO/mhcc6ll3gSwQvAI5LOkHQG8BDwfDwrl3S2pOWSiiVd38Qyn5G0RNJiSQ/GH3rqeGXpZrbvreai4/zZAedc+onnV0PfAaYDV4fxBUS/HDqocOdwG3AmUUN1syTNMLMlMcuMAG4ATjGz7ZL6tDD+lPDY7BL6FeTzkRG9kx2Kc861WLN3BKED+7eANUR9EXwMWBrHuicCxWa2ysyqgIeBqQ2WuQq4LdQ7YGab4w89NWzaWcFrK8r49PiBZGd5t87OufTT5B2BpCOAaeG1BXgEwMw+Gue6BwIlMeOlwAkNljkibOsNIBu4ycxeaCSW6UR3JQwZklodwD/5zjrqDC70YiHnXJo62B3BMqKr//PM7FQz+x1Q28rbzwFGAJOIEs5dkgobLmRmd5rZBDOb0Lt36hS/mBmPzSnh+KLuDPMG5pxzaepgieDTwAZgpqS7QkVxS8o+1gGxP6ofFKbFKgVmmFm1ma0GVhAlhrTwztrtrCrb488OOOfSWpOJwMyeNrOLgSOBmURNTfSR9HtJZ8Wx7lnACElDJeUBFwMzGizzNNHdAJJ6ERUVrWrhPiTNY7NL6ZibzeRj+yc7FOecO2TxVBbvMbMHQ9/Fg4C5RL8kau59NcA1wItElcuPmtliSTdLmhIWexHYKmkJUbL5tpltPcR9aVN7q2p4dsEGJh/Tny4d4vnxlXPOpaYWncHCr3vuDK94ln8OeK7BtBtjhg347/BKKy8s2sjuyhou8n4HnHNpzttKPkSPzS7lsJ6dOGFoj2SH4pxzH4ongkOwdute3ly1lQvHDyKm9Q3nnEtLnggOwePvlCLBBf7sgHOuHfBE0EJ1dcYTc0o59fBeDCjs2PwbnHMuxXkiaKE3V21lXfk+Lprgzw4459oHTwQt9MQ7pXTNz+GsUd45vXOuffBE0AIV1bW8tHgT5xzdj/zc7GSH45xzrcITQQvMXLaZ3ZU1TBkzMNmhOOdcq/FE0AIz5q+nV5cOnDS8Z7JDcc65VuOJIE67Kqp5Zdlmzj2mn/c74JxrVzwRxOnlJZuoqqljindO75xrZzwRxGnG/PUMLOzI+CHdkx2Kc861Kk8Ecdi2p4p/vbuF88cM8CYlnHPtjieCODy3cAM1dcb5Y7zfAedc++OJIA7PzF/P8N6dGdW/INmhOOdcq/NE0IyNOyp4e802powZ6MVCzrl2yRNBM55dsB4z/NdCzrl2yxNBM2bMX88xA7sxtFfnZIfinHMJ4YngIFZv2cOC0h1eSeyca9c8ERzEs/PXA3DesV4s5JxrvzwRNMHMmDF/PROLengHNM65di2hiUDS2ZKWSyqWdH0j86+QVCZpXnh9MZHxtMSyjbt4d/NuzvdKYudcO5eTqBVLygZuA84ESoFZkmaY2ZIGiz5iZtckKo5DNWP+erKzxOSj+yU7FOecS6hE3hFMBIrNbJWZVQEPA1MTuL1WY2Y8M389pxzei55dOiQ7HOecS6hEJoKBQEnMeGmY1tAFkhZIelxSox0BS5ouabak2WVlZYmI9QBzS8op3b6PKWO8WMg51/4lu7L4GaDIzI4FXgbubWwhM7vTzCaY2YTevXsnPKgZ89aTl5PFWaO9X2LnXPuXyESwDoi9wh8Upu1nZlvNrDKM3g0cl8B44lJbZ/xt4QY+OrI3Bfm5yQ7HOecSLpGJYBYwQtJQSXnAxcCM2AUkxT6pNQVYmsB44vKfVVsp21Xp/RI75zJGwn41ZGY1kq4BXgSygT+Z2WJJNwOzzWwG8FVJU4AaYBtwRaLiidcz89fTOS+bM47qk+xQnHOuTSQsEQCY2XPAcw2m3RgzfANwQyJjaImqmjqeX7SRs0b3Iz83O9nhOOdcm0h2ZXFKeX1FGTv2VfuvhZxzGcUTQYxXV2ymS4ccTjm8V7JDcc65NuOJIMa8knKOHdSNvBw/LM65zOFnvKCiupZlG3YxdnBhskNxzrk25YkgWLRuBzV15onAOZdxPBEE80rKARg7pDCpcTjnXFvzRBDMKylnYGFH+nTNT3YozjnXpjwRBPNKyr1YyDmXkTwRAFt2V1K6fZ8nAudcRvJEAMxbWw54/YBzLjN5IiAqFsrOEkcP6JbsUJxzrs15IiBKBEf260rHPG9fyDmXeTI+EdTVGfO9otg5l8EyPhGs2rKbXZU1ngiccxkr4xPB3FBRPM4rip1zGSrjE8G8knK65ucwrFeXZIfinHNJ4YmgpJwxgwrJylKyQ3HOuaTI6ESwr6qWZRu9xVHnXGbL6ESwaP0Oar3FUedchsvoRFD/RPEYTwTOuQyW0EQg6WxJyyUVS7r+IMtdIMkkTUhkPA3Vtzjau2uHttysc86llIQlAknZwG3AOcAoYJqkUY0s1xX4GvBWomJpyryScm9fyDmX8RJ5RzARKDazVWZWBTwMTG1kuR8APwUqEhjLB2zeVcG68n2M82Ih51yGS2QiGAiUxIyXhmn7SRoPDDazvyUwjkbtb3HUE4FzLsMlrbJYUhbwK+CbcSw7XdJsSbPLyspaZfvzSsrJyRJHD/QWR51zmS2RiWAdMDhmfFCYVq8rcDTwqqQ1wInAjMYqjM3sTjObYGYTevfu3SrBzSsp58j+XcnP9RZHnXOZLZGJYBYwQtJQSXnAxcCM+plmtsPMeplZkZkVAf8BppjZ7ATGBEBtnbGgdIcXCznnHAlMBGZWA1wDvAgsBR41s8WSbpY0JVHbjcfKst3srqxh7ODuyQzDOedSQk4iV25mzwHPNZh2YxPLTkpkLLG8otg5596XkU8Wz93f4mjnZIfinHNJl5GJYF7okcxbHHXOuQxMBHurali+cacXCznnXJBxiWBh6Q7qzOsHnHOuXsYlgnkl5YAnAuecq5eRiWBwj4707OItjjrnHGRoIvDnB5xz7n0ZlQg27axgw44KLxZyzrkYGZUI5vqDZM459wEZlQjmlZSTmy1GDyhIdijOOZcyMiwRbOeo/gXe4qhzzsXImERQW2cs9BZHnXPuAzImERRv3s2eqlpPBM4510DGJIJ5JdsBryh2zrmGMiYRdO+Ux5mj+jLUWxx1zrkDJLQ/glRy1uh+nDW6X7LDcM65lJMxdwTOOeca54nAOecynCcC55zLcJ4InHMuw3kicM65DOeJwDnnMpwnAuecy3CeCJxzLsPJzJIdQ4tIKgPei3PxXsCWBIaTCOkWc7rFCx5zW0m3mNMtXmhZzIeZWe/GZqRdImgJSbPNbEKy42iJdIs53eIFj7mtpFvM6RYvtF7MXjTknHMZzhOBc85luPaeCO5MdgCHIN1iTrd4wWNuK+kWc7rFC60Uc7uuI3DOOde89n5H4JxzrhmeCJxzLsO1y0Qg6WxJyyUVS7o+2fHUk/QnSZslLYqZ1kPSy5LeDX+7h+mS9NuwDwskjU9SzIMlzZS0RNJiSV9L5bgl5Ut6W9L8EO/3w/Shkt4KcT0iKS9M7xDGi8P8oraMt0Hs2ZLmSno2HWKWtEbSQknzJM0O01LyexETc6GkxyUtk7RU0kmpHLOkkeH41r92Svp6q8dsZu3qBWQDK4FhQB4wHxiV7LhCbKcB44FFMdN+Blwfhq8HfhqGJwPPAwJOBN5KUsz9gfFhuCuwAhiVqnGH7XYJw7nAWyGOR4GLw/Q7gP8Kw18G7gjDFwOPJPH78d/Ag8CzYTylYwbWAL0aTEvJ70VMfPcCXwzDeUBhqsccE3s2sBE4rLVjTtpOJfBgnQS8GDN+A3BDsuOKiaeoQSJYDvQPw/2B5WH4D8C0xpZLcvx/Bc5Mh7iBTsA7wAlET1/mNPyOAC8CJ4XhnLCckhDrIOAV4GPAs+EfOdVjbiwRpOz3AugGrG54rFI55gZxngW8kYiY22PR0ECgJGa8NExLVX3NbEMY3gj0DcMptx+hCGIc0VV2ysYdiljmAZuBl4nuEMvNrKaRmPbHG+bvAHq2ZbzBLcB1QF0Y70nqx2zAS5LmSJoepqXs9wIYCpQBfw5FcHdL6kxqxxzrYuChMNyqMbfHRJC2LErhKfl7XkldgCeAr5vZzth5qRa3mdWa2Viiq+yJwJHJjejgJJ0HbDazOcmOpYVONbPxwDnAVySdFjsz1b4XRHdP44Hfm9k4YA9Rscp+KRgzAKF+aArwWMN5rRFze0wE64DBMeODwrRUtUlSf4Dwd3OYnjL7ISmXKAk8YGZPhskpH7eZlQMziYpVCiXlNBLT/njD/G7A1raNlFOAKZLWAA8TFQ/9htSOGTNbF/5uBp4iSrqp/L0oBUrN7K0w/jhRYkjlmOudA7xjZpvCeKvG3B4TwSxgRPjFRR7R7dSMJMd0MDOAy8Pw5URl8PXTLwu/AjgR2BFzK9hmJAn4I7DUzH4VMysl45bUW1JhGO5IVJ+xlCghXNhEvPX7cSHwj3CF1WbM7AYzG2RmRUTf13+Y2aWkcMySOkvqWj9MVH69iBT9XgCY2UagRNLIMOkMYEkqxxxjGu8XC0Frx5ysio8EV6pMJvp1y0rgu8mOJyauh4ANQDXR1cmVRGW7rwDvAn8HeoRlBdwW9mEhMCFJMZ9KdNu5AJgXXpNTNW7gWGBuiHcRcGOYPgx4Gygmur3uEKbnh/HiMH9Ykr8jk3j/V0MpG3OIbX54La7/P0vV70VM3GOB2eH78TTQPQ1i7kx0x9ctZlqrxuxNTDjnXIZrj0VDzjnnWsATgXPOZThPBM45l+E8ETjnXIbzROCccxnOE4FrFZJM0i9jxr8l6aZWWvc9ki5sfskPvZ2LQouUMxtML1JMi7FxrOeTkkZ9iDiKJF1ykHn7GrRImXcI27hC0oBDjdG1L54IXGupBD4tqVeyA4kV82RuPK4ErjKzj37IzX6SqIXWQ1UENJoIgpVmNjbmVXUI27gCaFEiaOGxdGnEE4FrLTVE/ad+o+GMhlf0knaHv5MkvSbpr5JWSfqJpEsV9SewUNLwmNV8XNJsSStC2zz1jcv9XNKs0Pb6l2LW+09JM4ieHG0Yz7Sw/kWSfhqm3Uj08NwfJf08nh2WdFXY9nxJT0jqJOlkojZhfh6u1oeH1wuKGmf7p6QjY47LbyX9O+x//TH6CfCR8P4PHM8mYjlL0puS3pH0mKK2oZB0Y4hxkaQ7wxOnFwITgAfCNjoq6lugV3jPBEmvhuGbJN0n6Q3gvvDk9hNhnbMknRKWOz3mDmVu/VPHLk0k40k5f7W/F7AbKCBqmrgb8C3gpjDvHuDC2GXD30lAOVEzuh2I2kT5fpj3NeCWmPe/QHThMoLoqex8YDrwvbBMB6InRoeG9e4BhjYS5wBgLdCbqBGyfwCfDPNepZEnMWnQdHjM9J4xwz8Erm1if18BRoThE4iahKhf7rGwX6OA4pjj8mwTx7kI2Mf7T3nfBvQCXgc6h2W+w/tPVPeIee99wPmN7SsxTUoTJYlXw/BNwBygYxh/kKixOYAhRE2PADwDnBKGuxCaz/ZXerz8Vs+1GjPbKekvwFeJTlbxmGWhLRRJK4GXwvSFQGwRzaNmVge8K2kVUYuiZwHHxlxJdyNKFFXA22a2upHtHU90kisL23yAqMOgp+OMN9bRkn5I1LlJF6J+Ag4QrsxPBh6TVD+5Q8wiT4f9WiKpb8P3N2GlRa2r1m/jPKJE8kbYRh7wZpj9UUnXEfXN0IOoOYhn4txOvRlmVv95fhwYFbMvBWEf3wB+FY7nk2ZW2sJtuCTyROBa2y1EncH8OWZaDaEYUlIW0YmqXmXMcF3MeB0Hfj8btoViRO2qXGtmB5yAJU0iuiNItHuI7ibmS7qC6Eq+oSyifgXGNrGO2P1XE8s0R8DLZjbtgIlSPnA70ZV/iaLK+/wm1rH/M2pkmdhjmQWcaGYVDZb5iaS/EbVD9YakT5jZspbviksGryNwrcrMthF1sXhlzOQ1wHFheApRF5ItdZGkrFBvMIyo56UXgf9S1Ew2ko5Q1BLmwbwNnC6pl6RsolYdXzuEeCDqunND2P6lMdN3hXlY1HfDakkXhRglaUwz693//jj9BzhF0uFhG50lHcH7J/Qt4ao99pdXDbexhvc/owsOsq2XgGvrRySNDX+Hm9lCM/spUQvAKd0HhDuQJwKXCL8kKreudxfRyXc+Ud8Ah3K1vpboJP48cHW4Ir2bqDL4HUU/7/wDzdzlhmKo64maeJ4PzDGzvx7sPcFISaUxr4uA/yXqre0NIPbq92Hg26HSdDhRkrgy7P9iYGoz21oA1IZK6GYri0Mx1xXAQ5IWEBULHWlRfwx3EbXC+iLRCbrePcAd9ZXFwPeB3yjqhL72IJv7KjAhVM4vAa4O078eKqQXELWu+3xzcbvU4a2POudchvM7Auecy3CeCJxzLsN5InDOuQznicA55zKcJwLnnMtwngiccy7DeSJwzrkM9/8AhDn7h9Bsm2cAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "num_latent_feats = np.arange(10,700+10,20)\n",
    "sum_errs = []\n",
    "\n",
    "for k in num_latent_feats:\n",
    "    # restructure with k latent features\n",
    "    s_new, u_new, vt_new = np.diag(s[:k]), u[:, :k], vt[:k, :]\n",
    "    \n",
    "    # take dot product\n",
    "    user_item_est = np.around(np.dot(np.dot(u_new, s_new), vt_new))\n",
    "    \n",
    "    # compute error for each prediction to actual value\n",
    "    diffs = np.subtract(user_item, user_item_est)\n",
    "    \n",
    "    # total errors and keep track of them\n",
    "    err = np.sum(np.sum(np.abs(diffs)))\n",
    "    sum_errs.append(err)\n",
    "    \n",
    "    \n",
    "plt.plot(num_latent_feats, 1 - np.array(sum_errs)/df.shape[0]);\n",
    "plt.xlabel('Number of Latent Features');\n",
    "plt.ylabel('Accuracy');\n",
    "plt.title('Accuracy vs. Number of Latent Features');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`4.` From the above, we can't really be sure how many features to use, because simply having a better way to predict the 1's and 0's of the matrix doesn't exactly give us an indication of if we are able to make good recommendations.  Instead, we might split our dataset into a training and test set of data, as shown in the cell below.  \n",
    "\n",
    "Use the code from question 3 to understand the impact on accuracy of the training and test sets of data with different numbers of latent features. Using the split below: \n",
    "\n",
    "* How many users can we make predictions for in the test set?  \n",
    "* How many users are we not able to make predictions for because of the cold start problem?\n",
    "* How many articles can we make predictions for in the test set?  \n",
    "* How many articles are we not able to make predictions for because of the cold start problem?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of instances for training:  40000\n",
      "Number of instances for testing:  5993\n"
     ]
    }
   ],
   "source": [
    "# Train | Test Spilt\n",
    "\n",
    "df_train = df.iloc[0:40000,:].copy()\n",
    "df_test = df.iloc[40000:,:].copy()\n",
    "\n",
    "\n",
    "df_train.article_id = df_train.article_id.astype(int)\n",
    "df_test.article_id = df_test.article_id.astype(int)\n",
    "\n",
    "print(\"Number of instances for training: \", df_train.shape[0])\n",
    "print(\"Number of instances for testing: \", df_test.shape[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_test_and_train_user_item(df_train, df_test):\n",
    "    '''\n",
    "    INPUT:\n",
    "    df_train - training dataframe\n",
    "    df_test - test dataframe\n",
    "    \n",
    "    OUTPUT:\n",
    "    user_item_train - a user-item matrix of the training dataframe \n",
    "                      (unique users for each row and unique articles for each column)\n",
    "    user_item_test - a user-item matrix of the testing dataframe \n",
    "                    (unique users for each row and unique articles for each column)\n",
    "    test_idx - all of the test user ids\n",
    "    test_arts - all of the test article ids\n",
    "    \n",
    "    '''\n",
    "    # Your code here\n",
    "    user_item_train = create_user_item_matrix(df_train)\n",
    "    user_item_test = create_user_item_matrix(df_test)\n",
    "    \n",
    "    test_idx = user_item_test.index.to_list()\n",
    "    test_arts = user_item_test.columns.to_list()\n",
    "    \n",
    "    return user_item_train, user_item_test, test_idx, test_arts\n",
    "\n",
    "user_item_train, user_item_test, test_idx, test_arts = create_test_and_train_user_item(df_train, df_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "20\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "662"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Q: \n",
    "# How many users can we make predictions for in the test set?': \n",
    "# \n",
    "# that means how many intersected users in train that also found in test set.\n",
    "\n",
    "user_ids_intersected = np.intersect1d(user_item_train.index, user_item_test.index)\n",
    "\n",
    "number_of_user_ids_intersected = len(user_ids_intersected)\n",
    "print(number_of_user_ids_intersected)\n",
    "\n",
    "# Substract the number from number of test. (number of unique users different from train set)\n",
    "len(test_idx) - number_of_user_ids_intersected\n",
    "\n",
    "# Users can predict are the intersected / common users in train and test set. \n",
    "# (If users that are not in training set, then machine will not have affinity info for them)\n",
    "# since latent factor applied to users. Since we know these users' affinity, we might know how likely \n",
    "# he/she would like to interact each article (even tho he/she not interacted that article before). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Q:\n",
    "# How many articles can we make predictions for in the test set?\n",
    "# How many articles in the test set are we not able to make predictions for because of the cold start problem?\n",
    "# That means, are article ids that in test set, also appear in train set. Any unique article not found in train set.\n",
    "\n",
    "article_ids_intersected = np.intersect1d(user_item_train.columns, user_item_test.columns)\n",
    "\n",
    "np.sum(article_ids_intersected != np.array(user_item_test.columns))\n",
    "\n",
    "# zero. no unique found. Which means, ids in test set are all also there in train set. all movies in test will \n",
    "# have its latent factor. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(4487, 714)\n",
      "(682, 574)\n"
     ]
    }
   ],
   "source": [
    "print(user_item_train.shape)\n",
    "print(user_item_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Awesome job!  That's right!  All of the test movies are in the training data, but there are only 20 test users that were also in the training set.  All of the other users that are in the test set we have no data on.  Therefore, we cannot make predictions for these users using SVD.\n"
     ]
    }
   ],
   "source": [
    "# Replace the values in the dictionary below\n",
    "a = 662 \n",
    "b = 574 \n",
    "c = 20 \n",
    "d = 0 \n",
    "\n",
    "\n",
    "sol_4_dict = {\n",
    "    'How many users can we make predictions for in the test set?': c, \n",
    "    'How many users in the test set are we not able to make predictions for because of the cold start problem?': a, \n",
    "    'How many movies can we make predictions for in the test set?': b,\n",
    "    'How many movies in the test set are we not able to make predictions for because of the cold start problem?': d\n",
    "    }\n",
    "\n",
    "t.sol_4_test(sol_4_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`5.` Now use the **user_item_train** dataset from above to find U, S, and V transpose using SVD. Then find the subset of rows in the **user_item_test** dataset that you can predict using this matrix decomposition with different numbers of latent features to see how many features makes sense to keep based on the accuracy on the test data. This will require combining what was done in questions `2` - `4`.\n",
    "\n",
    "Use the cells below to explore how well SVD works towards making predictions for recommendations on the test data.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "# fit SVD on the user_item_train matrix\n",
    "u_train, s_train, vt_train = np.linalg.svd(user_item_train) # fit svd similar to above then use the cells below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use these cells to see how well you can use the training \n",
    "# decomposition to predict on test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(4487, 4487) (714,) (714, 714)\n",
      "Total users in train set: 4487\n",
      "Total articles in train set: 714\n",
      "Total number of latent factors: 714\n"
     ]
    }
   ],
   "source": [
    "print(u_train.shape, s_train.shape, vt_train.shape)\n",
    "\n",
    "print(f\"Total users in train set: {u_train.shape[0]}\")\n",
    "print(f\"Total articles in train set: {vt_train.shape[0]}\")\n",
    "print(f\"Total number of latent factors: {s_train.shape[0]}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.007516754093525902 0.6604198054917303\n",
      "0.05582851571825394 0.6328017631956505\n",
      "0.11007778268016816 0.6168364553780433\n",
      "0.16561571815696632 0.5848351225385319\n",
      "0.21981865806865664 0.5603945375754067\n",
      "0.27937356264867874 0.5309917623446467\n",
      "0.3322637852734185 0.5149353611791314\n",
      "0.3752365658946877 0.5038755724797407\n",
      "0.4179165888310912 0.49245474398574146\n",
      "0.46329130485256803 0.48191008966701776\n",
      "0.5025390623844941 0.4679728074370526\n",
      "0.5404423050485289 0.46091247367582017\n",
      "0.5811492088746872 0.4503678193570966\n",
      "0.6171027979107029 0.4433074855958643\n",
      "0.643502954189826 0.4378976194671278\n",
      "0.6678385930007024 0.4309289783521452\n",
      "0.6982067468163089 0.4309289783521452\n",
      "0.7221670545560043 0.42744465779465396\n",
      "0.7470141484284704 0.42744465779465396\n",
      "0.7799326215735457 0.42396033723716264\n",
      "0.7999294739708878 0.42396033723716264\n",
      "0.8196544380706139 0.42396033723716264\n",
      "0.8360005380978012 0.42396033723716264\n",
      "0.8582465833381127 0.42396033723716264\n",
      "0.8778089868944815 0.42396033723716264\n",
      "0.8927366188987824 0.42396033723716264\n",
      "0.9155420154647751 0.42396033723716264\n",
      "0.9371374843830138 0.42396033723716264\n",
      "0.9540702908796326 0.42396033723716264\n",
      "0.9680117988512403 0.42396033723716264\n",
      "0.9717484999139269 0.42396033723716264\n",
      "0.9803877859659246 0.42396033723716264\n",
      "0.9957973827069838 0.42396033723716264\n",
      "1.0 0.42396033723716264\n",
      "1.0 0.42396033723716264\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import r2_score\n",
    "\n",
    "#  Rows that match the test set\n",
    "test_idx = user_item_test.index\n",
    "row_idxs = user_item_train.index.isin(test_idx)\n",
    "u_test = u_train[row_idxs, :]\n",
    "\n",
    "\n",
    "#  Columns that match the test set\n",
    "test_col = user_item_test.columns\n",
    "col_idxs = user_item_train.columns.isin(test_col)\n",
    "vt_test = vt_train[:, col_idxs]\n",
    "\n",
    "# Test data (reshaping to the subset of those intersected users)\n",
    "train_idx = user_item_train.index\n",
    "row_idxs_2 = user_item_test.index.isin(train_idx)\n",
    "user_item_test_subset = user_item_test.loc[row_idxs_2]\n",
    "\n",
    "\n",
    "\n",
    "num_latent_feats = np.arange(10,700+10,20)\n",
    "\n",
    "scores_train = []\n",
    "scores_test = []\n",
    "\n",
    "\n",
    "for k in num_latent_feats:\n",
    "    \n",
    "    \n",
    "    # restructure with k latent features\n",
    "    s_train_lat, u_train_lat, vt_train_lat = np.diag(s_train[:k]), u_train[:, :k], vt_train[:k, :]\n",
    "    u_test_lat, vt_test_lat = u_test[:, :k], vt_test[:k, :]\n",
    "    \n",
    "    \n",
    "    # Take dot product to get prediction\n",
    "    \n",
    "    # predictions on train set\n",
    "    user_item_train_preds = np.around(np.dot(np.dot(u_train_lat, s_train_lat), vt_train_lat))\n",
    "    \n",
    "    # predictions on test set\n",
    "    user_item_test_preds = np.around(np.dot(np.dot(u_test_lat, s_train_lat), vt_test_lat))\n",
    "    \n",
    "    \n",
    "\n",
    "    # Calculate R2 score with \n",
    "    # sklearn.metrics.r2_score(y_true, y_pred)\n",
    "    \n",
    "    # R2 score as accuracy for train set. (true of train vs pred of train)\n",
    "    r2s_train_sets = r2_score(user_item_train, user_item_train_preds)\n",
    "    \n",
    "    # R2 score as accuracy for test set. (true ot test vs pred of test)\n",
    "    r2s_test_sets = r2_score(user_item_test_subset, user_item_test_preds)\n",
    "    \n",
    "    # Append each accuracy result\n",
    "    scores_train.append(r2s_train_sets)\n",
    "    scores_test.append(r2s_test_sets)\n",
    "    \n",
    "    print(r2s_train_sets, r2s_test_sets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAArGklEQVR4nO3deVxWZf7/8deHRZBFkEUUEcFEc98INbUsNbcyM2uyPWtsKiubZn5T35nWaZpqpmWaqUzNysqcss3K0hbNTFNxF0UBRQU3REWRHa7fH+e27hgVzBvOvXyej8f98L7PfYQ3Rm8O17nOucQYg1JKKc/nZ3cApZRSrqGFrpRSXkILXSmlvIQWulJKeQktdKWU8hIBdn3imJgYk5SUZNenV0opj7R69eqDxpjYk71nW6EnJSWRnp5u16dXSimPJCI7T/WeDrkopZSX0EJXSikvoYWulFJewrYx9JOprKwkLy+PsrIyu6M0qODgYBISEggMDLQ7ilLKi7hVoefl5REeHk5SUhIiYnecBmGMobCwkLy8PJKTk+2Oo5TyInUOuYjITBE5ICKbTvG+iMiLIpItIhtEpPevDVNWVkZ0dLTXljmAiBAdHe31v4UopRpffcbQ3wBGnOb9kUCK4zEJeOVsAnlzmZ/gC1+jUqrx1TnkYoxZIiJJp9nlcmCWse7D+6OIRIpIK2PMXleFVEopd1NWWc3B4nIKjpVzsLjC8Wc5VdU1df7dIZ3i6NEm0uWZXDGG3hrY7fQ6z7HtfwpdRCZhHcWTmJjogk/tWkeOHGH27NnceeedZ/T3Ro0axezZs4mMjGyYYEop21RW17A0+yBfbtzHzkPHKThmlfjRsqqT7l+fX8BbNAt220KvN2PMNGAaQGpqqtutrHHkyBFefvnl/yn0qqoqAgJO/U81f/78ho6mlGpE1TWGFdsL+XTDXr7YtJcjJZWEBwdwbstwOrYMZ2D7GGLDg35+hAUTGx5EdFgTAv3tmw3uikLPB9o4vU5wbPM4DzzwADk5OfTs2ZPAwECCg4Np3rw5mZmZbNu2jbFjx7J7927Kysq49957mTRpEvDzbQyKi4sZOXIkAwcOZNmyZbRu3ZpPPvmEpk2b2vyVKaXqUlNjWLPrMJ9t2MvnG/dScKyckCb+DOscx2Xd4xnUIYagAH+7Y56WKwp9HjBZROYAfYEiV4yfP/ZpBpv3HD3rcM46xzfjkcu6nPL9p556ik2bNrFu3ToWL17M6NGj2bRp00/TC2fOnElUVBSlpaWcd955XHnllURHR//iY2RlZfHuu+8yffp0rr76aj744AOuv/56l34dSqmzV1ldQ05BMRn5R9mYX8TCjH3sKSojKMCPi89twWU94rmoYwuaNnHvEndWZ6GLyLvAYCBGRPKAR4BAAGPMVGA+MArIBkqAWxoqbGNLS0v7xVzxF198kY8++giA3bt3k5WV9T+FnpycTM+ePQHo06cPubm5jRVXKXUKpRXVZO47Ssaeo2TsKSJjz1Ey9x2joso6gRkc6Mf558TwxxEdGdopjvBgz7zorz6zXCbU8b4B7nJZIofTHUk3ltDQ0J+eL168mK+//prly5cTEhLC4MGDTzqXPCgo6Kfn/v7+lJaWNkpWpXzR0bJKdhWWcOh4BYXHyyksrrCeF1dQeLyCQ8etGSh5h0uocZy1i2gaSJf4ZtzUvy1d4iPoEt+MdrFh+Pt5/nRit7pS1G7h4eEcO3bspO8VFRXRvHlzQkJCyMzM5Mcff2zkdEopgLzDJXy1eT9fbd7Pih2HqK755fyKAD8hKrQJUaFNiA5rQo82kYztGU9nR3knNG/qtdeCaKE7iY6OZsCAAXTt2pWmTZsSFxf303sjRoxg6tSpdOrUiY4dO9KvXz8bkyrlO4wxbN57lIUZVolv3mudW0tpEcbtF7Sje0IE0WFBRIc2ITo0iGZNA7y2sOsi1ohJ40tNTTW1F7jYsmULnTp1siVPY/Olr1WpM2WMYeWOQ3yxaR9fbd5P/pFSRCC1bXOGdY5jWOeWJMeE1v2BvJCIrDbGpJ7sPT1CV0q5jbzDJXywOp8P1uSx61AJQQF+DEqJ5d4hKVzcqQUxYUF1fxAfpoWulLJVaUU1X2bs5f30PJblFAIwoH009w1LYXiXloQ00ZqqL/2XUko1OmMMq3ceZu7qPD7bsJfi8ioSo0L4/bAOjOvdmoTmIXZH9Eha6EqpRlNVXcNnG/by8uJstu0vJqSJP6O7tWJ8nwTSkqN89mSmq2ihK6UaXHlVNR+uyWfqdznsLCyhQ1wYz4zvzuhurQgN0hpyFf2XVEo1mNKKauas2sW0JdvZW1RG94QIXr2hD8M6xeHnBRfyuBtdJNrJibst/hovvPACJSUlLk6klGc6VlbJy4uzGfj0tzz26WbaRIUwa2Ian9w1gOFdWmqZNxAtdCda6EqdnaLSSp7/ahsDnvqWZ77cStfWEbz/u/68d3t/LugQq2PkDUyHXJw43z532LBhtGjRgvfee4/y8nKuuOIKHnvsMY4fP87VV19NXl4e1dXVPPTQQ+zfv589e/Zw0UUXERMTw6JFi+z+UpRqVMfKKnn9h1xmfL+do2VVDO8Sx+SLUuiWEGF3NJ/ivoX+xQOwb6NrP2bLbjDyqVO+7Xz73IULFzJ37lxWrlyJMYYxY8awZMkSCgoKiI+P5/PPPwese7xERETw3HPPsWjRImJiYlybWSk3dry8ijeW5TL9++0cKalkWOc4pgxNoUu8Frkd3LfQbbZw4UIWLlxIr169ACguLiYrK4tBgwZx//3386c//YlLL72UQYMG2ZxUqcZXUlHFW8t38uqS7Rw6XsHF57bgvqEd9IjcZu5b6Kc5km4MxhgefPBBbr/99v95b82aNcyfP5+//OUvDBkyhIcfftiGhEo1vtKKat5ZsZOp3+VwsLiCCzvEct+wDvRsgPUx1Zlz30K3gfPtc4cPH85DDz3EddddR1hYGPn5+QQGBlJVVUVUVBTXX389kZGRzJgx4xd/V4dclLcprahm0dYDfL5xL4syD1BSUc3A9jHcNyyFPm2j7I6nnGihO3G+fe7IkSO59tpr6d+/PwBhYWG8/fbbZGdn88c//hE/Pz8CAwN55ZVXAJg0aRIjRowgPj5eT4oqj1dSUcWizALmb9zLt5kHKK2sJjq0CWN7tWZcr9akJmmRuyO9fa5NfOlrVZ6hrLKar7fsZ/7GvSzKLKC0spqYsCYM79KS0d1akZYcRYCNK9ori94+Vyl1SnuLSpm1fCfvrtzFkZJKYsKCuLJPa0Z1a0Xf5GivWJrNV2ihK+Wj1uw6zMylO/hi0z6MMVzSuSU39m9L33Za4p7K7QrdGOP1V5PZNcylVEVVDV9s2svMH3JZv/sI4cEBTByQxI39k2gTpbes9XRuVejBwcEUFhYSHR3ttaVujKGwsJDg4GC7oygfcvh4Be+s2MlbP+5k/9Fy2sWE8tfLuzCud4Le7dCLuNV/yYSEBPLy8igoKLA7SoMKDg4mISHB7hjKBxQWlzP9+x28tTyX4xXVDEqJ4akru3NhSqzeIMsLuVWhBwYGkpycbHcMpTzeweJypi/Zzls/7qS0sppLu8cz+aL2dGwZbnc01YDcqtCVUmfnwLEypn23nbdX7KSiqoYxPeKZfHEK7VuE2R1NNQItdKW8wP6jZUz9LofZK3ZRWV3D2F6tmXxRe9rFapH7Ei10pTxUdY1heU4hH661FlqurjFc4SjypJhQu+MpG2ihK+VhMvcd5aM1+Xy8Lp/9R8sJDwpgfJ8Ebr+gHW2jtch9mRa6Uh7gwNEyPlm3hw/X5rNl71EC/IQLO8Ty0KWtGdopjuBAf7sjKjegha6Um6quMXy1eT+zV+5iaVYBNQa6J0Tw6GWduaxHPNFhQXZHVG5GC10pN3OsrJL/rtrNG8tyyTtcSnxEMHcMPocreiXobBV1WvUqdBEZAfwL8AdmGGOeqvV+IvAmEOnY5wFjzHzXRlXKu+0qLOH1ZTt4Pz2P4vIqzktqzp9HdWJY5zi9y6GqlzoLXUT8gZeAYUAesEpE5hljNjvt9hfgPWPMKyLSGZgPJDVAXqW8ijGGFTsOMXPpDr7ash9/ES7t3oqJA5PpnhBpdzzlYepzhJ4GZBtjtgOIyBzgcsC50A3QzPE8AtjjypBKeZuKqho+27CH15buIGPPUZqHBHLX4Pbc0L8tcc30Pj/q16lPobcGdju9zgP61trnUWChiNwNhAJDT/aBRGQSMAkgMTHxTLMq5fGOlFTwzopdvLkslwPHyklpEcbfx3VjbM/WNG2iM1XU2XHVSdEJwBvGmGdFpD/wloh0NcbUOO9kjJkGTANrxSIXfW6l3N72gmJe/yGXuavzKK20bpL1j6t6cEFKjNfeWVQ1vvoUej7Qxul1gmObs1uBEQDGmOUiEgzEAAdcEVIpT3RifHzG9zv4JnM/gX5+jO0Vz8SByZzbslndH0CpM1SfQl8FpIhIMlaRXwNcW2ufXcAQ4A0R6QQEA959D1ylTiM99xCPfprBpvyjRIU24e6LU7ihX1tiw3XuuGo4dRa6MaZKRCYDC7CmJM40xmSIyONAujFmHnA/MF1E7sM6QXqz0WV5lA+qrjG8vCibF77JolVEMH8f140rerXWKzlVo6jXGLpjTvn8Wtsednq+GRjg2mhKeZb9R8uYMmcdy7cXcnnPeJ4Y25Xw4EC7YykfoleKKuUC32bu5w/vb6C0opp/jO/O+D4JerJTNTotdKXOQnlVNc98uZXXlu7g3Jbh/Ofa3np5vrKNFrpSv1LuwePc/e5aNuYXcVP/tjw4qpOOlStbaaEr9St8vDafP3+0kQB/P169oQ/Du7S0O5JSWuhKnYnMfUd5cn4mS7YVkJYUxQvX9CQ+sqndsZQCtNCVqpf9R8t4duFW5q7OIywogL+M7sTN5yfpXRCVW9FCV+o0isurmPZdDtO/30FVTQ0TByQz+eL2RIY0sTuaUv9DC12pk6iqruG99Dye+2obB4vLubR7K/7f8HNJjA6xO5pSp6SFrpQTYwyLth7gyfmZZB8o5ryk5ky/sQ+9EpvbHU2pOmmhK+Wweudhnv4yk5U7DpEcE8qrN/Thks5xeoGQ8hha6MrnZe0/xjMLtvLV5v3EhAXx18u7cE1aIoF6wlN5GC105bPyj5Ty/Ffb+HBNHqFNAvjDJR2YODCZkCb6v4XyTPqdq3zOoeMVvLQom7eW7wSBWwcmc+fg9jQP1ZkryrNpoSufUVZZzfQl23l1yXZKKqoY3yeBe4d2oLVeGKS8hBa68gnrdx/h/vfXk32gmOFd4vjDJR1JiQu3O5ZSLqWFrrxaRVUNL36TxSvf5RAbFsSbE9O4sEOs3bGUahBa6MprZewp4v731pO57xjj+yTw0KWdiWiqC04o76WFrrxOZXUNryzO4cVvsogMacKMG1MZ2jnO7lhKNTgtdOVVtu0/xv3vrWdjfhGX9Yjn8TFddPaK8hla6MorVFbX8NrSHTy3cBthwQG8fF1vRnVrZXcspRqV5xW6MVBdAQFBdidRbsAYw+cb9/LPBVvJLSxheJc4/nZFN2LC9PtD+R7PK/SN78N3T8PoZ6HdYLvTKBstyznI019ksj6viI5x4cy8OZWLOrbQe68on+V5hR4WBzXVMOty6Doehj8J4XrCy5ds2XuUp7/MZPHWAlpFBPOP8d0Z1zsBfz8tcuXbPK/Q210Id/4IS5+Hpc9B1lcw5CFInQh+ukCvN8s/UsqzC7fy0dp8woMCeHDkudx0fpIuzKyUgxhjbPnEqampJj09/ew+yMFsmH8/bF8M8b1g9HPQurdL8in3cbSskv98m80by3IBuOX8JO4c3J6IEJ1TrnyPiKw2xqSe7D3PO0J3FtMebvgYNn0AC/4Ppl8Mab+Fi/8CwRF2p1NnqabGMHd1Hs8syKTweAXjeiXw+0v03itKnYpnFzqACHQbDynD4NsnYNUM2PwJDH0UuoyDwGC7E6pfYe2uwzw6L4P1eUX0Tozk9ZvT6JagP6SVOh3PHnI5mT1r4bP7rD+DmkGnMdD9KkgapGPsHqDgWDlPf5nJ3NV5tAgP4sFR5zK2Z2uduaKUg/cOuZxMfC+47RvYscSa4rj5E1j3NoS1hK5XWuXeqqd1ZK/cRmV1DW8uy+VfX2dRVlXN7Re24+6LUwgL8r5vUaUaivcdoddWWQrbvoQN70PWQqiphOgU6H41dP8NNG/b8BnUaS3NOsijn2aQfaCYwR1jefjSzrSLDbM7llJu6XRH6N5f6M5KDllH7Bvfh50/gH8QDHsM0m4HP10/srFV1xj+uXArryzOoW10CA9f2pmLz9ULg5Q6ndMVer1aTERGiMhWEckWkQdOsc/VIrJZRDJEZPbZBG4wIVGQegvcMh+mbIRzLoYvH4B3xsOxfXan8ylHSiq45Y1VvLI4hwlpbVgw5QKGdIrTMlfqLNRZ6CLiD7wEjAQ6AxNEpHOtfVKAB4EBxpguwBTXR3WxyESY8K41d33nMnjlfMicb3cqn5C57yhj/vMDy3MO8uQV3fj7uO56cZBSLlCfI/Q0INsYs90YUwHMAS6vtc9vgZeMMYcBjDEHXBuzgYjAebfC7d9Bs3iYM8GaIVNRYncyr/X5hr2Me3kZpZXVzJnUj2v7JtodSSmvUZ9Cbw3sdnqd59jmrAPQQUR+EJEfRWTEyT6QiEwSkXQRSS8oKPh1iRtCbEdrZsz5d0P6TJh2Iexdb3cqr1JdY3j6y0zumr2Gji3D+ezugfRpG2V3LKW8iqvOBAYAKcBgYAIwXUQia+9kjJlmjEk1xqTGxrrZuo4BQXDJE9aVp+XHYPoQ+OFFqKmxO5nHqz1ePmdSP+Ka6QVfSrlafQo9H2jj9DrBsc1ZHjDPGFNpjNkBbMMqeM9zzkVwxzLoMBy+egjevBT2rLM7lcc62Xh5UICOlyvVEOpT6KuAFBFJFpEmwDXAvFr7fIx1dI6IxGANwWx3XcxGFhIFv3kbxvwbDmyxhmDm3gqHc+1O5lGWbCvQ8XKlGlGdhW6MqQImAwuALcB7xpgMEXlcRMY4dlsAFIrIZmAR8EdjTGFDhW4UItD7Rrh3HQy6HzI/h3+nwpcPWvPZ1Wl9si6fiW+sIjEqhE8n63i5Uo3Bty4sOhtH98CiJ2HdO9AkDAZOgb53QJMQu5O5ndeW7uCvn22mb3IU029KpVmw3uZWKVc56wuLFNa0xsv/Y42vtx0A3zwO/+4Da96yVlBSGGN46otM/vrZZkZ0acmbE9O0zJVqRFroZ6pFJ7h2Dtw8H5q1gnmT4dULrbs7+rDK6hr+8P4Gpn6Xw3V9E3nput56sZBSjUwL/ddKGmDNXb/qDTheYE1z/PpRqCyzO1mjK6moYtKsdD5Yk8eUoSk8Mbarru+plA200M+GCHS5Au5aAT0nWOucTh0Iu1bYnazRHD5ewXUzVvDdtgKeGNuVKUM76P1YlLKJFrorNI2Ey1+C6z+EqnKYORy++BNUHLc7WYPKP1LK+KnLyNhzlJev6831/fRWxErZSQvdldoPgTuXwXm3wYqp8HJ/awFrL7Qpv4grX17GgaPlzJqYxoiureyOpJTP00J3taBwGP1P66SpXwDMuhzm3QNlRXYnc5kFGfu4aupy/ATe+11/+rWLtjuSUgot9IaTNADu+AHOvwfWvgXPdYGP7oCcbz12mqMxhqnf5fC7t1fToWU4H08eQKdWzeyOpZRy0AUbG1JgU7jkr9Zapqumw+Z5sH62R65vWlFVw58/2sj7q/MY3b0Vz17VQ6clKuVm9ErRxlRZZq1vuvF92LbAWt80pgN0uxq6jYeoZLsTntTh4xX87u3VrNhxiHuGpDBlSAp+Oi1RKVvomqLuqPb6pgAtu0PSIGu4JrG/dZMwm+UUFDPxjVXsLSrjmSu7M7ZX7VvhK6Uakxa6uzuyCzbOhexvIG8VVJcDAnFdrNsMJA2w/gyNadRYP2Qf5I63VxPo78e0G/voDbaUcgNa6J6ksgzyV1tH7blLYfdKqCq13os9F5IGOo7iB0Fow80umb1iFw99solzYkN57abzaBOlNyFTyh1ooXuyqgrrPjE7l0LuD7DrR6h0XLDUogskD/p5mKZp87P+dDU1hmcWbGXqdzkM7hjLvyf0IlxvsKWU29BC9ybVlVbB71gCud9btxmoKgUEWnaD5Aus4ZnEfmc8Bl9eVc0f3t/Ap+v3cH2/RB69rAsB/jqzVSl3ooXuzarKrSGaHd9bBb97pWMMHmuIJrGfdYI1sR9Etj3lFMkjJRVMmrWalbmHeGDkudx+QTu9J4tSbkgL3ZdUlsGeNbBruTU8s2sFlDuuUg1v9XPBt7sIYjsAsPtQCTe9vpK8Q6U8e3UPLusRb+MXoJQ6ndMVul5Y5G0Cg6Ht+dYDoKYGCrZYBb9zufVnxkfWe9HtORA/hEc2t+FwdXvevq0vack6k0UpT6WF7u38/Kzpj3FdrJuGARzeCVkLObj6Y5pvmMFMqaaqaQwBG0dBxWhoN9j6waCU8iha6L6oeVtmVQ/j0V3x9G11D6/2P0yz3AWQ8TGsmQWBodD+YmjTD2I7QkwKRLQBP73UXyl3poXuY4wxPPVlJq9+t52hnVrw4oRehDQJgPOusaZI5n4PW+fD1i9gy6c//8WAYIhub5V7jKPkYzpYJ14Dmtj3BSmlfqInRX2IMYYnPt/Ca0t3cH2/RB4bU8dScccLoTALDm6Dgq1w0PH8yE4wNdY+AcHQOvXnk61tzoPgiMb5gpTyQXpSVGGM4cn5VpnffH4Sj1zWue5piaHR1iOx3y+3V5bBoRwoyIS81daJ1qXPg/kn1i0LujoK3lHyzeI94o6SSnk6PUL3AcYYnvoik1eXbOfG/m15bEwX188xLy+25sPv+tEq+LxVUFFsvde0uTU84/yI7WDNi9dxeaXOiB6h+zBjrEv5X12y3THM0gBlDhAUBu0utB4A1VWwfxPsXgEHtljDNdu+tBb7OME/CKLPscbjmydZJ14j2kBkG4hI0KEbpc6QFroXM8bwz4VbeWVxDtf2TeTxMV0b7+pP/wCI72k9nJUcgsJsayz+4DYo2Ab7NlknYasrfrlvUISj3B0lH3XOzydjm7W2pmQqpX6ihe6ljDE899U2XlqUw4S0NjxxeVf3WJQiJApC0qBN2i+319TA8QNQlGfdTrhoNxzZ7fhzl3XnyYpjP+8fGOI4uj8xjJNiFX5AHfPnRSA40srhrzcdU95FC91LvfB1Fv/+NpvfpLbhb2O7uUeZn46fH4S3tB4JJxkeNAaKD1hH9YVZP8+4yUuHTR8Cv+JcUFCE4wdMtNMjCppGgtQxti9+1rkB578XEm1t0/MCyiZa6F7oX19n8a9vshjfJ4G/j/OAMq8PEQiPsx7Jg375XmUpFObA4R3W3ShPx9RA6WHrUVL486N4HxzYbD2vLDmboNYPhKYnfjDosJA6iQH3QqfLXP5htdC9zL+/yeL5r7cxrndrnr6yu3eUeV0Cm0LLrtbDFarKrd8ITqemCsqOOP1QOOR4OP2QKCviV/3moLyfX8MM92mhe4kTs1leWZzDFb1a84/xPU5/0ZA6tYCg+u0XFGbNxlHKTdTr90ERGSEiW0UkW0QeOM1+V4qIEZGTzpFUDaOmxvDovAxeWZzDhLRE/nmVlrlSvqjOI3QR8QdeAoYBecAqEZlnjNlca79w4F5gRUMEVSdXVV3DAx9uZO7qPH47KJn/G9VJF6ZQykfV5wg9Dcg2xmw3xlQAc4DLT7LfX4GngTIX5lOnUVFVw71z1jF3dR5ThqZomSvl4+pT6K2B3U6v8xzbfiIivYE2xpjPT/eBRGSSiKSLSHpBQcEZh1U/K6us5va30vl8417+PKoTU4Z20DJXysed9ZwqEfEDngPur2tfY8w0Y0yqMSY1Njb2bD+1zyour+Lm11eyeFsBT17Rjd9e0M7uSEopN1CfWS75QBun1wmObSeEA12BxY4jxJbAPBEZY4zRu2+5WFFJJTe9vpKN+UU8f3VPxvZqXfdfUkr5hPoU+iogRUSSsYr8GuDaE28aY4qAmBOvRWQx8Actc9crOFbODa+tYHvBcV6+rjfDu7S0O5JSyo3UWejGmCoRmQwsAPyBmcaYDBF5HEg3xsxr6JAK9haVct2MFew5UsprN6cyKEWHrJRSv1SvC4uMMfOB+bW2PXyKfQeffSzlbPehEq6d8SOHj1cya2Jf0pKj7I6klHJDeqWom8spKOa66Ssorazmndv60qNNpN2RlFJuSgvdjW3Ze5QbXrOu05ozqR+dWjWzOZFSyp1pobup9buPcOPMlTQN9Oed3/blnNgwuyMppdycFrobWpV7iFteX0Xz0EBm39aPNlEhdkdSSnkALXQ3831WAb+dlU58ZFNm39aPlhF1rMCjlFIOWuhu5OvN+7nznTW0iw3lrVv7Ehtez9u4KqUUWuhuY/7Gvdzz7lq6xDfjzYlpRIY0sTuSUsrDaKG7gVW5h5gyZx0920Ty+i3nER6sixcrpc6cLnhos52Fx5k0K52E5k157SYtc6XUr6eFbqOikkpueWMVBph583lEhGiZK6V+PS10m1RW13DHO6vZfaiEaTekkhQTanckpZSH0zF0GxhjeOjjTSzLKeTZq3rovVmUUi6hR+g2mP79duas2s3dF7fnyj66arxSyjW00BvZl5v28fcvMhndvRX3De1gdxyllBfRQm9EG/OKmPLftfRIiOTZq3rg56drgCqlXEcLvZHsLSrl1jdXER0axPQbUwkO9Lc7klLKy+hJ0UZwvLyKW99Ip6Simg/u0Ev6lVINQwu9gRlj+P1769i6/xgzbz6Pji3D7Y6klPJSOuTSwF5buoMFGft5cOS5XNhB1wFVSjUcLfQGtGbXYZ76IpPhXeK4dWCy3XGUUl5OC72BHCmp4O7Za2kVGcwz43sgojNalFINS8fQG0BNjeH+99ZTcKycuXf0J6Kp3qNFKdXw9Ai9AUz/fjvfZB7gz6M70T0h0u44SikfoYXuYqtyD/HMgq2M7taKG/u3tTuOUsqHaKG7UGFxOXfPXkub5k156spuOm6ulGpUOobuIjU1hvveW8+hkgo+uvN8XahCKdXo9AjdRV75Locl2wp45LLOdImPsDuOUsoHaaG7wPKcQp5duJUxPeK5Ni3R7jhKKR+lhX6WCo6Vc8+ctSRFh/LkOB03V0rZR8fQz0JFVQ2TZ6/haGklb92aRliQ/nMqpexTryN0ERkhIltFJFtEHjjJ+78Xkc0iskFEvhERr5+vZ4zhkXkZrNhxiKev7M65LZvZHUkp5ePqLHQR8QdeAkYCnYEJItK51m5rgVRjTHdgLvCMq4O6mzeX5fLuyl3cOfgcxvZqbXccpZSq1xF6GpBtjNlujKkA5gCXO+9gjFlkjClxvPwR8OqFMpdsK+DxzzYzrHMcf7iko91xlFIKqF+htwZ2O73Oc2w7lVuBL84mlDvLKSjmrtlr6BAXzvO/6anLyCml3IZLz+KJyPVAKnDhKd6fBEwCSEz0vOl9RSWV/PbNdJr4+zHjplQ9CaqUciv1OULPB9o4vU5wbPsFERkK/BkYY4wpP9kHMsZMM8akGmNSY2M9a7GHquoa7pq9ht2HS5h6Qx8SmofYHUkppX6hPoW+CkgRkWQRaQJcA8xz3kFEegGvYpX5AdfHtN8Tn29hafZB/ja2G+clRdkdRyml/kedhW6MqQImAwuALcB7xpgMEXlcRMY4dvsHEAa8LyLrRGTeKT6cR3pnxU7eWJbLbQOTufq8NnX/BaWUskG9BoGNMfOB+bW2Pez0fKiLc7mN5TmFPPJJBoM7xvLgqE52x1FKqVPSS/9PY1dhCXe8s5qkmFBenNALf53RopRyY1rop1BcXsVts1YBMOPGVJrp7XCVUm5O592dhLUm6DpyCo4za2IaSTGhdkdSSqk66RH6Sbz4bRYLMvbzf6M6MaB9jN1xlFKqXrTQa1mQsY8Xvs7iyt4JTByQZHccpZSqNy10J9v2H+P3/11Hj4QI/nZFV723uVLKo2ihOxSVVDJpVjohQQG8ekMqwYH+dkdSSqkzooWOdVn/5HfXkH+klKnX96ZlRLDdkZRS6ozpLBfgmQVb+T7rIE+N60aftnpZv1LKM/n8EfrHa/OZtmQ7N/RryzW6wLNSyoP5dKFvzCviTx9sIC05iocvq70Ik1JKeRafLfSDxeXc/lY6MWFBvHxdbwL9ffafQinlJXxyDL28qpo73l7NoZIK5v7ufGLCguyOpJRSZ83nCt0Yw18+2sSq3MP859pedG0dYXckpZRyCZ8bZ3ht6Q7eX53HPUNSuLR7vN1xlFLKZXyq0BdlHuDJ+VsY2bUlU4ak2B1HKaVcymcKPWv/Me55dy3ntmzGs1f3wE/vba6U8jI+UeiHj1dw26x0ggL9mXFTKiFNfO7UgVLKB3h9s1VW13DnO2vYW1TGnEn9iI9sanckpZRqEF5/hP7Ypxks317IU+O60Tuxud1xlFKqwXh1ob+1PJe3f9zF7y48h3G9E+yOo5RSDcprC31p1kEe/XQzQzu14I/DO9odRymlGpxXFvr2gmLufGc17WPDeOGaXvjrjBallA/wukLfV1TGjTNXEuDvx4ybUgkL8vrzvkopBXhZoR8+XsENr63gSEklb96SRpuoELsjKaVUo/Gaw9fi8ipufmMVOw+V8OYtaXRL0Hu0KKV8i1ccoZdVVjNpVjqb8ot46dre9D8n2u5ISinV6Dy+0Kuqa7jn3bUsyynkH+O7M6xznN2RlFLKFh5d6DU1hgc+3MjCzft55LLOOtdcKeXTPLbQjTE88fkW5q7OY8rQFG4ZkGx3JKWUspXHFvp/vs1m5g87uGVAEvfqrXCVUqp+hS4iI0Rkq4hki8gDJ3k/SET+63h/hYgkuTypk1nLc3n2q22M692ah0Z3RkQvHFJKqToLXUT8gZeAkUBnYIKIdK61263AYWNMe+B54GlXBz3hk3X5PPxJBkM7xfHMld31vuZKKeVQnyP0NCDbGLPdGFMBzAEur7XP5cCbjudzgSHSQIfNLZsFM6xzHP+5thcB/h47YqSUUi5XnwuLWgO7nV7nAX1PtY8xpkpEioBo4KDzTiIyCZgEkJiY+KsC920XTd92Os9cKaVqa9RDXGPMNGNMqjEmNTY2tjE/tVJKeb36FHo+0MbpdYJj20n3EZEAIAIodEVApZRS9VOfQl8FpIhIsog0Aa4B5tXaZx5wk+P5eOBbY4xxXUyllFJ1qXMM3TEmPhlYAPgDM40xGSLyOJBujJkHvAa8JSLZwCGs0ldKKdWI6nW3RWPMfGB+rW0POz0vA65ybTSllFJnQuf9KaWUl9BCV0opL6GFrpRSXkLsmowiIgXAznruHkOti5Q8gKdl9rS8oJkbi6dl9rS8cGaZ2xpjTnohj22FfiZEJN0Yk2p3jjPhaZk9LS9o5sbiaZk9LS+4LrMOuSillJfQQldKKS/hKYU+ze4Av4KnZfa0vKCZG4unZfa0vOCizB4xhq6UUqpunnKErpRSqg5a6Eop5SXcutDrWsvULiIyU0QOiMgmp21RIvKViGQ5/mzu2C4i8qLja9ggIr1tytxGRBaJyGYRyRCRe905t4gEi8hKEVnvyPuYY3uyY93abMc6tk0c2xt1Xds6svuLyFoR+cwTMotIrohsFJF1IpLu2OaW3xdOmSNFZK6IZIrIFhHp786ZRaSj49/3xOOoiExxeWZjjFs+sO7smAO0A5oA64HOdudyZLsA6A1sctr2DPCA4/kDwNOO56OALwAB+gErbMrcCujteB4ObMNaI9Ytczs+b5jjeSCwwpHjPeAax/apwB2O53cCUx3PrwH+a+P3x++B2cBnjtdunRnIBWJqbXPL7wunfG8CtzmeNwEi3T2zU3Z/YB/Q1tWZbfui6vFF9wcWOL1+EHjQ7lxOeZJqFfpWoJXjeStgq+P5q8CEk+1nc/5PgGGekBsIAdZgLX14EAio/T2CdXvn/o7nAY79xIasCcA3wMXAZ47/Id0988kK3W2/L7AW0NlR+9/KnTPXynkJ8ENDZHbnIZeTrWXa2qYs9RFnjNnreL4PiHM8d7uvw/GrfS+so163ze0YulgHHAC+wvqN7YgxpuokmX6xri1wYl3bxvYC8P+AGsfraNw/swEWishqsdb9BTf+vgCSgQLgdcfQ1gwRCcW9Mzu7BnjX8dylmd250D2WsX6kuuV8UBEJAz4Aphhjjjq/5265jTHVxpieWEe9acC59iY6PRG5FDhgjFltd5YzNNAY0xsYCdwlIhc4v+lu3xdYv830Bl4xxvQCjmMNV/zEDTMD4Dh/MgZ4v/Z7rsjszoVen7VM3cl+EWkF4PjzgGO723wdIhKIVebvGGM+dGx2+9zGmCPAIqzhikix1q2tnckd1rUdAIwRkVxgDtawy79w78wYY/Idfx4APsL64enO3xd5QJ4xZoXj9VysgnfnzCeMBNYYY/Y7Xrs0szsXen3WMnUnzuuq3oQ1Rn1i+42Os9b9gCKnX7EajYgI1lKBW4wxzzm95Za5RSRWRCIdz5tijfdvwSr28afIa+u6tsaYB40xCcaYJKzv12+NMdfhxplFJFREwk88xxrf3YSbfl8AGGP2AbtFpKNj0xBgsztndjKBn4dbwNWZ7ToxUM+TB6OwZmPkAH+2O49TrneBvUAl1tHCrVhjn98AWcDXQJRjXwFecnwNG4FUmzIPxPp1bgOwzvEY5a65ge7AWkfeTcDDju3tgJVANtavrUGO7cGO19mO99vZ/D0ymJ9nubhtZke29Y5Hxon/z9z1+8Ipd08g3fH98THQ3AMyh2L9BhbhtM2lmfXSf6WU8hLuPOSilFLqDGihK6WUl9BCV0opL6GFrpRSXkILXSmlvIQWulJKeQktdKWU8hL/H0snHKKpdbqCAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plotting\n",
    "\n",
    "df_scores = pd.DataFrame({\n",
    "   'train': scores_train,\n",
    "   'test': scores_test\n",
    "   }, index=num_latent_feats)\n",
    "\n",
    "df_scores.plot.line()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`6.` Use the cell below to comment on the results you found in the previous question. Given the circumstances of your results, discuss what you might do to determine if the recommendations you make with any of the above recommendation systems are an improvement to how users currently find articles? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Intepreting the Result:\n",
    "- As we can see, while the k (the number of latent factors) increases, the accuracy improves, the amount of data points benefit the model, the latent factors/affinity gets more accurate.\n",
    "- In contrast, in the test set predict simulation, accuracy worsens when the number of latent factors increases. Possibly due to a very limited of data points (20 instances only) in the test set. The more latent factors, the more overfitted, making it even less predictable. (This is because the most front latent factors weight more, with respect  to the sigma values order. All latent factors are trained with large train set, the following factors (less weights) would confuse the test set (small set - 20 instances), as a result, accuracy keep decreasing for test set.)\n",
    "- To see the effectiveness of our recommendation, we could set up A/B testing to measure the outcome. For example, set up evaluation metrics, measure user engagement time, sessions timeline, the click-through rate of recommended items, survey how users like our recommendations, etc. By aggregating the statistics, we could inferent the actual performance. With t-test, check p-values that make sure if users prefer our recommendation is not by random chance. Once we got actual performances stats, we could then continue to adjust/improve our algorithms."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='conclusions'></a>\n",
    "### Extras\n",
    "Using your workbook, you could now save your recommendations for each user, develop a class to make new predictions and update your results, and make a flask app to deploy your results.  These tasks are beyond what is required for this project.  However, from what you learned in the lessons, you certainly capable of taking these tasks on to improve upon your work here!\n",
    "\n",
    "\n",
    "## Conclusion\n",
    "\n",
    "> Congratulations!  You have reached the end of the Recommendations with IBM project! \n",
    "\n",
    "> **Tip**: Once you are satisfied with your work here, check over your report to make sure that it is satisfies all the areas of the [rubric](https://review.udacity.com/#!/rubrics/2322/view). You should also probably remove all of the \"Tips\" like this one so that the presentation is as polished as possible.\n",
    "\n",
    "\n",
    "## Directions to Submit\n",
    "\n",
    "> Before you submit your project, you need to create a .html or .pdf version of this notebook in the workspace here. To do that, run the code cell below. If it worked correctly, you should get a return code of 0, and you should see the generated .html file in the workspace directory (click on the orange Jupyter icon in the upper left).\n",
    "\n",
    "> Alternatively, you can download this report as .html via the **File** > **Download as** submenu, and then manually upload it into the workspace directory by clicking on the orange Jupyter icon in the upper left, then using the Upload button.\n",
    "\n",
    "> Once you've done this, you can submit your project by clicking on the \"Submit Project\" button in the lower right here. This will create and submit a zip file with this .ipynb doc and the .html or .pdf version you created. Congratulations! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}