00_01-preprocess_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e401729b-a42a-4681-a868-a53a1c764dce",
   "metadata": {},
   "source": [
    "## Preprocess datasets\n",
    "\n",
    "This notebook preprocess the datasets used in this paper.  We perform preprocessing on the following datasets\n",
    "\n",
    "\n",
    "1. Movielens20m\n",
    "2. KuaiRec\n",
    "3. Goodreads\n",
    "\n",
    "\n",
    "The results can be found in the datasets folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "00037dbe-a13a-4958-8bcb-7d93ad75d42e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "623ee6de-64c9-4e72-8f73-11a6c7331e80",
   "metadata": {},
   "source": [
    "## 01 - Preprocessing MovieLens\n",
    "\n",
    "\n",
    "This step is meant for comparison with Steck base paper\n",
    "\n",
    "steps:\n",
    "\n",
    "1. Consider only movies with rating >= 4\n",
    "2. Remove movies with no gender\n",
    "3. Remove movies with less than 10 interactions\n",
    "4. Remove movies without any user rating (this is done by simply doing a inner join between movie and rating df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb4bf1f2-9065-4b87-9a60-56ca4fe9a8f4",
   "metadata": {},
   "source": [
    "ten interactions and users with less than 190 movies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "3dc0e7d9-9d24-4920-9111-585a9d87037e",
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocess_rating_df(ratings, sample_size=300_000):\n",
    "    \"\"\"\n",
    "        ins:\n",
    "            ratings: pd.DataFrame\n",
    "        outs:\n",
    "            preprocessed_ratigs: pd.DataFrame\n",
    "\n",
    "        -------------------------------------\n",
    "        Preprocess the ratings df following the procedures suggested in Steck (2018).\n",
    "        The following steps are applied:\n",
    "        1. Consider only movies with rating >= 4\n",
    "        2. Remove movies with less than 10 interactions (we'll consider movies with move than\n",
    "            10 reviews)\n",
    "    \"\"\"\n",
    "\n",
    "    over_4_df = ratings[ratings[\"rating\"] >= 4.0]\n",
    "    num_interactions = over_4_df.groupby(\"movieId\").size().reset_index(name=\"size\")\n",
    "    to_remove = list(num_interactions[num_interactions[\"size\"] <= 10].movieId.values)\n",
    "    df = over_4_df[~over_4_df[\"movieId\"].isin(to_remove)]\n",
    "\n",
    "    if sample_size is not None:\n",
    "        return df.sample(sample_size)\n",
    "    return dfs\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "5097a53a-ba30-40c1-be69-92537ce736fe",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2024-11-07 19:14:36--  https://files.grouplens.org/datasets/movielens/ml-20m.zip\n",
      "Resolvendo files.grouplens.org (files.grouplens.org)... 128.101.65.152\n",
      "Conectando-se a files.grouplens.org (files.grouplens.org)|128.101.65.152|:443... conectado.\n",
      "A requisição HTTP foi enviada, aguardando resposta... 200 OK\n",
      "Tamanho: 198702078 (189M) [application/zip]\n",
      "Salvando em: ‘ml-20m.zip’\n",
      "\n",
      "ml-20m.zip          100%[===================>] 189,50M  9,95MB/s    em 31s     \n",
      "\n",
      "2024-11-07 19:15:11 (6,05 MB/s) - ‘ml-20m.zip’ salvo [198702078/198702078]\n",
      "\n",
      "Archive:  ml-20m.zip\n",
      "   creating: ml-20m/\n",
      "  inflating: ml-20m/genome-scores.csv  \n",
      "  inflating: ml-20m/genome-tags.csv  \n",
      "  inflating: ml-20m/links.csv        \n",
      "  inflating: ml-20m/movies.csv       \n",
      "  inflating: ml-20m/ratings.csv      \n",
      "  inflating: ml-20m/README.txt       \n",
      "  inflating: ml-20m/tags.csv         \n"
     ]
    }
   ],
   "source": [
    "!wget https://files.grouplens.org/datasets/movielens/ml-20m.zip\n",
    "!unzip ml-20m.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "a3d53b32-d077-4a46-a730-195833b93b89",
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm ml-20m/links.csv\n",
    "!rm ml-20m/tags.csv\n",
    "!rm ml-20m/genome-scores.csv\n",
    "!rm ml-20m/genome-tags.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "aa4baed5-ff54-4622-af18-bda01ef04a5c",
   "metadata": {},
   "outputs": [],
   "source": [
    "movies = pd.read_csv(\"ml-20m/movies.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "178043e9-027d-4bc6-8ee0-74e86966ce91",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movieId</th>\n",
       "      <th>title</th>\n",
       "      <th>genres</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Toy Story (1995)</td>\n",
       "      <td>Adventure|Animation|Children|Comedy|Fantasy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>Jumanji (1995)</td>\n",
       "      <td>Adventure|Children|Fantasy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>Grumpier Old Men (1995)</td>\n",
       "      <td>Comedy|Romance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Waiting to Exhale (1995)</td>\n",
       "      <td>Comedy|Drama|Romance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>Father of the Bride Part II (1995)</td>\n",
       "      <td>Comedy</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   movieId                               title  \\\n",
       "0        1                    Toy Story (1995)   \n",
       "1        2                      Jumanji (1995)   \n",
       "2        3             Grumpier Old Men (1995)   \n",
       "3        4            Waiting to Exhale (1995)   \n",
       "4        5  Father of the Bride Part II (1995)   \n",
       "\n",
       "                                        genres  \n",
       "0  Adventure|Animation|Children|Comedy|Fantasy  \n",
       "1                   Adventure|Children|Fantasy  \n",
       "2                               Comedy|Romance  \n",
       "3                         Comedy|Drama|Romance  \n",
       "4                                       Comedy  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "d0cf1757-c72b-44da-8a74-68aa0cc1ab2e",
   "metadata": {},
   "outputs": [],
   "source": [
    "ratings = pd.read_csv(\"ml-20m/ratings.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "b7e75e91-5559-4609-bb64-cf1ee407281f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1112486027</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>29</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1112484676</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>32</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1112484819</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>47</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1112484727</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1112484580</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20000258</th>\n",
       "      <td>138493</td>\n",
       "      <td>68954</td>\n",
       "      <td>4.5</td>\n",
       "      <td>1258126920</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20000259</th>\n",
       "      <td>138493</td>\n",
       "      <td>69526</td>\n",
       "      <td>4.5</td>\n",
       "      <td>1259865108</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20000260</th>\n",
       "      <td>138493</td>\n",
       "      <td>69644</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1260209457</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20000261</th>\n",
       "      <td>138493</td>\n",
       "      <td>70286</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1258126944</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20000262</th>\n",
       "      <td>138493</td>\n",
       "      <td>71619</td>\n",
       "      <td>2.5</td>\n",
       "      <td>1255811136</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>20000263 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          userId  movieId  rating   timestamp\n",
       "0              1        2     3.5  1112486027\n",
       "1              1       29     3.5  1112484676\n",
       "2              1       32     3.5  1112484819\n",
       "3              1       47     3.5  1112484727\n",
       "4              1       50     3.5  1112484580\n",
       "...          ...      ...     ...         ...\n",
       "20000258  138493    68954     4.5  1258126920\n",
       "20000259  138493    69526     4.5  1259865108\n",
       "20000260  138493    69644     3.0  1260209457\n",
       "20000261  138493    70286     5.0  1258126944\n",
       "20000262  138493    71619     2.5  1255811136\n",
       "\n",
       "[20000263 rows x 4 columns]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "0198371f-a828-454c-ba5e-6afff2058a35",
   "metadata": {},
   "outputs": [],
   "source": [
    "filtered_ratings = preprocess_rating_df(ratings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "10315930-99d7-4e0a-bd69-16f0df6ec785",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>11053637</th>\n",
       "      <td>76414</td>\n",
       "      <td>704</td>\n",
       "      <td>5.0</td>\n",
       "      <td>843742726</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12827145</th>\n",
       "      <td>88641</td>\n",
       "      <td>4022</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1058636265</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19175697</th>\n",
       "      <td>132652</td>\n",
       "      <td>778</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1112551957</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11192242</th>\n",
       "      <td>77297</td>\n",
       "      <td>908</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1414671763</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5589270</th>\n",
       "      <td>38429</td>\n",
       "      <td>7438</td>\n",
       "      <td>4.5</td>\n",
       "      <td>1352481230</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1085726</th>\n",
       "      <td>7398</td>\n",
       "      <td>50</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1417206076</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17597631</th>\n",
       "      <td>121725</td>\n",
       "      <td>593</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1223982340</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6414965</th>\n",
       "      <td>44055</td>\n",
       "      <td>6552</td>\n",
       "      <td>4.5</td>\n",
       "      <td>1179697507</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10500314</th>\n",
       "      <td>72652</td>\n",
       "      <td>3147</td>\n",
       "      <td>4.5</td>\n",
       "      <td>1115027905</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14547160</th>\n",
       "      <td>100492</td>\n",
       "      <td>4509</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1254364689</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>300000 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          userId  movieId  rating   timestamp\n",
       "11053637   76414      704     5.0   843742726\n",
       "12827145   88641     4022     4.0  1058636265\n",
       "19175697  132652      778     5.0  1112551957\n",
       "11192242   77297      908     4.0  1414671763\n",
       "5589270    38429     7438     4.5  1352481230\n",
       "...          ...      ...     ...         ...\n",
       "1085726     7398       50     5.0  1417206076\n",
       "17597631  121725      593     4.0  1223982340\n",
       "6414965    44055     6552     4.5  1179697507\n",
       "10500314   72652     3147     4.5  1115027905\n",
       "14547160  100492     4509     4.0  1254364689\n",
       "\n",
       "[300000 rows x 4 columns]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filtered_ratings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "c3aea84f-4126-4d4f-a725-6acd12ebfb36",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = movies.merge(filtered_ratings, on=\"movieId\").drop(columns=[\"title\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "c21aae37-fa59-4f96-bc42-ad775059f92d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movieId</th>\n",
       "      <th>genres</th>\n",
       "      <th>userId</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Adventure|Animation|Children|Comedy|Fantasy</td>\n",
       "      <td>136670</td>\n",
       "      <td>4.0</td>\n",
       "      <td>862652804</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Adventure|Animation|Children|Comedy|Fantasy</td>\n",
       "      <td>83076</td>\n",
       "      <td>4.5</td>\n",
       "      <td>1234146518</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>Adventure|Animation|Children|Comedy|Fantasy</td>\n",
       "      <td>42831</td>\n",
       "      <td>4.0</td>\n",
       "      <td>979348139</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>Adventure|Animation|Children|Comedy|Fantasy</td>\n",
       "      <td>66181</td>\n",
       "      <td>5.0</td>\n",
       "      <td>939675916</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>Adventure|Animation|Children|Comedy|Fantasy</td>\n",
       "      <td>2758</td>\n",
       "      <td>4.0</td>\n",
       "      <td>833215908</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>299995</th>\n",
       "      <td>119145</td>\n",
       "      <td>Action|Adventure|Comedy|Crime</td>\n",
       "      <td>84136</td>\n",
       "      <td>4.5</td>\n",
       "      <td>1424865002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>299996</th>\n",
       "      <td>125916</td>\n",
       "      <td>Drama</td>\n",
       "      <td>24646</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1424274393</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>299997</th>\n",
       "      <td>127098</td>\n",
       "      <td>Comedy</td>\n",
       "      <td>69470</td>\n",
       "      <td>4.5</td>\n",
       "      <td>1422828040</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>299998</th>\n",
       "      <td>127098</td>\n",
       "      <td>Comedy</td>\n",
       "      <td>136684</td>\n",
       "      <td>4.5</td>\n",
       "      <td>1424133121</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>299999</th>\n",
       "      <td>129354</td>\n",
       "      <td>Comedy|Crime|Drama|Romance</td>\n",
       "      <td>116204</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1425718311</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>300000 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        movieId                                       genres  userId  rating  \\\n",
       "0             1  Adventure|Animation|Children|Comedy|Fantasy  136670     4.0   \n",
       "1             1  Adventure|Animation|Children|Comedy|Fantasy   83076     4.5   \n",
       "2             1  Adventure|Animation|Children|Comedy|Fantasy   42831     4.0   \n",
       "3             1  Adventure|Animation|Children|Comedy|Fantasy   66181     5.0   \n",
       "4             1  Adventure|Animation|Children|Comedy|Fantasy    2758     4.0   \n",
       "...         ...                                          ...     ...     ...   \n",
       "299995   119145                Action|Adventure|Comedy|Crime   84136     4.5   \n",
       "299996   125916                                        Drama   24646     4.0   \n",
       "299997   127098                                       Comedy   69470     4.5   \n",
       "299998   127098                                       Comedy  136684     4.5   \n",
       "299999   129354                   Comedy|Crime|Drama|Romance  116204     4.0   \n",
       "\n",
       "         timestamp  \n",
       "0        862652804  \n",
       "1       1234146518  \n",
       "2        979348139  \n",
       "3        939675916  \n",
       "4        833215908  \n",
       "...            ...  \n",
       "299995  1424865002  \n",
       "299996  1424274393  \n",
       "299997  1422828040  \n",
       "299998  1424133121  \n",
       "299999  1425718311  \n",
       "\n",
       "[300000 rows x 5 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "ad5f13db-f175-4981-a894-b6b70c5a7614",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_csv(\"ml20m_filtered_300k_sample.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d11aef6-722d-4ee8-8f32-e06ff12dde9b",
   "metadata": {},
   "source": [
    "## 02- Preprocessing movielens for comparison with saciloti et al\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "e7c75e3c-5fc4-46ce-bc4d-0a1fc939789e",
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocess_rating_for_sacilotti(ratings, sample_size=300_000):\n",
    "    \"\"\"\n",
    "        ins:\n",
    "            ratings: pd.DataFrame\n",
    "        outs:\n",
    "            preprocessed_ratigs: pd.DataFrame\n",
    "\n",
    "        -------------------------------------\n",
    "        Preprocess the ratings df following the procedures suggested in Steck (2018).\n",
    "        The following steps are applied:\n",
    "\n",
    "        1. Remove movies with less than 10 interactions (we'll consider movies with move than\n",
    "            10 reviews)\n",
    "        2. Consider only users who have interacted with over 190 movies.\n",
    "    \"\"\"\n",
    "    num_interactions = ratings.groupby(\"movieId\").size().reset_index(name=\"size\")\n",
    "    num_movies = ratings.groupby(\"userId\").size().reset_index(name=\"num_movies\")\n",
    "    m_to_remove = list(num_interactions[num_interactions[\"size\"] <= 10].movieId.values)\n",
    "    u_to_remove = list(num_movies[num_movies[\"num_movies\"] >= 190].userId.values)\n",
    "    \n",
    "    df =ratings[~ratings[\"movieId\"].isin(m_to_remove)]\n",
    "    df = df[~df[\"userId\"].isin(u_to_remove)]\n",
    "    if sample_size is not None:\n",
    "        return df.sample(sample_size)\n",
    "    return df\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "ffb18690-91e5-46d9-9da2-b0214e3b7f44",
   "metadata": {},
   "outputs": [],
   "source": [
    "filtered_ratings_s = preprocess_rating_for_sacilotti(ratings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "8ec89cdf-6f52-4789-b584-0a6071afc319",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7215026</th>\n",
       "      <td>49783</td>\n",
       "      <td>1591</td>\n",
       "      <td>3.0</td>\n",
       "      <td>919075246</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2363579</th>\n",
       "      <td>15987</td>\n",
       "      <td>648</td>\n",
       "      <td>3.0</td>\n",
       "      <td>848432405</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6509726</th>\n",
       "      <td>44724</td>\n",
       "      <td>2571</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1301528455</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1442246</th>\n",
       "      <td>9756</td>\n",
       "      <td>6370</td>\n",
       "      <td>4.5</td>\n",
       "      <td>1295733998</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10643158</th>\n",
       "      <td>73641</td>\n",
       "      <td>586</td>\n",
       "      <td>3.0</td>\n",
       "      <td>838430934</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19391246</th>\n",
       "      <td>134228</td>\n",
       "      <td>4128</td>\n",
       "      <td>4.0</td>\n",
       "      <td>982025349</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19048307</th>\n",
       "      <td>131804</td>\n",
       "      <td>5401</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1122202974</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10289899</th>\n",
       "      <td>71173</td>\n",
       "      <td>5452</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1081865905</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4435118</th>\n",
       "      <td>30314</td>\n",
       "      <td>319</td>\n",
       "      <td>5.0</td>\n",
       "      <td>832206541</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16259226</th>\n",
       "      <td>112527</td>\n",
       "      <td>2571</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1202672474</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>300000 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          userId  movieId  rating   timestamp\n",
       "7215026    49783     1591     3.0   919075246\n",
       "2363579    15987      648     3.0   848432405\n",
       "6509726    44724     2571     5.0  1301528455\n",
       "1442246     9756     6370     4.5  1295733998\n",
       "10643158   73641      586     3.0   838430934\n",
       "...          ...      ...     ...         ...\n",
       "19391246  134228     4128     4.0   982025349\n",
       "19048307  131804     5401     4.0  1122202974\n",
       "10289899   71173     5452     3.0  1081865905\n",
       "4435118    30314      319     5.0   832206541\n",
       "16259226  112527     2571     3.5  1202672474\n",
       "\n",
       "[300000 rows x 4 columns]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filtered_ratings_s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "e7c845b1-5927-4ad0-aa7a-475008c16e83",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_s = movies.merge(filtered_ratings_s, on=\"movieId\").drop(columns=[\"title\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "e5912619-0ff9-4d36-86f1-1ade1f6339b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_s.to_csv(\"ml20m_sacilotti_filtered_300k_sample.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "700fd0ec-7297-4b09-8338-99629b31a8ff",
   "metadata": {},
   "source": [
    "## 03 - Preprocessing kuairec dataset\n",
    "\n",
    "\n",
    "Downloaded from https://drive.google.com/file/d/1qe5hOSBxzIuxBb1G_Ih5X-O65QElollE/view on 30/10/2024\n",
    "\n",
    "\n",
    "Apparently, the authors of Lin et al 2024 used the small_matrix.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "8b0eedc0-8f50-4346-8269-50c813a0dcbf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Archive:  KuaiRec.zip\n",
      "   creating: KuaiRec 2.0/\n",
      "  inflating: KuaiRec 2.0/LICENSE     \n",
      "  inflating: KuaiRec 2.0/Statistics_KuaiRec.ipynb  \n",
      "   creating: KuaiRec 2.0/data/\n",
      "  inflating: KuaiRec 2.0/data/big_matrix.csv  \n",
      "  inflating: KuaiRec 2.0/data/item_categories.csv  \n",
      "  inflating: KuaiRec 2.0/data/item_daily_features.csv  \n",
      "  inflating: KuaiRec 2.0/data/kuairec_caption_category.csv  \n",
      "  inflating: KuaiRec 2.0/data/small_matrix.csv  \n",
      "  inflating: KuaiRec 2.0/data/social_network.csv  \n",
      "  inflating: KuaiRec 2.0/data/user_features.csv  \n",
      "   creating: KuaiRec 2.0/figs/\n",
      "  inflating: KuaiRec 2.0/figs/KuaiRec.png  \n",
      "  inflating: KuaiRec 2.0/figs/colab-badge.svg  \n",
      "  inflating: KuaiRec 2.0/loaddata.py  \n"
     ]
    }
   ],
   "source": [
    "!unzip KuaiRec.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "76552386-0b07-4490-8d62-70761f47e15b",
   "metadata": {},
   "outputs": [],
   "source": [
    "kuairec = pd.read_csv(\"KuaiRec 2.0/data/small_matrix.csv\").dropna(subset=[\"time\", \"timestamp\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "340aa972-3609-4284-9e71-6d98596b4a46",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>video_id</th>\n",
       "      <th>play_duration</th>\n",
       "      <th>video_duration</th>\n",
       "      <th>time</th>\n",
       "      <th>date</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>watch_ratio</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>14</td>\n",
       "      <td>148</td>\n",
       "      <td>4381</td>\n",
       "      <td>6067</td>\n",
       "      <td>2020-07-05 05:27:48.378</td>\n",
       "      <td>20200705.0</td>\n",
       "      <td>1.593898e+09</td>\n",
       "      <td>0.722103</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>14</td>\n",
       "      <td>183</td>\n",
       "      <td>11635</td>\n",
       "      <td>6100</td>\n",
       "      <td>2020-07-05 05:28:00.057</td>\n",
       "      <td>20200705.0</td>\n",
       "      <td>1.593898e+09</td>\n",
       "      <td>1.907377</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>14</td>\n",
       "      <td>3649</td>\n",
       "      <td>22422</td>\n",
       "      <td>10867</td>\n",
       "      <td>2020-07-05 05:29:09.479</td>\n",
       "      <td>20200705.0</td>\n",
       "      <td>1.593898e+09</td>\n",
       "      <td>2.063311</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>14</td>\n",
       "      <td>5262</td>\n",
       "      <td>4479</td>\n",
       "      <td>7908</td>\n",
       "      <td>2020-07-05 05:30:43.285</td>\n",
       "      <td>20200705.0</td>\n",
       "      <td>1.593898e+09</td>\n",
       "      <td>0.566388</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>14</td>\n",
       "      <td>8234</td>\n",
       "      <td>4602</td>\n",
       "      <td>11000</td>\n",
       "      <td>2020-07-05 05:35:43.459</td>\n",
       "      <td>20200705.0</td>\n",
       "      <td>1.593899e+09</td>\n",
       "      <td>0.418364</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4676370</th>\n",
       "      <td>7162</td>\n",
       "      <td>9177</td>\n",
       "      <td>5315</td>\n",
       "      <td>37205</td>\n",
       "      <td>2020-09-01 20:06:35.984</td>\n",
       "      <td>20200901.0</td>\n",
       "      <td>1.598962e+09</td>\n",
       "      <td>0.142857</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4676371</th>\n",
       "      <td>7162</td>\n",
       "      <td>4987</td>\n",
       "      <td>10085</td>\n",
       "      <td>8167</td>\n",
       "      <td>2020-09-02 14:44:51.342</td>\n",
       "      <td>20200902.0</td>\n",
       "      <td>1.599029e+09</td>\n",
       "      <td>1.234848</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4676372</th>\n",
       "      <td>7162</td>\n",
       "      <td>7988</td>\n",
       "      <td>50523</td>\n",
       "      <td>49319</td>\n",
       "      <td>2020-09-03 08:45:01.474</td>\n",
       "      <td>20200903.0</td>\n",
       "      <td>1.599094e+09</td>\n",
       "      <td>1.024412</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4676373</th>\n",
       "      <td>7162</td>\n",
       "      <td>6533</td>\n",
       "      <td>2190</td>\n",
       "      <td>8000</td>\n",
       "      <td>2020-09-04 22:56:32.021</td>\n",
       "      <td>20200904.0</td>\n",
       "      <td>1.599231e+09</td>\n",
       "      <td>0.273750</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4676374</th>\n",
       "      <td>7162</td>\n",
       "      <td>6523</td>\n",
       "      <td>11909</td>\n",
       "      <td>7255</td>\n",
       "      <td>2020-09-05 00:32:09.154</td>\n",
       "      <td>20200905.0</td>\n",
       "      <td>1.599237e+09</td>\n",
       "      <td>1.641489</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>4494578 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         user_id  video_id  play_duration  video_duration  \\\n",
       "0             14       148           4381            6067   \n",
       "1             14       183          11635            6100   \n",
       "2             14      3649          22422           10867   \n",
       "3             14      5262           4479            7908   \n",
       "4             14      8234           4602           11000   \n",
       "...          ...       ...            ...             ...   \n",
       "4676370     7162      9177           5315           37205   \n",
       "4676371     7162      4987          10085            8167   \n",
       "4676372     7162      7988          50523           49319   \n",
       "4676373     7162      6533           2190            8000   \n",
       "4676374     7162      6523          11909            7255   \n",
       "\n",
       "                            time        date     timestamp  watch_ratio  \n",
       "0        2020-07-05 05:27:48.378  20200705.0  1.593898e+09     0.722103  \n",
       "1        2020-07-05 05:28:00.057  20200705.0  1.593898e+09     1.907377  \n",
       "2        2020-07-05 05:29:09.479  20200705.0  1.593898e+09     2.063311  \n",
       "3        2020-07-05 05:30:43.285  20200705.0  1.593898e+09     0.566388  \n",
       "4        2020-07-05 05:35:43.459  20200705.0  1.593899e+09     0.418364  \n",
       "...                          ...         ...           ...          ...  \n",
       "4676370  2020-09-01 20:06:35.984  20200901.0  1.598962e+09     0.142857  \n",
       "4676371  2020-09-02 14:44:51.342  20200902.0  1.599029e+09     1.234848  \n",
       "4676372  2020-09-03 08:45:01.474  20200903.0  1.599094e+09     1.024412  \n",
       "4676373  2020-09-04 22:56:32.021  20200904.0  1.599231e+09     0.273750  \n",
       "4676374  2020-09-05 00:32:09.154  20200905.0  1.599237e+09     1.641489  \n",
       "\n",
       "[4494578 rows x 8 columns]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "kuairec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "df194a0c-7d48-42df-baff-e653f4b4f170",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    4.494578e+06\n",
       "mean     9.113757e-01\n",
       "std      1.356487e+00\n",
       "min      0.000000e+00\n",
       "25%      4.735875e-01\n",
       "50%      7.745146e-01\n",
       "75%      1.124976e+00\n",
       "max      5.715214e+02\n",
       "Name: watch_ratio, dtype: float64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "kuairec[\"watch_ratio\"].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d80c33a-7007-4acc-9d7d-4b843ab33d84",
   "metadata": {},
   "source": [
    "Notice that watch_ratio can be larger than one. That happens because watch_ratio is defined as (=play_duration/video_duration). So if a user repeatedely sees a video, then this can be larger than 1.\n",
    "\n",
    "In order to compare with Lin et al 2024, we'll label watch_ratio >= 0.9 as positive examples and watch_ratio < 0.9 as negative"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "f35fe291-ada4-4372-a169-3e8287fd2707",
   "metadata": {},
   "outputs": [],
   "source": [
    "kuairec['label'] = kuairec['watch_ratio'].apply(lambda x: 1 if x > 0.9 else 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "92708df7-0047-4a96-80e1-6be61b3504b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "kuairec.to_csv(\"datasets/kuairec_filtered.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1717f98-1f81-4a62-8294-bb1981b27c5a",
   "metadata": {},
   "source": [
    "## 04 - Preprocessing goodreads"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "02fb5dcc-4e3e-4ae8-b884-64f4478eefc9",
   "metadata": {},
   "outputs": [],
   "source": [
    "goodreads = pd.read_json('path_to_file.json.gz', lines=True, compression='gzip')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "recsysintro",
   "language": "python",
   "name": "recsysintro"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}