cap4_text_processing.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "cap4_text_processing.ipynb",
      "provenance": [],
      "authorship_tag": "ABX9TyOoV3Dl28q4jZQx9oBdP+J7",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/viniciusrpb/cic0193_machinelearning/blob/main/cap4_text_processing.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DGsf0RI4D5mu"
      },
      "source": [
        "#Capítulo 4\n",
        "\n",
        "## Processamento de Textos\n",
        "\n",
        "Neste notebook, vamos abordar a extração de características e pré-processamento de textos."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BnZkgWLVpMvP"
      },
      "source": [
        "import re\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import math\n",
        "import nltk\n",
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "from nltk.stem.porter import PorterStemmer\n",
        "from collections import Counter"
      ],
      "execution_count": 251,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ESF_Pz-ZETCc"
      },
      "source": [
        "## Visão geral do processo de pré-processamento e extração de características\n",
        "\n",
        "\n",
        "\n",
        "Vamos criar uma sentença:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IB2sOuzopOUx"
      },
      "source": [
        "sentence = \"O Athletico Paranaense eh o melhor time do pais dentre 54. Dito isso, nao existem questionamentos ou outro time que ira vencer #paz.\""
      ],
      "execution_count": 252,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aR7dDeY6EaAU"
      },
      "source": [
        "A partir da sentença, vamos extrair seus tokens:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LXU5jlNOpbtn",
        "outputId": "692150c8-7e43-411e-e6e8-e315328d6df3"
      },
      "source": [
        "tokens = str.split(sentence)\n",
        "\n",
        "tokens"
      ],
      "execution_count": 253,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['O',\n",
              " 'Athletico',\n",
              " 'Paranaense',\n",
              " 'eh',\n",
              " 'o',\n",
              " 'melhor',\n",
              " 'time',\n",
              " 'do',\n",
              " 'pais',\n",
              " 'dentre',\n",
              " '54.',\n",
              " 'Dito',\n",
              " 'isso,',\n",
              " 'nao',\n",
              " 'existem',\n",
              " 'questionamentos',\n",
              " 'ou',\n",
              " 'outro',\n",
              " 'time',\n",
              " 'que',\n",
              " 'ira',\n",
              " 'vencer',\n",
              " '#paz.']"
            ]
          },
          "metadata": {},
          "execution_count": 253
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rNBDvU-JHKYB"
      },
      "source": [
        "Agora vamos criar um vocabulário com base no documento. Para isso, precisamos ordenar alfabeticamente os tokens:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QgO3wptspm6M",
        "outputId": "0925da07-a121-4dd2-d2b5-811cb83b03ab"
      },
      "source": [
        "vocab = sorted(set(tokens))\n",
        "vocab"
      ],
      "execution_count": 254,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['#paz.',\n",
              " '54.',\n",
              " 'Athletico',\n",
              " 'Dito',\n",
              " 'O',\n",
              " 'Paranaense',\n",
              " 'dentre',\n",
              " 'do',\n",
              " 'eh',\n",
              " 'existem',\n",
              " 'ira',\n",
              " 'isso,',\n",
              " 'melhor',\n",
              " 'nao',\n",
              " 'o',\n",
              " 'ou',\n",
              " 'outro',\n",
              " 'pais',\n",
              " 'que',\n",
              " 'questionamentos',\n",
              " 'time',\n",
              " 'vencer']"
            ]
          },
          "metadata": {},
          "execution_count": 254
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SMbitGv1p_Uy"
      },
      "source": [
        "Agora vamos criar uma representação numérica dessa sentença baseada em one-hot encoding, em que as colunas se referem às palavras do vocabulário e as linhas aos tokens "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fY2_3r7ip6ib"
      },
      "source": [
        "num_tokens = len(tokens)\n",
        "\n",
        "vocab_size = len(vocab)\n",
        "\n",
        "onehot_encoding = np.zeros((num_tokens,vocab_size),int)"
      ],
      "execution_count": 255,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6XUs7uyrqfBB"
      },
      "source": [
        "for i,word in enumerate(tokens):\n",
        "  onehot_encoding[i,vocab.index(word)] = 1"
      ],
      "execution_count": 256,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wLOHfiGvq4E3",
        "outputId": "d9f4201d-b45d-46ea-a7c6-63778cf1d355"
      },
      "source": [
        "onehot_encoding"
      ],
      "execution_count": 257,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
              "       [0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
              "       [0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
              "       [0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
              "       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0],\n",
              "       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
              "       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0],\n",
              "       [0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
              "       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0],\n",
              "       [0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
              "       [0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
              "       [0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
              "       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
              "       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0],\n",
              "       [0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
              "       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0],\n",
              "       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0],\n",
              "       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0],\n",
              "       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0],\n",
              "       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0],\n",
              "       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
              "       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1],\n",
              "       [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]])"
            ]
          },
          "metadata": {},
          "execution_count": 257
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r6KFZsThgSQv"
      },
      "source": [
        "Colocando essa representação numérica em um DataFrame:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 758
        },
        "id": "D5T-NzFjrXRt",
        "outputId": "871628b0-d04a-4652-a104-5c0bcd35db48"
      },
      "source": [
        "pd.DataFrame(onehot_encoding,columns=vocab)"
      ],
      "execution_count": 258,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>#paz.</th>\n",
              "      <th>54.</th>\n",
              "      <th>Athletico</th>\n",
              "      <th>Dito</th>\n",
              "      <th>O</th>\n",
              "      <th>Paranaense</th>\n",
              "      <th>dentre</th>\n",
              "      <th>do</th>\n",
              "      <th>eh</th>\n",
              "      <th>existem</th>\n",
              "      <th>ira</th>\n",
              "      <th>isso,</th>\n",
              "      <th>melhor</th>\n",
              "      <th>nao</th>\n",
              "      <th>o</th>\n",
              "      <th>ou</th>\n",
              "      <th>outro</th>\n",
              "      <th>pais</th>\n",
              "      <th>que</th>\n",
              "      <th>questionamentos</th>\n",
              "      <th>time</th>\n",
              "      <th>vencer</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>20</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>21</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22</th>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "    #paz.  54.  Athletico  Dito  O  ...  pais  que  questionamentos  time  vencer\n",
              "0       0    0          0     0  1  ...     0    0                0     0       0\n",
              "1       0    0          1     0  0  ...     0    0                0     0       0\n",
              "2       0    0          0     0  0  ...     0    0                0     0       0\n",
              "3       0    0          0     0  0  ...     0    0                0     0       0\n",
              "4       0    0          0     0  0  ...     0    0                0     0       0\n",
              "5       0    0          0     0  0  ...     0    0                0     0       0\n",
              "6       0    0          0     0  0  ...     0    0                0     1       0\n",
              "7       0    0          0     0  0  ...     0    0                0     0       0\n",
              "8       0    0          0     0  0  ...     1    0                0     0       0\n",
              "9       0    0          0     0  0  ...     0    0                0     0       0\n",
              "10      0    1          0     0  0  ...     0    0                0     0       0\n",
              "11      0    0          0     1  0  ...     0    0                0     0       0\n",
              "12      0    0          0     0  0  ...     0    0                0     0       0\n",
              "13      0    0          0     0  0  ...     0    0                0     0       0\n",
              "14      0    0          0     0  0  ...     0    0                0     0       0\n",
              "15      0    0          0     0  0  ...     0    0                1     0       0\n",
              "16      0    0          0     0  0  ...     0    0                0     0       0\n",
              "17      0    0          0     0  0  ...     0    0                0     0       0\n",
              "18      0    0          0     0  0  ...     0    0                0     1       0\n",
              "19      0    0          0     0  0  ...     0    1                0     0       0\n",
              "20      0    0          0     0  0  ...     0    0                0     0       0\n",
              "21      0    0          0     0  0  ...     0    0                0     0       1\n",
              "22      1    0          0     0  0  ...     0    0                0     0       0\n",
              "\n",
              "[23 rows x 22 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 258
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TSj14ppKrl7G"
      },
      "source": [
        "## Pré-processamento\n",
        "\n",
        "Coloca todos os caracteres de um texto em minúsculo\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LBsbjPMossLX"
      },
      "source": [
        "lower_sentence = sentence.lower();"
      ],
      "execution_count": 259,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IewygdFBs3pu"
      },
      "source": [
        "Utilizamos expressões regulares para remover os símbolos de pontuação que podem estar acoplados às palavras:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zbasn0e6s3x4",
        "outputId": "c4011ee9-9772-46d2-98aa-9b33427c15ee"
      },
      "source": [
        "tokens = re.split(r'[-\\s.,;!#?@]+',lower_sentence)\n",
        "\n",
        "tokens"
      ],
      "execution_count": 260,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['o',\n",
              " 'athletico',\n",
              " 'paranaense',\n",
              " 'eh',\n",
              " 'o',\n",
              " 'melhor',\n",
              " 'time',\n",
              " 'do',\n",
              " 'pais',\n",
              " 'dentre',\n",
              " '54',\n",
              " 'dito',\n",
              " 'isso',\n",
              " 'nao',\n",
              " 'existem',\n",
              " 'questionamentos',\n",
              " 'ou',\n",
              " 'outro',\n",
              " 'time',\n",
              " 'que',\n",
              " 'ira',\n",
              " 'vencer',\n",
              " 'paz',\n",
              " '']"
            ]
          },
          "metadata": {},
          "execution_count": 260
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jWfdrHBjyYvF"
      },
      "source": [
        "Em seguida, removemos as stop words do texto, pois essas palavras não contribuem com informações relevantes na caracterização. Para isso, podemos definir uma lista de stop words denominada stop list:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ahTG4mVayY20"
      },
      "source": [
        "stop_list = ['eh','o','que','do','ou','se','e','para']"
      ],
      "execution_count": 261,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LyohH3_FytFD"
      },
      "source": [
        "Entretanto, incluir stop words uma a uma pode ser trabalhoso, apensar de ser apropriado dependendo da tarefa de aprendizado de máquina e a natureza do texto. Alternativamente, podemos fazer o download das stop words da língua Portuguesa utilizando a biblioteca nltk:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IuesbQGSytNy",
        "outputId": "429922f6-0b47-4b45-c49d-d77c53a606e4"
      },
      "source": [
        "nltk.download('stopwords')\n",
        "\n",
        "stop_list_nltk = nltk.corpus.stopwords.words('portuguese')"
      ],
      "execution_count": 262,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Package stopwords is already up-to-date!\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9j-TAz7Vy6-Z"
      },
      "source": [
        "Finalmente, geramos uma lista com as palavras remanescentes após a remoção de stop words:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5c4fL-Ljy7I5",
        "outputId": "0855e129-2335-4d6b-836d-ea330a904f22"
      },
      "source": [
        "important_words = [x for x in tokens if x not in stop_list_nltk]\n",
        "\n",
        "#important_words = [x for x in tokens if x not in stop_list]\n",
        "\n",
        "len(important_words)"
      ],
      "execution_count": 263,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "18"
            ]
          },
          "metadata": {},
          "execution_count": 263
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G3-V6zFC2m02"
      },
      "source": [
        "Mais um processamento adicional se trata da stemmização, em que tentaremos remover as terminações e flexão das palavras. Dentre os diversos Stemmers que podem ser encontrados para diferentes línguas, utilizamos a seguir o Stemmer de Porter:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Dlx7Hlqh2m88",
        "outputId": "d6439fb6-abe2-4ba0-f7e4-5a5f020b49aa"
      },
      "source": [
        "stemmer = PorterStemmer()\n",
        "\n",
        "tokens_final = [stemmer.stem(w).strip(\" \") for w in important_words]\n",
        "\n",
        "tokens_final"
      ],
      "execution_count": 264,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['athletico',\n",
              " 'paranaens',\n",
              " 'eh',\n",
              " 'melhor',\n",
              " 'time',\n",
              " 'pai',\n",
              " 'dentr',\n",
              " '54',\n",
              " 'dito',\n",
              " 'nao',\n",
              " 'existem',\n",
              " 'questionamento',\n",
              " 'outro',\n",
              " 'time',\n",
              " 'ira',\n",
              " 'vencer',\n",
              " 'paz',\n",
              " '']"
            ]
          },
          "metadata": {},
          "execution_count": 264
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a0JyR5CE4BH1"
      },
      "source": [
        "Finalmente, para facilitar o pré-processamento de vários textos, podemos criar uma função:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CgAxQbzu4Nfc"
      },
      "source": [
        "def preprocessing(text):\n",
        "\n",
        "  lower_sentence = text.lower();\n",
        "\n",
        "  tokens = re.split(r'[-\\s.,;!#?@]+',lower_sentence)\n",
        "\n",
        "  nltk.download('stopwords')\n",
        "\n",
        "  stop_list_nltk = nltk.corpus.stopwords.words('portuguese')\n",
        "\n",
        "  important_words = [x for x in tokens if x not in stop_list_nltk]\n",
        "\n",
        "  stemmer = PorterStemmer()\n",
        "\n",
        "  tokens_final = [stemmer.stem(w).strip(\" \") for w in important_words]\n",
        "\n",
        "  return tokens_final\n"
      ],
      "execution_count": 265,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "91dbffv33iVR"
      },
      "source": [
        "## Extração de características\n",
        "\n",
        "O objetivo dessa etapa consiste em transformar os documentos de textos de um corpus em vetores de características. Começaremos entendendo a abordagem de matriz de incidência binária, que foi introduzida na primeira seção desse notebook.\n",
        "\n",
        "Primeiramente, vamos montar um pequeno corpus, em que cada documento está associado com uma frase (string):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9b9xBlt636le"
      },
      "source": [
        "corpus = []\n",
        "\n",
        "corpus.append(\"O Athletico Paranaense eh o melhor time do pais dentre 54. Dito isso, nao existem questionamentos ou outro time que ira vencer #paz.\")\n",
        "corpus.append(\"O pais possui varios times e o questionamento eh se ha espaco para golfe\")\n",
        "corpus.append(\"Vasco eh time estranho, mas nao eh o melhor\")\n",
        "corpus.append(\"Vencer eh o que vale. Sera o melhor do pais\")"
      ],
      "execution_count": 266,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZFe9UTIy42CK"
      },
      "source": [
        "Em seguida, cada texto do corpus será tokenizado e pré-processado:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ocxRy-Il42LR",
        "outputId": "efca469c-9f46-4423-ce6b-b3fc8905050e"
      },
      "source": [
        "tokens = []\n",
        "\n",
        "for text in corpus:\n",
        "  tokens.append(preprocessing(text))"
      ],
      "execution_count": 267,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Package stopwords is already up-to-date!\n",
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Package stopwords is already up-to-date!\n",
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Package stopwords is already up-to-date!\n",
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Package stopwords is already up-to-date!\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fC6EPFEG5dLn"
      },
      "source": [
        "Agora vamos recriar a matriz de incidência binária para o corpus"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "O-Ct-6X45nq2",
        "outputId": "2ef0a270-8098-461e-95ed-85940b5cc91d"
      },
      "source": [
        "bow_binario = {}\n",
        "\n",
        "for i,doc_tokens in enumerate(tokens):\n",
        "  bow_binario['doc {}'.format(i)] = dict((token,1) for token in doc_tokens)\n",
        "\n",
        "bow_binario"
      ],
      "execution_count": 268,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'doc 0': {'': 1,\n",
              "  '54': 1,\n",
              "  'athletico': 1,\n",
              "  'dentr': 1,\n",
              "  'dito': 1,\n",
              "  'eh': 1,\n",
              "  'existem': 1,\n",
              "  'ira': 1,\n",
              "  'melhor': 1,\n",
              "  'nao': 1,\n",
              "  'outro': 1,\n",
              "  'pai': 1,\n",
              "  'paranaens': 1,\n",
              "  'paz': 1,\n",
              "  'questionamento': 1,\n",
              "  'time': 1,\n",
              "  'vencer': 1},\n",
              " 'doc 1': {'eh': 1,\n",
              "  'espaco': 1,\n",
              "  'golf': 1,\n",
              "  'ha': 1,\n",
              "  'pai': 1,\n",
              "  'possui': 1,\n",
              "  'questionamento': 1,\n",
              "  'time': 1,\n",
              "  'vario': 1},\n",
              " 'doc 2': {'eh': 1,\n",
              "  'estranho': 1,\n",
              "  'melhor': 1,\n",
              "  'nao': 1,\n",
              "  'time': 1,\n",
              "  'vasco': 1},\n",
              " 'doc 3': {'eh': 1, 'melhor': 1, 'pai': 1, 'sera': 1, 'vale': 1, 'vencer': 1}}"
            ]
          },
          "metadata": {},
          "execution_count": 268
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NG_Vqpr16Gb0"
      },
      "source": [
        "Colocamos a matriz de incidência binária em um data frame:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 259
        },
        "id": "vE7N1umm6Gj0",
        "outputId": "c5346c76-8775-4d2f-b4e6-93db0189cd79"
      },
      "source": [
        "df = pd.DataFrame.from_records(bow_binario).fillna(0).astype(int).T\n",
        "\n",
        "df"
      ],
      "execution_count": 269,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>athletico</th>\n",
              "      <th>paranaens</th>\n",
              "      <th>eh</th>\n",
              "      <th>melhor</th>\n",
              "      <th>time</th>\n",
              "      <th>pai</th>\n",
              "      <th>dentr</th>\n",
              "      <th>54</th>\n",
              "      <th>dito</th>\n",
              "      <th>nao</th>\n",
              "      <th>existem</th>\n",
              "      <th>questionamento</th>\n",
              "      <th>outro</th>\n",
              "      <th>ira</th>\n",
              "      <th>vencer</th>\n",
              "      <th>paz</th>\n",
              "      <th></th>\n",
              "      <th>possui</th>\n",
              "      <th>vario</th>\n",
              "      <th>ha</th>\n",
              "      <th>espaco</th>\n",
              "      <th>golf</th>\n",
              "      <th>vasco</th>\n",
              "      <th>estranho</th>\n",
              "      <th>vale</th>\n",
              "      <th>sera</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>doc 0</th>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>doc 1</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>doc 2</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>doc 3</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "       athletico  paranaens  eh  melhor  ...  vasco  estranho  vale  sera\n",
              "doc 0          1          1   1       1  ...      0         0     0     0\n",
              "doc 1          0          0   1       0  ...      0         0     0     0\n",
              "doc 2          0          0   1       1  ...      1         1     0     0\n",
              "doc 3          0          0   1       1  ...      0         0     1     1\n",
              "\n",
              "[4 rows x 26 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 269
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R4VX9Vz8GcFY"
      },
      "source": [
        "A matriz de incidência binária possui uma desvantagem: informações de frequência das palavras não são contabilizadas. Por isso, podemos alterar essa abordagem de BoW por matriz de incidência para uma matriz de contagens:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HzZ1ZqDCHJjt"
      },
      "source": [
        "bow_counter = {}\n",
        "\n",
        "for i,doc_tokens in enumerate(tokens):\n",
        "  \n",
        "  histograma = Counter(doc_tokens)\n",
        "  \n",
        "  bow_counter['doc {}'.format(i)] = histograma"
      ],
      "execution_count": 270,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DDigpQc4sgop",
        "outputId": "e6052cc0-be9a-4ab2-ef2f-8569af522266"
      },
      "source": [
        "bow_counter"
      ],
      "execution_count": 271,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'doc 0': Counter({'': 1,\n",
              "          '54': 1,\n",
              "          'athletico': 1,\n",
              "          'dentr': 1,\n",
              "          'dito': 1,\n",
              "          'eh': 1,\n",
              "          'existem': 1,\n",
              "          'ira': 1,\n",
              "          'melhor': 1,\n",
              "          'nao': 1,\n",
              "          'outro': 1,\n",
              "          'pai': 1,\n",
              "          'paranaens': 1,\n",
              "          'paz': 1,\n",
              "          'questionamento': 1,\n",
              "          'time': 2,\n",
              "          'vencer': 1}),\n",
              " 'doc 1': Counter({'eh': 1,\n",
              "          'espaco': 1,\n",
              "          'golf': 1,\n",
              "          'ha': 1,\n",
              "          'pai': 1,\n",
              "          'possui': 1,\n",
              "          'questionamento': 1,\n",
              "          'time': 1,\n",
              "          'vario': 1}),\n",
              " 'doc 2': Counter({'eh': 2,\n",
              "          'estranho': 1,\n",
              "          'melhor': 1,\n",
              "          'nao': 1,\n",
              "          'time': 1,\n",
              "          'vasco': 1}),\n",
              " 'doc 3': Counter({'eh': 1,\n",
              "          'melhor': 1,\n",
              "          'pai': 1,\n",
              "          'sera': 1,\n",
              "          'vale': 1,\n",
              "          'vencer': 1})}"
            ]
          },
          "metadata": {},
          "execution_count": 271
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 259
        },
        "id": "mXEHckBLsslJ",
        "outputId": "a8944472-ac20-4d1e-ee4e-2a02e0b2d2ee"
      },
      "source": [
        "df = pd.DataFrame.from_records(bow_counter).fillna(0).astype(int).T\n",
        "\n",
        "df"
      ],
      "execution_count": 272,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>athletico</th>\n",
              "      <th>paranaens</th>\n",
              "      <th>eh</th>\n",
              "      <th>melhor</th>\n",
              "      <th>time</th>\n",
              "      <th>pai</th>\n",
              "      <th>dentr</th>\n",
              "      <th>54</th>\n",
              "      <th>dito</th>\n",
              "      <th>nao</th>\n",
              "      <th>existem</th>\n",
              "      <th>questionamento</th>\n",
              "      <th>outro</th>\n",
              "      <th>ira</th>\n",
              "      <th>vencer</th>\n",
              "      <th>paz</th>\n",
              "      <th></th>\n",
              "      <th>possui</th>\n",
              "      <th>vario</th>\n",
              "      <th>ha</th>\n",
              "      <th>espaco</th>\n",
              "      <th>golf</th>\n",
              "      <th>vasco</th>\n",
              "      <th>estranho</th>\n",
              "      <th>vale</th>\n",
              "      <th>sera</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>doc 0</th>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>doc 1</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>doc 2</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>doc 3</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "       athletico  paranaens  eh  melhor  ...  vasco  estranho  vale  sera\n",
              "doc 0          1          1   1       1  ...      0         0     0     0\n",
              "doc 1          0          0   1       0  ...      0         0     0     0\n",
              "doc 2          0          0   2       1  ...      1         1     0     0\n",
              "doc 3          0          0   1       1  ...      0         0     1     1\n",
              "\n",
              "[4 rows x 26 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 272
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YDsVOLoYN-UW"
      },
      "source": [
        "## Term Frequency - Inverse Document Frequency (TF-IDF)\n",
        "\n",
        "Por fim, a abordagem Term Frequency-Inverse Document Frequency é calculada como:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KII8M9pKORxd"
      },
      "source": [
        ""
      ],
      "execution_count": 272,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rPTuZv1di6kQ"
      },
      "source": [
        "## Extração de Características e Pré-Processamento por meio da scikit-learn\n",
        "\n",
        "Aqui apresentamos como realizar a extração de características utilizando a abordagem Bag-of-Words por matriz de contagem. Para esse propósito, vamos utilizar a classe CountVectorizer da biblioteca scikit-learn.\n",
        "\n",
        "Clique [aqui](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html#sklearn.feature_extraction.text.CountVectorizer) para acessar a documentação."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KGKNhNnJi60Q",
        "outputId": "5cdb5eb0-2676-412e-ed11-140f331ca1d2"
      },
      "source": [
        "vectorizer = CountVectorizer()\n",
        "\n",
        "X = vectorizer.fit_transform(corpus)\n",
        "\n",
        "print(X.toarray())"
      ],
      "execution_count": 273,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[1 1 1 1 1 1 0 0 1 0 0 1 1 0 1 1 1 1 1 0 1 1 0 1 0 1 0 0 2 0 0 0 0 1]\n",
            " [0 0 0 0 0 1 1 0 0 1 1 0 0 0 0 0 0 0 1 1 0 0 1 0 1 0 1 0 0 1 0 1 0 0]\n",
            " [0 0 0 0 0 2 0 1 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 0]\n",
            " [0 0 0 0 1 1 0 0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 1 0 0 0 1 0 0 1 0 0 1]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NuLpBu32j96-"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ruBtIrrXj-Bv",
        "outputId": "01c1321a-c7e6-4ee8-bda0-e2d2b9105f93"
      },
      "source": [
        "print(vectorizer.get_feature_names())\n",
        "\n",
        "print(X.shape)"
      ],
      "execution_count": 274,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['54', 'athletico', 'dentre', 'dito', 'do', 'eh', 'espaco', 'estranho', 'existem', 'golfe', 'ha', 'ira', 'isso', 'mas', 'melhor', 'nao', 'ou', 'outro', 'pais', 'para', 'paranaense', 'paz', 'possui', 'que', 'questionamento', 'questionamentos', 'se', 'sera', 'time', 'times', 'vale', 'varios', 'vasco', 'vencer']\n",
            "(4, 34)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GPriP8d_jNhw"
      },
      "source": [
        "Clique [aqui](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html) para acessar a documentação."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pUi_KQYkjNn4"
      },
      "source": [
        "vectorizer = TfidfVectorizer()\n",
        "\n",
        "X = vectorizer.fit_transform(corpus)"
      ],
      "execution_count": 275,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zaBfPE55jkXe"
      },
      "source": [
        "Da mesma maneira como no CounterVectorizer, podemos extrair os termos que foram as colunas da representação TF-IDF:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ik57f5jJjkeH",
        "outputId": "7b56c4ae-c070-4e01-c568-cd0c89322861"
      },
      "source": [
        "print(vectorizer.get_feature_names())\n",
        "\n",
        "print(X.shape)"
      ],
      "execution_count": 276,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['54', 'athletico', 'dentre', 'dito', 'do', 'eh', 'espaco', 'estranho', 'existem', 'golfe', 'ha', 'ira', 'isso', 'mas', 'melhor', 'nao', 'ou', 'outro', 'pais', 'para', 'paranaense', 'paz', 'possui', 'que', 'questionamento', 'questionamentos', 'se', 'sera', 'time', 'times', 'vale', 'varios', 'vasco', 'vencer']\n",
            "(4, 34)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-zj5DfmMkvld"
      },
      "source": [
        "A classe TfIdfVectorizer possui em seu construtor alguns atributos que realizam pré-processamentos, como remoção de stop-words, padrões de extração dos termos e alterações na maneira com que o TF-IDF pode ser calculado, como a quantidade (ou proporção) mínima ou máxima em que a frequência dos documentos é considerada. Maiores detalhes consulte a documentação da classe. \n",
        "\n",
        "No exemplo abaixo, a matriz TF-IDF é obtida a partir dos termos obtidos de pré-processamento em que os termos são tratados como bi-grams. O atributo max_df ignora os termos que se apresentam em mais do 96% de todos os documentos do corpus, em que valores entre [0.7,1.0) são úteis para remoção de stop-words. Já o atributo min_df ignora termos que pouco aparecem em, no mínimo, dois documentos (altere esse valor para 1 para analisar como mais termos podem ser considerados)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "j45BErctqAUi",
        "outputId": "a6ed642d-323b-4034-afbc-7d6209171ea2"
      },
      "source": [
        "vectorizer = TfidfVectorizer(ngram_range=(2,2),max_df=0.96,min_df=2)\n",
        "\n",
        "X = vectorizer.fit_transform(corpus)\n",
        "\n",
        "print(vectorizer.get_feature_names())"
      ],
      "execution_count": 277,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['do pais', 'eh melhor']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F6_7r3ZtOSBV"
      },
      "source": [
        "## Similaridade entre textos\n",
        "\n",
        "O objetivo dessa seção é determinar a similaridade entre dois textos, representados pelos seus vetores TF-IDF, por meio da similaridade cosseno. Primeiramente, definimos uma função que implementa o cálculo da similaridade cosseno:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "r8IGYcOQ-K1T"
      },
      "source": [
        "def cosine_similarity(v1,v2):\n",
        "  \n",
        "  dot_prod = 0\n",
        "  for i,v in enumerate(v1):\n",
        "    dot_prod += v * v2[i]\n",
        "\n",
        "  mag_v1 = math.sqrt(sum([x**2 for x in v1]))\n",
        "  mag_v2 = math.sqrt(sum([x**2 for x in v2]))\n",
        "\n",
        "  return dot_prod / (mag_v1 * mag_v2)"
      ],
      "execution_count": 278,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H8oRRvftOokW"
      },
      "source": [
        "Vamos criar duas sentenças e determinar a similairdade entre elas:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4S3eRrEXBeW3"
      },
      "source": [
        "sentence1 = \"Positivo e negativo e neutro\"\n",
        "sentence2 = \"Positivo demais hoje\"\n",
        "\n",
        "corpus = []\n",
        "corpus.append(sentence1)\n",
        "corpus.append(sentence2)\n",
        "\n",
        "tfidf_obj = TfidfVectorizer()\n",
        "\n",
        "tfidf_vectors = tfidf_obj.fit_transform(corpus)"
      ],
      "execution_count": 279,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2XaOlFziOxWr"
      },
      "source": [
        "Separando os vetores de características em variáveis separadas para exemplificar o uso da similaridade cosseno:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cBv_I_oJCAaN"
      },
      "source": [
        "X = tfidf_vectors.toarray()\n",
        "\n",
        "vet1 = X[0]\n",
        "vet2 = X[1]"
      ],
      "execution_count": 280,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mVaUkNvOPBOT"
      },
      "source": [
        "Imprime a similaridade cosseno entre os vetores TF-IDF das sentenças 1 e 2:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "48KV0G2LCPsk",
        "outputId": "e94e2887-37c7-4e6d-ee2c-03d847cf9ed1"
      },
      "source": [
        "print(cosine_similarity(vet1,vet2))"
      ],
      "execution_count": 281,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "0.2019930924979184\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}