lectures/cap12_2_lang_models_rnn.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyM4vl0vdL2IET7Jxynmx9di",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/viniciusrpb/cic0269_natural_language_processing/blob/main/lectures/cap12_2_lang_models_rnn.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Capítulo 12 - Redes Neurais Recorrentes\n",
        "\n",
        "## 12.2. Modelos de Linguagem Baseados em Redes Neurais Recorrentes\n",
        "\n",
        "O objetivo deste notebook consiste em desenvolver modelos de linguagem baseados em redes neurais recorrentes. Iremos abordar dois tipos de modelos:\n",
        "\n",
        "*   word-to-word: trata cada palavra do *corpus* como um documento. O processo de treinamento consiste em pares (palavra,próxima palavra) como sendo o texto e o rótulo;\n",
        "*   sentence-to-word: considera as palavras anteriores de uma sentença (e suas relações de dependência) do *corpus* para prever a próxima palavra.\n",
        "\n"
      ],
      "metadata": {
        "id": "qsStPDjrehxC"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from tensorflow.keras.preprocessing.text import Tokenizer\n",
        "from keras.models import Sequential\n",
        "from keras.layers import Dense,LSTM,Embedding,Activation,Flatten,Dropout,Bidirectional\n",
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "from tensorflow.keras.optimizers import SGD\n",
        "from tensorflow.keras.preprocessing.text import Tokenizer\n",
        "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
        "from keras.utils.np_utils import to_categorical"
      ],
      "metadata": {
        "id": "-Kkjn6yWSVkn"
      },
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "o2ixGd2AQEgL"
      },
      "outputs": [],
      "source": [
        "corpus = \"Quero jogar futebol hoje\\n Hoje não tem futebol\\n\"\n",
        "corpus = corpus.lower()"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Modelo de Linguagem do tipo Word-to-Word"
      ],
      "metadata": {
        "id": "3_SU9CMTf50C"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "termos = {}\n",
        "tokens = []\n",
        "for sentence in corpus.split('\\n'):\n",
        "    for word in sentence.split():\n",
        "        tokens.append(word)\n",
        "        if word in termos:\n",
        "            termos[word]+=1\n",
        "        else:\n",
        "            termos[word]=1\n",
        "vocab_size = len(termos)"
      ],
      "metadata": {
        "id": "2s9NKeRvRujE"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "vocab_size"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "X7xZQihOSxEd",
        "outputId": "b3141376-511e-4745-a558-bbd63eed8712"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "6"
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "tokenizer = Tokenizer(num_words = vocab_size)\n",
        "tokenizer.fit_on_texts(termos)\n",
        "word"
      ],
      "metadata": {
        "id": "-Lutla8-SIG7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "7d15d849-a35b-4f54-e379-6a956aee466b"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'futebol'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "word2index = tokenizer.word_index\n",
        "word2index['<OOV>'] = 0\n",
        "index2word = {}\n",
        "for key in word2index:\n",
        "    value = word2index[key]\n",
        "    index2word[value] = key"
      ],
      "metadata": {
        "id": "um9s0lnETH5h"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "index2word"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jHo4LytNcdOQ",
        "outputId": "36b207f1-202a-4952-92a0-4bfad11e5371"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{1: 'quero',\n",
              " 2: 'jogar',\n",
              " 3: 'futebol',\n",
              " 4: 'hoje',\n",
              " 5: 'não',\n",
              " 6: 'tem',\n",
              " 0: '<OOV>'}"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "X_train = [0]\n",
        "y_train = [word2index[tokens[0]]]\n",
        "for i in range(0,len(tokens)-1):\n",
        "    X_train.append(word2index[tokens[i]])\n",
        "    y_train.append(word2index[tokens[i+1]])\n",
        "X_train.append(word2index[tokens[len(tokens)-1]])\n",
        "y_train.append(0)"
      ],
      "metadata": {
        "id": "cDiK0PFUTlFm"
      },
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "X_train"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "l91d3K15U6Fz",
        "outputId": "58c1ec46-a67e-4578-c86f-9bf1cbc6b91f"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0, 1, 2, 3, 4, 4, 5, 6, 3]"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "y_train"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qg1RD_1tU-js",
        "outputId": "10c04598-45cc-4089-c7c3-ba3fe383c00e"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[1, 2, 3, 4, 4, 5, 6, 3, 0]"
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model = Sequential()\n",
        "model.add(Embedding(input_dim=vocab_size+1,output_dim=32,input_length=1))\n",
        "model.add(Bidirectional(LSTM(256,activation='relu')))\n",
        "model.add(Dropout(0.5))\n",
        "model.add(Dense(vocab_size+1,activation='softmax'))"
      ],
      "metadata": {
        "id": "FO3__UIRVlrb"
      },
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "sgd = SGD(learning_rate = 0.001)\n",
        "model.compile(optimizer=sgd,loss='sparse_categorical_crossentropy',metrics=['accuracy'])\n",
        "model.fit(X_train,y_train,batch_size=16,epochs=3)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9wKHw4qkWeW5",
        "outputId": "4d9d32c7-7f6c-4b4a-c01b-a798c1d74d4c"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/3\n",
            "1/1 [==============================] - 3s 3s/step - loss: 1.9452 - accuracy: 0.2222\n",
            "Epoch 2/3\n",
            "1/1 [==============================] - 0s 20ms/step - loss: 1.9459 - accuracy: 0.2222\n",
            "Epoch 3/3\n",
            "1/1 [==============================] - 0s 31ms/step - loss: 1.9451 - accuracy: 0.3333\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x7f9cbbed4c90>"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "\n",
        "frase = 'futebol quero jogar'\n",
        "\n",
        "for w in frase.split():\n",
        "    idx = word2index[w]\n",
        "    prob = model.predict([idx])\n",
        "    pal = np.argmax(prob)\n",
        "    print(f'Palavra atual: {index2word[idx]} Proxima palavra: {index2word[pal]}')\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uxOXwZq6aXio",
        "outputId": "06757c9a-032b-4c74-d243-d2ca19b13f82"
      },
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Palavra atual: futebol Proxima palavra: futebol\n",
            "Palavra atual: quero Proxima palavra: quero\n",
            "Palavra atual: jogar Proxima palavra: futebol\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Modelo de Linguagem do tipo Sentence-2-Word\n",
        "\n",
        "Nesse tipo de modelo, as palavras são analisadas dentro das sentenças:"
      ],
      "metadata": {
        "id": "nTfcP50LdVMM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "novel_corpus = []\n",
        "y_train = []\n",
        "for sentence in corpus.split('\\n'):\n",
        "\n",
        "    novos_termos = sentence.split()\n",
        "\n",
        "    for i in range(0,len(novos_termos)):\n",
        "        lista = novos_termos[:i+1]\n",
        "        novel_corpus.append(lista)\n",
        "        if i < len(novos_termos)-1:\n",
        "            y_train.append(word2index[novos_termos[i+1]])\n",
        "        else:\n",
        "            y_train.append(0)"
      ],
      "metadata": {
        "id": "pqO1amVjdY-_"
      },
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "novel_corpus"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xTrActnGfEa7",
        "outputId": "6bcb2bd3-91dd-4918-d734-264f2defcd13"
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[['quero'],\n",
              " ['quero', 'jogar'],\n",
              " ['quero', 'jogar', 'futebol'],\n",
              " ['quero', 'jogar', 'futebol', 'hoje'],\n",
              " ['hoje'],\n",
              " ['hoje', 'não'],\n",
              " ['hoje', 'não', 'tem'],\n",
              " ['hoje', 'não', 'tem', 'futebol']]"
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "y_train"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "u2NzTefBnncE",
        "outputId": "4ecb1d77-4b9c-4ef2-cffc-9f936aaa7cde"
      },
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[2, 3, 4, 0, 5, 6, 3, 0]"
            ]
          },
          "metadata": {},
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "max_length = max([len(sentence) for sentence in novel_corpus])"
      ],
      "metadata": {
        "id": "_vwvNFvMjuZF"
      },
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "max_length"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rAEmLMVDkEzq",
        "outputId": "c2534177-85d2-4209-e6a5-78b05b881f98"
      },
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "4"
            ]
          },
          "metadata": {},
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "train_sequences = tokenizer.texts_to_sequences(novel_corpus)"
      ],
      "metadata": {
        "id": "KMXwYWfrkIRI"
      },
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "train_sequences"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AfvcajYQkY4B",
        "outputId": "80c5607d-a0b4-4f88-a9eb-94e6d658f088"
      },
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[[1], [1, 2], [1, 2, 3], [1, 2, 3, 4], [4], [4, 5], [4, 5], [4, 5, 3]]"
            ]
          },
          "metadata": {},
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "trunc_type = 'post'\n",
        "padding_type = 'pre'\n",
        "\n",
        "train_padded = pad_sequences(train_sequences, maxlen=max_length, padding=padding_type, truncating=trunc_type)"
      ],
      "metadata": {
        "id": "HXv9j9u7kld8"
      },
      "execution_count": 21,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "train_padded"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Y_eQgyb_kvMj",
        "outputId": "1aea3d9a-ee4a-4892-d853-34c4a7a81b6b"
      },
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0, 0, 0, 1],\n",
              "       [0, 0, 1, 2],\n",
              "       [0, 1, 2, 3],\n",
              "       [1, 2, 3, 4],\n",
              "       [0, 0, 0, 4],\n",
              "       [0, 0, 4, 5],\n",
              "       [0, 0, 4, 5],\n",
              "       [0, 4, 5, 3]], dtype=int32)"
            ]
          },
          "metadata": {},
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "y_train = to_categorical(y_train)"
      ],
      "metadata": {
        "id": "VvX7hMG5lsCw"
      },
      "execution_count": 23,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model2 = Sequential()\n",
        "model2.add(Embedding(input_dim=vocab_size+1,output_dim=32,input_length=max_length))\n",
        "model2.add(Bidirectional(LSTM(256,activation='relu')))\n",
        "model2.add(Dropout(0.5))\n",
        "model2.add(Dense(vocab_size+1,activation='softmax'))"
      ],
      "metadata": {
        "id": "AEND9pGmk8bB"
      },
      "execution_count": 24,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "sgd = SGD(learning_rate = 0.001)\n",
        "model.compile(optimizer=sgd,loss='categorical_crossentropy',metrics=['accuracy'])\n",
        "model.fit(train_padded,y_train,batch_size=16,epochs=3)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4C5wFJIzlaYA",
        "outputId": "32399594-9994-4fc4-c11f-e2fcfd0a5745"
      },
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/3\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "WARNING:tensorflow:Model was constructed with shape (None, 1) for input KerasTensor(type_spec=TensorSpec(shape=(None, 1), dtype=tf.float32, name='embedding_input'), name='embedding_input', description=\"created by layer 'embedding_input'\"), but it was called on an input with incompatible shape (None, 4).\n",
            "WARNING:tensorflow:Model was constructed with shape (None, 1) for input KerasTensor(type_spec=TensorSpec(shape=(None, 1), dtype=tf.float32, name='embedding_input'), name='embedding_input', description=\"created by layer 'embedding_input'\"), but it was called on an input with incompatible shape (None, 4).\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "1/1 [==============================] - 5s 5s/step - loss: 1.9455 - accuracy: 0.1250\n",
            "Epoch 2/3\n",
            "1/1 [==============================] - 0s 77ms/step - loss: 1.9462 - accuracy: 0.1250\n",
            "Epoch 3/3\n",
            "1/1 [==============================] - 0s 55ms/step - loss: 1.9441 - accuracy: 0.2500\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x7f9cbbd56050>"
            ]
          },
          "metadata": {},
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Modelo de Linguagem com Corpus da Reuters\n",
        "\n",
        "**Homework:** desenvolver um modelo de linguagem do tipo Sentence-to-Word utilizando o vocabulário do *corpus* (split do treinamento) da reuters. Aproveite os splits de validação e de testes para experimentos.\n",
        "\n",
        "Obs.: delimitar o ```vocab_size``` para que seja possível a execução desse notebook no Google Colab. "
      ],
      "metadata": {
        "id": "9E-MMLp9bbCp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from keras.datasets import reuters\n",
        "\n",
        "vocab_size = 3000\n",
        "\n",
        "(x_train,y_train_int),(x_test2,y_test2) = reuters.load_data(num_words=vocab_size,test_split=0.3)\n",
        "word2index = reuters.get_word_index()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "trppo9zHbf0Z",
        "outputId": "e5b9560b-5b87-4ce0-e559-79ca8cf6f187"
      },
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/reuters.npz\n",
            "2113536/2110848 [==============================] - 0s 0us/step\n",
            "2121728/2110848 [==============================] - 0s 0us/step\n",
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/reuters_word_index.json\n",
            "557056/550378 [==============================] - 0s 0us/step\n",
            "565248/550378 [==============================] - 0s 0us/step\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "index2word = {}\n",
        "\n",
        "for key,value in word2index.items():\n",
        "  index2word[value] = key\n",
        "\n",
        "print(' '.join([index2word[x] for x in x_train[0]]))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HZEHU8eob4jQ",
        "outputId": "2576a73a-ec30-4115-ec5e-37f2fe3437d7"
      },
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "the of of mln loss for plc said at only ended said of could 1 traders now april 0 a after said from 1985 and from foreign 000 april 0 prices its account year a but in this mln home an states earlier and rise and revs vs 000 its 16 vs 000 a but 3 of oils several and shareholders and dividend vs 000 its all 4 vs 000 1 mln agreed largely april 0 are 2 states will billion total and against 000 pct dlrs\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Lembre-se de construir as variáveis apropriadamente as variáveis ``` y_train ``` e ``` X_train ``` para treinamento dos modelos.\n",
        "\n"
      ],
      "metadata": {
        "id": "RLWLKziwcn1e"
      }
    }
  ]
}