lstm.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "lstm.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyOMrlknPT5IfrjngSJgDaQS"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "535ilqDV9kjd"
      },
      "source": [
        "# Word-Level Text Generation with LSTM\r\n",
        "\r\n",
        "In addition to making predictions, RNNs may also be used as generative models (can learn the sequences and then generate entirely new seqences). One of RNN variant, LSTM neural network has been recognized as a very successful tool when working with sequences of letters or words.\r\n",
        "\r\n",
        "Let's examine performance of basic LSTM model on generating text of fairy tales."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VFchVFdsCBEw",
        "outputId": "650a678d-435a-4a0e-ea1f-e34c56fe49f0"
      },
      "source": [
        "from google.colab import drive\r\n",
        "drive.mount('/content/drive', force_remount=True)\r\n",
        "%cd 'drive/MyDrive/Colab Notebooks/nlg_tales_generation'"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Mounted at /content/drive\n",
            "/content/drive/MyDrive/Colab Notebooks/nlg_tales_generation\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IbE3BBh68v55"
      },
      "source": [
        "import functions as f\r\n",
        "\r\n",
        "from Text import *\r\n",
        "from LSTM_class import *\r\n",
        "\r\n",
        "from keras import layers, models, optimizers"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aUqK-M-p1raQ"
      },
      "source": [
        "## Text preprocessing\r\n",
        "\r\n",
        "The loaded text file contains the content of tales scraped from websites. By creating the instance of Text object, the text is quickly preprocessed and tokenized; by creating the instance of Sequence object the text is prepared for use in LSTM model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JLHty8SmmueL"
      },
      "source": [
        "path_train, path_test = 'data/train.txt', 'data/test.txt'\r\n",
        "\r\n",
        "input_train = f.read_txt(path_train)"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5ZQU63Rz10_C",
        "outputId": "bfc2e6b8-8550-4ad8-f1c7-0dfd44de9600"
      },
      "source": [
        "max_len = 4\r\n",
        "step = 3\r\n",
        "\r\n",
        "text_train = Text(input_train)\r\n",
        "text_train.tokens_info()\r\n",
        "\r\n",
        "seq_train = Sequences(text_train, max_len, step)\r\n",
        "seq_train.sequences_info()"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "total tokens: 890750, distinct tokens: 25165\n",
            "number of sequences of length 4: 296916\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EXG4cBuABXM2"
      },
      "source": [
        "The text is split into sequences of length 4 (max_len parameter) with step 3. We can see that the first sequence of 4 words starts with the first (0-index) word and the second sequence starts after 3 words, so from the 4th word (3-index)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Er-A4Yv2DnoD",
        "outputId": "1c1f3e24-277b-47f3-ac1e-c96da5d1f023"
      },
      "source": [
        "print(text_train.tokens[:10])\r\n",
        "print(text_train.tokens_ind[:10], '\\n')\r\n",
        "\r\n",
        "np.array(seq_train.sequences[:2])"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['Once', 'upon', 'a', 'time', 'there', 'lived', 'a', 'sultan', 'who', 'loved']\n",
            "[23084, 5388, 8660, 263, 17275, 23956, 8660, 14335, 10634, 11630] \n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[23084,  5388,  8660,   263],\n",
              "       [  263, 17275, 23956,  8660]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dGR6r0WpBeC8"
      },
      "source": [
        "TextDataGenerator is a Python generator that outputs batches of data (sequences and corresponding next words). Since the vocabulary size is over 25k, it's impossible to fit all data to the memory and that's why batch generator is extremely useful."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2Hw3av9-v1TO"
      },
      "source": [
        "batch_size = 4096\r\n",
        "\r\n",
        "params = {\r\n",
        "  'sequence_length': max_len,\r\n",
        "  'vocab_size': len(text_train),\r\n",
        "  'batch_size': batch_size,\r\n",
        "  'shuffle': True\r\n",
        "}\r\n",
        "\r\n",
        "train_generator = TextDataGenerator(seq_train.sequences, seq_train.next_words, **params)"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SG0GSvi8_anC"
      },
      "source": [
        "## Training the LSTM model\r\n",
        "\r\n",
        "We'll build a simple model with one LSTM layer, dropout and dense layer with softmax activation (to return word probabilities)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JAzlW7zXSWcK"
      },
      "source": [
        "def lstm_model(sequence_length, vocab_size, layer_size, embedding=False):\r\n",
        "  model = models.Sequential()\r\n",
        "  if embedding:\r\n",
        "    model.add(layers.Embedding(vocab_size, layer_size))\r\n",
        "    model.add(layers.LSTM(layer_size))    \r\n",
        "  else:\r\n",
        "    model.add(layers.LSTM(layer_size, input_shape=(sequence_length, vocab_size)))\r\n",
        "  model.add(layers.Dropout(0.3))\r\n",
        "  model.add(layers.Dense(vocab_size, activation='softmax'))\r\n",
        "  return model"
      ],
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YZfL534BwLiO"
      },
      "source": [
        "model = lstm_model(max_len, len(text_train), 512)\r\n",
        "\r\n",
        "optimizer = optimizers.RMSprop(lr=0.01)\r\n",
        "model.compile(loss='categorical_crossentropy', optimizer=optimizer)"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lfRwajqPwWPi",
        "outputId": "6bc83bd2-20d9-4319-aa44-854ba4e21468"
      },
      "source": [
        "model.fit(train_generator,\r\n",
        "          steps_per_epoch=len(train_generator),\r\n",
        "          epochs=40,\r\n",
        "          verbose=1)"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/40\n",
            "72/72 [==============================] - 135s 2s/step - loss: 7.6544\n",
            "Epoch 2/40\n",
            "72/72 [==============================] - 130s 2s/step - loss: 5.5843\n",
            "Epoch 3/40\n",
            "72/72 [==============================] - 131s 2s/step - loss: 4.9627\n",
            "Epoch 4/40\n",
            "72/72 [==============================] - 131s 2s/step - loss: 4.4947\n",
            "Epoch 5/40\n",
            "72/72 [==============================] - 130s 2s/step - loss: 4.0160\n",
            "Epoch 6/40\n",
            "72/72 [==============================] - 131s 2s/step - loss: 3.5458\n",
            "Epoch 7/40\n",
            "72/72 [==============================] - 131s 2s/step - loss: 3.1083\n",
            "Epoch 8/40\n",
            "72/72 [==============================] - 131s 2s/step - loss: 2.7399\n",
            "Epoch 9/40\n",
            "72/72 [==============================] - 131s 2s/step - loss: 2.4194\n",
            "Epoch 10/40\n",
            "72/72 [==============================] - 131s 2s/step - loss: 2.1634\n",
            "Epoch 11/40\n",
            "72/72 [==============================] - 131s 2s/step - loss: 1.9340\n",
            "Epoch 12/40\n",
            "72/72 [==============================] - 131s 2s/step - loss: 1.7456\n",
            "Epoch 13/40\n",
            "72/72 [==============================] - 131s 2s/step - loss: 1.5934\n",
            "Epoch 14/40\n",
            "72/72 [==============================] - 131s 2s/step - loss: 1.4495\n",
            "Epoch 15/40\n",
            "72/72 [==============================] - 131s 2s/step - loss: 1.3233\n",
            "Epoch 16/40\n",
            "72/72 [==============================] - 131s 2s/step - loss: 1.2241\n",
            "Epoch 17/40\n",
            "72/72 [==============================] - 131s 2s/step - loss: 1.1297\n",
            "Epoch 18/40\n",
            "72/72 [==============================] - 131s 2s/step - loss: 1.0514\n",
            "Epoch 19/40\n",
            "72/72 [==============================] - 131s 2s/step - loss: 0.9821\n",
            "Epoch 20/40\n",
            "72/72 [==============================] - 131s 2s/step - loss: 0.9224\n",
            "Epoch 21/40\n",
            "72/72 [==============================] - 131s 2s/step - loss: 0.8682\n",
            "Epoch 22/40\n",
            "72/72 [==============================] - 130s 2s/step - loss: 0.8148\n",
            "Epoch 23/40\n",
            "72/72 [==============================] - 130s 2s/step - loss: 0.7777\n",
            "Epoch 24/40\n",
            "72/72 [==============================] - 131s 2s/step - loss: 0.7353\n",
            "Epoch 25/40\n",
            "72/72 [==============================] - 131s 2s/step - loss: 0.7032\n",
            "Epoch 26/40\n",
            "72/72 [==============================] - 131s 2s/step - loss: 0.6691\n",
            "Epoch 27/40\n",
            "72/72 [==============================] - 131s 2s/step - loss: 0.6445\n",
            "Epoch 28/40\n",
            "72/72 [==============================] - 131s 2s/step - loss: 0.6187\n",
            "Epoch 29/40\n",
            "72/72 [==============================] - 131s 2s/step - loss: 0.5959\n",
            "Epoch 30/40\n",
            "72/72 [==============================] - 131s 2s/step - loss: 0.5748\n",
            "Epoch 31/40\n",
            "72/72 [==============================] - 131s 2s/step - loss: 0.5548\n",
            "Epoch 32/40\n",
            "72/72 [==============================] - 131s 2s/step - loss: 0.5370\n",
            "Epoch 33/40\n",
            "72/72 [==============================] - 131s 2s/step - loss: 0.5261\n",
            "Epoch 34/40\n",
            "72/72 [==============================] - 131s 2s/step - loss: 0.5065\n",
            "Epoch 35/40\n",
            "72/72 [==============================] - 131s 2s/step - loss: 0.4993\n",
            "Epoch 36/40\n",
            "72/72 [==============================] - 131s 2s/step - loss: 0.4809\n",
            "Epoch 37/40\n",
            "72/72 [==============================] - 131s 2s/step - loss: 0.4723\n",
            "Epoch 38/40\n",
            "72/72 [==============================] - 131s 2s/step - loss: 0.4604\n",
            "Epoch 39/40\n",
            "72/72 [==============================] - 131s 2s/step - loss: 0.4504\n",
            "Epoch 40/40\n",
            "72/72 [==============================] - 131s 2s/step - loss: 0.4440\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.keras.callbacks.History at 0x7f5b7008de48>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tpRxjstuTMVL",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "eea09391-93e7-4b04-87fe-1a18a54d9d14"
      },
      "source": [
        "model.save('data/out/lstm_model')"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:absl:Found untraced functions such as lstm_cell_layer_call_fn, lstm_cell_layer_call_and_return_conditional_losses, lstm_cell_layer_call_fn, lstm_cell_layer_call_and_return_conditional_losses, lstm_cell_layer_call_and_return_conditional_losses while saving (showing 5 of 5). These functions will not be directly callable after loading.\n",
            "WARNING:absl:Found untraced functions such as lstm_cell_layer_call_fn, lstm_cell_layer_call_and_return_conditional_losses, lstm_cell_layer_call_fn, lstm_cell_layer_call_and_return_conditional_losses, lstm_cell_layer_call_and_return_conditional_losses while saving (showing 5 of 5). These functions will not be directly callable after loading.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Assets written to: data/out/lstm_model/assets\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Assets written to: data/out/lstm_model/assets\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oKx_NBOV_iey"
      },
      "source": [
        "## Text generation with LSTM model\r\n",
        "\r\n",
        "Generating text with LSTM model requires building the prediction loop which starts with choosing a prefix and setting the number of words to generate. Then we need to predict the next word using our LSTM model and use this word as part of the prefix for the next model input. The loop is executed until the expected number of words is generated."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oGU38wwtTN1J"
      },
      "source": [
        "#model = models.load_model('data/out/lstm_model')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mwJRWQIvnp1I"
      },
      "source": [
        "token2ind, ind2token = text_train.token2ind, text_train.ind2token\r\n",
        "\r\n",
        "input_prefix = 'Once upon a time'\r\n",
        "text_prefix = Text(input_prefix, token2ind, ind2token)"
      ],
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "B32EAv2mxsa1"
      },
      "source": [
        "pred = ModelPredict(model, text_prefix, token2ind, ind2token, max_len)"
      ],
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ccvYg3JO2xI1",
        "outputId": "220f2278-e049-44fa-f445-28df6861041a"
      },
      "source": [
        "temperatures = [1, 0.7, 0.4, 0.1]\r\n",
        "\r\n",
        "for temperature in temperatures:\r\n",
        "  print('temperature:', temperature)\r\n",
        "  print(pred.generate_sequence(100, temperature=0.7))\r\n",
        "  print('\\n')"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "temperature: 1\n",
            "Once upon a time there were a man and noble princess!' being so good- natured, and that morning, when he awoke, he found it in his hand it to the palace. The young wild hut a lived, on the bridge of two days. Every day when they had to be done for now. She is Wait an ass, and said to the merchant, and thou wilt have your bed has looked to take her up. A large man was lost in favor. The king be plunged and in the water\n",
            "\n",
            "\n",
            "temperature: 0.7\n",
            "Once upon a time there were a lot of mice, but could run see if he had been so good, for you will see someone there was. All the great wild Huldre; but I am very much mistaken. The is Moor- there. done as he, and his children wept, and he sat down in the forest. Go, and give They all the of people that I had gone his bed with the little giant, and she would draw none out of the mountain. This was the first thing that I was\n",
            "\n",
            "\n",
            "temperature: 0.4\n",
            "Once upon a time there was a father who had to be allowed to stay overnight, night as the far- there she could hardly tell from whether the king coming and he became angry, and when the prince took from the roof, I had a dream which had been question in the trick played to presently her, and he looked as if the king was sent breakfast with a child? he asked them. What could you do do as she ought. quite grand hope; then he would grant them if it wish to a river\n",
            "\n",
            "\n",
            "temperature: 0.1\n",
            "Once upon a time there were a man of the Feni, whether on straightway or or language of thy, followers, come to your father and your son.' And he went to the room, and lifted a prostrate. The king, who was pleased and her new full of money, and drive your long in a place where they might be. He found it all his friends, and one will see over it, and it was so petted that she could not see a step- mother said, you told done that the\n",
            "\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xe6ha7SayvQg"
      },
      "source": [
        "## Text generation with LSTM model with Embedding layer\r\n",
        "\r\n",
        "The previous model was taking as an input the sequences of words represented as one-hot vectors. In the second approach, we'll feed indexes of words to the model and train the Embedding layers which will create word representations."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-WiKwkcXyu6C"
      },
      "source": [
        "batch_size_emb = 4096\r\n",
        "\r\n",
        "params_emb = params.copy()\r\n",
        "params_emb['embedding'] = True\r\n",
        "\r\n",
        "train_generator_emb = TextDataGenerator(seq_train.sequences, seq_train.next_words, **params_emb)"
      ],
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EjeejnRMzEyq"
      },
      "source": [
        "model_emb = lstm_model(max_len, len(text_train), 512, embedding=True)\r\n",
        "model_emb.compile(loss='categorical_crossentropy', optimizer=optimizer)"
      ],
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "68tjpQ9tzXo7",
        "outputId": "0c31c62d-9e15-431c-93f0-f31fa2ee16cb"
      },
      "source": [
        "model_emb.fit(train_generator_emb,\r\n",
        "              steps_per_epoch=len(train_generator_emb),\r\n",
        "              epochs=40,\r\n",
        "              verbose=1)"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/40\n",
            "72/72 [==============================] - 31s 413ms/step - loss: 7.4319\n",
            "Epoch 2/40\n",
            "72/72 [==============================] - 30s 411ms/step - loss: 5.1867\n",
            "Epoch 3/40\n",
            "72/72 [==============================] - 29s 406ms/step - loss: 4.7109\n",
            "Epoch 4/40\n",
            "72/72 [==============================] - 29s 399ms/step - loss: 4.3334\n",
            "Epoch 5/40\n",
            "72/72 [==============================] - 29s 404ms/step - loss: 3.9703\n",
            "Epoch 6/40\n",
            "72/72 [==============================] - 29s 398ms/step - loss: 3.6161\n",
            "Epoch 7/40\n",
            "72/72 [==============================] - 29s 397ms/step - loss: 3.2614\n",
            "Epoch 8/40\n",
            "72/72 [==============================] - 29s 403ms/step - loss: 2.9379\n",
            "Epoch 9/40\n",
            "72/72 [==============================] - 29s 402ms/step - loss: 2.6587\n",
            "Epoch 10/40\n",
            "72/72 [==============================] - 29s 403ms/step - loss: 2.4155\n",
            "Epoch 11/40\n",
            "72/72 [==============================] - 29s 395ms/step - loss: 2.2170\n",
            "Epoch 12/40\n",
            "72/72 [==============================] - 29s 404ms/step - loss: 2.0368\n",
            "Epoch 13/40\n",
            "72/72 [==============================] - 29s 395ms/step - loss: 1.8822\n",
            "Epoch 14/40\n",
            "72/72 [==============================] - 29s 397ms/step - loss: 1.7618\n",
            "Epoch 15/40\n",
            "72/72 [==============================] - 29s 401ms/step - loss: 1.6465\n",
            "Epoch 16/40\n",
            "72/72 [==============================] - 29s 407ms/step - loss: 1.5579\n",
            "Epoch 17/40\n",
            "72/72 [==============================] - 29s 405ms/step - loss: 1.4756\n",
            "Epoch 18/40\n",
            "72/72 [==============================] - 29s 403ms/step - loss: 1.4188\n",
            "Epoch 19/40\n",
            "72/72 [==============================] - 30s 410ms/step - loss: 1.3593\n",
            "Epoch 20/40\n",
            "72/72 [==============================] - 30s 410ms/step - loss: 1.3167\n",
            "Epoch 21/40\n",
            "72/72 [==============================] - 29s 406ms/step - loss: 1.2660\n",
            "Epoch 22/40\n",
            "72/72 [==============================] - 30s 411ms/step - loss: 1.2212\n",
            "Epoch 23/40\n",
            "72/72 [==============================] - 29s 403ms/step - loss: 1.1985\n",
            "Epoch 24/40\n",
            "72/72 [==============================] - 29s 399ms/step - loss: 1.1584\n",
            "Epoch 25/40\n",
            "72/72 [==============================] - 29s 407ms/step - loss: 1.1417\n",
            "Epoch 26/40\n",
            "72/72 [==============================] - 30s 414ms/step - loss: 1.1192\n",
            "Epoch 27/40\n",
            "72/72 [==============================] - 30s 410ms/step - loss: 1.0904\n",
            "Epoch 28/40\n",
            "72/72 [==============================] - 30s 411ms/step - loss: 1.0799\n",
            "Epoch 29/40\n",
            "72/72 [==============================] - 29s 398ms/step - loss: 1.0706\n",
            "Epoch 30/40\n",
            "72/72 [==============================] - 30s 411ms/step - loss: 1.0514\n",
            "Epoch 31/40\n",
            "72/72 [==============================] - 30s 417ms/step - loss: 1.0390\n",
            "Epoch 32/40\n",
            "72/72 [==============================] - 30s 414ms/step - loss: 1.0294\n",
            "Epoch 33/40\n",
            "72/72 [==============================] - 30s 414ms/step - loss: 1.0177\n",
            "Epoch 34/40\n",
            "72/72 [==============================] - 30s 410ms/step - loss: 1.0060\n",
            "Epoch 35/40\n",
            "72/72 [==============================] - 29s 397ms/step - loss: 1.0068\n",
            "Epoch 36/40\n",
            "72/72 [==============================] - 29s 404ms/step - loss: 0.9970\n",
            "Epoch 37/40\n",
            "72/72 [==============================] - 31s 422ms/step - loss: 0.9881\n",
            "Epoch 38/40\n",
            "72/72 [==============================] - 28s 394ms/step - loss: 0.9880\n",
            "Epoch 39/40\n",
            "72/72 [==============================] - 29s 405ms/step - loss: 0.9860\n",
            "Epoch 40/40\n",
            "72/72 [==============================] - 29s 398ms/step - loss: 0.9774\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.keras.callbacks.History at 0x7f5b1e27a940>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "m_VbeU-vzeHm",
        "outputId": "6a3f0f67-4745-4338-9c77-2156ddcdc501"
      },
      "source": [
        "model_emb.save('data/out/lstm_model_emb')"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:absl:Found untraced functions such as lstm_cell_2_layer_call_fn, lstm_cell_2_layer_call_and_return_conditional_losses, lstm_cell_2_layer_call_fn, lstm_cell_2_layer_call_and_return_conditional_losses, lstm_cell_2_layer_call_and_return_conditional_losses while saving (showing 5 of 5). These functions will not be directly callable after loading.\n",
            "WARNING:absl:Found untraced functions such as lstm_cell_2_layer_call_fn, lstm_cell_2_layer_call_and_return_conditional_losses, lstm_cell_2_layer_call_fn, lstm_cell_2_layer_call_and_return_conditional_losses, lstm_cell_2_layer_call_and_return_conditional_losses while saving (showing 5 of 5). These functions will not be directly callable after loading.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Assets written to: data/out/lstm_model_emb/assets\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Assets written to: data/out/lstm_model_emb/assets\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yWlpBP6lzhpk"
      },
      "source": [
        "#model_emb = models.load_model('data/out/lstm_model_emb')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nea-1TkMzped"
      },
      "source": [
        "pred_emb = ModelPredict(model_emb, text_prefix, token2ind, ind2token, max_len, embedding=True)"
      ],
      "execution_count": 21,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qWnwmx8VzxNB",
        "outputId": "31ac6b6f-55ba-464e-f318-665d89cbc10d"
      },
      "source": [
        "temperatures = [1, 0.7, 0.4, 0.1]\r\n",
        "\r\n",
        "for temperature in temperatures:\r\n",
        "  print('temperature:', temperature)\r\n",
        "  print(pred_emb.generate_sequence(100, temperature=0.7))\r\n",
        "  print('\\n')"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "temperature: 1\n",
            "Once upon a time there a majesty a young ball my grand- good sister, this is good smile. I know a; if I am a minute to see under the tree back to the table.' So he soon I have a thicket. But if you behold me that I have not heard this they are my great will, and will we have already already already killed here and drink as a rock to; and, as He went, to look for the second, and told them to marry you.' Marya Morevna\n",
            "\n",
            "\n",
            "temperature: 0.7\n",
            "Once upon a time there was a great many people, a splendid food, and a man who were out, he returned to her hut and went out to his mouth and said : Little Muck, that I wanted to go to the king that I cannot refuse. And he bowed to the ground, and were about with the horse, and sang that he was thought to a request, and from it was stuck to her, you will see, and you will come young man with you,' he asked. What do you\n",
            "\n",
            "\n",
            "temperature: 0.4\n",
            "Once upon a time there was a great rock of a green fig, and they entered his garden and were looking. Little Muck asked Pinocchio cursed only one who had kissed her voice broken her upon her, and he returned, and he came to the chamber in a corner. Now, when it was not altogether. O King, my little fish. But my thoughts able to tell him half his hand and said to him, he thought the king's son-- he was called the first time, and he went to the palace\n",
            "\n",
            "\n",
            "temperature: 0.1\n",
            "Once upon a time there was a great say for them. The mother had a golden lion- one would only only more then he would have been able to find who he was so much that he would help her but whether she would not do not do this would not work any more to eat, he got his back, he was waiting for his old woman. In her eleven of a short and son. From I took my peace, and I have got it; for you always someone he does that things?' And\n",
            "\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MUtj7TdP2Npc"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}