30_analogy_using_embeddings.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QT2lDzMy4Mgh"
      },
      "source": [
        "<center><div style=\"direction:rtl;font-family:B Lotus, B Nazanin, Tahoma\">به نام خدا</div></center>\n",
        "<img src=\"https://github.com/Alireza-Akhavan/rnn-notebooks/blob/master/logo.png?raw=1\" alt=\"class.vision\" style=\"width: 200px;\"/>\n",
        "<h1><center><div style=\"direction:rtl;font-family:B Lotus, B Nazanin, Tahoma\">قیاس کلمات (Word analogies)</div></center></h1>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uc0zYfMM4Mgl"
      },
      "source": [
        "<div style=\"direction:rtl;text-align:right;font-family:Tahoma\">\n",
        "کدها  با تغییرات برگرفته از کورس Sequence Models پروفسور Andrew NG است.\n",
        "</div>\n",
        "\n",
        "[https://www.coursera.org/learn/nlp-sequence-models](https://www.coursera.org/learn/nlp-sequence-models)\n",
        "\n",
        "<div style=\"direction:rtl;text-align:right;font-family:Tahoma\">\n",
        "بردار از قبل آموزش داده شده را می‌توانید از اینجا دانلود کنید:</div>\n",
        "\n",
        "https://nlp.stanford.edu/projects/glove/\n",
        "\n",
        "http://nlp.stanford.edu/data/glove.6B.zip\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!wget https://downloads.cs.stanford.edu/nlp/data/glove.6B.zip"
      ],
      "metadata": {
        "id": "K5_fW67c4RSp",
        "outputId": "df1a07a6-78d6-4c80-b76f-a12099ef59e3",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2023-09-11 10:55:46--  https://downloads.cs.stanford.edu/nlp/data/glove.6B.zip\n",
            "Resolving downloads.cs.stanford.edu (downloads.cs.stanford.edu)... 171.64.64.22\n",
            "Connecting to downloads.cs.stanford.edu (downloads.cs.stanford.edu)|171.64.64.22|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 862182613 (822M) [application/zip]\n",
            "Saving to: ‘glove.6B.zip’\n",
            "\n",
            "glove.6B.zip        100%[===================>] 822.24M  5.00MB/s    in 2m 39s  \n",
            "\n",
            "2023-09-11 10:58:25 (5.18 MB/s) - ‘glove.6B.zip’ saved [862182613/862182613]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!unzip glove.6B.zip"
      ],
      "metadata": {
        "id": "9euvaAFE4ktH",
        "outputId": "9f17b54c-8613-46e8-fe15-9ae40ff4187e",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Archive:  glove.6B.zip\n",
            "  inflating: glove.6B.50d.txt        \n",
            "  inflating: glove.6B.100d.txt       \n",
            "  inflating: glove.6B.200d.txt       \n",
            "  inflating: glove.6B.300d.txt       \n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "ksZJprBL4Mgm"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "import os"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "OpFl3W1w4Mgo"
      },
      "outputs": [],
      "source": [
        "glove_dir = '.'\n",
        "\n",
        "embeddings_index = {}\n",
        "f = open(os.path.join(glove_dir, 'glove.6B.100d.txt'), encoding=\"utf8\")\n",
        "for line in f:\n",
        "    values = line.split()\n",
        "    word = values[0]\n",
        "    coefs = np.asarray(values[1:], dtype='float32')\n",
        "    embeddings_index[word] = coefs\n",
        "f.close()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QwMo9iyw4Mgr"
      },
      "source": [
        "# 1 - Cosine similarity\n",
        "\n",
        "To measure how similar two words are, we need a way to measure the degree of similarity between two embedding vectors for the two words. Given two vectors $u$ and $v$, cosine similarity is defined as follows:\n",
        "\n",
        "$$\\text{CosineSimilarity(u, v)} = \\frac {u . v} {||u||_2 ||v||_2} = cos(\\theta) \\tag{1}$$\n",
        "\n",
        "where $u.v$ is the dot product (or inner product) of two vectors, $||u||_2$ is the norm (or length) of the vector $u$, and $\\theta$ is the angle between $u$ and $v$. This similarity depends on the angle between $u$ and $v$. If $u$ and $v$ are very similar, their cosine similarity will be close to 1; if they are dissimilar, the cosine similarity will take a smaller value.\n",
        "\n",
        "<img src=\"https://github.com/Alireza-Akhavan/rnn-notebooks/blob/master/images/cosine_sim.png?raw=1\" style=\"width:800px;height:250px;\">\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "8oEhtKHM4Mgs"
      },
      "outputs": [],
      "source": [
        "from sklearn.metrics.pairwise import cosine_similarity\n",
        "\n",
        "def similarity(u, v):\n",
        "    return np.squeeze(cosine_similarity(u.reshape(1, -1), v.reshape(1, -1)))\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "FqKfIAkO4Mgt",
        "outputId": "c0fc3b3d-af3a-4314-fb9c-7350dcc615ff",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "cosine_similarity(father, mother) =  0.8656661\n",
            "cosine_similarity(ball, crocodile) =  0.15206575\n",
            "cosine_similarity(france - paris, tehran - iran) =  -0.6854124\n"
          ]
        }
      ],
      "source": [
        "father = embeddings_index[\"father\"]\n",
        "mother = embeddings_index[\"mother\"]\n",
        "ball = embeddings_index[\"ball\"]\n",
        "crocodile = embeddings_index[\"crocodile\"]\n",
        "france = embeddings_index[\"france\"]\n",
        "tehran = embeddings_index[\"tehran\"]\n",
        "paris = embeddings_index[\"paris\"]\n",
        "iran = embeddings_index[\"iran\"]\n",
        "print(\"cosine_similarity(father, mother) = \", similarity(father, mother))\n",
        "print(\"cosine_similarity(ball, crocodile) = \",similarity(ball, crocodile))\n",
        "print(\"cosine_similarity(france - paris, tehran - iran) = \",similarity(france - paris, tehran - iran))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Xcu-Q8TB4Mgu"
      },
      "source": [
        "## 2 - Word analogy task\n",
        "\n",
        "In the word analogy task, we complete the sentence <font color='brown'>\"*a* is to *b* as *c* is to **____**\"</font>. An example is <font color='brown'> '*man* is to *woman* as *king* is to *queen*' </font>. In detail, we are trying to find a word *d*, such that the associated word vectors $e_a, e_b, e_c, e_d$ are related in the following manner: $e_b - e_a \\approx e_d - e_c$. We will measure the similarity between $e_b - e_a$ and $e_d - e_c$ using cosine similarity.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "XMXqj44L4Mgu",
        "outputId": "76cbad90-ac93-4d1f-a169-d271d3e6dc65",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([ 0.64706 , -0.068067,  0.15468 , -0.17408 , -0.29134 ,  0.76999 ,\n",
              "       -0.3192  , -0.25663 , -0.25082 , -0.036737, -0.25509 ,  0.29636 ,\n",
              "        0.5776  ,  0.49641 ,  0.19167 , -0.83888 ,  0.58482 , -0.38717 ,\n",
              "       -0.71591 ,  0.9519  , -0.37966 , -0.1131  ,  0.47154 ,  0.20921 ,\n",
              "        0.38197 ,  0.067582, -0.92879 , -1.1237  ,  0.84831 ,  0.68744 ,\n",
              "       -0.15472 ,  0.92714 ,  0.53371 , -0.037392, -0.856   ,  0.19056 ,\n",
              "       -0.014594,  0.15186 ,  0.53514 , -0.20306 , -0.35164 ,  0.33152 ,\n",
              "        1.1306  , -0.72787 , -0.19724 ,  0.031659, -0.24041 , -0.057617,\n",
              "        0.60473 , -0.49233 , -0.24405 , -0.3184  ,  0.96156 ,  1.0895  ,\n",
              "        0.21534 , -2.0542  , -1.0615  ,  0.052439,  0.57958 ,  0.2748  ,\n",
              "        0.91587 ,  0.85195 ,  0.36113 , -0.31901 ,  0.7784  , -0.36865 ,\n",
              "        0.64387 ,  0.33104 , -0.27181 ,  0.58524 , -0.15143 ,  0.11121 ,\n",
              "        0.2126  , -0.60345 ,  0.16148 ,  0.32952 , -0.1354  , -0.30629 ,\n",
              "       -0.89143 ,  0.091912,  0.49753 ,  0.55932 ,  0.19329 ,  0.044859,\n",
              "       -1.0416  , -0.41566 , -0.54174 , -0.7244  , -0.57492 , -1.1188  ,\n",
              "        0.087097, -0.2992  ,  0.87227 ,  0.86996 , -0.89641 , -0.28259 ,\n",
              "       -0.47295 , -0.74062 , -0.39    , -0.78099 ], dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ],
      "source": [
        "embeddings_index[\"father\"]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "NMHUSp9e4Mgv"
      },
      "outputs": [],
      "source": [
        "def complete_analogy(word_a, word_b, word_c, embeddings_index):\n",
        "\n",
        "    # convert words to lower case\n",
        "    word_a, word_b, word_c = word_a.lower(), word_b.lower(), word_c.lower()\n",
        "\n",
        "    # Get the word embeddings v_a, v_b and v_c\n",
        "    e_a, e_b, e_c = embeddings_index[word_a], embeddings_index[word_b], embeddings_index[word_c]\n",
        "\n",
        "    words = embeddings_index.keys()\n",
        "    max_cosine_sim = -100              # Initialize max_cosine_sim to a large negative number\n",
        "    best_word = None                   # Initialize best_word with None, it will help keep track of the word to output\n",
        "\n",
        "    # loop over the whole word vector set\n",
        "    for w in words:\n",
        "        # to avoid best_word being one of the input words, pass on them.\n",
        "        if w in [word_a, word_b, word_c] :\n",
        "            continue\n",
        "\n",
        "        # Compute cosine similarity between the vector (e_b - e_a) and the vector ((w's vector representation) - e_c)  (≈1 line)\n",
        "        cosine_sim = similarity(e_b - e_a, embeddings_index[w] - e_c)\n",
        "\n",
        "        # If the cosine_sim is more than the max_cosine_sim seen so far,\n",
        "            # then: set the new max_cosine_sim to the current cosine_sim and the best_word to the current word (≈3 lines)\n",
        "        if cosine_sim > max_cosine_sim:\n",
        "            max_cosine_sim = cosine_sim\n",
        "            best_word = w\n",
        "\n",
        "    return best_word"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wU7Z20-x4Mgx"
      },
      "source": [
        "Run the cell below to test your code, this may take 1-2 minutes."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "eSnt9MfU4Mgx",
        "outputId": "3685ee2a-142a-4432-ea74-e59ad5e18580",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'iranian'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 11
        }
      ],
      "source": [
        "complete_analogy('china', 'chinese', 'iran', embeddings_index)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "flcJWZYo4Mgy",
        "outputId": "7b76b0d1-efa2-4694-95f1-fc80925c32b9"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "'tehran'"
            ]
          },
          "execution_count": 10,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "complete_analogy('india', 'delhi', 'iran', embeddings_index)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xZKa8txu4Mgy",
        "outputId": "c6dfe35f-2a08-4f31-9147-e347ffdaec09"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "'girl'"
            ]
          },
          "execution_count": 11,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "complete_analogy('man', 'woman', 'boy', embeddings_index)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MyxLbpRy4Mgz",
        "outputId": "7941dee4-e858-418d-f2b3-9d220516479c"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "'bigger'"
            ]
          },
          "execution_count": 12,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "complete_analogy('small', 'smaller', 'big', embeddings_index)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "BXnB01w94Mgz",
        "outputId": "b01cb5f9-2d5c-4981-f801-97f3125ea390"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "'inuktitut'"
            ]
          },
          "execution_count": 9,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "complete_analogy('iran', 'farsi', 'canada', embeddings_index)"
      ]
    }
  ],
  "metadata": {
    "coursera": {
      "course_slug": "nlp-sequence-models",
      "graded_item_id": "8hb5s",
      "launcher_item_id": "5NrJ6"
    },
    "kernelspec": {
      "display_name": "tf2-GPU",
      "language": "python",
      "name": "tf2"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.9"
    },
    "colab": {
      "provenance": []
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}