46-debiasing-word-vectors.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"http://akhavanpour.ir/notebook/images/srttu.gif\" alt=\"SRTTU\" style=\"width: 150px;\"/>\n",
    "\n",
    "[![Azure Notebooks](https://notebooks.azure.com/launch.png)](https://notebooks.azure.com/import/gh/Alireza-Akhavan/class.vision)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Debiasing word vectors\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"direction:rtl;text-align:right;font-family:Tahoma\">\n",
    "کدها  با تغییرات برگرفته از کورس Sequence Models پروفسور Andrew NG است.\n",
    "</div>\n",
    "\n",
    "[https://www.coursera.org/learn/nlp-sequence-models](https://www.coursera.org/learn/nlp-sequence-models)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "glove_dir = 'D:/data/'\n",
    "\n",
    "embeddings_index = {}\n",
    "f = open(os.path.join(glove_dir, 'glove.6B.50d.txt'), encoding=\"utf8\")\n",
    "for line in f:\n",
    "    values = line.split()\n",
    "    word = values[0]\n",
    "    coefs = np.asarray(values[1:], dtype='float32')\n",
    "    embeddings_index[word] = coefs\n",
    "f.close()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "\n",
    "def similarity(u, v):\n",
    "    return np.squeeze(cosine_similarity(u.reshape(1, -1), v.reshape(1, -1)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following exercise, you will examine gender biases that can be reflected in a word embedding, and explore algorithms for reducing the bias. In addition to learning about the topic of debiasing, this exercise will also help hone your intuition about what word vectors are doing. This section involves a bit of linear algebra, though you can probably complete it even without being expert in linear algebra, and we encourage you to give it a shot. This portion of the notebook is optional and is not graded. \n",
    "\n",
    "Lets first see how the GloVe word embeddings relate to gender. You will first compute a vector $g = e_{woman}-e_{man}$, where $e_{woman}$ represents the word vector corresponding to the word *woman*, and $e_{man}$ corresponds to the word vector corresponding to the word *man*. The resulting vector $g$ roughly encodes the concept of \"gender\". (You might get a more accurate representation if you compute $g_1 = e_{mother}-e_{father}$, $g_2 = e_{girl}-e_{boy}$, etc. and average over them. But just using $e_{woman}-e_{man}$ will give good enough results for now.) \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-0.087144    0.2182     -0.40985996 -0.03922001 -0.10320008  0.94165003\n",
      " -0.06042001  0.32988     0.46144    -0.35962     0.31102    -0.86824\n",
      "  0.96006     0.01073003  0.24337     0.08193001 -1.02722    -0.21122\n",
      "  0.695044   -0.00222     0.29106003  0.50530005 -0.099454    0.40445\n",
      "  0.30181003  0.1355002  -0.06060004 -0.07131001 -0.19245    -0.06115001\n",
      " -0.3204      0.07165    -0.13337001 -0.25068715 -0.14292999 -0.224957\n",
      " -0.14899999  0.048882    0.12191002 -0.27362    -0.16547601 -0.20426002\n",
      "  0.54376    -0.27142498 -0.10244995 -0.32108003  0.2516     -0.33454996\n",
      " -0.04371002  0.01258   ]\n"
     ]
    }
   ],
   "source": [
    "g = embeddings_index['woman'] - embeddings_index['man']\n",
    "print(g)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, you will consider the cosine similarity of different words with $g$. Consider what a positive value of similarity means vs a negative cosine similarity. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "List of names and their similarities with constructed vector:\n",
      "ali -0.27142075\n",
      "john -0.2316336\n",
      "sara 0.2272694\n",
      "reza -0.0793043\n",
      "hanie 0.18960176\n",
      "alireza -0.023223324\n",
      "reza -0.0793043\n",
      "katy 0.2831069\n",
      "yasmin 0.23313855\n"
     ]
    }
   ],
   "source": [
    "print ('List of names and their similarities with constructed vector:')\n",
    "\n",
    "# girls and boys name\n",
    "name_list = ['ali', 'john', 'sara', 'reza', 'hanie', 'alireza', 'reza', 'katy', 'yasmin']\n",
    "\n",
    "for w in name_list:\n",
    "    print (w, similarity(embeddings_index[w], g))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, female first names tend to have a positive cosine similarity with our constructed vector $g$, while male first names tend to have a negative cosine similarity. This is not suprising, and the result seems acceptable. \n",
    "\n",
    "But let's try with some other words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Other words and their similarities:\n",
      "lipstick 0.27691913\n",
      "guns -0.18884854\n",
      "science -0.060829073\n",
      "arts 0.008189303\n",
      "literature 0.06472502\n",
      "warrior -0.20920166\n",
      "doctor 0.11895285\n",
      "tree -0.07089399\n",
      "receptionist 0.33077937\n",
      "technology -0.13193733\n",
      "fashion 0.035638936\n",
      "teacher 0.1792092\n",
      "engineer -0.080392815\n",
      "pilot 0.001076434\n",
      "computer -0.10330359\n",
      "singer 0.18500516\n"
     ]
    }
   ],
   "source": [
    "print('Other words and their similarities:')\n",
    "word_list = ['lipstick', 'guns', 'science', 'arts', 'literature', 'warrior','doctor', 'tree', 'receptionist', \n",
    "             'technology',  'fashion', 'teacher', 'engineer', 'pilot', 'computer', 'singer']\n",
    "for w in word_list:\n",
    "    print (w, similarity(embeddings_index[w], g))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Do you notice anything surprising? It is astonishing how these results reflect certain unhealthy gender stereotypes. For example, \"computer\" is closer to \"man\" while \"literature\" is closer to \"woman\". Ouch! \n",
    "\n",
    "We'll see below how to reduce the bias of these vectors, using an algorithm due to [Boliukbasi et al., 2016](https://arxiv.org/abs/1607.06520). Note that some word pairs such as \"actor\"/\"actress\" or \"grandmother\"/\"grandfather\" should remain gender specific, while other words such as \"receptionist\" or \"technology\" should be neutralized, i.e. not be gender-related. You will have to treat these two type of words differently when debiasing.\n",
    "\n",
    "### 1 - Neutralize bias for non-gender specific words \n",
    "\n",
    "The figure below should help you visualize what neutralizing does. If you're using a 50-dimensional word embedding, the 50 dimensional space can be split into two parts: The bias-direction $g$, and the remaining 49 dimensions, which we'll call $g_{\\perp}$. In linear algebra, we say that the 49 dimensional $g_{\\perp}$ is perpendicular (or \"othogonal\") to $g$, meaning it is at 90 degrees to $g$. The neutralization step takes a vector such as $e_{receptionist}$ and zeros out the component in the direction of $g$, giving us $e_{receptionist}^{debiased}$. \n",
    "\n",
    "Even though $g_{\\perp}$ is 49 dimensional, given the limitations of what we can draw on a screen, we illustrate it using a 1 dimensional axis below. \n",
    "\n",
    "<img src=\"images/neutral.png\" style=\"width:800px;height:300px;\">\n",
    "<caption><center> **Figure 2**: The word vector for \"receptionist\" represented before and after applying the neutralize operation. </center></caption>\n",
    "\n",
    "**Exercise**: Implement `neutralize()` to remove the bias of words such as \"receptionist\" or \"scientist\". Given an input embedding $e$, you can use the following formulas to compute $e^{debiased}$: \n",
    "\n",
    "$$e^{bias\\_component} = \\frac{e \\cdot g}{||g||_2^2} * g\\tag{2}$$\n",
    "$$e^{debiased} = e - e^{bias\\_component}\\tag{3}$$\n",
    "\n",
    "If you are an expert in linear algebra, you may recognize $e^{bias\\_component}$ as the projection of $e$ onto the direction $g$. If you're not an expert in linear algebra, don't worry about this.\n",
    "\n",
    "<!-- \n",
    "**Reminder**: a vector $u$ can be split into two parts: its projection over a vector-axis $v_B$ and its projection over the axis orthogonal to $v$:\n",
    "$$u = u_B + u_{\\perp}$$\n",
    "where : $u_B = $ and $ u_{\\perp} = u - u_B $\n",
    "!--> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "def neutralize(word, g, embeddings_index):\n",
    "    \"\"\"\n",
    "    Removes the bias of \"word\" by projecting it on the space orthogonal to the bias axis. \n",
    "    This function ensures that gender neutral words are zero in the gender subspace.\n",
    "    \n",
    "    Arguments:\n",
    "        word -- string indicating the word to debias\n",
    "        g -- numpy-array of shape (50,), corresponding to the bias axis (such as gender)\n",
    "        word_to_vec_map -- dictionary mapping words to their corresponding vectors.\n",
    "    \n",
    "    Returns:\n",
    "        e_debiased -- neutralized word vector representation of the input \"word\"\n",
    "    \"\"\"\n",
    "    \n",
    "    # Select word vector representation of \"word\". Use word_to_vec_map. (≈ 1 line)\n",
    "    e = embeddings_index[word]\n",
    "    \n",
    "    # Compute e_biascomponent using the formula give above. (≈ 1 line)\n",
    "    e_biascomponent = np.dot(e ,g) / np.sum(g * g) * g\n",
    " \n",
    "    # Neutralize e by substracting e_biascomponent from it \n",
    "    # e_debiased should be equal to its orthogonal projection. (≈ 1 line)\n",
    "    e_debiased = e - e_biascomponent\n",
    "    \n",
    "    return e_debiased"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cosine similarity between receptionist and g, before neutralizing:  0.33077937\n",
      "cosine similarity between receptionist and g, after neutralizing:  6.949995e-08\n"
     ]
    }
   ],
   "source": [
    "e = \"receptionist\"\n",
    "print(\"cosine similarity between \" + e + \" and g, before neutralizing: \", similarity(embeddings_index[\"receptionist\"], g))\n",
    "\n",
    "e_debiased = neutralize(\"receptionist\", g, embeddings_index)\n",
    "print(\"cosine similarity between \" + e + \" and g, after neutralizing: \", similarity(e_debiased, g))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2 - Equalization algorithm for gender-specific words\n",
    "\n",
    "Next, lets see how debiasing can also be applied to word pairs such as \"actress\" and \"actor.\" Equalization is applied to pairs of words that you might want to have differ only through the gender property. As a concrete example, suppose that \"actress\" is closer to \"babysit\" than \"actor.\" By applying neutralizing to \"babysit\" we can reduce the gender-stereotype associated with babysitting. But this still does not guarantee that \"actor\" and \"actress\" are equidistant from \"babysit.\" The equalization algorithm takes care of this. \n",
    "\n",
    "The key idea behind equalization is to make sure that a particular pair of words are equi-distant from the 49-dimensional $g_\\perp$. The equalization step also ensures that the two equalized steps are now the same distance from $e_{receptionist}^{debiased}$, or from any other work that has been neutralized. In pictures, this is how equalization works: \n",
    "\n",
    "<img src=\"images/equalize10.png\" style=\"width:800px;height:400px;\">\n",
    "\n",
    "\n",
    "The derivation of the linear algebra to do this is a bit more complex. (See Bolukbasi et al., 2016 for details.) But the key equations are: \n",
    "\n",
    "$$ \\mu = \\frac{e_{w1} + e_{w2}}{2}\\tag{4}$$ \n",
    "\n",
    "$$ \\mu_{B} = \\frac {\\mu \\cdot \\text{bias_axis}}{||\\text{bias_axis}||_2^2} *\\text{bias_axis}\n",
    "\\tag{5}$$ \n",
    "\n",
    "$$\\mu_{\\perp} = \\mu - \\mu_{B} \\tag{6}$$\n",
    "\n",
    "$$ e_{w1B} = \\frac {e_{w1} \\cdot \\text{bias_axis}}{||\\text{bias_axis}||_2^2} *\\text{bias_axis}\n",
    "\\tag{7}$$ \n",
    "$$ e_{w2B} = \\frac {e_{w2} \\cdot \\text{bias_axis}}{||\\text{bias_axis}||_2^2} *\\text{bias_axis}\n",
    "\\tag{8}$$\n",
    "\n",
    "\n",
    "$$e_{w1B}^{corrected} = \\sqrt{ |{1 - ||\\mu_{\\perp} ||^2_2} |} * \\frac{e_{\\text{w1B}} - \\mu_B} {|(e_{w1} - \\mu_{\\perp}) - \\mu_B)|} \\tag{9}$$\n",
    "\n",
    "\n",
    "$$e_{w2B}^{corrected} = \\sqrt{ |{1 - ||\\mu_{\\perp} ||^2_2} |} * \\frac{e_{\\text{w2B}} - \\mu_B} {|(e_{w2} - \\mu_{\\perp}) - \\mu_B)|} \\tag{10}$$\n",
    "\n",
    "$$e_1 = e_{w1B}^{corrected} + \\mu_{\\perp} \\tag{11}$$\n",
    "$$e_2 = e_{w2B}^{corrected} + \\mu_{\\perp} \\tag{12}$$\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "def equalize(pair, bias_axis, embeddings_index):\n",
    "    \"\"\"\n",
    "    Debias gender specific words by following the equalize method described in the figure above.\n",
    "    \n",
    "    Arguments:\n",
    "    pair -- pair of strings of gender specific words to debias, e.g. (\"actress\", \"actor\") \n",
    "    bias_axis -- numpy-array of shape (50,), vector corresponding to the bias axis, e.g. gender\n",
    "    word_to_vec_map -- dictionary mapping words to their corresponding vectors\n",
    "    \n",
    "    Returns\n",
    "    e_1 -- word vector corresponding to the first word\n",
    "    e_2 -- word vector corresponding to the second word\n",
    "    \"\"\"\n",
    "    \n",
    "    # Step 1: Select word vector representation of \"word\". Use word_to_vec_map. (≈ 2 lines)\n",
    "    w1, w2 = pair\n",
    "    e_w1, e_w2 = embeddings_index[w1],embeddings_index[w2]\n",
    "    \n",
    "    # Step 2: Compute the mean of e_w1 and e_w2 (≈ 1 line)\n",
    "    mu = (e_w1 + e_w2) / 2\n",
    "\n",
    "    # Step 3: Compute the projections of mu over the bias axis and the orthogonal axis (≈ 2 lines)\n",
    "    mu_B = np.dot(mu, bias_axis) / np.sum(bias_axis * bias_axis) * bias_axis\n",
    "    mu_orth = mu - mu_B\n",
    "\n",
    "    # Step 4: Use equations (7) and (8) to compute e_w1B and e_w2B (≈2 lines)\n",
    "    e_w1B = np.dot(e_w1, bias_axis) / np.sum(bias_axis * bias_axis) * bias_axis\n",
    "    e_w2B = np.dot(e_w2, bias_axis) / np.sum(bias_axis * bias_axis) * bias_axis\n",
    "        \n",
    "    # Step 5: Adjust the Bias part of e_w1B and e_w2B using the formulas (9) and (10) given above (≈2 lines)\n",
    "    corrected_e_w1B = np.sqrt(np.abs(1 - np.sum(mu_orth * mu_orth))) * (e_w1B - mu_B) / np.linalg.norm(e_w1 - mu_orth - mu_B)\n",
    "    corrected_e_w2B = np.sqrt(np.abs(1 - np.sum(mu_orth * mu_orth))) * (e_w2B - mu_B) / np.linalg.norm(e_w2 - mu_orth - mu_B)\n",
    "\n",
    "    # Step 6: Debias by equalizing e1 and e2 to the sum of their corrected projections (≈2 lines)\n",
    "    e1 = corrected_e_w1B + mu_orth\n",
    "    e2 = corrected_e_w2B + mu_orth\n",
    "                                                                \n",
    "    \n",
    "    return e1, e2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cosine similarities before equalizing:\n",
      "cosine_similarity(word_to_vec_map[\"man\"], gender) =  -0.11711098\n",
      "cosine_similarity(word_to_vec_map[\"woman\"], gender) =  0.35666615\n",
      "\n",
      "cosine similarities after equalizing:\n",
      "cosine_similarity(e1, gender) =  -0.7004364\n",
      "cosine_similarity(e2, gender) =  0.7004364\n"
     ]
    }
   ],
   "source": [
    "print(\"cosine similarities before equalizing:\")\n",
    "print(\"cosine_similarity(word_to_vec_map[\\\"man\\\"], gender) = \", similarity(embeddings_index[\"man\"], g))\n",
    "print(\"cosine_similarity(word_to_vec_map[\\\"woman\\\"], gender) = \", similarity(embeddings_index[\"woman\"], g))\n",
    "print()\n",
    "e1, e2 = equalize((\"man\", \"woman\"), g, embeddings_index)\n",
    "print(\"cosine similarities after equalizing:\")\n",
    "print(\"cosine_similarity(e1, gender) = \", similarity(e1, g))\n",
    "print(\"cosine_similarity(e2, gender) = \", similarity(e2, g))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "\n",
    "These debiasing algorithms are very helpful for reducing bias, but are not perfect and do not eliminate all traces of bias. For example, one weakness of this implementation was that the bias direction $g$ was defined using only the pair of words _woman_ and _man_. As discussed earlier, if $g$ were defined by computing $g_1 = e_{woman} - e_{man}$; $g_2 = e_{mother} - e_{father}$; $g_3 = e_{girl} - e_{boy}$; and so on and averaging over them, you would obtain a better estimate of the \"gender\" dimension in the 50 dimensional word embedding space. Feel free to play with such variants as well.  \n",
    "                     "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**References**:\n",
    "- The debiasing algorithm is from Bolukbasi et al., 2016, [Man is to Computer Programmer as Woman is to\n",
    "Homemaker? Debiasing Word Embeddings](https://papers.nips.cc/paper/6228-man-is-to-computer-programmer-as-woman-is-to-homemaker-debiasing-word-embeddings.pdf)\n",
    "- The GloVe word embeddings were due to Jeffrey Pennington, Richard Socher, and Christopher D. Manning. (https://nlp.stanford.edu/projects/glove/)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<div style=\"direction:rtl;text-align:right;font-family:B Lotus, B Nazanin, Tahoma\"> دانشگاه تربیت دبیر شهید رجایی<br>مباحث ویژه 2 - یادگیری عمیق پیشرفته<br>علیرضا اخوان پور<br>97-98<br>\n",
    "</div>\n",
    "<a href=\"https://www.srttu.edu/\">SRTTU.edu</a> - <a href=\"http://class.vision\">Class.Vision</a> - <a href=\"http://AkhavanPour.ir\">AkhavanPour.ir</a>\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "coursera": {
   "course_slug": "nlp-sequence-models",
   "graded_item_id": "8hb5s",
   "launcher_item_id": "5NrJ6"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}