A4_chunker.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.090144,
     "end_time": "2020-12-11T20:29:48.834520",
     "exception": false,
     "start_time": "2020-12-11T20:29:48.744376",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Extracting syntactic groups using machine learning techniques\n",
    "Author: Pierre Nugues"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.086467,
     "end_time": "2020-12-11T20:29:49.009958",
     "exception": false,
     "start_time": "2020-12-11T20:29:48.923491",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "In this assignment, you will create a system to extract syntactic groups from a text. You will apply it to the CoNLL 2000 dataset. In addition, you will try to link a few extracted named entities to real things using wikipedia."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.094678,
     "end_time": "2020-12-11T20:29:49.191339",
     "exception": false,
     "start_time": "2020-12-11T20:29:49.096661",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Objectives"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.119434,
     "end_time": "2020-12-11T20:29:49.419450",
     "exception": false,
     "start_time": "2020-12-11T20:29:49.300016",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "The objectives of this assignment are to:\n",
    "* Write a program to detect partial syntactic structures\n",
    "* Extract named entities and link them to real things using Wikipedia\n",
    "* Understand the principles of supervised machine learning techniques applied to language processing\n",
    "* Use a popular machine learning toolkit: scikit-learn\n",
    "* Write a short report of 2 to 3 pages on the assignment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.091028,
     "end_time": "2020-12-11T20:29:49.606992",
     "exception": false,
     "start_time": "2020-12-11T20:29:49.515964",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Choosing a training and a test sets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.091128,
     "end_time": "2020-12-11T20:29:49.785720",
     "exception": false,
     "start_time": "2020-12-11T20:29:49.694592",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "* As annotated data and annotation scheme, you will use the data available from [CoNLL 2000](https://www.clips.uantwerpen.be/conll2000/chunking/).\n",
    "* Download both the training and test sets and decompress them.\n",
    "* Read the description of the CoNLL 2000 task"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.088922,
     "end_time": "2020-12-11T20:29:49.962848",
     "exception": false,
     "start_time": "2020-12-11T20:29:49.873926",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Loading the corpus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.086648,
     "end_time": "2020-12-11T20:29:50.144760",
     "exception": false,
     "start_time": "2020-12-11T20:29:50.058112",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### The datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.087952,
     "end_time": "2020-12-11T20:29:50.329370",
     "exception": false,
     "start_time": "2020-12-11T20:29:50.241418",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "You may need to adjust the paths to load the datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:29:50.507131Z",
     "iopub.status.busy": "2020-12-11T20:29:50.505958Z",
     "iopub.status.idle": "2020-12-11T20:29:50.510460Z",
     "shell.execute_reply": "2020-12-11T20:29:50.510958Z"
    },
    "papermill": {
     "duration": 0.095117,
     "end_time": "2020-12-11T20:29:50.511143",
     "exception": false,
     "start_time": "2020-12-11T20:29:50.416026",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "train_file = '../input/conll-corpora/conll2000/train.txt'\n",
    "test_file = '../input/conll-corpora/conll2000/test.txt'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.086855,
     "end_time": "2020-12-11T20:29:50.686264",
     "exception": false,
     "start_time": "2020-12-11T20:29:50.599409",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "#### Reading the files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.086295,
     "end_time": "2020-12-11T20:29:50.860761",
     "exception": false,
     "start_time": "2020-12-11T20:29:50.774466",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Read the functions below to load the datasets. They store the corpus in a list of sentences. Each sentence is a list of rows, where each row is a dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:29:51.039025Z",
     "iopub.status.busy": "2020-12-11T20:29:51.038012Z",
     "iopub.status.idle": "2020-12-11T20:29:51.043668Z",
     "shell.execute_reply": "2020-12-11T20:29:51.044163Z"
    },
    "papermill": {
     "duration": 0.095779,
     "end_time": "2020-12-11T20:29:51.044320",
     "exception": false,
     "start_time": "2020-12-11T20:29:50.948541",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def read_sentences(file):\n",
    "    \"\"\"\n",
    "    Creates a list of sentences from the corpus\n",
    "    Each sentence is a string\n",
    "    :param file:\n",
    "    :return:\n",
    "    \"\"\"\n",
    "    f = open(file).read().strip()\n",
    "    sentences = f.split('\\n\\n')\n",
    "    return sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:29:51.233171Z",
     "iopub.status.busy": "2020-12-11T20:29:51.232198Z",
     "iopub.status.idle": "2020-12-11T20:29:51.239256Z",
     "shell.execute_reply": "2020-12-11T20:29:51.239728Z"
    },
    "papermill": {
     "duration": 0.100543,
     "end_time": "2020-12-11T20:29:51.239900",
     "exception": false,
     "start_time": "2020-12-11T20:29:51.139357",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def split_rows(sentences, column_names):\n",
    "    \"\"\"\n",
    "    Creates a list of sentence where each sentence is a list of lines\n",
    "    Each line is a dictionary of columns\n",
    "    :param sentences:\n",
    "    :param column_names:\n",
    "    :return:\n",
    "    \"\"\"\n",
    "    new_sentences = []\n",
    "    for sentence in sentences:\n",
    "        rows = sentence.split('\\n')\n",
    "        sentence = [dict(zip(column_names, row.split())) for row in rows]\n",
    "        new_sentences.append(sentence)\n",
    "    return new_sentences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.086884,
     "end_time": "2020-12-11T20:29:51.415411",
     "exception": false,
     "start_time": "2020-12-11T20:29:51.328527",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Loading dictionaries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.090164,
     "end_time": "2020-12-11T20:29:51.593536",
     "exception": false,
     "start_time": "2020-12-11T20:29:51.503372",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "The CoNLL 2000 files have three columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:29:51.776815Z",
     "iopub.status.busy": "2020-12-11T20:29:51.775812Z",
     "iopub.status.idle": "2020-12-11T20:29:51.779321Z",
     "shell.execute_reply": "2020-12-11T20:29:51.778662Z"
    },
    "papermill": {
     "duration": 0.098049,
     "end_time": "2020-12-11T20:29:51.779435",
     "exception": false,
     "start_time": "2020-12-11T20:29:51.681386",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "column_names = ['form', 'pos', 'chunk']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.089283,
     "end_time": "2020-12-11T20:29:51.956877",
     "exception": false,
     "start_time": "2020-12-11T20:29:51.867594",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We load the corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:29:52.136562Z",
     "iopub.status.busy": "2020-12-11T20:29:52.135857Z",
     "iopub.status.idle": "2020-12-11T20:29:52.561646Z",
     "shell.execute_reply": "2020-12-11T20:29:52.562172Z"
    },
    "papermill": {
     "duration": 0.517874,
     "end_time": "2020-12-11T20:29:52.562311",
     "exception": false,
     "start_time": "2020-12-11T20:29:52.044437",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[{'form': 'Confidence', 'pos': 'NN', 'chunk': 'B-NP'},\n",
       "  {'form': 'in', 'pos': 'IN', 'chunk': 'B-PP'},\n",
       "  {'form': 'the', 'pos': 'DT', 'chunk': 'B-NP'},\n",
       "  {'form': 'pound', 'pos': 'NN', 'chunk': 'I-NP'},\n",
       "  {'form': 'is', 'pos': 'VBZ', 'chunk': 'B-VP'},\n",
       "  {'form': 'widely', 'pos': 'RB', 'chunk': 'I-VP'},\n",
       "  {'form': 'expected', 'pos': 'VBN', 'chunk': 'I-VP'},\n",
       "  {'form': 'to', 'pos': 'TO', 'chunk': 'I-VP'},\n",
       "  {'form': 'take', 'pos': 'VB', 'chunk': 'I-VP'},\n",
       "  {'form': 'another', 'pos': 'DT', 'chunk': 'B-NP'},\n",
       "  {'form': 'sharp', 'pos': 'JJ', 'chunk': 'I-NP'},\n",
       "  {'form': 'dive', 'pos': 'NN', 'chunk': 'I-NP'},\n",
       "  {'form': 'if', 'pos': 'IN', 'chunk': 'B-SBAR'},\n",
       "  {'form': 'trade', 'pos': 'NN', 'chunk': 'B-NP'},\n",
       "  {'form': 'figures', 'pos': 'NNS', 'chunk': 'I-NP'},\n",
       "  {'form': 'for', 'pos': 'IN', 'chunk': 'B-PP'},\n",
       "  {'form': 'September', 'pos': 'NNP', 'chunk': 'B-NP'},\n",
       "  {'form': ',', 'pos': ',', 'chunk': 'O'},\n",
       "  {'form': 'due', 'pos': 'JJ', 'chunk': 'B-ADJP'},\n",
       "  {'form': 'for', 'pos': 'IN', 'chunk': 'B-PP'},\n",
       "  {'form': 'release', 'pos': 'NN', 'chunk': 'B-NP'},\n",
       "  {'form': 'tomorrow', 'pos': 'NN', 'chunk': 'B-NP'},\n",
       "  {'form': ',', 'pos': ',', 'chunk': 'O'},\n",
       "  {'form': 'fail', 'pos': 'VB', 'chunk': 'B-VP'},\n",
       "  {'form': 'to', 'pos': 'TO', 'chunk': 'I-VP'},\n",
       "  {'form': 'show', 'pos': 'VB', 'chunk': 'I-VP'},\n",
       "  {'form': 'a', 'pos': 'DT', 'chunk': 'B-NP'},\n",
       "  {'form': 'substantial', 'pos': 'JJ', 'chunk': 'I-NP'},\n",
       "  {'form': 'improvement', 'pos': 'NN', 'chunk': 'I-NP'},\n",
       "  {'form': 'from', 'pos': 'IN', 'chunk': 'B-PP'},\n",
       "  {'form': 'July', 'pos': 'NNP', 'chunk': 'B-NP'},\n",
       "  {'form': 'and', 'pos': 'CC', 'chunk': 'I-NP'},\n",
       "  {'form': 'August', 'pos': 'NNP', 'chunk': 'I-NP'},\n",
       "  {'form': \"'s\", 'pos': 'POS', 'chunk': 'B-NP'},\n",
       "  {'form': 'near-record', 'pos': 'JJ', 'chunk': 'I-NP'},\n",
       "  {'form': 'deficits', 'pos': 'NNS', 'chunk': 'I-NP'},\n",
       "  {'form': '.', 'pos': '.', 'chunk': 'O'}],\n",
       " [{'form': 'Chancellor', 'pos': 'NNP', 'chunk': 'O'},\n",
       "  {'form': 'of', 'pos': 'IN', 'chunk': 'B-PP'},\n",
       "  {'form': 'the', 'pos': 'DT', 'chunk': 'B-NP'},\n",
       "  {'form': 'Exchequer', 'pos': 'NNP', 'chunk': 'I-NP'},\n",
       "  {'form': 'Nigel', 'pos': 'NNP', 'chunk': 'B-NP'},\n",
       "  {'form': 'Lawson', 'pos': 'NNP', 'chunk': 'I-NP'},\n",
       "  {'form': \"'s\", 'pos': 'POS', 'chunk': 'B-NP'},\n",
       "  {'form': 'restated', 'pos': 'VBN', 'chunk': 'I-NP'},\n",
       "  {'form': 'commitment', 'pos': 'NN', 'chunk': 'I-NP'},\n",
       "  {'form': 'to', 'pos': 'TO', 'chunk': 'B-PP'},\n",
       "  {'form': 'a', 'pos': 'DT', 'chunk': 'B-NP'},\n",
       "  {'form': 'firm', 'pos': 'NN', 'chunk': 'I-NP'},\n",
       "  {'form': 'monetary', 'pos': 'JJ', 'chunk': 'I-NP'},\n",
       "  {'form': 'policy', 'pos': 'NN', 'chunk': 'I-NP'},\n",
       "  {'form': 'has', 'pos': 'VBZ', 'chunk': 'B-VP'},\n",
       "  {'form': 'helped', 'pos': 'VBN', 'chunk': 'I-VP'},\n",
       "  {'form': 'to', 'pos': 'TO', 'chunk': 'I-VP'},\n",
       "  {'form': 'prevent', 'pos': 'VB', 'chunk': 'I-VP'},\n",
       "  {'form': 'a', 'pos': 'DT', 'chunk': 'B-NP'},\n",
       "  {'form': 'freefall', 'pos': 'NN', 'chunk': 'I-NP'},\n",
       "  {'form': 'in', 'pos': 'IN', 'chunk': 'B-PP'},\n",
       "  {'form': 'sterling', 'pos': 'NN', 'chunk': 'B-NP'},\n",
       "  {'form': 'over', 'pos': 'IN', 'chunk': 'B-PP'},\n",
       "  {'form': 'the', 'pos': 'DT', 'chunk': 'B-NP'},\n",
       "  {'form': 'past', 'pos': 'JJ', 'chunk': 'I-NP'},\n",
       "  {'form': 'week', 'pos': 'NN', 'chunk': 'I-NP'},\n",
       "  {'form': '.', 'pos': '.', 'chunk': 'O'}]]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_sentences = read_sentences(train_file)\n",
    "train_corpus = split_rows(train_sentences, column_names)\n",
    "train_corpus[:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.090566,
     "end_time": "2020-12-11T20:29:52.742430",
     "exception": false,
     "start_time": "2020-12-11T20:29:52.651864",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Baseline chunker"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.089821,
     "end_time": "2020-12-11T20:29:52.922202",
     "exception": false,
     "start_time": "2020-12-11T20:29:52.832381",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Most statistical algorithms for language processing start with a so-called baseline. The baseline performance corresponds to the application of a minimal technique that is used to assess the difficulty of a task and for comparison with further programs.\n",
    "\n",
    "You will implement the baseline proposed by the organizers of the\n",
    "        <a href=\"https://www.clips.uantwerpen.be/conll2000/chunking/\">CoNLL 2000 shared task</a>, Sect. <i>Results</i>.\n",
    "1. Read it;\n",
    "2. In the report you will tell what do you think of it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.092946,
     "end_time": "2020-12-11T20:29:53.103439",
     "exception": false,
     "start_time": "2020-12-11T20:29:53.010493",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Auxiliary functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.089756,
     "end_time": "2020-12-11T20:29:53.318414",
     "exception": false,
     "start_time": "2020-12-11T20:29:53.228658",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "A function to count the parts of speech"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:29:53.503872Z",
     "iopub.status.busy": "2020-12-11T20:29:53.503158Z",
     "iopub.status.idle": "2020-12-11T20:29:53.506537Z",
     "shell.execute_reply": "2020-12-11T20:29:53.505879Z"
    },
    "papermill": {
     "duration": 0.099298,
     "end_time": "2020-12-11T20:29:53.506655",
     "exception": false,
     "start_time": "2020-12-11T20:29:53.407357",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def count_pos(corpus):\n",
    "    \"\"\"\n",
    "    Computes the part-of-speech distribution\n",
    "    in a CoNLL 2000 file\n",
    "    :param corpus:\n",
    "    :return:\n",
    "    \"\"\"\n",
    "    pos_cnt = {}\n",
    "    for sentence in corpus:\n",
    "        for row in sentence:\n",
    "            if row['pos'] in pos_cnt:\n",
    "                pos_cnt[row['pos']] += 1\n",
    "            else:\n",
    "                pos_cnt[row['pos']] = 1\n",
    "    return pos_cnt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.089835,
     "end_time": "2020-12-11T20:29:53.687661",
     "exception": false,
     "start_time": "2020-12-11T20:29:53.597826",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We first collect all the parts of speech and we count them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:29:53.873681Z",
     "iopub.status.busy": "2020-12-11T20:29:53.872438Z",
     "iopub.status.idle": "2020-12-11T20:29:53.952405Z",
     "shell.execute_reply": "2020-12-11T20:29:53.951852Z"
    },
    "papermill": {
     "duration": 0.17541,
     "end_time": "2020-12-11T20:29:53.952515",
     "exception": false,
     "start_time": "2020-12-11T20:29:53.777105",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'NN': 30147,\n",
       " 'IN': 22764,\n",
       " 'DT': 18335,\n",
       " 'VBZ': 4648,\n",
       " 'RB': 6607,\n",
       " 'VBN': 4763,\n",
       " 'TO': 5081,\n",
       " 'VB': 6017,\n",
       " 'JJ': 13085,\n",
       " 'NNS': 13619,\n",
       " 'NNP': 19884,\n",
       " ',': 10770,\n",
       " 'CC': 5372,\n",
       " 'POS': 1769,\n",
       " '.': 8827,\n",
       " 'VBP': 2868,\n",
       " 'VBG': 3272,\n",
       " 'PRP$': 1881,\n",
       " 'CD': 8315,\n",
       " '``': 1531,\n",
       " \"''\": 1493,\n",
       " 'VBD': 6745,\n",
       " 'EX': 206,\n",
       " 'MD': 2167,\n",
       " '#': 36,\n",
       " '(': 274,\n",
       " '$': 1750,\n",
       " ')': 281,\n",
       " 'NNPS': 420,\n",
       " 'PRP': 3820,\n",
       " 'JJS': 374,\n",
       " 'WP': 529,\n",
       " 'RBR': 321,\n",
       " 'JJR': 853,\n",
       " 'WDT': 955,\n",
       " 'WRB': 478,\n",
       " 'RBS': 191,\n",
       " 'PDT': 55,\n",
       " 'RP': 83,\n",
       " ':': 1047,\n",
       " 'FW': 38,\n",
       " 'WP$': 35,\n",
       " 'SYM': 6,\n",
       " 'UH': 15}"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pos_cnt = count_pos(train_corpus)\n",
    "pos_cnt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.089313,
     "end_time": "2020-12-11T20:29:54.130026",
     "exception": false,
     "start_time": "2020-12-11T20:29:54.040713",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Chunk distribution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.088803,
     "end_time": "2020-12-11T20:29:54.319326",
     "exception": false,
     "start_time": "2020-12-11T20:29:54.230523",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "You will compute the chunk distribution for each part of speech. You will use the training file to derive the distribution and you will store the results in a dictionary. Below, you have an excerpt of the expected results:\n",
    "```\n",
    "{'JJR':\n",
    "{'I-ADVP': 17, 'I-ADJP': 45, 'I-NP': 204, 'B-ADVP': 63,\n",
    "'B-PP': 2, 'B-ADJP': 111, 'B-NP': 382, 'B-VP': 2,\n",
    "'I-VP': 11, 'O': 16},\n",
    "'CC':\n",
    "{'B-ADVP': 3, 'O': 3676, 'I-VP': 104, 'B-CONJP': 6,\n",
    "'I-ADVP': 30, 'I-UCP': 2, 'I-PP': 24, 'I-ADJP': 26,\n",
    "'I-NP': 1409, 'B-ADJP': 2, 'B-NP': 18, 'B-PP': 70,\n",
    "'I-PRT': 1, 'B-VP': 1},\n",
    "'NN':\n",
    "{'B-LST': 2, 'I-INTJ': 2, 'B-ADVP': 38, 'O': 37,\n",
    "'I-ADVP': 11, 'B-INTJ': 1, 'I-UCP': 2, 'B-UCP': 2,\n",
    "'I-VP': 77, 'B-PRT': 2, 'I-ADJP': 41, 'I-NP': 24456,\n",
    "'B-ADJP': 44, 'B-NP': 5160, 'B-PP': 15, 'B-VP': 257},\n",
    "...\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:29:54.501140Z",
     "iopub.status.busy": "2020-12-11T20:29:54.500118Z",
     "iopub.status.idle": "2020-12-11T20:29:54.624319Z",
     "shell.execute_reply": "2020-12-11T20:29:54.624820Z"
    },
    "papermill": {
     "duration": 0.216632,
     "end_time": "2020-12-11T20:29:54.624978",
     "exception": false,
     "start_time": "2020-12-11T20:29:54.408346",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def get_chunk_dist(corpus, pos_counts):\n",
    "    \n",
    "    pos_dist = {pos: {} for pos in pos_counts.keys()}\n",
    "    \n",
    "    for sentence in corpus:\n",
    "        for row in sentence:\n",
    "\n",
    "            if row['chunk'] not in pos_dist[row['pos']].keys():\n",
    "                pos_dist[row['pos']][row['chunk']] = 1\n",
    "            else:\n",
    "                pos_dist[row['pos']][row['chunk']] += 1\n",
    "\n",
    "    return pos_dist\n",
    "\n",
    "chunk_dist = get_chunk_dist(train_corpus, pos_cnt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:29:54.808362Z",
     "iopub.status.busy": "2020-12-11T20:29:54.807088Z",
     "iopub.status.idle": "2020-12-11T20:29:54.813226Z",
     "shell.execute_reply": "2020-12-11T20:29:54.813679Z"
    },
    "papermill": {
     "duration": 0.099602,
     "end_time": "2020-12-11T20:29:54.813838",
     "exception": false,
     "start_time": "2020-12-11T20:29:54.714236",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'B-NP': 5160,\n",
       " 'I-NP': 24456,\n",
       " 'B-VP': 257,\n",
       " 'B-ADJP': 44,\n",
       " 'B-ADVP': 38,\n",
       " 'O': 37,\n",
       " 'B-PP': 15,\n",
       " 'I-ADVP': 11,\n",
       " 'I-ADJP': 41,\n",
       " 'I-VP': 77,\n",
       " 'B-INTJ': 1,\n",
       " 'B-LST': 2,\n",
       " 'B-UCP': 2,\n",
       " 'I-UCP': 2,\n",
       " 'B-PRT': 2,\n",
       " 'I-INTJ': 2}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chunk_dist['NN']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.089069,
     "end_time": "2020-12-11T20:29:54.992621",
     "exception": false,
     "start_time": "2020-12-11T20:29:54.903552",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Selecting the POS-chunk associations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.089302,
     "end_time": "2020-12-11T20:29:55.171326",
     "exception": false,
     "start_time": "2020-12-11T20:29:55.082024",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "For each part of speech, select the best association. In the example above, you will have (NN, I-NP) as it is the most frequent. You will store the results in a dictionary that you will call `pos_chunk`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:29:55.354127Z",
     "iopub.status.busy": "2020-12-11T20:29:55.353110Z",
     "iopub.status.idle": "2020-12-11T20:29:55.358165Z",
     "shell.execute_reply": "2020-12-11T20:29:55.358630Z"
    },
    "papermill": {
     "duration": 0.097903,
     "end_time": "2020-12-11T20:29:55.358795",
     "exception": false,
     "start_time": "2020-12-11T20:29:55.260892",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "pos_chunk = {pos: max(chunk_dist[pos], key=chunk_dist[pos].get) for pos in chunk_dist.keys()}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:29:55.544171Z",
     "iopub.status.busy": "2020-12-11T20:29:55.543190Z",
     "iopub.status.idle": "2020-12-11T20:29:55.548394Z",
     "shell.execute_reply": "2020-12-11T20:29:55.548874Z"
    },
    "papermill": {
     "duration": 0.099143,
     "end_time": "2020-12-11T20:29:55.549017",
     "exception": false,
     "start_time": "2020-12-11T20:29:55.449874",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'I-NP'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pos_chunk['NN']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.092738,
     "end_time": "2020-12-11T20:29:55.732488",
     "exception": false,
     "start_time": "2020-12-11T20:29:55.639750",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Prediction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.091133,
     "end_time": "2020-12-11T20:29:55.916337",
     "exception": false,
     "start_time": "2020-12-11T20:29:55.825204",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Using the resulting associations, apply your chunker to the test file. You will write a `predict(model, corpus)` function, where `model` will be your associations and `corpus`, the test corpus. You will format the test corpus as a dictionary, where you will add a `pchunk` key for each row with a value that will correspond to the predicted chunk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:29:56.101713Z",
     "iopub.status.busy": "2020-12-11T20:29:56.100735Z",
     "iopub.status.idle": "2020-12-11T20:29:56.106501Z",
     "shell.execute_reply": "2020-12-11T20:29:56.106982Z"
    },
    "papermill": {
     "duration": 0.099615,
     "end_time": "2020-12-11T20:29:56.107125",
     "exception": false,
     "start_time": "2020-12-11T20:29:56.007510",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def predict(model, corpus):\n",
    "    \n",
    "    updated = corpus\n",
    "    \n",
    "    for ind_s, sentence in enumerate(corpus):\n",
    "        for ind_r, row in enumerate(sentence):\n",
    "            updated[ind_s][ind_r]['pchunk'] = model[row['pos']]\n",
    "    \n",
    "    return updated"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.089684,
     "end_time": "2020-12-11T20:29:56.287311",
     "exception": false,
     "start_time": "2020-12-11T20:29:56.197627",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We load the test corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:29:56.473089Z",
     "iopub.status.busy": "2020-12-11T20:29:56.472071Z",
     "iopub.status.idle": "2020-12-11T20:29:56.579395Z",
     "shell.execute_reply": "2020-12-11T20:29:56.578826Z"
    },
    "papermill": {
     "duration": 0.202161,
     "end_time": "2020-12-11T20:29:56.579507",
     "exception": false,
     "start_time": "2020-12-11T20:29:56.377346",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[{'form': 'Rockwell', 'pos': 'NNP', 'chunk': 'B-NP'},\n",
       "  {'form': 'International', 'pos': 'NNP', 'chunk': 'I-NP'},\n",
       "  {'form': 'Corp.', 'pos': 'NNP', 'chunk': 'I-NP'},\n",
       "  {'form': \"'s\", 'pos': 'POS', 'chunk': 'B-NP'},\n",
       "  {'form': 'Tulsa', 'pos': 'NNP', 'chunk': 'I-NP'},\n",
       "  {'form': 'unit', 'pos': 'NN', 'chunk': 'I-NP'},\n",
       "  {'form': 'said', 'pos': 'VBD', 'chunk': 'B-VP'},\n",
       "  {'form': 'it', 'pos': 'PRP', 'chunk': 'B-NP'},\n",
       "  {'form': 'signed', 'pos': 'VBD', 'chunk': 'B-VP'},\n",
       "  {'form': 'a', 'pos': 'DT', 'chunk': 'B-NP'},\n",
       "  {'form': 'tentative', 'pos': 'JJ', 'chunk': 'I-NP'},\n",
       "  {'form': 'agreement', 'pos': 'NN', 'chunk': 'I-NP'},\n",
       "  {'form': 'extending', 'pos': 'VBG', 'chunk': 'B-VP'},\n",
       "  {'form': 'its', 'pos': 'PRP$', 'chunk': 'B-NP'},\n",
       "  {'form': 'contract', 'pos': 'NN', 'chunk': 'I-NP'},\n",
       "  {'form': 'with', 'pos': 'IN', 'chunk': 'B-PP'},\n",
       "  {'form': 'Boeing', 'pos': 'NNP', 'chunk': 'B-NP'},\n",
       "  {'form': 'Co.', 'pos': 'NNP', 'chunk': 'I-NP'},\n",
       "  {'form': 'to', 'pos': 'TO', 'chunk': 'B-VP'},\n",
       "  {'form': 'provide', 'pos': 'VB', 'chunk': 'I-VP'},\n",
       "  {'form': 'structural', 'pos': 'JJ', 'chunk': 'B-NP'},\n",
       "  {'form': 'parts', 'pos': 'NNS', 'chunk': 'I-NP'},\n",
       "  {'form': 'for', 'pos': 'IN', 'chunk': 'B-PP'},\n",
       "  {'form': 'Boeing', 'pos': 'NNP', 'chunk': 'B-NP'},\n",
       "  {'form': \"'s\", 'pos': 'POS', 'chunk': 'B-NP'},\n",
       "  {'form': '747', 'pos': 'CD', 'chunk': 'I-NP'},\n",
       "  {'form': 'jetliners', 'pos': 'NNS', 'chunk': 'I-NP'},\n",
       "  {'form': '.', 'pos': '.', 'chunk': 'O'}]]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_sentences = read_sentences(test_file)\n",
    "test_corpus = split_rows(test_sentences, column_names)\n",
    "test_corpus[:1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.09126,
     "end_time": "2020-12-11T20:29:56.762393",
     "exception": false,
     "start_time": "2020-12-11T20:29:56.671133",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We predict the groups. You should have added a `pchunk` key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:29:56.974090Z",
     "iopub.status.busy": "2020-12-11T20:29:56.965110Z",
     "iopub.status.idle": "2020-12-11T20:29:56.977514Z",
     "shell.execute_reply": "2020-12-11T20:29:56.978246Z"
    },
    "papermill": {
     "duration": 0.123921,
     "end_time": "2020-12-11T20:29:56.978395",
     "exception": false,
     "start_time": "2020-12-11T20:29:56.854474",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[{'form': 'Rockwell', 'pos': 'NNP', 'chunk': 'B-NP', 'pchunk': 'I-NP'},\n",
       "  {'form': 'International', 'pos': 'NNP', 'chunk': 'I-NP', 'pchunk': 'I-NP'},\n",
       "  {'form': 'Corp.', 'pos': 'NNP', 'chunk': 'I-NP', 'pchunk': 'I-NP'},\n",
       "  {'form': \"'s\", 'pos': 'POS', 'chunk': 'B-NP', 'pchunk': 'B-NP'},\n",
       "  {'form': 'Tulsa', 'pos': 'NNP', 'chunk': 'I-NP', 'pchunk': 'I-NP'},\n",
       "  {'form': 'unit', 'pos': 'NN', 'chunk': 'I-NP', 'pchunk': 'I-NP'},\n",
       "  {'form': 'said', 'pos': 'VBD', 'chunk': 'B-VP', 'pchunk': 'B-VP'},\n",
       "  {'form': 'it', 'pos': 'PRP', 'chunk': 'B-NP', 'pchunk': 'B-NP'},\n",
       "  {'form': 'signed', 'pos': 'VBD', 'chunk': 'B-VP', 'pchunk': 'B-VP'},\n",
       "  {'form': 'a', 'pos': 'DT', 'chunk': 'B-NP', 'pchunk': 'B-NP'},\n",
       "  {'form': 'tentative', 'pos': 'JJ', 'chunk': 'I-NP', 'pchunk': 'I-NP'},\n",
       "  {'form': 'agreement', 'pos': 'NN', 'chunk': 'I-NP', 'pchunk': 'I-NP'},\n",
       "  {'form': 'extending', 'pos': 'VBG', 'chunk': 'B-VP', 'pchunk': 'B-VP'},\n",
       "  {'form': 'its', 'pos': 'PRP$', 'chunk': 'B-NP', 'pchunk': 'B-NP'},\n",
       "  {'form': 'contract', 'pos': 'NN', 'chunk': 'I-NP', 'pchunk': 'I-NP'},\n",
       "  {'form': 'with', 'pos': 'IN', 'chunk': 'B-PP', 'pchunk': 'B-PP'},\n",
       "  {'form': 'Boeing', 'pos': 'NNP', 'chunk': 'B-NP', 'pchunk': 'I-NP'},\n",
       "  {'form': 'Co.', 'pos': 'NNP', 'chunk': 'I-NP', 'pchunk': 'I-NP'},\n",
       "  {'form': 'to', 'pos': 'TO', 'chunk': 'B-VP', 'pchunk': 'B-PP'},\n",
       "  {'form': 'provide', 'pos': 'VB', 'chunk': 'I-VP', 'pchunk': 'I-VP'},\n",
       "  {'form': 'structural', 'pos': 'JJ', 'chunk': 'B-NP', 'pchunk': 'I-NP'},\n",
       "  {'form': 'parts', 'pos': 'NNS', 'chunk': 'I-NP', 'pchunk': 'I-NP'},\n",
       "  {'form': 'for', 'pos': 'IN', 'chunk': 'B-PP', 'pchunk': 'B-PP'},\n",
       "  {'form': 'Boeing', 'pos': 'NNP', 'chunk': 'B-NP', 'pchunk': 'I-NP'},\n",
       "  {'form': \"'s\", 'pos': 'POS', 'chunk': 'B-NP', 'pchunk': 'B-NP'},\n",
       "  {'form': '747', 'pos': 'CD', 'chunk': 'I-NP', 'pchunk': 'I-NP'},\n",
       "  {'form': 'jetliners', 'pos': 'NNS', 'chunk': 'I-NP', 'pchunk': 'I-NP'},\n",
       "  {'form': '.', 'pos': '.', 'chunk': 'O', 'pchunk': 'O'}]]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predicted_test_corpus = predict(pos_chunk, test_corpus)\n",
    "predicted_test_corpus[:1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.092495,
     "end_time": "2020-12-11T20:29:57.163415",
     "exception": false,
     "start_time": "2020-12-11T20:29:57.070920",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.091817,
     "end_time": "2020-12-11T20:29:57.347328",
     "exception": false,
     "start_time": "2020-12-11T20:29:57.255511",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We can evaluate the performance of the baseline with the tag accuracy: the percentage of words that receive the correct tag."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:29:57.536634Z",
     "iopub.status.busy": "2020-12-11T20:29:57.535897Z",
     "iopub.status.idle": "2020-12-11T20:29:57.541416Z",
     "shell.execute_reply": "2020-12-11T20:29:57.542071Z"
    },
    "papermill": {
     "duration": 0.102844,
     "end_time": "2020-12-11T20:29:57.542228",
     "exception": false,
     "start_time": "2020-12-11T20:29:57.439384",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def eval(predicted):\n",
    "    \"\"\"\n",
    "    Evaluates the predicted chunk accuracy\n",
    "    :param predicted:\n",
    "    :return:\n",
    "    \"\"\"\n",
    "    word_cnt = 0\n",
    "    correct = 0\n",
    "    for sentence in predicted:\n",
    "        for row in sentence:\n",
    "            word_cnt += 1\n",
    "            if row['chunk'] == row['pchunk']:\n",
    "                correct += 1\n",
    "    return correct / word_cnt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:29:57.731980Z",
     "iopub.status.busy": "2020-12-11T20:29:57.731248Z",
     "iopub.status.idle": "2020-12-11T20:29:57.748092Z",
     "shell.execute_reply": "2020-12-11T20:29:57.748625Z"
    },
    "papermill": {
     "duration": 0.113273,
     "end_time": "2020-12-11T20:29:57.748764",
     "exception": false,
     "start_time": "2020-12-11T20:29:57.635491",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7729066846782194"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy = eval(predicted_test_corpus)\n",
    "accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.093105,
     "end_time": "2020-12-11T20:29:57.936644",
     "exception": false,
     "start_time": "2020-12-11T20:29:57.843539",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### The CoNLL evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.093477,
     "end_time": "2020-12-11T20:29:58.127024",
     "exception": false,
     "start_time": "2020-12-11T20:29:58.033547",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "The accuracy is very misleading as it is biased by the most frequent tags. It is not a good way to evaluate chunking. Instead, CoNLL compute the F1 score of all the chunks with a specific evaluation script."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.091848,
     "end_time": "2020-12-11T20:29:58.311855",
     "exception": false,
     "start_time": "2020-12-11T20:29:58.220007",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "#### Saving the corpus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.092639,
     "end_time": "2020-12-11T20:29:58.497325",
     "exception": false,
     "start_time": "2020-12-11T20:29:58.404686",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "To use the CoNLL evaluation script, you will store your results in an output file that has four columns. The three first columns will be the input columns from the test file: \n",
    "* word, \n",
    "* part of speech, and \n",
    "* gold-standard chunk. \n",
    "\n",
    "You will append the predicted chunk as the 4th column. Your output file should look like the excerpt below:\n",
    "```\n",
    "Rockwell NNP B-NP I-NP\n",
    "International NNP I-NP I-NP\n",
    "Corp. NNP I-NP I-NP\n",
    "'s POS B-NP B-NP\n",
    "Tulsa NNP I-NP I-NP\n",
    "unit NN I-NP I-NP\n",
    "said VBD B-VP B-VP\n",
    "it PRP B-NP B-NP\n",
    "```\n",
    "The separator is the space."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.093146,
     "end_time": "2020-12-11T20:29:58.685123",
     "exception": false,
     "start_time": "2020-12-11T20:29:58.591977",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "You will use a `save_results(output_dict, keys, output_file)` function, where the keys will be `['form', 'pos', 'chunk', 'pchunk']`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:29:58.876180Z",
     "iopub.status.busy": "2020-12-11T20:29:58.875466Z",
     "iopub.status.idle": "2020-12-11T20:29:58.878727Z",
     "shell.execute_reply": "2020-12-11T20:29:58.878186Z"
    },
    "papermill": {
     "duration": 0.10083,
     "end_time": "2020-12-11T20:29:58.878871",
     "exception": false,
     "start_time": "2020-12-11T20:29:58.778041",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "keys = ['form', 'pos', 'chunk', 'pchunk']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:29:59.074468Z",
     "iopub.status.busy": "2020-12-11T20:29:59.073713Z",
     "iopub.status.idle": "2020-12-11T20:29:59.076479Z",
     "shell.execute_reply": "2020-12-11T20:29:59.075880Z"
    },
    "papermill": {
     "duration": 0.104235,
     "end_time": "2020-12-11T20:29:59.076602",
     "exception": false,
     "start_time": "2020-12-11T20:29:58.972367",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def save_results(output_dict, keys, output_file):\n",
    "    f_out = open(output_file, 'w')\n",
    "    # We write the word (form), part of speech (pos),\n",
    "    # gold-standard chunk (chunk), and predicted chunk (pchunk)\n",
    "    for sentence in output_dict:\n",
    "        for row in sentence:\n",
    "            for key in keys:\n",
    "                f_out.write(row[key] + ' ')\n",
    "            f_out.write('\\n')\n",
    "        f_out.write('\\n')\n",
    "    f_out.close()\n",
    "    return"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:29:59.269301Z",
     "iopub.status.busy": "2020-12-11T20:29:59.268598Z",
     "iopub.status.idle": "2020-12-11T20:29:59.357186Z",
     "shell.execute_reply": "2020-12-11T20:29:59.357721Z"
    },
    "papermill": {
     "duration": 0.187729,
     "end_time": "2020-12-11T20:29:59.357898",
     "exception": false,
     "start_time": "2020-12-11T20:29:59.170169",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "save_results(predicted_test_corpus, keys, 'out')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.093974,
     "end_time": "2020-12-11T20:29:59.544434",
     "exception": false,
     "start_time": "2020-12-11T20:29:59.450460",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "The CoNLL 2000 evaluation script will use these two last columns, chunk and predicted chunk, to compute the performance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.092501,
     "end_time": "2020-12-11T20:29:59.730736",
     "exception": false,
     "start_time": "2020-12-11T20:29:59.638235",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.093234,
     "end_time": "2020-12-11T20:29:59.919361",
     "exception": false,
     "start_time": "2020-12-11T20:29:59.826127",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "To evaluate your results, you have two options:\n",
    "1. Use the original conlleval script here:  <a href=\"https://www.clips.uantwerpen.be/conll2000/chunking/\"><tt>conlleval.txt</tt></a>.\n",
    "2. Use a Python translation of it. You will use this option."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.091622,
     "end_time": "2020-12-11T20:30:00.104941",
     "exception": false,
     "start_time": "2020-12-11T20:30:00.013319",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "#### The Python translation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.092019,
     "end_time": "2020-12-11T20:30:00.289933",
     "exception": false,
     "start_time": "2020-12-11T20:30:00.197914",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Install the script with:\n",
    "```\n",
    "pip install conlleval\n",
    "```\n",
    "from https://github.com/kaniblu/conlleval\n",
    "and run the cell below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:30:00.477549Z",
     "iopub.status.busy": "2020-12-11T20:30:00.476931Z",
     "iopub.status.idle": "2020-12-11T20:30:10.592956Z",
     "shell.execute_reply": "2020-12-11T20:30:10.592365Z"
    },
    "papermill": {
     "duration": 10.210851,
     "end_time": "2020-12-11T20:30:10.593092",
     "exception": false,
     "start_time": "2020-12-11T20:30:00.382241",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting conlleval\r\n",
      "  Downloading conlleval-0.2-py3-none-any.whl (5.4 kB)\r\n",
      "Installing collected packages: conlleval\r\n",
      "Successfully installed conlleval-0.2\r\n"
     ]
    }
   ],
   "source": [
    "!pip install conlleval"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:30:10.791227Z",
     "iopub.status.busy": "2020-12-11T20:30:10.790499Z",
     "iopub.status.idle": "2020-12-11T20:30:11.312945Z",
     "shell.execute_reply": "2020-12-11T20:30:11.312201Z"
    },
    "papermill": {
     "duration": 0.624205,
     "end_time": "2020-12-11T20:30:11.313073",
     "exception": false,
     "start_time": "2020-12-11T20:30:10.688868",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import conlleval\n",
    "lines = open('out').read().splitlines()\n",
    "res = conlleval.evaluate(lines)\n",
    "baseline_score = res['overall']['chunks']['evals']['f1']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:30:11.511128Z",
     "iopub.status.busy": "2020-12-11T20:30:11.510074Z",
     "iopub.status.idle": "2020-12-11T20:30:11.514660Z",
     "shell.execute_reply": "2020-12-11T20:30:11.514022Z"
    },
    "papermill": {
     "duration": 0.106302,
     "end_time": "2020-12-11T20:30:11.514795",
     "exception": false,
     "start_time": "2020-12-11T20:30:11.408493",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.770671072299583"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "baseline_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.095113,
     "end_time": "2020-12-11T20:30:11.707609",
     "exception": false,
     "start_time": "2020-12-11T20:30:11.612496",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### The official script"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.098081,
     "end_time": "2020-12-11T20:30:11.902930",
     "exception": false,
     "start_time": "2020-12-11T20:30:11.804849",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "You may want to double-check your results with the original CoNLL script. It is more conplex to use however:\n",
    "* <tt>conlleval.txt</tt> is the official CoNLL Perl script. It expects the two last columns of the test set to be the manually assigned chunk (gold standard) and the predicted chunk.\n",
    "* <tt>conlleval.txt</tt> was written for Unix and if you run Windows, you will have to use a terminal command. In the File menu of the notebook, select New and then Terminal.\n",
    "* Start it like this: ` $ conlleval.txt <out` where the `out` file contains both the gold and predicted chunk tags. `conlleval.txt` is a Perl script.\n",
    "* Perl is installed on most Unix distributions. If it is not installed on your machine, you need to install it. Make also sure that you have the execution rights. Otherwise change them with: `$ chmod +x conlleval.txt`\n",
    "* The `conlleval.txt` script expects the new lines to be `\\n` as in Unix. If you run your Python program on Windows, your new lines will be `\\r\\n`. To have the correct new lines, add this parameter to `open()`: `newline='\\n’` like this: `f_out = open('out', ‘w’, newline='\\n’)`\n",
    "* The complete description of the CoNLL 2000 evaluation script is available here: [https://www.clips.uantwerpen.be/conll2000/chunking/output.html](https://www.clips.uantwerpen.be/conll2000/chunking/output.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.097971,
     "end_time": "2020-12-11T20:30:12.099646",
     "exception": false,
     "start_time": "2020-12-11T20:30:12.001675",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "You will describe the results you obtained in your report"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.09723,
     "end_time": "2020-12-11T20:30:12.296316",
     "exception": false,
     "start_time": "2020-12-11T20:30:12.199086",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Using Machine Learning: A first ML program"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.095953,
     "end_time": "2020-12-11T20:30:12.489526",
     "exception": false,
     "start_time": "2020-12-11T20:30:12.393573",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "In this exercise, you will apply and explore a machine-learning program.\n",
    "\n",
    "The program that won the CoNLL 2000 shared task (Kudoh and Matsumoto, 2000) used a window of five words around the chunk tag to identify, $c_i$. They built a feature vector consisting of:\n",
    "1. The values of the five words in this window: $w_{i-2}, w_{i-1}, w_{i}, w_{i+1}, w_{i+2}$\n",
    "2. The values of the five parts of speech in this window: $t_{i-2}, t_{i-1}, t_{i}, t_{i+1}, t_{i+2}$\n",
    "3. The values of the two previous chunk tags in the first part of the window: $c_{i-2}, c_{i-1}$\n",
    "\n",
    "The two last parameters (3.) are said to be dynamic because the program computes them at run-time. Read [Kudoh and Matsumoto's paper](http://www.clips.uantwerpen.be/conll2000/pdf/14244kud.pdf) and the [Yamcha](http://www.chasen.org/~taku/software/yamcha/) software site.\n",
    "\n",
    "You will start with a given code that uses the two first set of features (1. and 2.) and add yourself the last one (3.) to improve the performance of your chunker. Kudoh and Matsumoto trained a classifier based on support vector machines. You will use logistic regression."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.142741,
     "end_time": "2020-12-11T20:30:12.729492",
     "exception": false,
     "start_time": "2020-12-11T20:30:12.586751",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:30:12.933061Z",
     "iopub.status.busy": "2020-12-11T20:30:12.932292Z",
     "iopub.status.idle": "2020-12-11T20:30:14.537544Z",
     "shell.execute_reply": "2020-12-11T20:30:14.536748Z"
    },
    "papermill": {
     "duration": 1.706982,
     "end_time": "2020-12-11T20:30:14.537666",
     "exception": false,
     "start_time": "2020-12-11T20:30:12.830684",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import bs4\n",
    "import os\n",
    "import requests\n",
    "from sklearn.feature_extraction import DictVectorizer\n",
    "from sklearn import svm\n",
    "from sklearn import linear_model\n",
    "from sklearn import metrics\n",
    "from sklearn import tree\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "import time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.096859,
     "end_time": "2020-12-11T20:30:14.732172",
     "exception": false,
     "start_time": "2020-12-11T20:30:14.635313",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Feature extraction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.09661,
     "end_time": "2020-12-11T20:30:14.929186",
     "exception": false,
     "start_time": "2020-12-11T20:30:14.832576",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "#### Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.114922,
     "end_time": "2020-12-11T20:30:15.139216",
     "exception": false,
     "start_time": "2020-12-11T20:30:15.024294",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "A first function to extract features from one sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:30:15.372669Z",
     "iopub.status.busy": "2020-12-11T20:30:15.371754Z",
     "iopub.status.idle": "2020-12-11T20:30:15.374738Z",
     "shell.execute_reply": "2020-12-11T20:30:15.374181Z"
    },
    "papermill": {
     "duration": 0.111415,
     "end_time": "2020-12-11T20:30:15.374883",
     "exception": false,
     "start_time": "2020-12-11T20:30:15.263468",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def extract_features_sent_static(sentence, w_size, feature_names):\n",
    "    \"\"\"\n",
    "    Extract the features from one sentence\n",
    "    returns X and y, where X is a list of dictionaries and\n",
    "    y is a list of symbols\n",
    "    :param sentence: string containing the CoNLL structure of a sentence\n",
    "    :param w_size:\n",
    "    :return:\n",
    "    \"\"\"\n",
    "\n",
    "    # We pad the sentence to extract the context window more easily\n",
    "    start = [{'form': 'BOS', 'pos': 'BOS', 'chunk': 'BOS'}]\n",
    "    end = [{'form': 'EOS', 'pos': 'EOS', 'chunk': 'EOS'}]\n",
    "    start *= w_size\n",
    "    end *= w_size\n",
    "    padded_sentence = start + sentence\n",
    "    padded_sentence += end\n",
    "\n",
    "    # We extract the features and the classes\n",
    "    # X contains is a list of features, where each feature vector is a dictionary\n",
    "    # y is the list of classes\n",
    "    X = list()\n",
    "    y = list()\n",
    "    for i in range(len(padded_sentence) - 2 * w_size):\n",
    "        # x is a row of X\n",
    "        x = list()\n",
    "        # The words in lower case\n",
    "        for j in range(2 * w_size + 1):\n",
    "            x.append(padded_sentence[i + j]['form'].lower())\n",
    "        # The POS\n",
    "        for j in range(2 * w_size + 1):\n",
    "            x.append(padded_sentence[i + j]['pos'])\n",
    "        # The chunks (Up to the word)\n",
    "        \"\"\"\n",
    "        for j in range(w_size):\n",
    "            feature_line.append(padded_sentence[i + j]['chunk'])\n",
    "        \"\"\"\n",
    "        # We represent the feature vector as a dictionary\n",
    "        X.append(dict(zip(feature_names, x)))\n",
    "        # The classes are stored in a list\n",
    "        y.append(padded_sentence[i + w_size]['chunk'])\n",
    "    return X, y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.09571,
     "end_time": "2020-12-11T20:30:15.569340",
     "exception": false,
     "start_time": "2020-12-11T20:30:15.473630",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "And from all the sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:30:15.770216Z",
     "iopub.status.busy": "2020-12-11T20:30:15.769209Z",
     "iopub.status.idle": "2020-12-11T20:30:15.771958Z",
     "shell.execute_reply": "2020-12-11T20:30:15.771423Z"
    },
    "papermill": {
     "duration": 0.106108,
     "end_time": "2020-12-11T20:30:15.772078",
     "exception": false,
     "start_time": "2020-12-11T20:30:15.665970",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def extract_features_static(sentences, w_size, feature_names):\n",
    "    \"\"\"\n",
    "    Builds X matrix and y vector\n",
    "    X is a list of dictionaries and y is a list\n",
    "    :param sentences:\n",
    "    :param w_size:\n",
    "    :return:\n",
    "    \"\"\"\n",
    "    X_l = []\n",
    "    y_l = []\n",
    "    for sentence in sentences:\n",
    "        X, y = extract_features_sent_static(sentence, w_size, feature_names)\n",
    "        X_l.extend(X)\n",
    "        y_l.extend(y)\n",
    "    return X_l, y_l"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.095257,
     "end_time": "2020-12-11T20:30:15.965414",
     "exception": false,
     "start_time": "2020-12-11T20:30:15.870157",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "#### Applying the feature extraction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.096013,
     "end_time": "2020-12-11T20:30:16.157456",
     "exception": false,
     "start_time": "2020-12-11T20:30:16.061443",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "The size of the window and the names of the features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:30:16.356138Z",
     "iopub.status.busy": "2020-12-11T20:30:16.355287Z",
     "iopub.status.idle": "2020-12-11T20:30:16.358635Z",
     "shell.execute_reply": "2020-12-11T20:30:16.357963Z"
    },
    "papermill": {
     "duration": 0.104532,
     "end_time": "2020-12-11T20:30:16.358751",
     "exception": false,
     "start_time": "2020-12-11T20:30:16.254219",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "w_size = 2  # The size of the context window to the left and right of the word\n",
    "feature_names = ['word_n2', 'word_n1', 'word', 'word_p1', 'word_p2',\n",
    "                 'pos_n2', 'pos_n1', 'pos', 'pos_p1', 'pos_p2']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.095502,
     "end_time": "2020-12-11T20:30:16.549698",
     "exception": false,
     "start_time": "2020-12-11T20:30:16.454196",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We read the corpus and format it as a dictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:30:16.746231Z",
     "iopub.status.busy": "2020-12-11T20:30:16.745386Z",
     "iopub.status.idle": "2020-12-11T20:30:17.111676Z",
     "shell.execute_reply": "2020-12-11T20:30:17.111071Z"
    },
    "papermill": {
     "duration": 0.466429,
     "end_time": "2020-12-11T20:30:17.111809",
     "exception": false,
     "start_time": "2020-12-11T20:30:16.645380",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "train_sentences = read_sentences(train_file)\n",
    "train_corpus = split_rows(train_sentences, column_names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:30:17.311288Z",
     "iopub.status.busy": "2020-12-11T20:30:17.310621Z",
     "iopub.status.idle": "2020-12-11T20:30:17.322566Z",
     "shell.execute_reply": "2020-12-11T20:30:17.323046Z"
    },
    "papermill": {
     "duration": 0.115895,
     "end_time": "2020-12-11T20:30:17.323201",
     "exception": false,
     "start_time": "2020-12-11T20:30:17.207306",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[{'form': 'Confidence', 'pos': 'NN', 'chunk': 'B-NP'},\n",
       "  {'form': 'in', 'pos': 'IN', 'chunk': 'B-PP'},\n",
       "  {'form': 'the', 'pos': 'DT', 'chunk': 'B-NP'},\n",
       "  {'form': 'pound', 'pos': 'NN', 'chunk': 'I-NP'},\n",
       "  {'form': 'is', 'pos': 'VBZ', 'chunk': 'B-VP'},\n",
       "  {'form': 'widely', 'pos': 'RB', 'chunk': 'I-VP'},\n",
       "  {'form': 'expected', 'pos': 'VBN', 'chunk': 'I-VP'},\n",
       "  {'form': 'to', 'pos': 'TO', 'chunk': 'I-VP'},\n",
       "  {'form': 'take', 'pos': 'VB', 'chunk': 'I-VP'},\n",
       "  {'form': 'another', 'pos': 'DT', 'chunk': 'B-NP'},\n",
       "  {'form': 'sharp', 'pos': 'JJ', 'chunk': 'I-NP'},\n",
       "  {'form': 'dive', 'pos': 'NN', 'chunk': 'I-NP'},\n",
       "  {'form': 'if', 'pos': 'IN', 'chunk': 'B-SBAR'},\n",
       "  {'form': 'trade', 'pos': 'NN', 'chunk': 'B-NP'},\n",
       "  {'form': 'figures', 'pos': 'NNS', 'chunk': 'I-NP'},\n",
       "  {'form': 'for', 'pos': 'IN', 'chunk': 'B-PP'},\n",
       "  {'form': 'September', 'pos': 'NNP', 'chunk': 'B-NP'},\n",
       "  {'form': ',', 'pos': ',', 'chunk': 'O'},\n",
       "  {'form': 'due', 'pos': 'JJ', 'chunk': 'B-ADJP'},\n",
       "  {'form': 'for', 'pos': 'IN', 'chunk': 'B-PP'},\n",
       "  {'form': 'release', 'pos': 'NN', 'chunk': 'B-NP'},\n",
       "  {'form': 'tomorrow', 'pos': 'NN', 'chunk': 'B-NP'},\n",
       "  {'form': ',', 'pos': ',', 'chunk': 'O'},\n",
       "  {'form': 'fail', 'pos': 'VB', 'chunk': 'B-VP'},\n",
       "  {'form': 'to', 'pos': 'TO', 'chunk': 'I-VP'},\n",
       "  {'form': 'show', 'pos': 'VB', 'chunk': 'I-VP'},\n",
       "  {'form': 'a', 'pos': 'DT', 'chunk': 'B-NP'},\n",
       "  {'form': 'substantial', 'pos': 'JJ', 'chunk': 'I-NP'},\n",
       "  {'form': 'improvement', 'pos': 'NN', 'chunk': 'I-NP'},\n",
       "  {'form': 'from', 'pos': 'IN', 'chunk': 'B-PP'},\n",
       "  {'form': 'July', 'pos': 'NNP', 'chunk': 'B-NP'},\n",
       "  {'form': 'and', 'pos': 'CC', 'chunk': 'I-NP'},\n",
       "  {'form': 'August', 'pos': 'NNP', 'chunk': 'I-NP'},\n",
       "  {'form': \"'s\", 'pos': 'POS', 'chunk': 'B-NP'},\n",
       "  {'form': 'near-record', 'pos': 'JJ', 'chunk': 'I-NP'},\n",
       "  {'form': 'deficits', 'pos': 'NNS', 'chunk': 'I-NP'},\n",
       "  {'form': '.', 'pos': '.', 'chunk': 'O'}],\n",
       " [{'form': 'Chancellor', 'pos': 'NNP', 'chunk': 'O'},\n",
       "  {'form': 'of', 'pos': 'IN', 'chunk': 'B-PP'},\n",
       "  {'form': 'the', 'pos': 'DT', 'chunk': 'B-NP'},\n",
       "  {'form': 'Exchequer', 'pos': 'NNP', 'chunk': 'I-NP'},\n",
       "  {'form': 'Nigel', 'pos': 'NNP', 'chunk': 'B-NP'},\n",
       "  {'form': 'Lawson', 'pos': 'NNP', 'chunk': 'I-NP'},\n",
       "  {'form': \"'s\", 'pos': 'POS', 'chunk': 'B-NP'},\n",
       "  {'form': 'restated', 'pos': 'VBN', 'chunk': 'I-NP'},\n",
       "  {'form': 'commitment', 'pos': 'NN', 'chunk': 'I-NP'},\n",
       "  {'form': 'to', 'pos': 'TO', 'chunk': 'B-PP'},\n",
       "  {'form': 'a', 'pos': 'DT', 'chunk': 'B-NP'},\n",
       "  {'form': 'firm', 'pos': 'NN', 'chunk': 'I-NP'},\n",
       "  {'form': 'monetary', 'pos': 'JJ', 'chunk': 'I-NP'},\n",
       "  {'form': 'policy', 'pos': 'NN', 'chunk': 'I-NP'},\n",
       "  {'form': 'has', 'pos': 'VBZ', 'chunk': 'B-VP'},\n",
       "  {'form': 'helped', 'pos': 'VBN', 'chunk': 'I-VP'},\n",
       "  {'form': 'to', 'pos': 'TO', 'chunk': 'I-VP'},\n",
       "  {'form': 'prevent', 'pos': 'VB', 'chunk': 'I-VP'},\n",
       "  {'form': 'a', 'pos': 'DT', 'chunk': 'B-NP'},\n",
       "  {'form': 'freefall', 'pos': 'NN', 'chunk': 'I-NP'},\n",
       "  {'form': 'in', 'pos': 'IN', 'chunk': 'B-PP'},\n",
       "  {'form': 'sterling', 'pos': 'NN', 'chunk': 'B-NP'},\n",
       "  {'form': 'over', 'pos': 'IN', 'chunk': 'B-PP'},\n",
       "  {'form': 'the', 'pos': 'DT', 'chunk': 'B-NP'},\n",
       "  {'form': 'past', 'pos': 'JJ', 'chunk': 'I-NP'},\n",
       "  {'form': 'week', 'pos': 'NN', 'chunk': 'I-NP'},\n",
       "  {'form': '.', 'pos': '.', 'chunk': 'O'}]]"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_corpus[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:30:17.561320Z",
     "iopub.status.busy": "2020-12-11T20:30:17.555990Z",
     "iopub.status.idle": "2020-12-11T20:30:18.745560Z",
     "shell.execute_reply": "2020-12-11T20:30:18.746075Z"
    },
    "papermill": {
     "duration": 1.326542,
     "end_time": "2020-12-11T20:30:18.746228",
     "exception": false,
     "start_time": "2020-12-11T20:30:17.419686",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'word_n2': 'bos',\n",
       "  'word_n1': 'bos',\n",
       "  'word': 'confidence',\n",
       "  'word_p1': 'in',\n",
       "  'word_p2': 'the',\n",
       "  'pos_n2': 'BOS',\n",
       "  'pos_n1': 'BOS',\n",
       "  'pos': 'NN',\n",
       "  'pos_p1': 'IN',\n",
       "  'pos_p2': 'DT'},\n",
       " {'word_n2': 'bos',\n",
       "  'word_n1': 'confidence',\n",
       "  'word': 'in',\n",
       "  'word_p1': 'the',\n",
       "  'word_p2': 'pound',\n",
       "  'pos_n2': 'BOS',\n",
       "  'pos_n1': 'NN',\n",
       "  'pos': 'IN',\n",
       "  'pos_p1': 'DT',\n",
       "  'pos_p2': 'NN'}]"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_dict, y = extract_features_static(train_corpus, w_size, feature_names)\n",
    "X_dict[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:30:18.947974Z",
     "iopub.status.busy": "2020-12-11T20:30:18.947007Z",
     "iopub.status.idle": "2020-12-11T20:30:18.951953Z",
     "shell.execute_reply": "2020-12-11T20:30:18.951343Z"
    },
    "papermill": {
     "duration": 0.106841,
     "end_time": "2020-12-11T20:30:18.952076",
     "exception": false,
     "start_time": "2020-12-11T20:30:18.845235",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['B-NP', 'B-PP']"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y[:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.0986,
     "end_time": "2020-12-11T20:30:19.149969",
     "exception": false,
     "start_time": "2020-12-11T20:30:19.051369",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Feature encoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:30:19.379210Z",
     "iopub.status.busy": "2020-12-11T20:30:19.368951Z",
     "iopub.status.idle": "2020-12-11T20:30:23.419598Z",
     "shell.execute_reply": "2020-12-11T20:30:23.418911Z"
    },
    "papermill": {
     "duration": 4.169431,
     "end_time": "2020-12-11T20:30:23.419706",
     "exception": false,
     "start_time": "2020-12-11T20:30:19.250275",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Vectorize the feature matrix and carry out a one-hot encoding\n",
    "vec = DictVectorizer(sparse=True)\n",
    "X = vec.fit_transform(X_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.09901,
     "end_time": "2020-12-11T20:30:23.619410",
     "exception": false,
     "start_time": "2020-12-11T20:30:23.520400",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Training the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:30:23.821712Z",
     "iopub.status.busy": "2020-12-11T20:30:23.821038Z",
     "iopub.status.idle": "2020-12-11T20:31:41.710062Z",
     "shell.execute_reply": "2020-12-11T20:31:41.709452Z"
    },
    "papermill": {
     "duration": 77.992459,
     "end_time": "2020-12-11T20:31:41.710187",
     "exception": false,
     "start_time": "2020-12-11T20:30:23.717728",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/sklearn/linear_model/_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "LogisticRegression()"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "classifier = linear_model.LogisticRegression()\n",
    "model = classifier.fit(X, y)\n",
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.101135,
     "end_time": "2020-12-11T20:31:41.912638",
     "exception": false,
     "start_time": "2020-12-11T20:31:41.811503",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Predicting the test set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.100842,
     "end_time": "2020-12-11T20:31:42.119245",
     "exception": false,
     "start_time": "2020-12-11T20:31:42.018403",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We read the sentences and create a dictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:31:42.325573Z",
     "iopub.status.busy": "2020-12-11T20:31:42.324798Z",
     "iopub.status.idle": "2020-12-11T20:31:42.392724Z",
     "shell.execute_reply": "2020-12-11T20:31:42.393205Z"
    },
    "papermill": {
     "duration": 0.171971,
     "end_time": "2020-12-11T20:31:42.393353",
     "exception": false,
     "start_time": "2020-12-11T20:31:42.221382",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[{'form': 'Rockwell', 'pos': 'NNP', 'chunk': 'B-NP'},\n",
       "  {'form': 'International', 'pos': 'NNP', 'chunk': 'I-NP'},\n",
       "  {'form': 'Corp.', 'pos': 'NNP', 'chunk': 'I-NP'},\n",
       "  {'form': \"'s\", 'pos': 'POS', 'chunk': 'B-NP'},\n",
       "  {'form': 'Tulsa', 'pos': 'NNP', 'chunk': 'I-NP'},\n",
       "  {'form': 'unit', 'pos': 'NN', 'chunk': 'I-NP'},\n",
       "  {'form': 'said', 'pos': 'VBD', 'chunk': 'B-VP'},\n",
       "  {'form': 'it', 'pos': 'PRP', 'chunk': 'B-NP'},\n",
       "  {'form': 'signed', 'pos': 'VBD', 'chunk': 'B-VP'},\n",
       "  {'form': 'a', 'pos': 'DT', 'chunk': 'B-NP'},\n",
       "  {'form': 'tentative', 'pos': 'JJ', 'chunk': 'I-NP'},\n",
       "  {'form': 'agreement', 'pos': 'NN', 'chunk': 'I-NP'},\n",
       "  {'form': 'extending', 'pos': 'VBG', 'chunk': 'B-VP'},\n",
       "  {'form': 'its', 'pos': 'PRP$', 'chunk': 'B-NP'},\n",
       "  {'form': 'contract', 'pos': 'NN', 'chunk': 'I-NP'},\n",
       "  {'form': 'with', 'pos': 'IN', 'chunk': 'B-PP'},\n",
       "  {'form': 'Boeing', 'pos': 'NNP', 'chunk': 'B-NP'},\n",
       "  {'form': 'Co.', 'pos': 'NNP', 'chunk': 'I-NP'},\n",
       "  {'form': 'to', 'pos': 'TO', 'chunk': 'B-VP'},\n",
       "  {'form': 'provide', 'pos': 'VB', 'chunk': 'I-VP'},\n",
       "  {'form': 'structural', 'pos': 'JJ', 'chunk': 'B-NP'},\n",
       "  {'form': 'parts', 'pos': 'NNS', 'chunk': 'I-NP'},\n",
       "  {'form': 'for', 'pos': 'IN', 'chunk': 'B-PP'},\n",
       "  {'form': 'Boeing', 'pos': 'NNP', 'chunk': 'B-NP'},\n",
       "  {'form': \"'s\", 'pos': 'POS', 'chunk': 'B-NP'},\n",
       "  {'form': '747', 'pos': 'CD', 'chunk': 'I-NP'},\n",
       "  {'form': 'jetliners', 'pos': 'NNS', 'chunk': 'I-NP'},\n",
       "  {'form': '.', 'pos': '.', 'chunk': 'O'}],\n",
       " [{'form': 'Rockwell', 'pos': 'NNP', 'chunk': 'B-NP'},\n",
       "  {'form': 'said', 'pos': 'VBD', 'chunk': 'B-VP'},\n",
       "  {'form': 'the', 'pos': 'DT', 'chunk': 'B-NP'},\n",
       "  {'form': 'agreement', 'pos': 'NN', 'chunk': 'I-NP'},\n",
       "  {'form': 'calls', 'pos': 'VBZ', 'chunk': 'B-VP'},\n",
       "  {'form': 'for', 'pos': 'IN', 'chunk': 'B-SBAR'},\n",
       "  {'form': 'it', 'pos': 'PRP', 'chunk': 'B-NP'},\n",
       "  {'form': 'to', 'pos': 'TO', 'chunk': 'B-VP'},\n",
       "  {'form': 'supply', 'pos': 'VB', 'chunk': 'I-VP'},\n",
       "  {'form': '200', 'pos': 'CD', 'chunk': 'B-NP'},\n",
       "  {'form': 'additional', 'pos': 'JJ', 'chunk': 'I-NP'},\n",
       "  {'form': 'so-called', 'pos': 'JJ', 'chunk': 'I-NP'},\n",
       "  {'form': 'shipsets', 'pos': 'NNS', 'chunk': 'I-NP'},\n",
       "  {'form': 'for', 'pos': 'IN', 'chunk': 'B-PP'},\n",
       "  {'form': 'the', 'pos': 'DT', 'chunk': 'B-NP'},\n",
       "  {'form': 'planes', 'pos': 'NNS', 'chunk': 'I-NP'},\n",
       "  {'form': '.', 'pos': '.', 'chunk': 'O'}]]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_sentences = read_sentences(test_file)\n",
    "test_corpus = split_rows(test_sentences, column_names)\n",
    "test_corpus[:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.117906,
     "end_time": "2020-12-11T20:31:42.610047",
     "exception": false,
     "start_time": "2020-12-11T20:31:42.492141",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We extract the features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:31:42.825573Z",
     "iopub.status.busy": "2020-12-11T20:31:42.824876Z",
     "iopub.status.idle": "2020-12-11T20:31:43.102543Z",
     "shell.execute_reply": "2020-12-11T20:31:43.103110Z"
    },
    "papermill": {
     "duration": 0.383481,
     "end_time": "2020-12-11T20:31:43.103284",
     "exception": false,
     "start_time": "2020-12-11T20:31:42.719803",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'word_n2': 'bos',\n",
       "  'word_n1': 'bos',\n",
       "  'word': 'rockwell',\n",
       "  'word_p1': 'international',\n",
       "  'word_p2': 'corp.',\n",
       "  'pos_n2': 'BOS',\n",
       "  'pos_n1': 'BOS',\n",
       "  'pos': 'NNP',\n",
       "  'pos_p1': 'NNP',\n",
       "  'pos_p2': 'NNP'},\n",
       " {'word_n2': 'bos',\n",
       "  'word_n1': 'rockwell',\n",
       "  'word': 'international',\n",
       "  'word_p1': 'corp.',\n",
       "  'word_p2': \"'s\",\n",
       "  'pos_n2': 'BOS',\n",
       "  'pos_n1': 'NNP',\n",
       "  'pos': 'NNP',\n",
       "  'pos_p1': 'NNP',\n",
       "  'pos_p2': 'POS'}]"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_test_dict, y_test = extract_features_static(test_corpus, w_size, feature_names)\n",
    "X_test_dict[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:31:43.315666Z",
     "iopub.status.busy": "2020-12-11T20:31:43.314912Z",
     "iopub.status.idle": "2020-12-11T20:31:43.318559Z",
     "shell.execute_reply": "2020-12-11T20:31:43.319069Z"
    },
    "papermill": {
     "duration": 0.111414,
     "end_time": "2020-12-11T20:31:43.319228",
     "exception": false,
     "start_time": "2020-12-11T20:31:43.207814",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['B-NP', 'I-NP']"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_test[:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.102007,
     "end_time": "2020-12-11T20:31:43.522827",
     "exception": false,
     "start_time": "2020-12-11T20:31:43.420820",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We vectorize the features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:31:43.766469Z",
     "iopub.status.busy": "2020-12-11T20:31:43.765846Z",
     "iopub.status.idle": "2020-12-11T20:31:44.608468Z",
     "shell.execute_reply": "2020-12-11T20:31:44.609425Z"
    },
    "papermill": {
     "duration": 0.975319,
     "end_time": "2020-12-11T20:31:44.609643",
     "exception": false,
     "start_time": "2020-12-11T20:31:43.634324",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "X_test = vec.transform(X_test_dict)  # Possible to add: .toarray()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.103102,
     "end_time": "2020-12-11T20:31:44.822603",
     "exception": false,
     "start_time": "2020-12-11T20:31:44.719501",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "And we predict the test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:31:45.039694Z",
     "iopub.status.busy": "2020-12-11T20:31:45.038601Z",
     "iopub.status.idle": "2020-12-11T20:31:45.077906Z",
     "shell.execute_reply": "2020-12-11T20:31:45.077361Z"
    },
    "papermill": {
     "duration": 0.151598,
     "end_time": "2020-12-11T20:31:45.078023",
     "exception": false,
     "start_time": "2020-12-11T20:31:44.926425",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['B-NP', 'I-NP'], dtype='<U7')"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_test_predicted = classifier.predict(X_test)\n",
    "y_test_predicted[:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.102996,
     "end_time": "2020-12-11T20:31:45.282383",
     "exception": false,
     "start_time": "2020-12-11T20:31:45.179387",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We now add the predicted chunks to the sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:31:45.510148Z",
     "iopub.status.busy": "2020-12-11T20:31:45.504647Z",
     "iopub.status.idle": "2020-12-11T20:31:45.544600Z",
     "shell.execute_reply": "2020-12-11T20:31:45.543862Z"
    },
    "papermill": {
     "duration": 0.157563,
     "end_time": "2020-12-11T20:31:45.544726",
     "exception": false,
     "start_time": "2020-12-11T20:31:45.387163",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "inx = 0\n",
    "for sentence in test_corpus:\n",
    "    for word in sentence:\n",
    "        word['pchunk'] = y_test_predicted[inx]\n",
    "        inx += 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.112096,
     "end_time": "2020-12-11T20:31:45.761926",
     "exception": false,
     "start_time": "2020-12-11T20:31:45.649830",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "The index sould be equal to the length of the prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:31:45.978837Z",
     "iopub.status.busy": "2020-12-11T20:31:45.978116Z",
     "iopub.status.idle": "2020-12-11T20:31:45.984809Z",
     "shell.execute_reply": "2020-12-11T20:31:45.984102Z"
    },
    "papermill": {
     "duration": 0.119285,
     "end_time": "2020-12-11T20:31:45.984931",
     "exception": false,
     "start_time": "2020-12-11T20:31:45.865646",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "47377\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "47377"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(inx)\n",
    "len(y_test_predicted)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:31:46.208736Z",
     "iopub.status.busy": "2020-12-11T20:31:46.207641Z",
     "iopub.status.idle": "2020-12-11T20:31:46.212515Z",
     "shell.execute_reply": "2020-12-11T20:31:46.211826Z"
    },
    "papermill": {
     "duration": 0.122283,
     "end_time": "2020-12-11T20:31:46.212634",
     "exception": false,
     "start_time": "2020-12-11T20:31:46.090351",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[{'form': 'Rockwell', 'pos': 'NNP', 'chunk': 'B-NP', 'pchunk': 'B-NP'},\n",
       "  {'form': 'International', 'pos': 'NNP', 'chunk': 'I-NP', 'pchunk': 'I-NP'},\n",
       "  {'form': 'Corp.', 'pos': 'NNP', 'chunk': 'I-NP', 'pchunk': 'I-NP'},\n",
       "  {'form': \"'s\", 'pos': 'POS', 'chunk': 'B-NP', 'pchunk': 'B-NP'},\n",
       "  {'form': 'Tulsa', 'pos': 'NNP', 'chunk': 'I-NP', 'pchunk': 'I-NP'},\n",
       "  {'form': 'unit', 'pos': 'NN', 'chunk': 'I-NP', 'pchunk': 'I-NP'},\n",
       "  {'form': 'said', 'pos': 'VBD', 'chunk': 'B-VP', 'pchunk': 'B-VP'},\n",
       "  {'form': 'it', 'pos': 'PRP', 'chunk': 'B-NP', 'pchunk': 'B-NP'},\n",
       "  {'form': 'signed', 'pos': 'VBD', 'chunk': 'B-VP', 'pchunk': 'B-VP'},\n",
       "  {'form': 'a', 'pos': 'DT', 'chunk': 'B-NP', 'pchunk': 'B-NP'},\n",
       "  {'form': 'tentative', 'pos': 'JJ', 'chunk': 'I-NP', 'pchunk': 'I-NP'},\n",
       "  {'form': 'agreement', 'pos': 'NN', 'chunk': 'I-NP', 'pchunk': 'I-NP'},\n",
       "  {'form': 'extending', 'pos': 'VBG', 'chunk': 'B-VP', 'pchunk': 'B-VP'},\n",
       "  {'form': 'its', 'pos': 'PRP$', 'chunk': 'B-NP', 'pchunk': 'B-NP'},\n",
       "  {'form': 'contract', 'pos': 'NN', 'chunk': 'I-NP', 'pchunk': 'I-NP'},\n",
       "  {'form': 'with', 'pos': 'IN', 'chunk': 'B-PP', 'pchunk': 'B-PP'},\n",
       "  {'form': 'Boeing', 'pos': 'NNP', 'chunk': 'B-NP', 'pchunk': 'B-NP'},\n",
       "  {'form': 'Co.', 'pos': 'NNP', 'chunk': 'I-NP', 'pchunk': 'I-NP'},\n",
       "  {'form': 'to', 'pos': 'TO', 'chunk': 'B-VP', 'pchunk': 'B-VP'},\n",
       "  {'form': 'provide', 'pos': 'VB', 'chunk': 'I-VP', 'pchunk': 'I-VP'},\n",
       "  {'form': 'structural', 'pos': 'JJ', 'chunk': 'B-NP', 'pchunk': 'B-NP'},\n",
       "  {'form': 'parts', 'pos': 'NNS', 'chunk': 'I-NP', 'pchunk': 'I-NP'},\n",
       "  {'form': 'for', 'pos': 'IN', 'chunk': 'B-PP', 'pchunk': 'B-PP'},\n",
       "  {'form': 'Boeing', 'pos': 'NNP', 'chunk': 'B-NP', 'pchunk': 'B-NP'},\n",
       "  {'form': \"'s\", 'pos': 'POS', 'chunk': 'B-NP', 'pchunk': 'B-NP'},\n",
       "  {'form': '747', 'pos': 'CD', 'chunk': 'I-NP', 'pchunk': 'I-NP'},\n",
       "  {'form': 'jetliners', 'pos': 'NNS', 'chunk': 'I-NP', 'pchunk': 'I-NP'},\n",
       "  {'form': '.', 'pos': '.', 'chunk': 'O', 'pchunk': 'O'}],\n",
       " [{'form': 'Rockwell', 'pos': 'NNP', 'chunk': 'B-NP', 'pchunk': 'B-NP'},\n",
       "  {'form': 'said', 'pos': 'VBD', 'chunk': 'B-VP', 'pchunk': 'B-VP'},\n",
       "  {'form': 'the', 'pos': 'DT', 'chunk': 'B-NP', 'pchunk': 'B-NP'},\n",
       "  {'form': 'agreement', 'pos': 'NN', 'chunk': 'I-NP', 'pchunk': 'I-NP'},\n",
       "  {'form': 'calls', 'pos': 'VBZ', 'chunk': 'B-VP', 'pchunk': 'B-VP'},\n",
       "  {'form': 'for', 'pos': 'IN', 'chunk': 'B-SBAR', 'pchunk': 'B-SBAR'},\n",
       "  {'form': 'it', 'pos': 'PRP', 'chunk': 'B-NP', 'pchunk': 'B-NP'},\n",
       "  {'form': 'to', 'pos': 'TO', 'chunk': 'B-VP', 'pchunk': 'B-VP'},\n",
       "  {'form': 'supply', 'pos': 'VB', 'chunk': 'I-VP', 'pchunk': 'I-VP'},\n",
       "  {'form': '200', 'pos': 'CD', 'chunk': 'B-NP', 'pchunk': 'B-NP'},\n",
       "  {'form': 'additional', 'pos': 'JJ', 'chunk': 'I-NP', 'pchunk': 'I-NP'},\n",
       "  {'form': 'so-called', 'pos': 'JJ', 'chunk': 'I-NP', 'pchunk': 'I-NP'},\n",
       "  {'form': 'shipsets', 'pos': 'NNS', 'chunk': 'I-NP', 'pchunk': 'I-NP'},\n",
       "  {'form': 'for', 'pos': 'IN', 'chunk': 'B-PP', 'pchunk': 'B-PP'},\n",
       "  {'form': 'the', 'pos': 'DT', 'chunk': 'B-NP', 'pchunk': 'B-NP'},\n",
       "  {'form': 'planes', 'pos': 'NNS', 'chunk': 'I-NP', 'pchunk': 'I-NP'},\n",
       "  {'form': '.', 'pos': '.', 'chunk': 'O', 'pchunk': 'O'}]]"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_corpus[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:31:46.444899Z",
     "iopub.status.busy": "2020-12-11T20:31:46.444149Z",
     "iopub.status.idle": "2020-12-11T20:31:46.518882Z",
     "shell.execute_reply": "2020-12-11T20:31:46.518271Z"
    },
    "papermill": {
     "duration": 0.194036,
     "end_time": "2020-12-11T20:31:46.519020",
     "exception": false,
     "start_time": "2020-12-11T20:31:46.324984",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "save_results(test_corpus, keys, 'out')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.104727,
     "end_time": "2020-12-11T20:31:46.728265",
     "exception": false,
     "start_time": "2020-12-11T20:31:46.623538",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "#### Evaluating the performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:31:46.963891Z",
     "iopub.status.busy": "2020-12-11T20:31:46.952957Z",
     "iopub.status.idle": "2020-12-11T20:31:47.472294Z",
     "shell.execute_reply": "2020-12-11T20:31:47.471568Z"
    },
    "papermill": {
     "duration": 0.637218,
     "end_time": "2020-12-11T20:31:47.472412",
     "exception": false,
     "start_time": "2020-12-11T20:31:46.835194",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "lines = open('out').read().splitlines()\n",
    "res = conlleval.evaluate(lines)\n",
    "simple_ml_score = res['overall']['chunks']['evals']['f1']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:31:47.693276Z",
     "iopub.status.busy": "2020-12-11T20:31:47.692234Z",
     "iopub.status.idle": "2020-12-11T20:31:47.696937Z",
     "shell.execute_reply": "2020-12-11T20:31:47.696273Z"
    },
    "papermill": {
     "duration": 0.114256,
     "end_time": "2020-12-11T20:31:47.697048",
     "exception": false,
     "start_time": "2020-12-11T20:31:47.582792",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9157688948047087"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "simple_ml_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.108362,
     "end_time": "2020-12-11T20:31:47.917418",
     "exception": false,
     "start_time": "2020-12-11T20:31:47.809056",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Question on the ML program"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.105047,
     "end_time": "2020-12-11T20:31:48.127046",
     "exception": false,
     "start_time": "2020-12-11T20:31:48.021999",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "1. What is the feature vector that corresponds to the <tt>ml_chunker.py</tt> program? Is it the same Kudoh\n",
    "    and Matsumoto used in their experiment?\n",
    "2. What is the performance of the chunker?\n",
    "3. Remove the lexical features (the words) from the feature vector and measure the performance. You should\n",
    "    observe a decrease.\n",
    "4. What is the classifier used in the program? Try two other classifiers from sklean and measure their performance:\n",
    "    decision trees, perceptron, support vector machines, etc..\n",
    "    Be aware that support vector machines take a long time to train: up to one hour."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.108479,
     "end_time": "2020-12-11T20:31:48.340747",
     "exception": false,
     "start_time": "2020-12-11T20:31:48.232268",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Using Machine Learning: Adding all the features from Kudoh and Matsumoto"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.110907,
     "end_time": "2020-12-11T20:31:48.560569",
     "exception": false,
     "start_time": "2020-12-11T20:31:48.449662",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Complement the feature vector used in the previous section with the two dynamic features, $c_{i-2}, c_{i-1}$, and train a new model. You will need to write a new `extract_features_sent_dyn` and `predict` functions. \n",
    "In his experiments, your teacher obtained a F1 score of 92.65 with logistic regression and a lbfgs solver and automatic multiclass;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.105876,
     "end_time": "2020-12-11T20:31:48.773960",
     "exception": false,
     "start_time": "2020-12-11T20:31:48.668084",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "**A frequent mistake in the labs** is to use the gold-standard chunks from the test set. Be aware that  when you predict the test set, you do not know the dynamic features in advance and you must  not use the ones from the test file. You will use the two previous chunk tags that you have predicted."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.104712,
     "end_time": "2020-12-11T20:31:48.983100",
     "exception": false,
     "start_time": "2020-12-11T20:31:48.878388",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "You need to reach a global F1 score of 92 to pass this laboratory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:31:49.203363Z",
     "iopub.status.busy": "2020-12-11T20:31:49.202284Z",
     "iopub.status.idle": "2020-12-11T20:31:49.205416Z",
     "shell.execute_reply": "2020-12-11T20:31:49.204860Z"
    },
    "papermill": {
     "duration": 0.119462,
     "end_time": "2020-12-11T20:31:49.205532",
     "exception": false,
     "start_time": "2020-12-11T20:31:49.086070",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def extract_features_sent_dyn(sentence, w_size, feature_names):\n",
    "    \n",
    "    # We pad the sentence to extract the context window more easily\n",
    "    start = [{'form': 'BOS', 'pos': 'BOS', 'chunk': 'BOS'}]\n",
    "    end = [{'form': 'EOS', 'pos': 'EOS', 'chunk': 'EOS'}]\n",
    "    start *= w_size\n",
    "    end *= w_size\n",
    "    padded_sentence = start + sentence\n",
    "    padded_sentence += end\n",
    "\n",
    "    # We extract the features and the classes\n",
    "    # X contains is a list of features, where each feature vector is a dictionary\n",
    "    # y is the list of classes\n",
    "    X = list()\n",
    "    y = list()\n",
    "    for i in range(len(padded_sentence) - 2 * w_size):\n",
    "        # x is a row of X\n",
    "        x = list()\n",
    "        # The words in lower case\n",
    "        for j in range(2 * w_size + 1):\n",
    "            x.append(padded_sentence[i + j]['form'].lower())\n",
    "        # The POS\n",
    "        for j in range(2 * w_size + 1):\n",
    "            x.append(padded_sentence[i + j]['pos'])\n",
    "        # The chunks (Up to the word)\n",
    "        for j in range(w_size):\n",
    "            x.append(padded_sentence[i + j]['chunk'])\n",
    "            \n",
    "        # We represent the feature vector as a dictionary\n",
    "        X.append(dict(zip(feature_names, x)))\n",
    "        # The classes are stored in a list\n",
    "        y.append(padded_sentence[i + w_size]['chunk'])\n",
    "    return X, y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:31:49.421050Z",
     "iopub.status.busy": "2020-12-11T20:31:49.420385Z",
     "iopub.status.idle": "2020-12-11T20:31:49.423654Z",
     "shell.execute_reply": "2020-12-11T20:31:49.423046Z"
    },
    "papermill": {
     "duration": 0.113635,
     "end_time": "2020-12-11T20:31:49.423864",
     "exception": false,
     "start_time": "2020-12-11T20:31:49.310229",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def extract_features_dyn(sentences, w_size, feature_names):\n",
    "    \"\"\"\n",
    "    Builds X matrix and y vector\n",
    "    X is a list of dictionaries and y is a list\n",
    "    :param sentences:\n",
    "    :param w_size:\n",
    "    :return:\n",
    "    \"\"\"\n",
    "    X_l = []\n",
    "    y_l = []\n",
    "    for sentence in sentences:\n",
    "        X, y = extract_features_sent_dyn(sentence, w_size, feature_names)\n",
    "        X_l.extend(X)\n",
    "        y_l.extend(y)\n",
    "    return X_l, y_l"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:31:49.635744Z",
     "iopub.status.busy": "2020-12-11T20:31:49.635121Z",
     "iopub.status.idle": "2020-12-11T20:31:49.639246Z",
     "shell.execute_reply": "2020-12-11T20:31:49.639734Z"
    },
    "papermill": {
     "duration": 0.110972,
     "end_time": "2020-12-11T20:31:49.639895",
     "exception": false,
     "start_time": "2020-12-11T20:31:49.528923",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "feature_names_dyn = ['word_n2', 'word_n1', 'word', 'word_p1', 'word_p2',\n",
    "                     'pos_n2', 'pos_n1', 'pos', 'pos_p1', 'pos_p2', 'chunk_n2',\n",
    "                     'chunk_n1']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:31:49.851314Z",
     "iopub.status.busy": "2020-12-11T20:31:49.850631Z",
     "iopub.status.idle": "2020-12-11T20:31:50.179065Z",
     "shell.execute_reply": "2020-12-11T20:31:50.178490Z"
    },
    "papermill": {
     "duration": 0.43483,
     "end_time": "2020-12-11T20:31:50.179191",
     "exception": false,
     "start_time": "2020-12-11T20:31:49.744361",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[{'form': 'Confidence', 'pos': 'NN', 'chunk': 'B-NP'},\n",
       "  {'form': 'in', 'pos': 'IN', 'chunk': 'B-PP'},\n",
       "  {'form': 'the', 'pos': 'DT', 'chunk': 'B-NP'},\n",
       "  {'form': 'pound', 'pos': 'NN', 'chunk': 'I-NP'},\n",
       "  {'form': 'is', 'pos': 'VBZ', 'chunk': 'B-VP'},\n",
       "  {'form': 'widely', 'pos': 'RB', 'chunk': 'I-VP'},\n",
       "  {'form': 'expected', 'pos': 'VBN', 'chunk': 'I-VP'},\n",
       "  {'form': 'to', 'pos': 'TO', 'chunk': 'I-VP'},\n",
       "  {'form': 'take', 'pos': 'VB', 'chunk': 'I-VP'},\n",
       "  {'form': 'another', 'pos': 'DT', 'chunk': 'B-NP'},\n",
       "  {'form': 'sharp', 'pos': 'JJ', 'chunk': 'I-NP'},\n",
       "  {'form': 'dive', 'pos': 'NN', 'chunk': 'I-NP'},\n",
       "  {'form': 'if', 'pos': 'IN', 'chunk': 'B-SBAR'},\n",
       "  {'form': 'trade', 'pos': 'NN', 'chunk': 'B-NP'},\n",
       "  {'form': 'figures', 'pos': 'NNS', 'chunk': 'I-NP'},\n",
       "  {'form': 'for', 'pos': 'IN', 'chunk': 'B-PP'},\n",
       "  {'form': 'September', 'pos': 'NNP', 'chunk': 'B-NP'},\n",
       "  {'form': ',', 'pos': ',', 'chunk': 'O'},\n",
       "  {'form': 'due', 'pos': 'JJ', 'chunk': 'B-ADJP'},\n",
       "  {'form': 'for', 'pos': 'IN', 'chunk': 'B-PP'},\n",
       "  {'form': 'release', 'pos': 'NN', 'chunk': 'B-NP'},\n",
       "  {'form': 'tomorrow', 'pos': 'NN', 'chunk': 'B-NP'},\n",
       "  {'form': ',', 'pos': ',', 'chunk': 'O'},\n",
       "  {'form': 'fail', 'pos': 'VB', 'chunk': 'B-VP'},\n",
       "  {'form': 'to', 'pos': 'TO', 'chunk': 'I-VP'},\n",
       "  {'form': 'show', 'pos': 'VB', 'chunk': 'I-VP'},\n",
       "  {'form': 'a', 'pos': 'DT', 'chunk': 'B-NP'},\n",
       "  {'form': 'substantial', 'pos': 'JJ', 'chunk': 'I-NP'},\n",
       "  {'form': 'improvement', 'pos': 'NN', 'chunk': 'I-NP'},\n",
       "  {'form': 'from', 'pos': 'IN', 'chunk': 'B-PP'},\n",
       "  {'form': 'July', 'pos': 'NNP', 'chunk': 'B-NP'},\n",
       "  {'form': 'and', 'pos': 'CC', 'chunk': 'I-NP'},\n",
       "  {'form': 'August', 'pos': 'NNP', 'chunk': 'I-NP'},\n",
       "  {'form': \"'s\", 'pos': 'POS', 'chunk': 'B-NP'},\n",
       "  {'form': 'near-record', 'pos': 'JJ', 'chunk': 'I-NP'},\n",
       "  {'form': 'deficits', 'pos': 'NNS', 'chunk': 'I-NP'},\n",
       "  {'form': '.', 'pos': '.', 'chunk': 'O'}],\n",
       " [{'form': 'Chancellor', 'pos': 'NNP', 'chunk': 'O'},\n",
       "  {'form': 'of', 'pos': 'IN', 'chunk': 'B-PP'},\n",
       "  {'form': 'the', 'pos': 'DT', 'chunk': 'B-NP'},\n",
       "  {'form': 'Exchequer', 'pos': 'NNP', 'chunk': 'I-NP'},\n",
       "  {'form': 'Nigel', 'pos': 'NNP', 'chunk': 'B-NP'},\n",
       "  {'form': 'Lawson', 'pos': 'NNP', 'chunk': 'I-NP'},\n",
       "  {'form': \"'s\", 'pos': 'POS', 'chunk': 'B-NP'},\n",
       "  {'form': 'restated', 'pos': 'VBN', 'chunk': 'I-NP'},\n",
       "  {'form': 'commitment', 'pos': 'NN', 'chunk': 'I-NP'},\n",
       "  {'form': 'to', 'pos': 'TO', 'chunk': 'B-PP'},\n",
       "  {'form': 'a', 'pos': 'DT', 'chunk': 'B-NP'},\n",
       "  {'form': 'firm', 'pos': 'NN', 'chunk': 'I-NP'},\n",
       "  {'form': 'monetary', 'pos': 'JJ', 'chunk': 'I-NP'},\n",
       "  {'form': 'policy', 'pos': 'NN', 'chunk': 'I-NP'},\n",
       "  {'form': 'has', 'pos': 'VBZ', 'chunk': 'B-VP'},\n",
       "  {'form': 'helped', 'pos': 'VBN', 'chunk': 'I-VP'},\n",
       "  {'form': 'to', 'pos': 'TO', 'chunk': 'I-VP'},\n",
       "  {'form': 'prevent', 'pos': 'VB', 'chunk': 'I-VP'},\n",
       "  {'form': 'a', 'pos': 'DT', 'chunk': 'B-NP'},\n",
       "  {'form': 'freefall', 'pos': 'NN', 'chunk': 'I-NP'},\n",
       "  {'form': 'in', 'pos': 'IN', 'chunk': 'B-PP'},\n",
       "  {'form': 'sterling', 'pos': 'NN', 'chunk': 'B-NP'},\n",
       "  {'form': 'over', 'pos': 'IN', 'chunk': 'B-PP'},\n",
       "  {'form': 'the', 'pos': 'DT', 'chunk': 'B-NP'},\n",
       "  {'form': 'past', 'pos': 'JJ', 'chunk': 'I-NP'},\n",
       "  {'form': 'week', 'pos': 'NN', 'chunk': 'I-NP'},\n",
       "  {'form': '.', 'pos': '.', 'chunk': 'O'}]]"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_sentences = read_sentences(train_file)\n",
    "train_corpus = split_rows(train_sentences, column_names)\n",
    "train_corpus[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:31:50.419816Z",
     "iopub.status.busy": "2020-12-11T20:31:50.399169Z",
     "iopub.status.idle": "2020-12-11T20:31:51.874004Z",
     "shell.execute_reply": "2020-12-11T20:31:51.873383Z"
    },
    "papermill": {
     "duration": 1.588842,
     "end_time": "2020-12-11T20:31:51.874118",
     "exception": false,
     "start_time": "2020-12-11T20:31:50.285276",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "X_dict, y = extract_features_dyn(train_corpus, w_size, feature_names_dyn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:31:52.096564Z",
     "iopub.status.busy": "2020-12-11T20:31:52.095704Z",
     "iopub.status.idle": "2020-12-11T20:31:52.100039Z",
     "shell.execute_reply": "2020-12-11T20:31:52.099403Z"
    },
    "papermill": {
     "duration": 0.118344,
     "end_time": "2020-12-11T20:31:52.100150",
     "exception": false,
     "start_time": "2020-12-11T20:31:51.981806",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'word_n2': 'bos',\n",
       "  'word_n1': 'bos',\n",
       "  'word': 'confidence',\n",
       "  'word_p1': 'in',\n",
       "  'word_p2': 'the',\n",
       "  'pos_n2': 'BOS',\n",
       "  'pos_n1': 'BOS',\n",
       "  'pos': 'NN',\n",
       "  'pos_p1': 'IN',\n",
       "  'pos_p2': 'DT',\n",
       "  'chunk_n2': 'BOS',\n",
       "  'chunk_n1': 'BOS'},\n",
       " {'word_n2': 'bos',\n",
       "  'word_n1': 'confidence',\n",
       "  'word': 'in',\n",
       "  'word_p1': 'the',\n",
       "  'word_p2': 'pound',\n",
       "  'pos_n2': 'BOS',\n",
       "  'pos_n1': 'NN',\n",
       "  'pos': 'IN',\n",
       "  'pos_p1': 'DT',\n",
       "  'pos_p2': 'NN',\n",
       "  'chunk_n2': 'BOS',\n",
       "  'chunk_n1': 'B-NP'},\n",
       " {'word_n2': 'confidence',\n",
       "  'word_n1': 'in',\n",
       "  'word': 'the',\n",
       "  'word_p1': 'pound',\n",
       "  'word_p2': 'is',\n",
       "  'pos_n2': 'NN',\n",
       "  'pos_n1': 'IN',\n",
       "  'pos': 'DT',\n",
       "  'pos_p1': 'NN',\n",
       "  'pos_p2': 'VBZ',\n",
       "  'chunk_n2': 'B-NP',\n",
       "  'chunk_n1': 'B-PP'}]"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_dict[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.112333,
     "end_time": "2020-12-11T20:31:52.319959",
     "exception": false,
     "start_time": "2020-12-11T20:31:52.207626",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "You will now vectorize the training set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:31:52.552158Z",
     "iopub.status.busy": "2020-12-11T20:31:52.546884Z",
     "iopub.status.idle": "2020-12-11T20:31:57.351702Z",
     "shell.execute_reply": "2020-12-11T20:31:57.351005Z"
    },
    "papermill": {
     "duration": 4.923801,
     "end_time": "2020-12-11T20:31:57.351843",
     "exception": false,
     "start_time": "2020-12-11T20:31:52.428042",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "X = vec.fit_transform(X_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.106016,
     "end_time": "2020-12-11T20:31:57.563743",
     "exception": false,
     "start_time": "2020-12-11T20:31:57.457727",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "And fit the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:31:57.781898Z",
     "iopub.status.busy": "2020-12-11T20:31:57.780976Z",
     "iopub.status.idle": "2020-12-11T20:33:18.101092Z",
     "shell.execute_reply": "2020-12-11T20:33:18.102416Z"
    },
    "papermill": {
     "duration": 80.433189,
     "end_time": "2020-12-11T20:33:18.102641",
     "exception": false,
     "start_time": "2020-12-11T20:31:57.669452",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/sklearn/linear_model/_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n"
     ]
    }
   ],
   "source": [
    "model = classifier.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:33:18.323205Z",
     "iopub.status.busy": "2020-12-11T20:33:18.322270Z",
     "iopub.status.idle": "2020-12-11T20:33:18.325893Z",
     "shell.execute_reply": "2020-12-11T20:33:18.326524Z"
    },
    "papermill": {
     "duration": 0.114636,
     "end_time": "2020-12-11T20:33:18.326655",
     "exception": false,
     "start_time": "2020-12-11T20:33:18.212019",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression()"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.105666,
     "end_time": "2020-12-11T20:33:18.538610",
     "exception": false,
     "start_time": "2020-12-11T20:33:18.432944",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Prediction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.105616,
     "end_time": "2020-12-11T20:33:18.752670",
     "exception": false,
     "start_time": "2020-12-11T20:33:18.647054",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "You will finally predict the test set. We load the corpus again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:33:18.971034Z",
     "iopub.status.busy": "2020-12-11T20:33:18.970041Z",
     "iopub.status.idle": "2020-12-11T20:33:19.049669Z",
     "shell.execute_reply": "2020-12-11T20:33:19.050258Z"
    },
    "papermill": {
     "duration": 0.191216,
     "end_time": "2020-12-11T20:33:19.050396",
     "exception": false,
     "start_time": "2020-12-11T20:33:18.859180",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[{'form': 'Rockwell', 'pos': 'NNP', 'chunk': 'B-NP'},\n",
       "  {'form': 'International', 'pos': 'NNP', 'chunk': 'I-NP'},\n",
       "  {'form': 'Corp.', 'pos': 'NNP', 'chunk': 'I-NP'},\n",
       "  {'form': \"'s\", 'pos': 'POS', 'chunk': 'B-NP'},\n",
       "  {'form': 'Tulsa', 'pos': 'NNP', 'chunk': 'I-NP'},\n",
       "  {'form': 'unit', 'pos': 'NN', 'chunk': 'I-NP'},\n",
       "  {'form': 'said', 'pos': 'VBD', 'chunk': 'B-VP'},\n",
       "  {'form': 'it', 'pos': 'PRP', 'chunk': 'B-NP'},\n",
       "  {'form': 'signed', 'pos': 'VBD', 'chunk': 'B-VP'},\n",
       "  {'form': 'a', 'pos': 'DT', 'chunk': 'B-NP'},\n",
       "  {'form': 'tentative', 'pos': 'JJ', 'chunk': 'I-NP'},\n",
       "  {'form': 'agreement', 'pos': 'NN', 'chunk': 'I-NP'},\n",
       "  {'form': 'extending', 'pos': 'VBG', 'chunk': 'B-VP'},\n",
       "  {'form': 'its', 'pos': 'PRP$', 'chunk': 'B-NP'},\n",
       "  {'form': 'contract', 'pos': 'NN', 'chunk': 'I-NP'},\n",
       "  {'form': 'with', 'pos': 'IN', 'chunk': 'B-PP'},\n",
       "  {'form': 'Boeing', 'pos': 'NNP', 'chunk': 'B-NP'},\n",
       "  {'form': 'Co.', 'pos': 'NNP', 'chunk': 'I-NP'},\n",
       "  {'form': 'to', 'pos': 'TO', 'chunk': 'B-VP'},\n",
       "  {'form': 'provide', 'pos': 'VB', 'chunk': 'I-VP'},\n",
       "  {'form': 'structural', 'pos': 'JJ', 'chunk': 'B-NP'},\n",
       "  {'form': 'parts', 'pos': 'NNS', 'chunk': 'I-NP'},\n",
       "  {'form': 'for', 'pos': 'IN', 'chunk': 'B-PP'},\n",
       "  {'form': 'Boeing', 'pos': 'NNP', 'chunk': 'B-NP'},\n",
       "  {'form': \"'s\", 'pos': 'POS', 'chunk': 'B-NP'},\n",
       "  {'form': '747', 'pos': 'CD', 'chunk': 'I-NP'},\n",
       "  {'form': 'jetliners', 'pos': 'NNS', 'chunk': 'I-NP'},\n",
       "  {'form': '.', 'pos': '.', 'chunk': 'O'}],\n",
       " [{'form': 'Rockwell', 'pos': 'NNP', 'chunk': 'B-NP'},\n",
       "  {'form': 'said', 'pos': 'VBD', 'chunk': 'B-VP'},\n",
       "  {'form': 'the', 'pos': 'DT', 'chunk': 'B-NP'},\n",
       "  {'form': 'agreement', 'pos': 'NN', 'chunk': 'I-NP'},\n",
       "  {'form': 'calls', 'pos': 'VBZ', 'chunk': 'B-VP'},\n",
       "  {'form': 'for', 'pos': 'IN', 'chunk': 'B-SBAR'},\n",
       "  {'form': 'it', 'pos': 'PRP', 'chunk': 'B-NP'},\n",
       "  {'form': 'to', 'pos': 'TO', 'chunk': 'B-VP'},\n",
       "  {'form': 'supply', 'pos': 'VB', 'chunk': 'I-VP'},\n",
       "  {'form': '200', 'pos': 'CD', 'chunk': 'B-NP'},\n",
       "  {'form': 'additional', 'pos': 'JJ', 'chunk': 'I-NP'},\n",
       "  {'form': 'so-called', 'pos': 'JJ', 'chunk': 'I-NP'},\n",
       "  {'form': 'shipsets', 'pos': 'NNS', 'chunk': 'I-NP'},\n",
       "  {'form': 'for', 'pos': 'IN', 'chunk': 'B-PP'},\n",
       "  {'form': 'the', 'pos': 'DT', 'chunk': 'B-NP'},\n",
       "  {'form': 'planes', 'pos': 'NNS', 'chunk': 'I-NP'},\n",
       "  {'form': '.', 'pos': '.', 'chunk': 'O'}]]"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_sentences = read_sentences(test_file)\n",
    "test_corpus = split_rows(test_sentences, column_names)\n",
    "test_corpus[:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.107889,
     "end_time": "2020-12-11T20:33:19.265048",
     "exception": false,
     "start_time": "2020-12-11T20:33:19.157159",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Let us extract the static features from one sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:33:19.483656Z",
     "iopub.status.busy": "2020-12-11T20:33:19.482659Z",
     "iopub.status.idle": "2020-12-11T20:33:19.506967Z",
     "shell.execute_reply": "2020-12-11T20:33:19.507494Z"
    },
    "papermill": {
     "duration": 0.135391,
     "end_time": "2020-12-11T20:33:19.507639",
     "exception": false,
     "start_time": "2020-12-11T20:33:19.372248",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'word_n2': 'bos',\n",
       "  'word_n1': 'bos',\n",
       "  'word': 'rockwell',\n",
       "  'word_p1': 'international',\n",
       "  'word_p2': 'corp.',\n",
       "  'pos_n2': 'BOS',\n",
       "  'pos_n1': 'BOS',\n",
       "  'pos': 'NNP',\n",
       "  'pos_p1': 'NNP',\n",
       "  'pos_p2': 'NNP'},\n",
       " {'word_n2': 'bos',\n",
       "  'word_n1': 'rockwell',\n",
       "  'word': 'international',\n",
       "  'word_p1': 'corp.',\n",
       "  'word_p2': \"'s\",\n",
       "  'pos_n2': 'BOS',\n",
       "  'pos_n1': 'NNP',\n",
       "  'pos': 'NNP',\n",
       "  'pos_p1': 'NNP',\n",
       "  'pos_p2': 'POS'}]"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_test_dict, y_test = extract_features_static([test_corpus[0]], w_size, feature_names)\n",
    "X_test_dict[:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.107112,
     "end_time": "2020-12-11T20:33:19.725546",
     "exception": false,
     "start_time": "2020-12-11T20:33:19.618434",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "This $\\mathbf{X}\\_{\\textrm{dict}}$ is incomplete. For the prediction, we need to reinject dynamically the two previously predicted tags to have the full feature vector. Write this code here. \n",
    "\n",
    "This part is probably the most difficult of the lab. You may want to write it first for one sentence, and then for the test corpus. The prediction will take a longer time and you may want to include a progress bar with this snippet: \n",
    "```\n",
    "from tqdm import tqdm\n",
    "for test_sentence in tqdm(test_corpus):\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:33:19.944923Z",
     "iopub.status.busy": "2020-12-11T20:33:19.943907Z",
     "iopub.status.idle": "2020-12-11T20:33:19.950622Z",
     "shell.execute_reply": "2020-12-11T20:33:19.950105Z"
    },
    "papermill": {
     "duration": 0.118354,
     "end_time": "2020-12-11T20:33:19.950736",
     "exception": false,
     "start_time": "2020-12-11T20:33:19.832382",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "47377"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_test_predicted_dyn = []\n",
    "len(y_test_predicted)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:33:20.190239Z",
     "iopub.status.busy": "2020-12-11T20:33:20.184821Z",
     "iopub.status.idle": "2020-12-11T20:33:20.438110Z",
     "shell.execute_reply": "2020-12-11T20:33:20.437483Z"
    },
    "papermill": {
     "duration": 0.37836,
     "end_time": "2020-12-11T20:33:20.438221",
     "exception": false,
     "start_time": "2020-12-11T20:33:20.059861",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "X_test_dict, y_test = extract_features_static(test_corpus, w_size, feature_names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T20:33:20.670312Z",
     "iopub.status.busy": "2020-12-11T20:33:20.669486Z",
     "iopub.status.idle": "2020-12-11T21:05:15.435418Z",
     "shell.execute_reply": "2020-12-11T21:05:15.433972Z"
    },
    "papermill": {
     "duration": 1914.889174,
     "end_time": "2020-12-11T21:05:15.435582",
     "exception": false,
     "start_time": "2020-12-11T20:33:20.546408",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 2012/2012 [31:54<00:00,  1.05it/s]\n"
     ]
    }
   ],
   "source": [
    "from tqdm import tqdm\n",
    "\n",
    "\n",
    "def append_prev_preds(corpus, dict_feats, preds):\n",
    "    X_test_dict = dict_feats\n",
    "    s = 0\n",
    "    for test_sentence in tqdm(corpus):\n",
    "        \n",
    "        for i in range(len(test_sentence)):\n",
    "            for j in range(-2, 0):\n",
    "                if s + j > -1:\n",
    "                    X_test_dict[s][f'chunk_n{abs(j)}'] = list(preds)[s + j]\n",
    "                else:\n",
    "                    X_test_dict[s][f'chunk_n{abs(j)}'] = 'BOS'\n",
    "            s += 1\n",
    "            \n",
    "    return X_test_dict\n",
    "\n",
    "X_append = append_prev_preds(test_corpus, X_test_dict, y_test_predicted)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T21:05:17.020783Z",
     "iopub.status.busy": "2020-12-11T21:05:17.015651Z",
     "iopub.status.idle": "2020-12-11T21:05:18.103518Z",
     "shell.execute_reply": "2020-12-11T21:05:18.102870Z"
    },
    "papermill": {
     "duration": 1.864061,
     "end_time": "2020-12-11T21:05:18.103635",
     "exception": false,
     "start_time": "2020-12-11T21:05:16.239574",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "X_test = vec.transform(X_append)\n",
    "y_test_predicted_dyn = classifier.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T21:05:19.687221Z",
     "iopub.status.busy": "2020-12-11T21:05:19.686504Z",
     "iopub.status.idle": "2020-12-11T21:05:19.690354Z",
     "shell.execute_reply": "2020-12-11T21:05:19.689707Z"
    },
    "papermill": {
     "duration": 0.774319,
     "end_time": "2020-12-11T21:05:19.690465",
     "exception": false,
     "start_time": "2020-12-11T21:05:18.916146",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['B-NP', 'I-NP', 'I-NP'], dtype='<U7')"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_test_predicted_dyn[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T21:05:21.267439Z",
     "iopub.status.busy": "2020-12-11T21:05:21.263321Z",
     "iopub.status.idle": "2020-12-11T21:05:21.270937Z",
     "shell.execute_reply": "2020-12-11T21:05:21.270243Z"
    },
    "papermill": {
     "duration": 0.814079,
     "end_time": "2020-12-11T21:05:21.271051",
     "exception": false,
     "start_time": "2020-12-11T21:05:20.456972",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "inx = 0\n",
    "for sentence in test_corpus:\n",
    "    for word in sentence:\n",
    "        word['pchunk'] = y_test_predicted_dyn[inx]\n",
    "        inx += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T21:05:22.887173Z",
     "iopub.status.busy": "2020-12-11T21:05:22.807071Z",
     "iopub.status.idle": "2020-12-11T21:05:22.890611Z",
     "shell.execute_reply": "2020-12-11T21:05:22.889868Z"
    },
    "papermill": {
     "duration": 0.851468,
     "end_time": "2020-12-11T21:05:22.890736",
     "exception": false,
     "start_time": "2020-12-11T21:05:22.039268",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "save_results(test_corpus, keys, 'out')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.766092,
     "end_time": "2020-12-11T21:05:24.421360",
     "exception": false,
     "start_time": "2020-12-11T21:05:23.655268",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "#### Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T21:05:25.953653Z",
     "iopub.status.busy": "2020-12-11T21:05:25.953004Z",
     "iopub.status.idle": "2020-12-11T21:05:26.472557Z",
     "shell.execute_reply": "2020-12-11T21:05:26.471955Z"
    },
    "papermill": {
     "duration": 1.289878,
     "end_time": "2020-12-11T21:05:26.472677",
     "exception": false,
     "start_time": "2020-12-11T21:05:25.182799",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9222800091902165"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lines = open('out').read().splitlines()\n",
    "res = conlleval.evaluate(lines)\n",
    "improved_ml_score = res['overall']['chunks']['evals']['f1']\n",
    "improved_ml_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.812358,
     "end_time": "2020-12-11T21:05:28.175118",
     "exception": false,
     "start_time": "2020-12-11T21:05:27.362760",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Optional improvement"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.764862,
     "end_time": "2020-12-11T21:05:29.706587",
     "exception": false,
     "start_time": "2020-12-11T21:05:28.941725",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "If you know what beam search is, apply it using the probability output of logistic regression or the score if you use support vector machines.<br/>With the same classifier and a beam diameter of 5, your teacher obtained 92.87."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.768353,
     "end_time": "2020-12-11T21:05:31.248162",
     "exception": false,
     "start_time": "2020-12-11T21:05:30.479809",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Collecting the entities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.766114,
     "end_time": "2020-12-11T21:05:32.785975",
     "exception": false,
     "start_time": "2020-12-11T21:05:32.019861",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "You will now collect all the named entities from the training set, defined as NP chunks and starting with a `NNP` (proper noun) or a `NNPS` (proper noun, plural) tag. As an example, in the first sentence of `train_corpus`, you will extract `('September', )` and `('July', 'and', 'August')`. You will set all the tuples in a set that you will call `ne_set`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T21:05:34.337099Z",
     "iopub.status.busy": "2020-12-11T21:05:34.336248Z",
     "iopub.status.idle": "2020-12-11T21:05:34.340619Z",
     "shell.execute_reply": "2020-12-11T21:05:34.340110Z"
    },
    "papermill": {
     "duration": 0.780831,
     "end_time": "2020-12-11T21:05:34.340737",
     "exception": false,
     "start_time": "2020-12-11T21:05:33.559906",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'form': 'Confidence', 'pos': 'NN', 'chunk': 'B-NP'},\n",
       " {'form': 'in', 'pos': 'IN', 'chunk': 'B-PP'},\n",
       " {'form': 'the', 'pos': 'DT', 'chunk': 'B-NP'},\n",
       " {'form': 'pound', 'pos': 'NN', 'chunk': 'I-NP'},\n",
       " {'form': 'is', 'pos': 'VBZ', 'chunk': 'B-VP'},\n",
       " {'form': 'widely', 'pos': 'RB', 'chunk': 'I-VP'},\n",
       " {'form': 'expected', 'pos': 'VBN', 'chunk': 'I-VP'},\n",
       " {'form': 'to', 'pos': 'TO', 'chunk': 'I-VP'},\n",
       " {'form': 'take', 'pos': 'VB', 'chunk': 'I-VP'},\n",
       " {'form': 'another', 'pos': 'DT', 'chunk': 'B-NP'},\n",
       " {'form': 'sharp', 'pos': 'JJ', 'chunk': 'I-NP'},\n",
       " {'form': 'dive', 'pos': 'NN', 'chunk': 'I-NP'},\n",
       " {'form': 'if', 'pos': 'IN', 'chunk': 'B-SBAR'},\n",
       " {'form': 'trade', 'pos': 'NN', 'chunk': 'B-NP'},\n",
       " {'form': 'figures', 'pos': 'NNS', 'chunk': 'I-NP'},\n",
       " {'form': 'for', 'pos': 'IN', 'chunk': 'B-PP'},\n",
       " {'form': 'September', 'pos': 'NNP', 'chunk': 'B-NP'},\n",
       " {'form': ',', 'pos': ',', 'chunk': 'O'},\n",
       " {'form': 'due', 'pos': 'JJ', 'chunk': 'B-ADJP'},\n",
       " {'form': 'for', 'pos': 'IN', 'chunk': 'B-PP'},\n",
       " {'form': 'release', 'pos': 'NN', 'chunk': 'B-NP'},\n",
       " {'form': 'tomorrow', 'pos': 'NN', 'chunk': 'B-NP'},\n",
       " {'form': ',', 'pos': ',', 'chunk': 'O'},\n",
       " {'form': 'fail', 'pos': 'VB', 'chunk': 'B-VP'},\n",
       " {'form': 'to', 'pos': 'TO', 'chunk': 'I-VP'},\n",
       " {'form': 'show', 'pos': 'VB', 'chunk': 'I-VP'},\n",
       " {'form': 'a', 'pos': 'DT', 'chunk': 'B-NP'},\n",
       " {'form': 'substantial', 'pos': 'JJ', 'chunk': 'I-NP'},\n",
       " {'form': 'improvement', 'pos': 'NN', 'chunk': 'I-NP'},\n",
       " {'form': 'from', 'pos': 'IN', 'chunk': 'B-PP'},\n",
       " {'form': 'July', 'pos': 'NNP', 'chunk': 'B-NP'},\n",
       " {'form': 'and', 'pos': 'CC', 'chunk': 'I-NP'},\n",
       " {'form': 'August', 'pos': 'NNP', 'chunk': 'I-NP'},\n",
       " {'form': \"'s\", 'pos': 'POS', 'chunk': 'B-NP'},\n",
       " {'form': 'near-record', 'pos': 'JJ', 'chunk': 'I-NP'},\n",
       " {'form': 'deficits', 'pos': 'NNS', 'chunk': 'I-NP'},\n",
       " {'form': '.', 'pos': '.', 'chunk': 'O'}]"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_corpus[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.845516,
     "end_time": "2020-12-11T21:05:35.950183",
     "exception": false,
     "start_time": "2020-12-11T21:05:35.104667",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "You can write a two-pass procedure. For each sentence of the corpus:\n",
    "1. In the first pass, you will collect the start indices of the noun groups which are also proper nouns. For the first sentence, it will result in the list `[16, 30]`;\n",
    "2. In the second pass, you will collect the segments, starting at each index. For the first sentence, it will result in the tuples `('September',)`and `('July', 'and', 'August')`\n",
    "\n",
    "Should you have a better solution, please use it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T21:05:37.530417Z",
     "iopub.status.busy": "2020-12-11T21:05:37.529687Z",
     "iopub.status.idle": "2020-12-11T21:05:37.661725Z",
     "shell.execute_reply": "2020-12-11T21:05:37.662322Z"
    },
    "papermill": {
     "duration": 0.905343,
     "end_time": "2020-12-11T21:05:37.662494",
     "exception": false,
     "start_time": "2020-12-11T21:05:36.757151",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def get_entities(corpus):\n",
    "    entities = []\n",
    "    for sentence in corpus:\n",
    "        indices = [ind for ind, row in enumerate(sentence) if (row['pos'] == 'NNP' or  row['pos'] == 'NNPS') and row['chunk'] =='B-NP']\n",
    "        \n",
    "        if len(indices) > 0:\n",
    "            for index in indices:\n",
    "                segment = ()\n",
    "                #if sentence[index]['form'] not in segment:\n",
    "                segment = (sentence[index]['form'],)\n",
    "                \n",
    "                i = index + 1\n",
    "                while i < len(sentence) - 1:\n",
    "                    if sentence[i]['chunk'] == 'I-NP':\n",
    "                            segment += (sentence[i]['form'],)\n",
    "                            i += 1\n",
    "                    else:\n",
    "                        i = len(sentence)\n",
    "                \n",
    "                if len(segment) > 0:    \n",
    "                    entities.append(segment)\n",
    "        \n",
    "    return set(entities)\n",
    "\n",
    "ne_set = get_entities(train_corpus)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T21:05:39.268589Z",
     "iopub.status.busy": "2020-12-11T21:05:39.267941Z",
     "iopub.status.idle": "2020-12-11T21:05:39.272326Z",
     "shell.execute_reply": "2020-12-11T21:05:39.272837Z"
    },
    "papermill": {
     "duration": 0.840384,
     "end_time": "2020-12-11T21:05:39.273002",
     "exception": false,
     "start_time": "2020-12-11T21:05:38.432618",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4349"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(ne_set)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T21:05:40.825012Z",
     "iopub.status.busy": "2020-12-11T21:05:40.824318Z",
     "iopub.status.idle": "2020-12-11T21:05:40.833736Z",
     "shell.execute_reply": "2020-12-11T21:05:40.833072Z"
    },
    "papermill": {
     "duration": 0.790674,
     "end_time": "2020-12-11T21:05:40.833866",
     "exception": false,
     "start_time": "2020-12-11T21:05:40.043192",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Aug.', '31'),\n",
       " ('Erie', 'and', 'Ottawa', 'counties'),\n",
       " ('Seton', 'Hall', 'University'),\n",
       " ('Arthur', 'Fleischer', 'Jr.'),\n",
       " ('Co.',),\n",
       " ('Germany',),\n",
       " ('Mr.', 'Helmsley'),\n",
       " ('Mr.', 'Cockburn'),\n",
       " ('East', 'Germans'),\n",
       " ('Justice', 'Department', 'attorney', 'Grover', 'Hartt', 'III')]"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(ne_set)[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.794635,
     "end_time": "2020-12-11T21:05:42.411872",
     "exception": false,
     "start_time": "2020-12-11T21:05:41.617237",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Creating a small set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.770199,
     "end_time": "2020-12-11T21:05:43.964489",
     "exception": false,
     "start_time": "2020-12-11T21:05:43.194290",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "To run the subsequent experiments faster, you will limit the dataset to the entities starting with letter _K_. I choose this letter, because this created one of the smallest set. You will call the resulting set `ne_small_set`. Feel free to use the full set after you have completed this assigment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T21:05:45.608710Z",
     "iopub.status.busy": "2020-12-11T21:05:45.608042Z",
     "iopub.status.idle": "2020-12-11T21:05:45.621570Z",
     "shell.execute_reply": "2020-12-11T21:05:45.620996Z"
    },
    "papermill": {
     "duration": 0.871784,
     "end_time": "2020-12-11T21:05:45.621686",
     "exception": false,
     "start_time": "2020-12-11T21:05:44.749902",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Kevin', 'Logan'),\n",
       " ('Kansas', 'Power'),\n",
       " ('Kremlin', 'wrangling'),\n",
       " ('King', 'Broadcasting', 'Co.'),\n",
       " ('Ke', 'Zaishuo'),\n",
       " ('Ko', 'Shioya'),\n",
       " ('Kenneth', 'T.', 'Rosen'),\n",
       " ('Kathy', 'Stanwick'),\n",
       " ('Kuala', 'Lumpur'),\n",
       " ('Kentucky',),\n",
       " ('Kenneth', 'H.', 'Olsen'),\n",
       " ('Karen', 'Olshan'),\n",
       " ('Kary', 'Moss'),\n",
       " ('Kean', 'forces'),\n",
       " ('Kajima',),\n",
       " ('Kenneth', 'A.', 'Eldred'),\n",
       " ('Kidder', 'Peabody'),\n",
       " ('Kathie', 'Huff'),\n",
       " ('Katonah',),\n",
       " ('K-H', 'Corp.'),\n",
       " ('Kobe', 'Steel', 'Ltd.'),\n",
       " ('K', 'mart', 'Corp.', 'Chairman', 'Joseph', 'E.', 'Antonini'),\n",
       " ('Kobe', 'Steel'),\n",
       " ('Kemper',),\n",
       " ('Kacy', 'McClelland'),\n",
       " ('Kenneth', 'Abraham'),\n",
       " ('Keefe', ',', 'Bruyette', '&', 'Woods', 'Inc.'),\n",
       " ('Kathryn', 'McGrath'),\n",
       " ('Kent', 'Neal'),\n",
       " ('KIM',),\n",
       " ('Kitchen',),\n",
       " ('Kenton',),\n",
       " ('Keith', 'L.', 'Fogg'),\n",
       " ('Knoxville',),\n",
       " ('Kollmorgen',),\n",
       " ('Khost',),\n",
       " ('Kleinwort', 'Benson', 'North', 'America'),\n",
       " ('Kyocera',),\n",
       " ('Kurt', 'Hager'),\n",
       " ('Kathie', 'Roberts'),\n",
       " ('Kidder', ',', 'Peabody', '&', 'Co'),\n",
       " ('Kurds',),\n",
       " ('Kimberly', 'Ann', 'Smith'),\n",
       " ('Ky.',),\n",
       " ('Kirin', 'Brewery'),\n",
       " ('Keihin', 'Electric', 'Express', 'Railway', 'Co'),\n",
       " ('Kentucky', 'Fried', 'Chicken', 'stores'),\n",
       " ('Khmer', 'Rouge', 'camps'),\n",
       " ('Kate', 'Michelman'),\n",
       " ('Kroger', 'Co'),\n",
       " ('KLM', 'Royal', 'Dutch', 'Airlines'),\n",
       " ('Kringle', 'fares'),\n",
       " ('Kawasaki', 'Steel'),\n",
       " ('Kansas',),\n",
       " ('Keizaikai', 'Corp.'),\n",
       " ('Krenz',),\n",
       " ('Kim', 'Schwartz'),\n",
       " ('Keizaikai',),\n",
       " ('Kidder', ',', 'Peabody', '&', 'Co.'),\n",
       " ('Kursk', 'and', 'Smolensk'),\n",
       " ('Kathe', 'Dillmann'),\n",
       " ('Kleinwort', 'Benson', 'Government', 'Securities', 'Inc.'),\n",
       " ('KTXL',),\n",
       " ('Kumagai-Gumi',),\n",
       " ('Kasler', 'Corp.'),\n",
       " ('Kodak',),\n",
       " ('Ky',),\n",
       " ('Kinney', 'and', 'Foot', 'Locker', 'shoe', 'stores'),\n",
       " ('Kraft', 'General', 'Foods'),\n",
       " ('KPMG', 'Peat', 'Marwick'),\n",
       " ('Kwek', 'Hong', 'Png'),\n",
       " ('Kaitaia',),\n",
       " ('Kabul',),\n",
       " ('Keith', 'Mulrooney'),\n",
       " ('Kirin',),\n",
       " ('Kansas', 'and', 'Texas')]"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ne_small_set = [ent for ent in ne_set if ent[0].startswith('K')]\n",
    "ne_small_set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T21:05:47.178202Z",
     "iopub.status.busy": "2020-12-11T21:05:47.177184Z",
     "iopub.status.idle": "2020-12-11T21:05:47.203129Z",
     "shell.execute_reply": "2020-12-11T21:05:47.203594Z"
    },
    "papermill": {
     "duration": 0.807684,
     "end_time": "2020-12-11T21:05:47.203740",
     "exception": false,
     "start_time": "2020-12-11T21:05:46.396056",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "76\n",
      "76\n"
     ]
    }
   ],
   "source": [
    "hello = [('Kathie', 'Roberts'),\n",
    " ('Keith', 'L.', 'Fogg'),\n",
    " ('Kansas', 'and', 'Texas'),\n",
    " ('Kringle', 'fares'),\n",
    " ('Ky.',),\n",
    " ('Keith', 'Mulrooney'),\n",
    " ('Kirin', 'Brewery'),\n",
    " ('Kent', 'Neal'),\n",
    " ('Kenneth', 'T.', 'Rosen'),\n",
    " ('Kuala', 'Lumpur'),\n",
    " ('Keizaikai', 'Corp.'),\n",
    " ('Kidder', ',', 'Peabody', '&', 'Co.'),\n",
    " ('Kathryn', 'McGrath'),\n",
    " ('Kobe', 'Steel', 'Ltd.'),\n",
    " ('Kathie', 'Huff'),\n",
    " ('Kary', 'Moss'),\n",
    " ('Kathy', 'Stanwick'),\n",
    " ('Khost',),\n",
    " ('Ke', 'Zaishuo'),\n",
    " ('Kidder', ',', 'Peabody', '&', 'Co'),\n",
    " ('Kirin',),\n",
    " ('KTXL',),\n",
    " ('Kabul',),\n",
    " ('Kaitaia',),\n",
    " ('Kidder', 'Peabody'),\n",
    " ('Kevin', 'Logan'),\n",
    " ('King', 'Broadcasting', 'Co.'),\n",
    " ('Kinney', 'and', 'Foot', 'Locker', 'shoe', 'stores'),\n",
    " ('Kimberly', 'Ann', 'Smith'),\n",
    " ('Kwek', 'Hong', 'Png'),\n",
    " ('Keihin', 'Electric', 'Express', 'Railway', 'Co'),\n",
    " ('Kraft', 'General', 'Foods'),\n",
    " ('Kenneth', 'A.', 'Eldred'),\n",
    " ('K', 'mart', 'Corp.', 'Chairman', 'Joseph', 'E.', 'Antonini'),\n",
    " ('KLM', 'Royal', 'Dutch', 'Airlines'),\n",
    " ('Kacy', 'McClelland'),\n",
    " ('Kitchen',),\n",
    " ('Kathe', 'Dillmann'),\n",
    " ('Keizaikai',),\n",
    " ('Knoxville',),\n",
    " ('KPMG', 'Peat', 'Marwick'),\n",
    " ('Krenz',),\n",
    " ('Katonah',),\n",
    " ('K-H', 'Corp.'),\n",
    " ('Kemper',),\n",
    " ('Kremlin', 'wrangling'),\n",
    " ('Kentucky', 'Fried', 'Chicken', 'stores'),\n",
    " ('Kate', 'Michelman'),\n",
    " ('Kleinwort', 'Benson', 'North', 'America'),\n",
    " ('Kobe', 'Steel'),\n",
    " ('Kansas',),\n",
    " ('Kleinwort', 'Benson', 'Government', 'Securities', 'Inc.'),\n",
    " ('Kawasaki', 'Steel'),\n",
    " ('Kurds',),\n",
    " ('Kollmorgen',),\n",
    " ('Karen', 'Olshan'),\n",
    " ('Kroger', 'Co'),\n",
    " ('Kenton',),\n",
    " ('Kurt', 'Hager'),\n",
    " ('Kodak',),\n",
    " ('Kentucky',),\n",
    " ('Keefe', ',', 'Bruyette', '&', 'Woods', 'Inc.'),\n",
    " ('KIM',),\n",
    " ('Kursk', 'and', 'Smolensk'),\n",
    " ('Kenneth', 'H.', 'Olsen'),\n",
    " ('Kim', 'Schwartz'),\n",
    " ('Kean', 'forces'),\n",
    " ('Ky',),\n",
    " ('Kyocera',),\n",
    " ('Kumagai-Gumi',),\n",
    " ('Kenneth', 'Abraham'),\n",
    " ('Ko', 'Shioya'),\n",
    " ('Kansas', 'Power'),\n",
    " ('Khmer', 'Rouge', 'camps'),\n",
    " ('Kajima',),\n",
    " ('Kasler', 'Corp.')]\n",
    "print(len(sorted(list(set(sorted(ne_small_set)) & set(sorted(hello))))))\n",
    "print(len(hello))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T21:05:48.758426Z",
     "iopub.status.busy": "2020-12-11T21:05:48.757756Z",
     "iopub.status.idle": "2020-12-11T21:05:48.760933Z",
     "shell.execute_reply": "2020-12-11T21:05:48.761561Z"
    },
    "papermill": {
     "duration": 0.788002,
     "end_time": "2020-12-11T21:05:48.761705",
     "exception": false,
     "start_time": "2020-12-11T21:05:47.973703",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Kevin', 'Logan'),\n",
       " ('Kansas', 'Power'),\n",
       " ('Kremlin', 'wrangling'),\n",
       " ('King', 'Broadcasting', 'Co.'),\n",
       " ('Ke', 'Zaishuo'),\n",
       " ('Ko', 'Shioya'),\n",
       " ('Kenneth', 'T.', 'Rosen'),\n",
       " ('Kathy', 'Stanwick'),\n",
       " ('Kuala', 'Lumpur'),\n",
       " ('Kentucky',),\n",
       " ('Kenneth', 'H.', 'Olsen'),\n",
       " ('Karen', 'Olshan'),\n",
       " ('Kary', 'Moss'),\n",
       " ('Kean', 'forces'),\n",
       " ('Kajima',),\n",
       " ('Kenneth', 'A.', 'Eldred'),\n",
       " ('Kidder', 'Peabody'),\n",
       " ('Kathie', 'Huff'),\n",
       " ('Katonah',),\n",
       " ('K-H', 'Corp.'),\n",
       " ('Kobe', 'Steel', 'Ltd.'),\n",
       " ('K', 'mart', 'Corp.', 'Chairman', 'Joseph', 'E.', 'Antonini'),\n",
       " ('Kobe', 'Steel'),\n",
       " ('Kemper',),\n",
       " ('Kacy', 'McClelland'),\n",
       " ('Kenneth', 'Abraham'),\n",
       " ('Keefe', ',', 'Bruyette', '&', 'Woods', 'Inc.'),\n",
       " ('Kathryn', 'McGrath'),\n",
       " ('Kent', 'Neal'),\n",
       " ('KIM',),\n",
       " ('Kitchen',),\n",
       " ('Kenton',),\n",
       " ('Keith', 'L.', 'Fogg'),\n",
       " ('Knoxville',),\n",
       " ('Kollmorgen',),\n",
       " ('Khost',),\n",
       " ('Kleinwort', 'Benson', 'North', 'America'),\n",
       " ('Kyocera',),\n",
       " ('Kurt', 'Hager'),\n",
       " ('Kathie', 'Roberts'),\n",
       " ('Kidder', ',', 'Peabody', '&', 'Co'),\n",
       " ('Kurds',),\n",
       " ('Kimberly', 'Ann', 'Smith'),\n",
       " ('Ky.',),\n",
       " ('Kirin', 'Brewery'),\n",
       " ('Keihin', 'Electric', 'Express', 'Railway', 'Co'),\n",
       " ('Kentucky', 'Fried', 'Chicken', 'stores'),\n",
       " ('Khmer', 'Rouge', 'camps'),\n",
       " ('Kate', 'Michelman'),\n",
       " ('Kroger', 'Co'),\n",
       " ('KLM', 'Royal', 'Dutch', 'Airlines'),\n",
       " ('Kringle', 'fares'),\n",
       " ('Kawasaki', 'Steel'),\n",
       " ('Kansas',),\n",
       " ('Keizaikai', 'Corp.'),\n",
       " ('Krenz',),\n",
       " ('Kim', 'Schwartz'),\n",
       " ('Keizaikai',),\n",
       " ('Kidder', ',', 'Peabody', '&', 'Co.'),\n",
       " ('Kursk', 'and', 'Smolensk'),\n",
       " ('Kathe', 'Dillmann'),\n",
       " ('Kleinwort', 'Benson', 'Government', 'Securities', 'Inc.'),\n",
       " ('KTXL',),\n",
       " ('Kumagai-Gumi',),\n",
       " ('Kasler', 'Corp.'),\n",
       " ('Kodak',),\n",
       " ('Ky',),\n",
       " ('Kinney', 'and', 'Foot', 'Locker', 'shoe', 'stores'),\n",
       " ('Kraft', 'General', 'Foods'),\n",
       " ('KPMG', 'Peat', 'Marwick'),\n",
       " ('Kwek', 'Hong', 'Png'),\n",
       " ('Kaitaia',),\n",
       " ('Kabul',),\n",
       " ('Keith', 'Mulrooney'),\n",
       " ('Kirin',),\n",
       " ('Kansas', 'and', 'Texas')]"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ne_small_set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.771193,
     "end_time": "2020-12-11T21:05:50.358060",
     "exception": false,
     "start_time": "2020-12-11T21:05:49.586867",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Resolving the entities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.78217,
     "end_time": "2020-12-11T21:05:51.914420",
     "exception": false,
     "start_time": "2020-12-11T21:05:51.132250",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "You will now implement a simple method to find the named entities from the previous exercise in Wikipedia and Wikidata."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.770477,
     "end_time": "2020-12-11T21:05:53.457241",
     "exception": false,
     "start_time": "2020-12-11T21:05:52.686764",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "First, look at a few entities in your set and find a few entities that you think are in wikipedia, entities that will not be in wikipedia, and entities that you think are ambiguous: An entity may correspond to two or more things. You will describe your findings in the report."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.773443,
     "end_time": "2020-12-11T21:05:55.043921",
     "exception": false,
     "start_time": "2020-12-11T21:05:54.270478",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### A function to lookup entities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.769783,
     "end_time": "2020-12-11T21:05:56.589310",
     "exception": false,
     "start_time": "2020-12-11T21:05:55.819527",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Read the function below and try to understand what it means. You will describe it in your report."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T21:05:58.144093Z",
     "iopub.status.busy": "2020-12-11T21:05:58.143338Z",
     "iopub.status.idle": "2020-12-11T21:05:58.146954Z",
     "shell.execute_reply": "2020-12-11T21:05:58.146224Z"
    },
    "papermill": {
     "duration": 0.784013,
     "end_time": "2020-12-11T21:05:58.147070",
     "exception": false,
     "start_time": "2020-12-11T21:05:57.363057",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def wikipedia_lookup(ner, base_url='https://en.wikipedia.org/wiki/'):\n",
    "    try:\n",
    "        url_en = base_url + '_'.join(ner)\n",
    "        html_doc = requests.get(url_en).text\n",
    "        parse_tree = bs4.BeautifulSoup(html_doc, 'html.parser')\n",
    "        entity_id = parse_tree.find(\"a\", {\"accesskey\": \"g\"})['href']\n",
    "        head_id, entity_id = os.path.split(entity_id)\n",
    "        \n",
    "        return entity_id\n",
    "    except:\n",
    "        pass\n",
    "        # print('Not found in: ', base_url)\n",
    "    entity_id = 'UNK'\n",
    "    return entity_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.783009,
     "end_time": "2020-12-11T21:05:59.751065",
     "exception": false,
     "start_time": "2020-12-11T21:05:58.968056",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Write a function to run the lookup and keep the resolved entities (only the resolved entities). You will call it `ne_ids_en`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T21:06:01.346735Z",
     "iopub.status.busy": "2020-12-11T21:06:01.345814Z",
     "iopub.status.idle": "2020-12-11T21:06:52.467118Z",
     "shell.execute_reply": "2020-12-11T21:06:52.466499Z"
    },
    "papermill": {
     "duration": 51.904503,
     "end_time": "2020-12-11T21:06:52.467233",
     "exception": false,
     "start_time": "2020-12-11T21:06:00.562730",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 76/76 [00:51<00:00,  1.49it/s]\n"
     ]
    }
   ],
   "source": [
    "def resolve_lookup(base_url):\n",
    "    ids = []\n",
    "    for ent in tqdm(ne_small_set):\n",
    "        ent_id = wikipedia_lookup(ent, base_url)\n",
    "        if ent_id != \"UNK\":\n",
    "            t = (ent, ent_id)\n",
    "            ids.append(t)\n",
    "            \n",
    "    return ids\n",
    "\n",
    "ne_ids_en = resolve_lookup('https://en.wikipedia.org/wiki/')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T21:06:54.136031Z",
     "iopub.status.busy": "2020-12-11T21:06:54.135236Z",
     "iopub.status.idle": "2020-12-11T21:06:54.140970Z",
     "shell.execute_reply": "2020-12-11T21:06:54.140336Z"
    },
    "papermill": {
     "duration": 0.817639,
     "end_time": "2020-12-11T21:06:54.141088",
     "exception": false,
     "start_time": "2020-12-11T21:06:53.323449",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(('KIM',), 'Q224736'), (('KLM', 'Royal', 'Dutch', 'Airlines'), 'Q181912'), (('KPMG', 'Peat', 'Marwick'), 'Q493751'), (('KTXL',), 'Q6339344'), (('Kabul',), 'Q5838'), (('Kaitaia',), 'Q257417'), (('Kajima',), 'Q1081154'), (('Kansas',), 'Q1558'), (('Kate', 'Michelman'), 'Q785671'), (('Katonah',), 'Q2888777'), (('Kawasaki', 'Steel'), 'Q6379829'), (('Kemper',), 'Q126993'), (('Kenneth', 'Abraham'), 'Q59268486'), (('Kenneth', 'H.', 'Olsen'), 'Q454315'), (('Kenton',), 'Q358393'), (('Kentucky',), 'Q1603'), (('Khost',), 'Q386682'), (('Kidder', 'Peabody'), 'Q3196386'), (('Kirin',), 'Q297659'), (('Kirin', 'Brewery'), 'Q13403399'), (('Kitchen',), 'Q43164'), (('Knoxville',), 'Q185582'), (('Kobe', 'Steel'), 'Q1730802'), (('Kodak',), 'Q486269'), (('Kraft', 'General', 'Foods'), 'Q327751'), (('Krenz',), 'Q21512656'), (('Kuala', 'Lumpur'), 'Q1865'), (('Kurds',), 'Q12223'), (('Kurt', 'Hager'), 'Q95367'), (('Kwek', 'Hong', 'Png'), 'Q10559663'), (('Ky',), 'Q225951'), (('Ky.',), 'Q225951'), (('Kyocera',), 'Q745099')]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "33"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(sorted(ne_ids_en))\n",
    "len(ne_ids_en)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.839347,
     "end_time": "2020-12-11T21:06:55.785789",
     "exception": false,
     "start_time": "2020-12-11T21:06:54.946442",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Sometimes, entities need a confirmation. You will apply the resolution with the Swedish wikipedia."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T21:06:57.402042Z",
     "iopub.status.busy": "2020-12-11T21:06:57.400969Z",
     "iopub.status.idle": "2020-12-11T21:07:38.185079Z",
     "shell.execute_reply": "2020-12-11T21:07:38.184035Z"
    },
    "papermill": {
     "duration": 41.594672,
     "end_time": "2020-12-11T21:07:38.185207",
     "exception": false,
     "start_time": "2020-12-11T21:06:56.590535",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 76/76 [00:40<00:00,  1.86it/s]\n"
     ]
    }
   ],
   "source": [
    "ne_ids_sv = resolve_lookup('https://sv.wikipedia.org/wiki/')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T21:07:39.896682Z",
     "iopub.status.busy": "2020-12-11T21:07:39.895756Z",
     "iopub.status.idle": "2020-12-11T21:07:39.900405Z",
     "shell.execute_reply": "2020-12-11T21:07:39.899436Z"
    },
    "papermill": {
     "duration": 0.883448,
     "end_time": "2020-12-11T21:07:39.900611",
     "exception": false,
     "start_time": "2020-12-11T21:07:39.017163",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(('KIM',), 'Q224736'), (('KLM', 'Royal', 'Dutch', 'Airlines'), 'Q181912'), (('Kabul',), 'Q5838'), (('Kansas',), 'Q1558'), (('Kenton',), 'Q358393'), (('Kentucky',), 'Q1603'), (('Khost',), 'Q386682'), (('Kirin',), 'Q297659'), (('Kitchen',), 'Q952431'), (('Knoxville',), 'Q232749'), (('Kodak',), 'Q486269'), (('Kraft', 'General', 'Foods'), 'Q12857502'), (('Kroger', 'Co'), 'Q153417'), (('Kuala', 'Lumpur'), 'Q1865')]\n",
      "14\n"
     ]
    }
   ],
   "source": [
    "print(sorted(ne_ids_sv))\n",
    "print(len(ne_ids_sv))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.829196,
     "end_time": "2020-12-11T21:07:41.612930",
     "exception": false,
     "start_time": "2020-12-11T21:07:40.783734",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "You will compute the intersection of the two sets. You will assign it to a list that you will sort and that you will call: `confirmed_ne_en_sv`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T21:07:43.314836Z",
     "iopub.status.busy": "2020-12-11T21:07:43.313797Z",
     "iopub.status.idle": "2020-12-11T21:07:43.317277Z",
     "shell.execute_reply": "2020-12-11T21:07:43.316680Z"
    },
    "papermill": {
     "duration": 0.844252,
     "end_time": "2020-12-11T21:07:43.317395",
     "exception": false,
     "start_time": "2020-12-11T21:07:42.473143",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "confirmed_ne_en_sv = sorted([ent for ent in ne_ids_en if ent[0] in [sv_ent[0] for sv_ent in ne_ids_sv]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T21:07:45.006458Z",
     "iopub.status.busy": "2020-12-11T21:07:45.005497Z",
     "iopub.status.idle": "2020-12-11T21:07:45.010204Z",
     "shell.execute_reply": "2020-12-11T21:07:45.010656Z"
    },
    "papermill": {
     "duration": 0.848815,
     "end_time": "2020-12-11T21:07:45.010826",
     "exception": false,
     "start_time": "2020-12-11T21:07:44.162011",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(('KIM',), 'Q224736'),\n",
       " (('KLM', 'Royal', 'Dutch', 'Airlines'), 'Q181912'),\n",
       " (('Kabul',), 'Q5838'),\n",
       " (('Kansas',), 'Q1558'),\n",
       " (('Kenton',), 'Q358393'),\n",
       " (('Kentucky',), 'Q1603'),\n",
       " (('Khost',), 'Q386682'),\n",
       " (('Kirin',), 'Q297659'),\n",
       " (('Kitchen',), 'Q43164'),\n",
       " (('Knoxville',), 'Q185582'),\n",
       " (('Kodak',), 'Q486269'),\n",
       " (('Kraft', 'General', 'Foods'), 'Q327751'),\n",
       " (('Kuala', 'Lumpur'), 'Q1865')]"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "confirmed_ne_en_sv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.839388,
     "end_time": "2020-12-11T21:07:46.673627",
     "exception": false,
     "start_time": "2020-12-11T21:07:45.834239",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "The first items in your list should look like:\n",
    "```\n",
    "[(('KIM',), 'Q224736'),\n",
    " (('KLM', 'Royal', 'Dutch', 'Airlines'), 'Q181912'),\n",
    " ...\n",
    "]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.823022,
     "end_time": "2020-12-11T21:07:48.326543",
     "exception": false,
     "start_time": "2020-12-11T21:07:47.503521",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "The submission code will send your answer. It consists of the predicted languages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-11T21:07:49.999606Z",
     "iopub.status.busy": "2020-12-11T21:07:49.998886Z",
     "iopub.status.idle": "2020-12-11T21:07:50.004091Z",
     "shell.execute_reply": "2020-12-11T21:07:50.004682Z"
    },
    "papermill": {
     "duration": 0.8528,
     "end_time": "2020-12-11T21:07:50.004915",
     "exception": false,
     "start_time": "2020-12-11T21:07:49.152115",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'{\"baseline_score\": 0.770671072299583, \"improved_ml_score\": 0.9222800091902165, \"confirmed_ne_en_sv\": [[[\"KIM\"], \"Q224736\"], [[\"KLM\", \"Royal\", \"Dutch\", \"Airlines\"], \"Q181912\"], [[\"Kabul\"], \"Q5838\"], [[\"Kansas\"], \"Q1558\"], [[\"Kenton\"], \"Q358393\"], [[\"Kentucky\"], \"Q1603\"], [[\"Khost\"], \"Q386682\"], [[\"Kirin\"], \"Q297659\"], [[\"Kitchen\"], \"Q43164\"], [[\"Knoxville\"], \"Q185582\"], [[\"Kodak\"], \"Q486269\"], [[\"Kraft\", \"General\", \"Foods\"], \"Q327751\"], [[\"Kuala\", \"Lumpur\"], \"Q1865\"]]}'"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import json\n",
    "ANSWER = json.dumps({'baseline_score': baseline_score,\n",
    "                    'improved_ml_score': improved_ml_score,\n",
    "                    'confirmed_ne_en_sv': confirmed_ne_en_sv})\n",
    "ANSWER"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.873736,
     "end_time": "2020-12-11T21:07:51.756048",
     "exception": false,
     "start_time": "2020-12-11T21:07:50.882312",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Reading"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.842268,
     "end_time": "2020-12-11T21:07:53.428503",
     "exception": false,
     "start_time": "2020-12-11T21:07:52.586235",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "You will read the article, <a href=\"https://www.aclweb.org/anthology/C18-1139\"><i>Contextual String\n",
    "            Embeddings for Sequence Labeling</i></a> by Akbik et al. (2018)\n",
    "            and you will outline the main differences between their system and yours. A LSTM is a type of\n",
    "            recurrent neural network, while CRF is a sort of beam search. You will tell the performance\n",
    "            they reach on the corpus you used in this laboratory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "papermill": {
     "duration": 0.855234,
     "end_time": "2020-12-11T21:07:55.133473",
     "exception": false,
     "start_time": "2020-12-11T21:07:54.278239",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "papermill": {
   "duration": 2292.453086,
   "end_time": "2020-12-11T21:07:56.075234",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2020-12-11T20:29:43.622148",
   "version": "2.1.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}