Exercise Solution Analysis.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import glob\n",
    "import json\n",
    "import pandas as pd\n",
    "import datetime\n",
    "import numpy as np\n",
    "import scipy.stats as sp\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# gets every log from directory\n",
    "all_logs = glob.glob(\"../logs/*.json\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Solution \n",
    "\n",
    "Excercise 1: \n",
    "```python\n",
    "df.intent = [\"MedianDebt\",\"Region=New England|Southeast|Far West\"]\n",
    "df\n",
    "```\n",
    "---------------------------------------------\n",
    "\n",
    "Excercise 2.1: \n",
    "```python\n",
    "Vis([\"Season\",\"AthleteRegion\"],df)\n",
    "```\n",
    "---------------------------------------------\n",
    "Excercise 2.2: \n",
    "```python\n",
    "physical_characteristics = [\"Age\", \"Weight\",\"Height\"]\n",
    "VisList([\"Year\", physical_characteristics],df)\n",
    "```\n",
    "------------------\n",
    "Bonus:\n",
    "\n",
    "```python\n",
    "from lux.vis.VisList import VisList\n",
    "vcLst = []\n",
    "for attribute in ['Weight','Age','Height']: \n",
    "    vis = Vis([lux.Clause(\"Year\"), lux.Clause(attribute)])\n",
    "    vcLst.append(vis)\n",
    "vc = VisList(vcLst,df)\n",
    "vc\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# formats all logs properly \n",
    "# key = session ID\n",
    "# value = json w/ same format as what was originally held in nb metadata\n",
    "formatted_logs = {}\n",
    "\n",
    "# all_logs = [\"logs/85b319c4eaae7abe4a4ce5275bee6447_flvg4gvka.json\"]\n",
    "for log in all_logs:\n",
    "    f = open(log, 'r')\n",
    "    \n",
    "    # merges body sent from various HTTP Posts\n",
    "    entries = f.read().split('}{')\n",
    "    for i in range(len(entries)):\n",
    "        entries[i] = '{' + entries[i] + '}'\n",
    "    entries[0] = entries[0][1:]\n",
    "    entries[-1] = entries[-1][:-1]\n",
    "    formatted_log = json.loads(entries[0])\n",
    "    for i in range(1,len(entries)):\n",
    "        j = json.loads(entries[i])\n",
    "        history = j['history']\n",
    "        formatted_log['history'].extend(history)\n",
    "    \n",
    "    # get time user started ipynb\n",
    "    formatted_log['startTime'] = formatted_log['history'][0]['time']\n",
    "    \n",
    "    # separates user hash from file name (sessionID)\n",
    "    names = log.split('_')\n",
    "    formatted_log['userHash'] = names[0][5:]\n",
    "    \n",
    "    # excluding Jerry and Doris and potentially others\n",
    "    exclude_list = ['bca87887a1cc89312f7d073fd007ea68', '1a735d0ee6a6f9d7fdab573b50851da7']\n",
    "    if names[0][5:] not in exclude_list:\n",
    "        formatted_logs[names[1][:-5]] = formatted_log\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame(formatted_logs).transpose().sort_values('nbName')\n",
    "df.index = df.index.rename(\"sessionHash\")\n",
    "df = df.reset_index()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Analyzing Exercise #1 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save this CSV and open in Excel for qualitative coding\n",
    "exercise1_submissions = df[df[\"nbName\"]==\"1-Specify-Intent.ipynb\"]\n",
    "# exercise1_submissions[['sessionHash', 'userHash']].to_csv(\"exercise1_grading.csv\",index=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Modify index number HERE to step through and print through every single solution\n",
    "excercise1_submission = exercise1_submissions.iloc[9]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "incorrect = ['3ea0843857adef1f9fd1734eb9c0a562', '85b319c4eaae7abe4a4ce5275bee6447', '88933303083562c5a59c05d1aa78cfba', 'c2f6a6bec33711664fa1cfeaac9003cf']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "excercise1_submission = exercise1_submissions[exercise1_submissions[\"userHash\"]=='c2f6a6bec33711664fa1cfeaac9003cf'].iloc[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "exercise1_log = excercise1_submission[\"history\"]\n",
    "print (\"Verify sessionHash on recorded row:\", excercise1_submission[\"sessionHash\"])\n",
    "\n",
    "# Print out all possible attempts\n",
    "excercise1_str = \"# Write code to specify the intent here\"#\\n# Hint: Try printing out your dataframe to see if your intent is specified correctly\"\n",
    "\n",
    "relevant_event = []\n",
    "for event in exercise1_log: \n",
    "    if \"code\" in event:\n",
    "        if excercise1_str in event[\"code\"]:\n",
    "            relevant_event.append(event[\"code\"])\n",
    "            \n",
    "# lots of repetitive code, deduplicate event\n",
    "deduplicated_relevant_event = set(relevant_event)\n",
    "print (\"Number of unique number of attempts:\", len(deduplicated_relevant_event))\n",
    "for idx,event in enumerate(deduplicated_relevant_event):\n",
    "    print (f'\\n--Attempt {idx+1}--')\n",
    "    print (event)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "exercise1_log"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Qualitative Codes: \n",
    "`value_correct`: User correctly specified intent Clause involving values\n",
    "\n",
    "`attr_correct`: User correctly specified intent Clause involving attribute\n",
    "\n",
    "For errors, if any of the attempts before first success contain the error, we record this, even if the solution is correct (since this is a mistaken concept in the iteration process):\n",
    "\n",
    "`concept_1`: Concept Error #1 : Mistaken OR with AND\n",
    "```python\n",
    "df.intent = [\"Region=New England\", \"Region=Southeast\", \"Region=Far West\"]\n",
    "df\n",
    "```\n",
    "\n",
    "`concept_2`: Concept Error #2 : Only specified one value of interest --> user did not understand how to specify OR relation\n",
    "```python\n",
    "df.intent = [\"Region=New England\"]\n",
    "```\n",
    "\n",
    "`concept_3`: Missed/Unable to specify attribute MedianDebt alongside values\n",
    "\n",
    "e.g.) Confused `|` with list notation, unclear how to specify AND\n",
    "```python\n",
    "df.intent = ['MedianDebt'|'Region=New England|Southeast|Far West']\n",
    "```\n",
    "\n",
    "`minor_1`: Minor Error #1: Missed one of the values, but able to use `|` successfully\n",
    "```python\n",
    "df.intent = ['MedianDebt', 'Region=New England |Southeast']\n",
    "```\n",
    "\n",
    "`blank`: Participant did not attempt at solution\n",
    "\n",
    "`num_attempts`: Number of \"qualitatively different\" attempts\n",
    "\n",
    "`num_attempts_1st_success`: Number of \"qualitatively different\" attempts until first success or total attempts if failure."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Analyzing Exercise #2.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save this CSV and open in Excel for qualitative coding\n",
    "exercise_submissions = df[df[\"nbName\"]==\"2-Quick-Vis.ipynb\"]\n",
    "exercise_submissions[['sessionHash', 'userHash']].to_csv(\"exercise2-1_grading.csv\",index=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Modify index number HERE to step through and print through every single solution\n",
    "excercise_submission = exercise_submissions.iloc[21]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "exercise_log = excercise_submission[\"history\"]\n",
    "print (\"Verify sessionHash on recorded row:\", excercise_submission[\"sessionHash\"])\n",
    "\n",
    "# Print out all possible attempts\n",
    "excercise1_str = \"# Hint: Inspect the dataframe columns to identify what are the relevant attributes\"\n",
    "\n",
    "relevant_event = []\n",
    "for event in exercise_log: \n",
    "    if \"code\" in event:\n",
    "        #print (event[\"code\"]) # Check if subsequent cells had relevant attempts if empty\n",
    "        if excercise1_str in event[\"code\"]:\n",
    "            relevant_event.append(event[\"code\"])\n",
    "            \n",
    "# lots of repetitive code, deduplicate event\n",
    "deduplicated_relevant_event = set(relevant_event)\n",
    "print (\"Number of unique number of iterations:\", len(deduplicated_relevant_event))\n",
    "for idx,event in enumerate(deduplicated_relevant_event):\n",
    "    print (f'\\n--Attempt {idx+1}--')\n",
    "    print (event)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Qualitative Codes: \n",
    "\n",
    "For errors, if any of the attempts before first success contain the error, we record this, even if the solution is correct (since this is a mistaken concept in the iteration process):\n",
    "\n",
    "`concept_1`: Concept Error #1 : Mistaken OR with AND\n",
    "\n",
    "From `9bb81b2aceab8a4f4b8fd07301654b65_r4tje5gfc.json`:\n",
    "```python\n",
    "Vis([\"Season|AthleteRegion\"],df)\n",
    "```\n",
    "From `3ea0843857adef1f9fd1734eb9c0a562_6nmstubyl.json`: \n",
    "```python\n",
    "vis.intent = [lux.Clause(\"NOC\", \"Games\")]\n",
    "```\n",
    "\n",
    "`concept_2`: Concept Error #2: Confusing intent notattion with dataframe notation\n",
    "```python\n",
    "intent = [\"Season\"]\n",
    "vis = Vis(intent,df[\"AthleteRegion\"])\n",
    "vis\n",
    "```\n",
    "\n",
    "`concept_3`: Confusing attribute with values (plotting syntax)\n",
    "\n",
    "From `1vpb2o1j0`:\n",
    "```python \n",
    "x = [\"winter\"]\n",
    "y = [\"summer\"] \n",
    "intent = [\"AthleteRegion\",\"Season\"]\n",
    "vis = vis(x,y)\n",
    "vis\n",
    "```\n",
    "`blank`: Participant did not attempt at solution\n",
    "\n",
    "`num_iterations`: Number of \"qualitatively different\" attempts\n",
    "\n",
    "`num_attempts_1st_success`: Number of \"qualitatively different\" attempts until first success or total attempts if failure."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Analyzing Exercise #2.2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save this CSV and open in Excel for qualitative coding\n",
    "exercise_submissions = df[df[\"nbName\"]==\"2-Quick-Vis.ipynb\"]\n",
    "# exercise_submissions[['sessionHash', 'userHash']].to_csv(\"exercise2-2_grading.csv\",index=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Modify index number HERE to step through and print through every single solution\n",
    "excercise_submission = exercise_submissions.iloc[7]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "exercise_log = excercise_submission[\"history\"]\n",
    "print (\"Verify sessionHash on recorded row:\", excercise_submission[\"sessionHash\"])\n",
    "\n",
    "# Print out all possible attempts\n",
    "excercise1_str = \"# Write code for generating the desired VisList\"\n",
    "\n",
    "relevant_event = []\n",
    "for event in exercise_log: \n",
    "    if \"code\" in event:\n",
    "        #print (event[\"code\"]) # Check if subsequent cells had relevant attempts if empty\n",
    "        if excercise1_str in event[\"code\"]:\n",
    "            relevant_event.append(event[\"code\"])\n",
    "            \n",
    "# lots of repetitive code, deduplicate event\n",
    "deduplicated_relevant_event = set(relevant_event)\n",
    "print (\"Number of unique number of attempts:\", len(deduplicated_relevant_event))\n",
    "for idx,event in enumerate(deduplicated_relevant_event):\n",
    "    print (f'\\n--Attempt {idx+1}--')\n",
    "    print (event)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Bonus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "exercise_log = excercise_submission[\"history\"]\n",
    "print (\"Verify sessionHash on recorded row:\", excercise_submission[\"sessionHash\"])\n",
    "\n",
    "# Print out all possible attempts\n",
    "excercise1_str = \"# Now try rewriting this code for by creating individual Vis objects\"\n",
    "\n",
    "relevant_event = []\n",
    "for event in exercise_log: \n",
    "    if \"code\" in event:\n",
    "        #print (event[\"code\"]) # Check if subsequent cells had relevant attempts if empty\n",
    "        if excercise1_str in event[\"code\"]:\n",
    "            relevant_event.append(event[\"code\"])\n",
    "            \n",
    "# lots of repetitive code, deduplicate event\n",
    "deduplicated_relevant_event = set(relevant_event)\n",
    "print (\"Number of unique number of attempts:\", len(deduplicated_relevant_event))\n",
    "for idx,event in enumerate(deduplicated_relevant_event):\n",
    "    print (f'\\n--Attempt {idx+1}--')\n",
    "    print (event)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Qualitative Codes: \n",
    "\n",
    "For errors, if any of the attempts before first success contain the error, we record this, even if the solution is correct (since this is a mistaken concept in the iteration process):\n",
    "\n",
    "`concept_1`: Concept Error #1 : Mistaken OR with AND\n",
    "```python\n",
    "intent = [\"Age\",\"Weight\",\"Height\",\"Year\"]\n",
    "vis = VisList(intent,df)\n",
    "vis\n",
    "```\n",
    "\n",
    "`concept_2`: Concept Error #2 : Confused about use of wildcards\n",
    "```python\n",
    "physical = \"Weight\",\"?\", \"Sex\", \"?\", \"Age\", \"?\", \"Height\", \"?\", \"Weight\n",
    "vc = VisList([\"Weight\",\"?\", \"Sex\", \"?\", \"Age\", \"?\", \"Height\", \"?\", \"Weight\"],df)\n",
    "```\n",
    "For `igjafm0x4`:\n",
    "```python\n",
    "vc = VisList([\"Age\",\"?\"],[\"Weight\",\"?\"],[\"Height\",\"?\"],df)\n",
    "vc\n",
    "```\n",
    "`concept_3`: Concept Error #3: Unclear how to combine `Year` and other physical characteristics\n",
    "```python\n",
    "from lux.vis.VisList import VisList\n",
    "vc = VisList([\"Weight\", \"Age\", \"Height\"],df , intent=\"Year\")\n",
    "vc\n",
    "```\n",
    "\n",
    "`blank`: Participant did not attempt at solution\n",
    "\n",
    "`num_attempts`: Number of \"qualitatively different\" attempts\n",
    "\n",
    "`num_attempts_1st_success`: Number of \"qualitatively different\" attempts until first success or total attempts if failure."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Comparison across Users"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Excercise labelled as a,b,c respectively\n",
    "s1 = pd.read_csv(\"grading/exercise1_grading.csv\").iloc[:-1,1:]\n",
    "# Discard all empty submissions\n",
    "s1 = s1[s1[\"blank\"]!=1].drop(columns=\"blank\")\n",
    "s1 = s1.groupby(\"userHash\").sum().reset_index() # get unique userHash\n",
    "s1.columns = s1.columns+\"_a\"\n",
    "s1 = s1.rename(columns={\"userHash_a\":\"userHash\"})\n",
    "\n",
    "s2 = pd.read_csv(\"grading/exercise2-1_grading.csv\").iloc[:-1,1:]\n",
    "s2 = s2[s2[\"blank\"]!=1].drop(columns=\"blank\")\n",
    "s2 = s2.groupby(\"userHash\").sum().reset_index() # get unique userHash\n",
    "s2.columns = s2.columns+\"_b\"\n",
    "s2 = s2.rename(columns={\"userHash_b\":\"userHash\"})\n",
    "\n",
    "s3 = pd.read_csv(\"grading/exercise2-2_grading.csv\").iloc[:-1,1:]\n",
    "s3 = s3[s3[\"blank\"]!=1].drop(columns=\"blank\")\n",
    "s3 = s3.groupby(\"userHash\").sum().reset_index() # get unique userHash\n",
    "s3.columns = s3.columns+\"_c\"\n",
    "s3 = s3.rename(columns={\"userHash_c\":\"userHash\"})\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print (len(s1),len(s2),len(s3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "merged = pd.merge(s1,s2,on=\"userHash\")\n",
    "merged = pd.merge(merged,s3,on=\"userHash\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "groups = pd.read_csv(\"user_groups.csv\")\n",
    "merged = merged.merge(groups,on=\"userHash\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "merged[[\"userHash\",\"correct_a\",\"correct_b\",\"correct_c\"]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "merged.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "merged[[\"userHash\",\"correct_a\",\"correct_b\",\"correct_c\",'userGroup','tried_0', 'tried_1', 'tried_2', 'tried_3', 'tried_4']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "merged[merged[\"userHash\"]==\"3ea0843857adef1f9fd1734eb9c0a562\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Corrected this on exercise1_grading.csv\n",
    "\n",
    "- user `fc836b6c537cc69b6eb6ab9cb7cb6b05` got part a conceptually correct, except for minor error. For part c this user didn't get enough time to complete the task, only set physical characteristics but no major error.\n",
    "\n",
    "- user `e0157f975b2aecbe652c4706db61a112` pretty much got part a correct (set intent done it in another cell)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Overall, it looks like most users who are able to accomplish the tasks are able to complete all three task, with the exception of a few users."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "s1[s1[\"correct_a\"]==0].merge(groups,on=\"userHash\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list(s1[s1[\"correct_a\"]==0].merge(groups,on=\"userHash\")[\"userHash\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# cat logs/88933303083562c5a59c05d1aa78cfba_g55it8gd5.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- user `88933303083562c5a59c05d1aa78cfba` only tried one thing and then stopped following along all together"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s2[s2[\"correct_b\"]==0].merge(groups,on=\"userHash\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s3[s3[\"correct_c\"]==0].merge(groups,on=\"userHash\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Overall percentages, minus empty submission, aggregate across users (i.e. combine submissions from the same users)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "correct = len(s1[s1[\"correct_a\"]>=1])\n",
    "print (f\"{correct/len(s1)} success rate, {len(s1)-correct} incorrect\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "correct = len(s2[s2[\"correct_b\"]>=1])\n",
    "print (f\"{correct/len(s2)} success rate, {len(s2)-correct} incorrect\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "correct = len(s3[s3[\"correct_c\"]>=1])\n",
    "print (f\"{correct/len(s3)} success rate, {len(s3)-correct} incorrect\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def printcodeonly(exercise_log):\n",
    "    relevant_event = []\n",
    "    for event in exercise_log: \n",
    "        if \"code\" in event:\n",
    "            relevant_event.append(event[\"code\"])\n",
    "\n",
    "    # lots of repetitive code, deduplicate event\n",
    "    deduplicated_relevant_event = set(relevant_event)\n",
    "    print (\"Number of unique number of attempts:\", len(deduplicated_relevant_event))\n",
    "    for idx,event in enumerate(deduplicated_relevant_event):\n",
    "        print (f'\\n--Attempt {idx+1}--')\n",
    "        print (event)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "userHash=\"3ea0843857adef1f9fd1734eb9c0a562\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[df[\"userHash\"]==userHash]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# printcodeonly(df[df[\"userHash\"]==userHash][\"history\"].iloc[0])\n",
    "# printcodeonly(df[df[\"userHash\"]==userHash][\"history\"].iloc[1])\n",
    "# printcodeonly(df[df[\"userHash\"]==userHash][\"history\"].iloc[3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "userHash=\"85b319c4eaae7abe4a4ce5275bee6447\"\n",
    "df[df[\"userHash\"]==userHash]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# printcodeonly(df[df[\"userHash\"]==userHash][\"history\"].iloc[0])\n",
    "printcodeonly(df[df[\"userHash\"]==userHash][\"history\"].iloc[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- `3ea0843857adef1f9fd1734eb9c0a562`: showed conceptual errors, seemed proficient with Python.\n",
    "- `85b319c4eaae7abe4a4ce5275bee6447`: probably a Python novice, did things like \n",
    "```python\n",
    "df.intent = ['MedianDebt'|'Region=New England|Southeast|]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I was able to aggregate response across users who had multiple session and many of them did have at least one correct response (This could have been caused by tab closed or session restart). \n",
    "Accounting for this, the new scores are: \n",
    "\n",
    "    0.76%, 4 incorrect\n",
    "    0.93%, 1 incorrect\n",
    "    0.69%, 5 incorrect\n",
    "    \n",
    "There were only 2 users that consistently made mistakes across more than one exercises. One seemed proficient at Python but genuinely confused about the AND and OR syntax. The other user seemed like a Python novice, so the responses was a barely syntaxically correct as Python code. \n",
    "The VisList construct was difficult to interpret resulting in the lower score.\n",
    "Only 3 errors came from USF course participants, the remaining errors came from RISE Camp participants.\n",
    "\n",
    "For Exercise 1, main error is `concept_3`: Missed/Unable to specify attribute MedianDebt alongside values\n",
    "\n",
    "For Exercise 3, main error is `concept_1`:  Mistaken OR with AND\n",
    "```python\n",
    "intent = [\"Age\",\"Weight\",\"Height\",\"Year\"]\n",
    "vis = VisList(intent,df)\n",
    "vis\n",
    "```\n",
    "Although interestingly, 2 of the participants who didn't make concept_1 was able to express the VisList as a list of `Vis`s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}