5-1_Working_with_files.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Lecture 5-1\n",
    "\n",
    "# Working with files\n",
    "\n",
    "# Week 5 Monday\n",
    "\n",
    "## Miles Chen, PhD\n",
    "\n",
    "Adapted from Chapter 14 of Think Python by Allen B Downey"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Persistence\n",
    "\n",
    "Most of the code we have run so far are transient. They will run during your session and after producing some output, their data disappears.\n",
    "\n",
    "Other programs need to be **persistent**, meaning that they need to store (at least some of) their data.\n",
    "\n",
    "This chapter will focus on reading and writing files to safe data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Opening a file\n",
    "\n",
    "You can open a file with `open()`\n",
    "\n",
    "You can write a to a file using `'w'` as a second parameter: `open('filename.txt', 'w')`\n",
    "\n",
    "If the file exists, running this command will replace the file. If the file does not exist, it will be created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "fout = open('output.txt', 'w')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The function `open()` returns a file object that allows you to write to the file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "32"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "line1 = \"This is the first line of text.\\n\"\n",
    "fout.write(line1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The method returns the number of characters that were written."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "If you call write again, it adds the new data to the end of the file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "38"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "line2 = \"This will be the second line of text.\\n\"\n",
    "fout.write(line2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "When you are done writing, you should close the file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "fout.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Closing the file frees up the system resources dedicated to tracking that file. If you forget to close the file and Python sees no object names bound to the file object, Python's garbage collector will automatically close the connection."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## with\n",
    "\n",
    "You can use the `with` keyword to have operations run with a particular file. When the operations finish, Python will automatically close the file. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# opening a file and writing lines to it:\n",
    "with open('output.txt', 'w') as file:\n",
    "    file.write(line1)\n",
    "    file.write(line2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is the first line of text.\n",
      "\n",
      "This will be the second line of text.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# opening the file and printing each line\n",
    "with open('output.txt', 'r') as file:\n",
    "    for i in file:\n",
    "        print(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## The Format Operator\n",
    "\n",
    "When using `write`, the argument has to be a string.\n",
    "\n",
    "The easiest way to convert non-strings to strings is with `str()`. If you don't convert the value to a string, you can run into problems."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": [
     "raises-exception"
    ]
   },
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "unsupported operand type(s) for +: 'float' and 'str'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m~\\AppData\\Local\\Temp/ipykernel_2080/3272603710.py\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[0mx\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;36m42.3\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 2\u001b[1;33m \u001b[0my\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mx\u001b[0m \u001b[1;33m+\u001b[0m \u001b[1;34m\" is my favorite number\"\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mTypeError\u001b[0m: unsupported operand type(s) for +: 'float' and 'str'"
     ]
    }
   ],
   "source": [
    "x = 42.3\n",
    "y = x + \" is my favorite number\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "42.3 is my favorite number\n"
     ]
    }
   ],
   "source": [
    "x = 42.3\n",
    "y = str(x) + \" is my favorite number\"\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Another option is to use the format method and curly brace `{}` replacement fields as placeholders for the variables to be inserted into the string.\n",
    "\n",
    "Python has a **format specification mini language** which is used to specify the format of the string.\n",
    "\n",
    "See: https://docs.python.org/3/library/string.html#format-string-syntax"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "12.34 is my favorite number\n"
     ]
    }
   ],
   "source": [
    "x = 12.34\n",
    "y = '{} is my favorite number'.format(x)\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "12.34 is my favorite number and 37.8 is my second favorite number\n"
     ]
    }
   ],
   "source": [
    "x = 12.34\n",
    "x2 = 37.8\n",
    "# you can use multiple replacement fields and python will insert value positionally\n",
    "y = '{} is my favorite number and {} is my second favorite number'.format(x, x2)\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "37.8 is my favorite number and 12.34 is my second favorite number.\n",
      "  Again, my favorite number is 37.8.\n"
     ]
    }
   ],
   "source": [
    "x = 12.34\n",
    "x2 = 37.8\n",
    "# you can set index values for replacement fields and python will insert values\n",
    "# based on the index of the tuple given to .format( )\n",
    "y = '''{1} is my favorite number and {0} is my second favorite number.\n",
    "  Again, my favorite number is {1}.'''.format(x, x2)\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.000123457 is my favorite number. I also like 1.2345679e-05.\n"
     ]
    }
   ],
   "source": [
    "# You can add :g in the replacement field.\n",
    "# This prints floating values in general format\n",
    "# Default is round to 6 significant figures.\n",
    "# You can specify my signficant figures with .8g for 8 significant figures.\n",
    "x  = 0.000123456789\n",
    "x2 = 0.0000123456789\n",
    "y  = '{0:g} is my favorite number. I also like {1:.8g}.'.format(x, x2)\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.000123 is my favorite number. I also like 0.00001235\n"
     ]
    }
   ],
   "source": [
    "# :f in the replacement field prints in fixed point format.\n",
    "# This prints floating values in fixed point format, default 6 fixed places after the decimal.\n",
    "# If you write .8f then it will provide 8 places after the decimal\n",
    "x = 0.000123456789\n",
    "x2 = 0.0000123456789\n",
    "y = '{0:f} is my favorite number. I also like {1:.8f}'.format(x, x2)\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "My age is 85, and 12.345 is my favorite number\n"
     ]
    }
   ],
   "source": [
    "# alternative method using + operator\n",
    "x = 12.345\n",
    "age = 85\n",
    "y = 'My age is ' + str(age) + ', and ' + str(x) + ' is my favorite number'\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "My name is Joe Bruin, I am 100 years old, and 83 is my favorite number\n"
     ]
    }
   ],
   "source": [
    "# placeholder indices are optional.\n",
    "y = 'My name is {:s}, I am {:g} years old, and {:g} is my favorite number'.format(\"Joe Bruin\", 100, 83)\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "A few more examples, directly from the Python documentation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'a, b, c'"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'{0}, {1}, {2}'.format('a', 'b', 'c')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'a, b, c'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'{}, {}, {}'.format('a', 'b', 'c')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'c, b, a'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'{2}, {1}, {0}'.format('a', 'b', 'c')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'c, b, a'"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'{2}, {1}, {0}'.format(*'abc')      # unpacking argument sequence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'abracadabra'"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'{0}{1}{0}'.format('abra', 'cad')   # arguments' indices can be repeated"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "Accessing arguments by name:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Coordinates: 37.24N, -115.81W'"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'Coordinates: {latitude}, {longitude}'.format(latitude='37.24N', longitude='-115.81W')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "coord = {'latitude': '37.24N', 'longitude': '-115.81W'} # create a dictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Coordinates: 37.24N, -115.81W'"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'Coordinates: {latitude}, {longitude}'.format(**coord) # kwarg unpacking"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "Aligning the text and specifying a width with `<` , `>` , and `^` and the number of spaces"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'left aligned                  '"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'{:<30}'.format('left aligned')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'                 right aligned'"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'{:>30}'.format('right aligned')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'           centered           '"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'{:^30}'.format('centered')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Filenames and Paths\n",
    "\n",
    "Python's `os` module allows you to work with your computer's file system in case you need to work with directories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'C:\\\\Users\\\\miles\\\\OneDrive\\\\Teaching\\\\21\\\\2022-wi-stats21'"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Use os.getcwd() to get the current working directory\n",
    "cwd = os.getcwd()\n",
    "cwd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "When you provide a filename to Python, it is generally a relative path because it relates to the current working directory.\n",
    "\n",
    "You can ask for absolute paths with `os.path.abspath()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'C:\\\\Users\\\\miles\\\\OneDrive\\\\Teaching\\\\21\\\\2022-wi-stats21\\\\output.txt'"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.path.abspath('output.txt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### more os functions\n",
    "\n",
    "You can read more about Python's OS functions at: https://docs.python.org/3/library/os.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check if a file exists\n",
    "os.path.exists('memo.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.path.exists('output.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check if something is a directory\n",
    "os.path.isdir('output.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.path.isdir('.git')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['.git',\n",
       " '.gitignore',\n",
       " '.ipynb_checkpoints',\n",
       " '1-1_Class_Welcome.pdf',\n",
       " '1-1_Class_Welcome.Rmd',\n",
       " '1-2_git_basics.pdf',\n",
       " '1-3_fetching_new_content_from_github.txt',\n",
       " '1-3_Jupyter_basics.ipynb',\n",
       " '1-3_Jupyter_basics.pdf',\n",
       " '1-3_Jupyter_basics.slides.html',\n",
       " '2-1_Basic_Data_Types.ipynb',\n",
       " '2-1_Basic_Data_Types.pdf',\n",
       " '2-1_Basic_Data_Types.slides.html',\n",
       " '2-2_Functions_in_Python.ipynb',\n",
       " '2-2_Functions_in_Python.pdf',\n",
       " '2-2_Functions_in_Python.slides.html',\n",
       " '2-3_Flow_control_Lists.ipynb',\n",
       " '2-3_Flow_control_Lists.pdf',\n",
       " '2-3_Flow_control_Lists.slides.html',\n",
       " '3-2_Lists_and_strings.ipynb',\n",
       " '3-2_Lists_and_strings.pdf',\n",
       " '3-2_Lists_and_strings.slides.html',\n",
       " '3-3_Dictionaries.ipynb',\n",
       " '3-3_Dictionaries.pdf',\n",
       " '3-3_Dictionaries.slides.html',\n",
       " '4-1_Tuples.ipynb',\n",
       " '4-1_Tuples.pdf',\n",
       " '4-1_Tuples.slides.html',\n",
       " '4-2_Numpy_Basics.ipynb',\n",
       " '4-2_Numpy_Basics.pdf',\n",
       " '4-2_Numpy_Basics.slides.html',\n",
       " '4-3_Numpy_Part2.ipynb',\n",
       " '4-3_Numpy_Part2.pdf',\n",
       " '4-3_Numpy_Part2.slides.html',\n",
       " '5-1_Working_with_files.ipynb',\n",
       " '5-1_Working_with_files.slides.html',\n",
       " 'captions.bak',\n",
       " 'captions.dat',\n",
       " 'captions.dir',\n",
       " 'demo.txt',\n",
       " 'demo2.txt',\n",
       " 'header_template.tex',\n",
       " 'objects.pkl',\n",
       " 'output.txt',\n",
       " 'script01.py',\n",
       " 'script02.py',\n",
       " 'ucla-std-blu-cmyk.jpg',\n",
       " 'Untitled.ipynb']"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# list the contents of the current working directory\n",
    "os.listdir(cwd)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Catching Exceptions\n",
    "\n",
    "When dealing with files, and also just in programming in general, a lot of things can go wrong and Python will throw an exception.\n",
    "\n",
    "You can often catch many of the exceptions with `try ... except` which works in a manner similar to `if ... else`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": [
     "raises-exception"
    ]
   },
   "outputs": [
    {
     "ename": "FileNotFoundError",
     "evalue": "[Errno 2] No such file or directory: 'bad_file'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mFileNotFoundError\u001b[0m                         Traceback (most recent call last)",
      "\u001b[1;32m~\\AppData\\Local\\Temp/ipykernel_2080/1899840290.py\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mfin\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mopen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'bad_file'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: 'bad_file'"
     ]
    }
   ],
   "source": [
    "fin = open('bad_file')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Something went wrong.\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    fin = open('bad_file')\n",
    "except:\n",
    "    print('Something went wrong.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Python first tries the commands in the `try:` block. If everything goes well, it skips the commands in the `except:` block. If it encounters an exception in the `try` block, it immediately exits the block and executes the code in the `except` block.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Python dbm databases\n",
    "\n",
    "A Python dbm database is like a Python dictionary but saved to a file on the harddrive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# You first have to import the dbm library\n",
    "import dbm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "`dbm.open()` will open the database connection\n",
    "\n",
    "The 'c' option will create it if it doesn't yet exist. It will not overwrite an existing file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "\n",
    "db = dbm.open('captions', 'c')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Add keys and values much like a dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "db['cleese.png'] = 'Photo of John Cleese'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The key here is the string 'cleese.png'. It is not a file, just the string of a filename (that may or may not exist). The value is another string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'Photo of John Cleese'"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# retrieve values like a dictionary\n",
    "db['cleese.png']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# I add another key-value pair.\n",
    "db['jones.png'] = 'Photo of Terry Jones'\n",
    "# if you need to, you can also delete an entry using del(db['jones.png'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b'cleese.png' b'Photo of John Cleese'\n",
      "b'jones.png' b'Photo of Terry Jones'\n",
      "b'list' b'\\x80\\x04\\x95\\x0b\\x00\\x00\\x00\\x00\\x00\\x00\\x00]\\x94(K\\x01K\\x02K\\x03e.'\n"
     ]
    }
   ],
   "source": [
    "# you can iterate through a database\n",
    "for key in db:\n",
    "    print(key, db[key])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# you should close when done\n",
    "db.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "captions.bak\n",
      "captions.dat\n",
      "captions.dir\n"
     ]
    }
   ],
   "source": [
    "# creating the captions dbm will add a few files to your working directory:\n",
    "for i in os.listdir(cwd):\n",
    "    if os.path.isfile(i) and 'captions' in i:\n",
    "        print(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Python Pickles\n",
    "\n",
    "A major limitation with dbm is that it can only store strings or byte objects.\n",
    "\n",
    "If you want to store other objects you created in Python, you'll want to `pickle` them. This converts the Python object into a string that can then be written to the dbm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "list"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t = [1 ,2, 3]\n",
    "type(t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "db = dbm.open('captions', 'c')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": [
     "raises-exception"
    ]
   },
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "values must be bytes or strings",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m~\\AppData\\Local\\Temp/ipykernel_2080/3564516374.py\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[1;31m# if I try to save the list t to the dbm, it causes an error.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 2\u001b[1;33m \u001b[0mdb\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'list'\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mt\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;32m~\\anaconda3\\lib\\dbm\\dumb.py\u001b[0m in \u001b[0;36m__setitem__\u001b[1;34m(self, key, val)\u001b[0m\n\u001b[0;32m    194\u001b[0m             \u001b[0mval\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mval\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mencode\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'utf-8'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    195\u001b[0m         \u001b[1;32melif\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[0misinstance\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mval\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m(\u001b[0m\u001b[0mbytes\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mbytearray\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 196\u001b[1;33m             \u001b[1;32mraise\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"values must be bytes or strings\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    197\u001b[0m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_verify_open\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    198\u001b[0m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_modified\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;32mTrue\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mTypeError\u001b[0m: values must be bytes or strings"
     ]
    }
   ],
   "source": [
    "# if I try to save the list t to the dbm, it causes an error.\n",
    "db['list'] = t"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import pickle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# pickle.dumps converts the python object to a string\n",
    "s = pickle.dumps(t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'\\x80\\x04\\x95\\x0b\\x00\\x00\\x00\\x00\\x00\\x00\\x00]\\x94(K\\x01K\\x02K\\x03e.'"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# the string representation of the list t\n",
    "# This is not meant to be human-readable. It is simply a format that can be stored in the dbm.\n",
    "s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "db['list'] = s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "db.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now that we have saved the \"pickled\" list to the dbm, let's see if we can retreive it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# open the dbm\n",
    "db = dbm.open('captions', 'c')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'\\x80\\x04\\x95\\x0b\\x00\\x00\\x00\\x00\\x00\\x00\\x00]\\x94(K\\x01K\\x02K\\x03e.'"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# retrieve the string associated with the key 'list'\n",
    "db['list']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1, 2, 3]"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# to conver the string back to a python object, you can use pickle.loads()\n",
    "pickle.loads(db['list'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "You can pickle objects directly to a pickle file.\n",
    "\n",
    "You can pickle multiple objects by grouping them together in a list or tuple."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "obj0 = ['file1', 'something else']\n",
    "obj1 = [1, 2, 3]\n",
    "obj2 = \"hello world!\"\n",
    "\n",
    "with open('objects.pkl', 'wb') as file:\n",
    "    pickle.dump([obj0, obj1, obj2], file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Loading the pickle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "with open('objects.pkl', \"rb\") as file:\n",
    "    x1, x2, x3 = pickle.load(file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['file1', 'something else']"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1, 2, 3]"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'hello world!'"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Read more about pickling:\n",
    "\n",
    "https://docs.python.org/3/library/pickle.html#what-can-be-pickled-and-unpickled"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}