Python_and_Unix_Commands_Weeks_1_4.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# This is a reference sheet for Python and Unix commands covered in weeks 1-4 of HUMBIO51 \n",
    "\n",
    " ## UNIX \n",
    "<ol>\n",
    " <li><a href=#13>Unix Commands </a></li>\n",
    " <li><a href=#14>Unix Operators </a></li>\n",
    "</ol>\n",
    "\n",
    " ## PYTHON\n",
    "<ol>\n",
    " <li><a href=#0>Miscellaneous essential concepts </a></li>\n",
    " <li><a href=#1>Moving around the file system </a></li>\n",
    " <li><a href=#2>Reading and writing files</a></li>\n",
    " <li><a href=#3>Strings</a></li>\n",
    " <li><a href=#4>Lists</a></li>\n",
    " <li><a href=#5>Tuples</a></li>\n",
    " <li><a href=#6>Dictionaries</a></li>\n",
    " <li><a href=#7>For loops</a></li>\n",
    " <li><a href=#8>If statements</a></li>\n",
    " <li><a href=#9>Functions</a></li>\n",
    " <li><a href=#10>Importing Packages</a></li>   \n",
    " <li><a href=#11>Working with Pandas Dataframes</a></li>\n",
    " <li><a href=#12>Making plots</a></li>     \n",
    "</ol>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# UNIX"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### UNIX COMMANDS <a name='13' />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pwd  #lists the name of the current directory \n",
    "!ls #displays the contents of the current directory \n",
    "!cat filename #prints the contents of a file (quotes around the filename are not required)\n",
    "!head -n10 # prints the first 10 lines of a file. Note: if your file is large, use the head or tail command rather than the cat\n",
    "           # command to examine its contents "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### UNIX OPERATORS <a name='14' />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "> write to a file\n",
    "| send the output of one command to the input of the next"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get help with a UNIX command you can type the command and use the --h or -help flags. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PYTHON "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Miscelaneous essential concepts <a name='0' />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is a comment. Comments are lines that begin with a hash(#) symbol. They are not executed and are used as notes for the programmer. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#The print function prints text to the screen. \n",
    "#Make sure you surround the text you wish to print in quotes \n",
    "print ('Hello World')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the help function when confused about a variable / function \n",
    "help(print)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Moving around the file system  <a name='1' />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#the import statement adds python libraries to your PYTHONPATH and allows the interpreter to find them \n",
    "#os library has not been imported: \n",
    "\n",
    "os.getcwd() #will fail because the os library has not been imported. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os  #to fix the error, first import the os library. \n",
    "os.getcwd() #We execute the getcwd function to print the current working directory "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.chdir(\"/home/jovyan/humbio51_instructor\") #changes your working directory. There are shortcuts we can use: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.chdir(\"/home/jovyan/humbio51_instructor/helpers\") #starting the directory name with \"/\" indicates an absolute path. \n",
    "os.chdir(\"..\") #moves one directory up (to the parent directory)\n",
    "os.chdir(\"helpers\") #omitting the parent directories indicates a relative path. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#lists files in the current directory, a single period (.)  stands for the current directory \n",
    "os.listdir('.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#create a new directory \n",
    "os.mkdir('mydir')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reading and writing files <a name='2' />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#open a file for writing \n",
    "f=open(\"myfile.txt\",'w') "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#write text to a file \n",
    "f.write(\"hello world! \\n goodbye world\\n\") #use \\n to indicate newline characters. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#open a file for reading \n",
    "f=open('myfile.txt','r')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#read an open file \n",
    "contents=f.read()\n",
    "print(contents)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#read the lines contained in a file, split by newlines \n",
    "contents=f.readlines()\n",
    "print(contents) #note! this can be done only once, contents will be empty the second time you iterate through. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#close a file\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#alternative to open a file without a separate close command\n",
    "with open('myfile.txt') as f: \n",
    "    data = f.read()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Working with strings <a name='3' />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#you can concatenate strings with the \"+\" sign \n",
    "a=\"hello\"\n",
    "b=\"world\"\n",
    "c=a+b\n",
    "print(c)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#We can print variables by changing them to strings. For example \n",
    "a=1 \n",
    "print(str(a))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#We can use the \"+\" sign to join strings for printing \n",
    "print(\"a is:\"+str(a))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#you can find and replace string characters \n",
    "c=c.replace(\"hello\",\"goodbye\") #make sure you assign the new string to a new variable, so your edits stick. \n",
    "print(c)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#string indexing \n",
    "sequence=\"ACGTACGT\"\n",
    "print(sequence[1]) #we can select a single character at a numeric index (0-based)\n",
    "print(sequence[1:3])#we can slice the string to select several contiguous characters\n",
    "print(sequence[::-1]) #string reversal\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Working with lists <a name='4' />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#both of the below are valid ways to create an empty list \n",
    "a=[] \n",
    "b=list()\n",
    "print(\"a\"+str(a))\n",
    "print(\"b\"+str(b))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#creating lists with some values in them \n",
    "a=[1,2,3,4]\n",
    "b=['a','b','c','d']\n",
    "print(a)\n",
    "print(b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#use the \"append\" command to add values to a list \n",
    "a.append(5)\n",
    "print(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#we can update list values with new values \n",
    "a[0]=7\n",
    "print(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#use the \"join\" command to join the elements in a list \n",
    "c='.'.join(b)\n",
    "print(c)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#use the \"split\" command to split a string into a list using a specified delimiter \n",
    "d=c.split('.')\n",
    "print(d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#use the \"len\" command to get the length of a list \n",
    "print(len(d))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tuples <a name='5' />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Unlike lists, tuples are immutable. \n",
    "#Tuples can be defined as follows: \n",
    "a=(1,2)\n",
    "print(a)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#you can index into a tuple, just like a list \n",
    "print(a[0])\n",
    "print(a[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#However, you cannot reassign values to a tuple \n",
    "a[0]=5 #should give an error "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dictionaries <a name='6' />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Dictionaries are key value pairs that are efficient to use \n",
    "#Dictionaries can be defined in two ways: \n",
    "\n",
    "my_dict={} \n",
    "my_dict=dict() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#We can populate a dictionary by assigning keys and values \n",
    "my_dict['a']=1\n",
    "my_dict['b']=2\n",
    "my_dict['c']=3\n",
    "print(my_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#We can query a dictionary by looking up specific keys \n",
    "my_dict['a']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#we can also print all keys or all values in a dictionary \n",
    "my_dict.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_dict.values()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## For loops <a name='7' />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#iterate by values \n",
    "sequence='AGCCCTCCA'\n",
    "for i in sequence:\n",
    "    print (i)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#iterate by index \n",
    "for i in [1,2,3,4]: \n",
    "    print(sequence[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#use the range command to get all integers in a given range \n",
    "list(range(0,3)) #first number is included, second number is excluded"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#if only one integer is provided to range, the first integer is assumed to be 0 \n",
    "list(range(3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#A common pattern is to combine range with len to iterate through all indices in a list \n",
    "for i in range(len(sequence)):\n",
    "    print(sequence[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## If statement logic <a name='8' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Equals: a == b\n",
    "\n",
    "Not Equals: a != b\n",
    "\n",
    "Less than: a < b\n",
    "\n",
    "Less than or equal to: a <= b\n",
    "\n",
    "Greater than: a > b\n",
    "\n",
    "Greater than or equal to: a >= b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#An \"if statement\" is written by using the if keyword.\n",
    "a=5\n",
    "b=10\n",
    "if a>b: \n",
    "    print(str(a) + \" is greater than \" + str(b))\n",
    "else: \n",
    "    print(str(b) + \" is greater than \"+ str(a))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#an elif can be used to perform multiple comparisons \n",
    "if a>b: \n",
    "    print(str(a) + \" is greater than \" + str(b))\n",
    "elif a==b: \n",
    "    print(str(a) + \" is equal to \" + str(b))\n",
    "else: \n",
    "    print(str(b) + \" is greater than \"+ str(a))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using functions <a name='9' />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Functions are re-usable chunks of code that can be called with a single command \n",
    "#functions are defined as follows \n",
    "\n",
    "#def my_function(arg1,arg2):\n",
    "#    #do something with arg1, arg2\n",
    "#    return output  #return an output value \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#For example: \n",
    "def multiply(a,b): \n",
    "    return a*b\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Functions can be executed after they are defined by passing arguments: \n",
    "product=multiply(2,3)\n",
    "print(product)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing Packages <a name='10' />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#import packages using the import cammand\n",
    "import Bio \n",
    "#the print command can be used to verify the version that is installed\n",
    "print(Bio.__version__) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Working with Pandas Dataframes <a name='11' />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#importing .tsv file into Python using pandas\n",
    "import pandas\n",
    "df = pd.read_table(\n",
    "     filepath_or_buffer='filename.tsv', \n",
    "     header=0,\n",
    "     index_col=0\n",
    ")\n",
    "\n",
    "\n",
    "#importing .csv file into Python using pandas\n",
    "df = pd.read_csv(\n",
    "     filepath_or_buffer='filename.csv', \n",
    "     header=0,\n",
    "     index_col=0\n",
    ")\n",
    "\n",
    "#specifying rows or columns in a dataframe\n",
    "x=df['columnname']\n",
    "x=df.loc[rowname]\n",
    "x=df.iloc[rownumber(s),columnnumber(s)]\n",
    "\n",
    "#sorting pandas dataframe\n",
    "df_sorted=pd.DataFrame.sort_values(df,by=\"columnname\",ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Make a scatter plot, barplot or histogram <a name='12'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plotting package  \n",
    "\n",
    "Plotnine: https://plotnine.readthedocs.io/en/stable/api.html <br> \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#load the necessary modules for plotnine\n",
    "from plotnine import * "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Make a Bar plot using plotnine\n",
    "x=df['columnname']\n",
    "qplot(x=x,\n",
    "      geom=\"bar\",\n",
    "      xlab=\"xlabel\",\n",
    "      ylab=\"ylabel\")  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Make a Bar plot using plotnine with the x and y-axes flipped\n",
    "x=df['columnname']\n",
    "qplot(x=x,\n",
    "      geom=\"bar\",\n",
    "      xlab=\"xlabel\",\n",
    "      ylab=\"ylabel\")+coord_flip()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Make a histogram using plotnine  \n",
    "x=df['columnname']\n",
    "qplot(x=x,\n",
    "      geom=\"histogram\",\n",
    "      xlab=\"xlabel\",\n",
    "      ylab=\"ylabel\")  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Make a Scatter Plot using plotnine\n",
    "x=df['columnname1']\n",
    "y=df['columnname2']\n",
    "qplot(x=x,\n",
    "      y=y,\n",
    "      geom=\"point\",\n",
    "      xlab=\"xlabel\",\n",
    "      ylab=\"ylabel\")  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Make a Scatter Plot using plotnine with colored points\n",
    "qplot(x=x,\n",
    "      y=y,\n",
    "      geom=\"point\",\n",
    "      xlab=\"PC1\",\n",
    "      ylab=\"PC2\",\n",
    "      color=list(metadata_subset['System']))+scale_color_discrete(name=\"System\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Make a Scatter Plot using plotnine labeling points\n",
    "x=df['columnname1']\n",
    "y=df['columnname2']\n",
    "label=df['columnname3']\n",
    "print(qplot(x=x,\n",
    "      y=y,\n",
    "      label=label,\n",
    "      geom=[\"point\",\"text\"],\n",
    "      xlab=\"xlabel\",\n",
    "      ylab=\"ylabel\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Make a Scatter Plot using plotnine labeling a single point\n",
    "x_subset=df.iloc[i,j]\n",
    "y_subset=df.iloc[i,k]\n",
    "\n",
    "qplot(x=x,\n",
    "      y=y,\n",
    "      xlab=xlab,\n",
    "      ylab=ylab)\n",
    "      + annotate('text',x=x_subset, y=y_subset, label=label_subset, size=7, color='blue', va='bottom', ha='left', angle=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Make a Scatter Plot using plotnine labeling a single point and setting x- and y- axis limits\n",
    "\n",
    "x_subset=df.iloc[i,j]\n",
    "y_subset=df.iloc[i,k]\n",
    "\n",
    "print(qplot(x=x,\n",
    "      y=y,\n",
    "      xlab=\"xlabel\",\n",
    "      ylab=\"ylabel\") + annotate('text',x=x_subset, y=y_subset,label=label_subset, size=7, color='blue', va='bottom', ha='left') \n",
    "      + xlim(0,10)\n",
    "      + ylim(0,70))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}