tutorials/03-Calcium Imaging Computed Tables.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Working with automated computations: Computed tables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Welcome back! In this session, we are going to continue working with the pipeline for the mouse calcium imaging example. \n",
    "\n",
    "In this session, we will learn to:\n",
    "\n",
    "* compute various statistics for each neuron by defining a `Computed` table\n",
    "* define a `Lookup` table to store parameters for computation\n",
    "* define another `Computed` table to perform spike detection and store the detected spikes\n",
    "* automatically trigger computations for all missing entries with `populate`\n",
    "* define a `Part` table to save the results computed with the master `Computed` table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's import `datajoint` again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import datajoint as dj"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we are going to perform some computations, let's go ahead and import NumPy and Matplotlib."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly as before, to continue working with the tables we defined in the previous notebook, we can either redefine the classes for each table `Mouse`, `Session`, `Scan`, `AverageFrame` and populate them. Or, again for your convenience, we can import them from the `tutorial_pipeline.imaging` module. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tutorial_pipeline.imaging import schema, Mouse, Session, Scan, AverageFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Session()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Scan()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "AverageFrame()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `imaging.py` also fill each table by inserting manually and loading data from the external tiff files."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Computations in data pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now it's time to perform more complicated analyses. \n",
    "\n",
    "When you perform computations in the DataJoint data pipeline, you focus and design tables in terms of **what** is it that you are computing rather than the **how**. You should think in terms of the \"things\" that you are computing!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we would like to detect cells from the average image. The final product we would like is the binary **mask** for each individual cell, with the 1 in the region of interest (ROI) and 0 in other places."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So the new \"thing\" or entity here is `Roi`, where each entry corresponds the mask of one ROI. Let's start designing the table, paying special attention to the dependencies."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Detect cells from the average fluorescence image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's perform the segmentation to isolate ROIs. Let's start by taking a look at one average fluoresence image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "keys = AverageFrame.fetch('KEY')\n",
    "\n",
    "# pick one key\n",
    "key = keys[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ENTER YOUR CODE! - preview an AverageFrame for a particular key\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "avg_image = AverageFrame.fetch('average_frame')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "avg_image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's a bit subtle, but `fetch` returns a NumPy array of the attribute, even if the attribute contains a NumPy array. So here, we actually got a NumPy array of NumPy array. We can of course just index into it,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "avg_image[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "but if we knew that there was only one item, we can use `fetch1` instead to save some trouble"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "avg_image = (AverageFrame & key).fetch1('average_frame')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "avg_image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's plot to take a quick look:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.imshow(avg_image, cmap=plt.cm.gray)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are multiple ways to perform the segmentation. To keep it simple, we just detect the cells by setting up the threshold on the average image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "threshold = 50\n",
    "mask = avg_image > threshold\n",
    "plt.imshow(mask)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The outcome is different across different thresholds we set. Therefore, this threshold is a parameter we could potentially tweak."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "threshold = 60\n",
    "mask = avg_image > threshold\n",
    "plt.imshow(mask)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we could use scipy.ndimage to detect the blobs from this binary mask\n",
    "\n",
    "For the detailed tutorial, please refer to https://scipy-lectures.org/advanced/image_processing/index.html#segmentation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy import ndimage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The method `label` marks each blob with different number, `label_im` is the image with different blobs marked with different number and `nb_labels` is the number of blobs that are detected"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "label_im, nb_labels = ndimage.label(mask)\n",
    "print(nb_labels)\n",
    "plt.imshow(label_im) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.unique(label_im)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "0 marks the background and 1 to 10 mark the detected blobs. Some of these are too small to be cells. We could set a cutoff on size to filter out the small blobs. Here the cutoff is another parameter that could be tweaked."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "size_cutoff = 50\n",
    "sizes = np.array([np.sum(label_im==i) for i in np.unique(label_im)])\n",
    "\n",
    "small_size_filter = sizes < size_cutoff\n",
    "pixel_to_remove = small_size_filter[label_im]\n",
    "\n",
    "label_im[pixel_to_remove] = 0\n",
    "plt.imshow(label_im)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now there are only two blobs left."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next let's separate out each mask."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rois = []\n",
    "for i in np.unique(label_im)[1:]: # 0 is the background\n",
    "    rois.append(label_im == i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are two special notes about the analyses we performed above.\n",
    "1. The segmentation results are dependent on the parameters `threshold` and `size_cutoff`. Rather than fixing the value of the threshold, we might want to try different values and see what works well.\n",
    "2. For the segemtation analyses of each `AverageFrame`, the result is a number of ROIs instead of one. We need to perform the analyses on the level of each `AverageFrame`, but save the result on the granularity of each `Roi`.\n",
    "\n",
    "Next we would like to introduce two DataJoint table tiers to help handling these two needs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parameter `Lookup` table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We would like to perform the segmentation for a **combination** of `AverageFrame`s and different set of parameters of `threshold` and `size_cutoff` values. To do this while still taking advantage of the `make` and `populate` logic, you would want to define a table to house parameters for segmentation in a `Lookup` table!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's define `Param` table to hold different parameter configuration for our spike detection algorithm. We are going to define this table as a `Lookup` table, rather than a `Manual` table. By now, you know that `Lookup` must be yet another **table tier** in DataJoint. `Lookup` tables are depicted by gray boxes in the Diagram.\n",
    "\n",
    "This tier indicates that the table will contain information:\n",
    "* that will be referenced by other tables\n",
    "* that doesn't change much - usually contains a few pre-known entries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "@schema\n",
    "class SegmentationParam(dj.Lookup):\n",
    "    definition = \"\"\"\n",
    "    seg_param_id        : int      # unique id for cell segmentation parameter set\n",
    "    ---\n",
    "    threshold           : float\n",
    "    size_cutoff         : float\n",
    "    \"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dj.Diagram(schema)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So far the `SegmentationParam` is an extra table that did not relate with any of the existing, but the `Segmentation` will depend on this table."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Master `Computed` table `Segmentation` and `Part` table for `Roi`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As mentioned above, we performed the segmentation processing on each `AverageFrame`, but the product is the masks of each ROI. In this case, we could create a `Computed` table `Segmentation` containing the `make` method to drive the processing, while using a `Part` table to save results into `Roi`.\n",
    "\n",
    "`Computed` table and `Part` table are another two table tiers like `Manual`, `Lookup`, and `Imported` we introduced previously.  \n",
    "\n",
    "`Computed` table is very similar to the `Imported` table, which also supports the definition of `make` function and `populate`. The only difference is that the computation in an `Imported` table is dependent on external data, while the computation in a `Computed` table only depends on data inside the database.\n",
    "\n",
    "Contents in a `Part` table is dependent on its **master** table and the master table could be any type of table. This current example is a very typical usage of Part table. The master `Computed` serves as the driver for computation and the major results with a smaller granularity are saved in the `Part` table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "@schema\n",
    "class Segmentation(dj.Computed):\n",
    "    definition = \"\"\"\n",
    "    -> AverageFrame\n",
    "    -> SegmentationParam\n",
    "    ---\n",
    "    segmented_masks         : longblob   # overview of segmented masks\n",
    "    \"\"\"\n",
    "    class Roi(dj.Part):\n",
    "        definition = \"\"\"\n",
    "        -> master\n",
    "        roi_idx             : int        # index of an roi\n",
    "        ---\n",
    "        mask                : longblob   # mask of this roi\n",
    "        \"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that in the definition of `Roi`, apart from inheriting the primary from its master table `Segmentation`, the Roi has another primary key attribute `roi_idx`. The relationship between `Segmentation` and `Roi` is **one-to-many**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dj.Diagram(schema)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `Computed` table is labeled as a pink oval and the `Part` table is bare text. We see that `Segmentation` is a `Computed` table that depends on **both AverageFrame and SegmentationParam**. Finally, let's go ahead and implement the `make` method for the `Segmentation` table. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "@schema\n",
    "class Segmentation(dj.Computed):\n",
    "    definition = \"\"\"\n",
    "    -> AverageFrame\n",
    "    -> SegmentationParam\n",
    "    ---\n",
    "    segmented_masks         : longblob   # overview of segmented masks\n",
    "    \"\"\"\n",
    "    class Roi(dj.Part):\n",
    "        definition = \"\"\"\n",
    "        -> master\n",
    "        roi_idx             : int        # index of an roi\n",
    "        ---\n",
    "        mask                : longblob   # mask of this roi\n",
    "        \"\"\"\n",
    "    \n",
    "    def make(self, key): # key is one of the primary keys of the join product of AverageFrame and ParameterSet\n",
    "        \n",
    "        print('Populating for: ', key)\n",
    "        \n",
    "        # fetch average image from the previous table AverageFrame\n",
    "        avg_image = (AverageFrame & key).fetch1('average_frame')\n",
    "        \n",
    "        # fetch the parameters threshold and size_cutoff\n",
    "        threshold, size_cutoff = (SegmentationParam & key).fetch1(\n",
    "            'threshold', 'size_cutoff')\n",
    "        \n",
    "        # perform the thresholding and blob detection\n",
    "        mask = avg_image > threshold\n",
    "        label_im, nb_labels = ndimage.label(mask)\n",
    "        sizes = np.array([np.sum(label_im==i) for i in np.unique(label_im)])\n",
    "\n",
    "        small_size_filter = sizes < size_cutoff\n",
    "        pixel_to_remove = small_size_filter[label_im]\n",
    "\n",
    "        label_im[pixel_to_remove] = 0\n",
    "        \n",
    "        rois = []\n",
    "        for i in np.unique(label_im)[1:]: # 0 is the background\n",
    "            rois.append(\n",
    "                dict(**key,       # inherit primary key from master table\n",
    "                     roi_idx=i,    \n",
    "                     mask=label_im==i))\n",
    "        \n",
    "        # insert into the master table first\n",
    "        self.insert1(\n",
    "            dict(**key, segmented_masks=label_im)\n",
    "        )\n",
    "        print('Detected {} ROIs!\\n'.format(len(rois)))\n",
    "        # then insert into the part table\n",
    "        self.Roi.insert(rois)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The implementation of the segmentation is pretty much what we had above, except that we now fetch the value of `threshold` and `size_cutoff` from the `SegmentationParam` table.\n",
    "\n",
    "**Important note: always insert into the master table first and then insert the corresponding entries in the part table.** If a master table entry does not exist, its corresponding entries would not be valid."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at the `Segmentation` table, we see that it indeed inherits the primary key attributes from **both AverageFrame (`mouse_id`, `session_date`, `scan_idx`) and SegmentationParam (`seg_param_id`)**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Segmentation()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And for the part table `Segmentation.Roi`, there was an additional primary key attribute `roi_idx`:`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Segmentation.Roi()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Populating `Segmentation` table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now ready to populate! When we call `populate` on `Segmentation`, DataJoint will automatically call `make` on **every valid combination of the parent tables - AverageFrame and SegmentationParam**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ENTER YOUR CODE! - populate the Segmentation table\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hm... `populate` doesn't seem to be doing anything... What could be the cause?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at `SegmentationParam` reveals the issue:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SegmentationParam()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's right! We have not added a parameter set yet. Let's go ahead and add one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SegmentationParam.insert1((0, 50, 50))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SegmentationParam()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we should really be ready to perform the computation..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ENTER YOUR CODE! - populate the Segmentation table for real!\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Segmentation()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...and we now have spike detection running!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Trying out other parameter values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how different thresholds affect the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SegmentationParam.insert1((1, 60, 50))  # add another threshold and size cutoff"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SegmentationParam()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ENTER YOUR CODE! - populate the \"missing\" entry in Segmentation table\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Segmentation()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see that the results of segmentation under different parameter settings can live happily next to each other, without any confusion as to what is what."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deleting entries from \"upstream\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's say that we decided that we don't like the first threshold of `50`. While there is really nothing wrong keeping those results around, you might decide that you'd rather delete all computations performed with that threshold to keep your tables clean."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While you can restrict `Segmentation` table to the specific parameter id (i.e. `seg_param_id = 0`) and delete the entries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select 'No' when it pops up\n",
    "(Segmentation & 'seg_param_id = 0').delete()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can simply delete the unwanted parameter from the `SegmentationParam` table, and let DataJoint cascade the deletion:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SegmentationParam & 'seg_param_id = 0'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(SegmentationParam() & 'seg_param_id = 0').delete()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Segmentation()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualize ROIs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have all ROI masks saved in the table `Segmentation.Roi`, let's quickly look at an example by fetching the `mask` from the table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# show one example ROI\n",
    "masks = (Segmentation.Roi).fetch('mask')\n",
    "plt.imshow(masks[4])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fluorescence trace of each segmented ROI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we got masks of ROIs in the table `Segmetation.Roi` obtained with different parameter combinations. We would like to extract the fluorescence trace of each segmentation. \n",
    "\n",
    "The table design is similar to the `Segmentation` and `Roi`. The master table `Fluorescence` is the driver for the computation, with a secondary attribute `time` shared across traces of all ROIs. The part table `Trace` saves the extracted trace for each ROI over time (frame). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from skimage import io\n",
    "import os\n",
    "@schema\n",
    "class Fluorescence(dj.Imported): # imported table because it also rely on the external tiff file.\n",
    "    definition = \"\"\"\n",
    "    -> Segmentation\n",
    "    ---\n",
    "    time    : longblob    # time for each frame\n",
    "    \"\"\"\n",
    "    \n",
    "    class Trace(dj.Part):\n",
    "        definition = \"\"\"\n",
    "        -> master\n",
    "        -> Segmentation.Roi\n",
    "        ---\n",
    "        trace      :  longblob    # fluorescence trace of each ROI\n",
    "        \"\"\"\n",
    "          \n",
    "    # the master table is mainly to perform the computation, while the part table contains the result\n",
    "    def make(self, key):\n",
    "        \n",
    "        print('Populating: {}'.format(key))\n",
    "        # fetch data directory from table Session\n",
    "        data_path = (Session & key).fetch1('data_path')\n",
    "        \n",
    "        # fetch data file name from table Scan\n",
    "        file_name = (Scan & key).fetch1('file_name')\n",
    "        \n",
    "        # load the file\n",
    "        im = io.imread(os.path.join(data_path, file_name))\n",
    "        \n",
    "        # get dimensions of the image and reshape\n",
    "        n, w, h = np.shape(im)\n",
    "        im_reshaped = np.reshape(im, [n, w*h])\n",
    "        \n",
    "        # get frames per second to compute time\n",
    "        fps = (Scan & key).fetch1('fps')\n",
    "        \n",
    "        # insert into master table first\n",
    "        self.insert1(dict(**key, time=np.array(range(n))/fps))\n",
    "        \n",
    "        \n",
    "        # extract traces\n",
    "        roi_keys, masks = (Segmentation.Roi & key).fetch('KEY', 'mask')\n",
    "        \n",
    "        traces = []\n",
    "        for roi_key, mask in zip(roi_keys, masks):\n",
    "            \n",
    "            # reshape mask\n",
    "            mask_reshaped = np.reshape(mask, [w*h])\n",
    "            trace = np.mean(im_reshaped[:, mask_reshaped], axis=1)\n",
    "            \n",
    "            traces.append(dict(**roi_key, trace=trace))\n",
    "        \n",
    "        self.Trace.insert(traces)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ENTER YOUR CODE! - populate the Fluorescence table\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we could plot the traces of an example scan"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "key = dict(mouse_id=0, session_number=1, scan_idx=1, seg_param_id=1)\n",
    "\n",
    "# ENTER YOUR CODE! - fetch 'time' from the Fluorescence table using fetch1()\n",
    "time = \n",
    "\n",
    "# ENTER YOUR CODE! - fetch 'trace' from the Fluorescence.Trace table using fetch()\n",
    "traces = \n",
    "\n",
    "plt.plot(time, np.vstack(traces).T)\n",
    "plt.xlabel('Time (s)')\n",
    "plt.ylabel('Fluorescence')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations! You have successfully extended your pipeline with all types of tables. This pipeline is simple, but reflecting a typical preprocessin pipeline of calcium imaging. We have all the table tiers and major dependencies between DataJoint tables.\n",
    "\n",
    "**Table tiers**:  \n",
    "Manual table: green box  \n",
    "Lookup table: gray box  \n",
    "Imported table: blue oval  \n",
    "Computed table: red circle  \n",
    "Part table: plain text\n",
    "\n",
    "**Dependencies**:  \n",
    "One-to-one primary: thick solid line, share the exact same primary key  \n",
    "One-to-many primary: thin solid line, inherit the primary key from the parent table, but have additional field(s) as part of the primary key as well"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dj.Diagram(schema)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have covered most of the building elements of data pipeline design. Using these elements, we could design more sophiscated pipelines that facillitates your experimental recordings and data analyses. To see these concepts in the context of another methodology, see the next folder on [Electrophysiology](../02-Electrophysiology)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.16"
  },
  "vscode": {
   "interpreter": {
    "hash": "949777d72b0d2535278d3dc13498b2535136f6dfe0678499012e853ee9abcab1"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}