scientific_libraries/Scientific_Libraries_Dask.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e26ab074-25ed-45ad-9331-3c939e1345a4",
   "metadata": {},
   "source": [
    "## Basic Dask\n",
    "\n",
    "What is an efficient way to process very large data? What if we do not have enough RAM to perform the calculations? Dask to the rescue! Dask allows for fast pre-compiled operations. It's a little more code and overhead to run, but for large computations will be worth the effort. It also allows reading and processing data larger than the available RAM memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "37aa82a1-072b-412b-9dd6-d1204834595a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import dask.array as da  # Convention is to import as da"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39b73745-0932-4cba-8e62-96c53bc79b6b",
   "metadata": {},
   "source": [
    "Create a Dask data array with dask module, similar to Numpy. We define chunks to help manage multiprocessing and allow for a smaller RAM memory footprint. We could process data larger than the RAM memory we have available to us. Jupyter and Dask play well together and Jupyter will print a nice graphical output of the Dask array size/shape/type. Notice when we print it does not print the values, only information about the Dask array. That is because the values have not been created yet, only the steps to perform the operation are created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "237dd05c-b547-43fe-a4d9-aeb59ccda465",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "    <tr>\n",
       "        <td>\n",
       "            <table style=\"border-collapse: collapse;\">\n",
       "                <thead>\n",
       "                    <tr>\n",
       "                        <td> </td>\n",
       "                        <th> Array </th>\n",
       "                        <th> Chunk </th>\n",
       "                    </tr>\n",
       "                </thead>\n",
       "                <tbody>\n",
       "                    \n",
       "                    <tr>\n",
       "                        <th> Bytes </th>\n",
       "                        <td> 762.94 MiB </td>\n",
       "                        <td> 7.63 MiB </td>\n",
       "                    </tr>\n",
       "                    \n",
       "                    <tr>\n",
       "                        <th> Shape </th>\n",
       "                        <td> (10000, 10000) </td>\n",
       "                        <td> (1000, 1000) </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <th> Dask graph </th>\n",
       "                        <td colspan=\"2\"> 100 chunks in 1 graph layer </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <th> Data type </th>\n",
       "                        <td colspan=\"2\"> float64 numpy.ndarray </td>\n",
       "                    </tr>\n",
       "                </tbody>\n",
       "            </table>\n",
       "        </td>\n",
       "        <td>\n",
       "        <svg width=\"170\" height=\"170\" style=\"stroke:rgb(0,0,0);stroke-width:1\" >\n",
       "\n",
       "  <!-- Horizontal lines -->\n",
       "  <line x1=\"0\" y1=\"0\" x2=\"120\" y2=\"0\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"0\" y1=\"12\" x2=\"120\" y2=\"12\" />\n",
       "  <line x1=\"0\" y1=\"24\" x2=\"120\" y2=\"24\" />\n",
       "  <line x1=\"0\" y1=\"36\" x2=\"120\" y2=\"36\" />\n",
       "  <line x1=\"0\" y1=\"48\" x2=\"120\" y2=\"48\" />\n",
       "  <line x1=\"0\" y1=\"60\" x2=\"120\" y2=\"60\" />\n",
       "  <line x1=\"0\" y1=\"72\" x2=\"120\" y2=\"72\" />\n",
       "  <line x1=\"0\" y1=\"84\" x2=\"120\" y2=\"84\" />\n",
       "  <line x1=\"0\" y1=\"96\" x2=\"120\" y2=\"96\" />\n",
       "  <line x1=\"0\" y1=\"108\" x2=\"120\" y2=\"108\" />\n",
       "  <line x1=\"0\" y1=\"120\" x2=\"120\" y2=\"120\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Vertical lines -->\n",
       "  <line x1=\"0\" y1=\"0\" x2=\"0\" y2=\"120\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"12\" y1=\"0\" x2=\"12\" y2=\"120\" />\n",
       "  <line x1=\"24\" y1=\"0\" x2=\"24\" y2=\"120\" />\n",
       "  <line x1=\"36\" y1=\"0\" x2=\"36\" y2=\"120\" />\n",
       "  <line x1=\"48\" y1=\"0\" x2=\"48\" y2=\"120\" />\n",
       "  <line x1=\"60\" y1=\"0\" x2=\"60\" y2=\"120\" />\n",
       "  <line x1=\"72\" y1=\"0\" x2=\"72\" y2=\"120\" />\n",
       "  <line x1=\"84\" y1=\"0\" x2=\"84\" y2=\"120\" />\n",
       "  <line x1=\"96\" y1=\"0\" x2=\"96\" y2=\"120\" />\n",
       "  <line x1=\"108\" y1=\"0\" x2=\"108\" y2=\"120\" />\n",
       "  <line x1=\"120\" y1=\"0\" x2=\"120\" y2=\"120\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Colored Rectangle -->\n",
       "  <polygon points=\"0.0,0.0 120.0,0.0 120.0,120.0 0.0,120.0\" style=\"fill:#ECB172A0;stroke-width:0\"/>\n",
       "\n",
       "  <!-- Text -->\n",
       "  <text x=\"60.000000\" y=\"140.000000\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" >10000</text>\n",
       "  <text x=\"140.000000\" y=\"60.000000\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" transform=\"rotate(-90,140.000000,60.000000)\">10000</text>\n",
       "</svg>\n",
       "        </td>\n",
       "    </tr>\n",
       "</table>"
      ],
      "text/plain": [
       "dask.array<random_sample, shape=(10000, 10000), dtype=float64, chunksize=(1000, 1000), chunktype=numpy.ndarray>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = da.random.random((10000, 10000), chunks=(1000, 1000))\n",
    "x"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b682b28f-4ee5-48a8-b5e2-d086eac06fa7",
   "metadata": {},
   "source": [
    "Perform math similar to Numpy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "2d97377c-780b-425a-8877-c1324defa39a",
   "metadata": {},
   "outputs": [],
   "source": [
    "y = x + x.T  # .T means transpose the array"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1831b0c-b021-4e0e-bbcb-11a0e446fd25",
   "metadata": {},
   "source": [
    "Here we calculate the mean along an axis.\n",
    "\n",
    "But notice when we print z it does not show any values, just information about the dask object. At this time no computation has occured, just the information about what we want to do."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "479e6265-a9a5-4f24-8f89-3891db2a1b53",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "    <tr>\n",
       "        <td>\n",
       "            <table style=\"border-collapse: collapse;\">\n",
       "                <thead>\n",
       "                    <tr>\n",
       "                        <td> </td>\n",
       "                        <th> Array </th>\n",
       "                        <th> Chunk </th>\n",
       "                    </tr>\n",
       "                </thead>\n",
       "                <tbody>\n",
       "                    \n",
       "                    <tr>\n",
       "                        <th> Bytes </th>\n",
       "                        <td> 39.06 kiB </td>\n",
       "                        <td> 3.91 kiB </td>\n",
       "                    </tr>\n",
       "                    \n",
       "                    <tr>\n",
       "                        <th> Shape </th>\n",
       "                        <td> (5000,) </td>\n",
       "                        <td> (500,) </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <th> Dask graph </th>\n",
       "                        <td colspan=\"2\"> 10 chunks in 7 graph layers </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <th> Data type </th>\n",
       "                        <td colspan=\"2\"> float64 numpy.ndarray </td>\n",
       "                    </tr>\n",
       "                </tbody>\n",
       "            </table>\n",
       "        </td>\n",
       "        <td>\n",
       "        <svg width=\"170\" height=\"75\" style=\"stroke:rgb(0,0,0);stroke-width:1\" >\n",
       "\n",
       "  <!-- Horizontal lines -->\n",
       "  <line x1=\"0\" y1=\"0\" x2=\"120\" y2=\"0\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"0\" y1=\"25\" x2=\"120\" y2=\"25\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Vertical lines -->\n",
       "  <line x1=\"0\" y1=\"0\" x2=\"0\" y2=\"25\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"12\" y1=\"0\" x2=\"12\" y2=\"25\" />\n",
       "  <line x1=\"24\" y1=\"0\" x2=\"24\" y2=\"25\" />\n",
       "  <line x1=\"36\" y1=\"0\" x2=\"36\" y2=\"25\" />\n",
       "  <line x1=\"48\" y1=\"0\" x2=\"48\" y2=\"25\" />\n",
       "  <line x1=\"60\" y1=\"0\" x2=\"60\" y2=\"25\" />\n",
       "  <line x1=\"72\" y1=\"0\" x2=\"72\" y2=\"25\" />\n",
       "  <line x1=\"84\" y1=\"0\" x2=\"84\" y2=\"25\" />\n",
       "  <line x1=\"96\" y1=\"0\" x2=\"96\" y2=\"25\" />\n",
       "  <line x1=\"108\" y1=\"0\" x2=\"108\" y2=\"25\" />\n",
       "  <line x1=\"120\" y1=\"0\" x2=\"120\" y2=\"25\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Colored Rectangle -->\n",
       "  <polygon points=\"0.0,0.0 120.0,0.0 120.0,25.412616514582485 0.0,25.412616514582485\" style=\"fill:#ECB172A0;stroke-width:0\"/>\n",
       "\n",
       "  <!-- Text -->\n",
       "  <text x=\"60.000000\" y=\"45.412617\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" >5000</text>\n",
       "  <text x=\"140.000000\" y=\"12.706308\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" transform=\"rotate(0,140.000000,12.706308)\">1</text>\n",
       "</svg>\n",
       "        </td>\n",
       "    </tr>\n",
       "</table>"
      ],
      "text/plain": [
       "dask.array<mean_agg-aggregate, shape=(5000,), dtype=float64, chunksize=(500,), chunktype=numpy.ndarray>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z = y[::2, 5000:].mean(axis=1)\n",
    "z"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a260d6f-2bb1-404e-877f-a5fadca5df38",
   "metadata": {},
   "source": [
    "We will need to compute the values and convert to Numpy before printing the values to the screen. .compute() on the object will return a Numpy array of the values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "7ecb755f-7132-4bee-9560-128eff43df6d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1.00450745, 0.99041878, 0.99831558, ..., 0.99420789, 1.00546017,\n",
       "       0.99417984])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z = z.compute()\n",
    "z"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40c34d7e-4def-4194-b411-fdec6f990638",
   "metadata": {},
   "source": [
    "Once we call .compute() Jupyter will treat z as a normal Numpy array. We could have also delayed all the processing and rolled the greater than and .any() computation into Dask for faster overall computation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "fe5b25d8-74b2-4b6c-965f-e61673f80544",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = z > 0  # Return array of boolean values where value is greater than zero\n",
    "b = a.any()  # Are any values set to True?\n",
    "b"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8db3f297-7edd-4989-a983-7e96648f2d74",
   "metadata": {},
   "source": [
    "Can also perform all computations delayed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "df6fb719-6f9c-4bf0-b42b-21dda4e2b377",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b: dask.array<any-aggregate, shape=(), dtype=bool, chunksize=(), chunktype=numpy.ndarray>\n",
      "b: True\n"
     ]
    }
   ],
   "source": [
    "x = da.random.random((10000, 10000), chunks=(1000, 1000))\n",
    "y = x + x.T  # .T means transpose the array\n",
    "z = y[::2, 5000:].mean(axis=1)\n",
    "a = z > 0  # Return array of boolean values where value greater than zero\n",
    "b = a.any()  # Are any values set to True\n",
    "print('b:', b)\n",
    "b = b.compute()\n",
    "print('b:', b)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "263a2f61-d33b-4e64-8055-e911c1cfedac",
   "metadata": {},
   "source": [
    "Most of the Numpy operations are available in Dask computations. Notice that the nanmean() method is not actually run yet. The printed variable \"c\" is a placeholder for the operation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "bbc5df8b-b9e7-47d7-9b97-9025376032a6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "    <tr>\n",
       "        <td>\n",
       "            <table style=\"border-collapse: collapse;\">\n",
       "                <thead>\n",
       "                    <tr>\n",
       "                        <td> </td>\n",
       "                        <th> Array </th>\n",
       "                        <th> Chunk </th>\n",
       "                    </tr>\n",
       "                </thead>\n",
       "                <tbody>\n",
       "                    \n",
       "                    <tr>\n",
       "                        <th> Bytes </th>\n",
       "                        <td> 8 B </td>\n",
       "                        <td> 8 B </td>\n",
       "                    </tr>\n",
       "                    \n",
       "                    <tr>\n",
       "                        <th> Shape </th>\n",
       "                        <td> () </td>\n",
       "                        <td> () </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <th> Dask graph </th>\n",
       "                        <td colspan=\"2\"> 1 chunks in 6 graph layers </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <th> Data type </th>\n",
       "                        <td colspan=\"2\"> float64 numpy.ndarray </td>\n",
       "                    </tr>\n",
       "                </tbody>\n",
       "            </table>\n",
       "        </td>\n",
       "        <td>\n",
       "        \n",
       "        </td>\n",
       "    </tr>\n",
       "</table>"
      ],
      "text/plain": [
       "dask.array<mean_agg-aggregate, shape=(), dtype=float64, chunksize=(), chunktype=numpy.ndarray>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = da.random.random(1000, chunks=100)\n",
    "b = da.ones(1000, chunks=100)\n",
    "\n",
    "c = b - a\n",
    "c = da.nanmean(c)\n",
    "c"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d435c2cb-c108-4d93-b124-9f3488fb97bd",
   "metadata": {},
   "source": [
    "We need to tell Dask to perform the computation to produce the value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "97892514-9c6a-4727-87c5-75fd944bffe1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.4979883783977874"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c = c.compute()\n",
    "c"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8991d1af-c412-42f3-b72a-fcd99b3928da",
   "metadata": {},
   "source": [
    "## How much faster is dask than Numpy at some calculations?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f7ab4d5-8f1f-4c1f-8fc2-ce5aada90fad",
   "metadata": {},
   "source": [
    "Let's make a large array.\n",
    "\n",
    "We will import a different library to do the timing to demonstrate the difference in computation speeds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "8b431df4-082d-4e3d-82b7-5ef27bd42ba4",
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e021751-38dd-458f-beef-76a14ffb56fe",
   "metadata": {},
   "source": [
    "Let's create two functions that perform the same task. One uses Dask and one uses Numpy. They return the number of seconds to make the computations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "bb3a4c3e-2121-40df-99be-b8222cfbe534",
   "metadata": {},
   "outputs": [],
   "source": [
    "def numpy_computation(num):\n",
    "    start_time = time.time()\n",
    "    b = np.ones(num) - np.random.random(num)\n",
    "    b[np.random.randint(0, num, int(num/10))] = np.nan\n",
    "    b = np.nanmean(b)\n",
    "\n",
    "    return time.time() - start_time\n",
    "    \n",
    "def dask_computation(num):\n",
    "    start_time = time.time()\n",
    "    chunks = int(num/10)\n",
    "    b = da.ones(num, chunks=chunks) - da.random.random(num, chunks=chunks)\n",
    "    b[da.random.randint(0, 1, num, chunks=chunks).astype(bool)] = np.nan\n",
    "    b = da.nanmean(b)\n",
    "    b = b.compute()\n",
    "\n",
    "    return time.time() - start_time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "0e18ff1e-b630-4d43-9500-f13b999fb3ba",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Numpy Elapsed Time: 1.7535278797149658 seconds\n",
      "Dask Elapsed Time: 1.1301848888397217 seconds\n",
      "\n",
      "Dask is about 1.5515407231423743 times faster than Numpy for this operation.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "num = 100_000_000\n",
    "numpy_time = numpy_computation(num)\n",
    "dask_time = dask_computation(num)\n",
    "\n",
    "print(f'Numpy Elapsed Time: {numpy_time} seconds')\n",
    "print(f'Dask Elapsed Time: {dask_time} seconds')\n",
    "\n",
    "print(f'\\nDask is about {numpy_time/dask_time} times faster than Numpy for this operation.\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "114b27c2-4a63-40f0-9679-b7f27051c47b",
   "metadata": {},
   "source": [
    "What about a smaller array. It takes overhead operations to perform the work so for small operations Dask can be slower than normal Numpy operations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "cb8a8a47-0c52-4474-9498-53ccf704fea2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Numpy Elapsed Time: 0.00017404556274414062 seconds\n",
      "\n",
      "Dask Elapsed Time: 0.008721113204956055 seconds\n",
      "\n",
      "\n",
      "Numpy is about 50.108219178082194 times faster than Dask for this much smaller operation.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "num = 1000\n",
    "numpy_time = numpy_computation(num)\n",
    "dask_time = dask_computation(num)\n",
    "\n",
    "print(f'Numpy Elapsed Time: {numpy_time} seconds\\n')\n",
    "print(f'Dask Elapsed Time: {dask_time} seconds\\n')\n",
    "\n",
    "print(f'\\nNumpy is about {dask_time/numpy_time} times faster than Dask for this much smaller operation.\\n')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}