part-3/lecture-slides.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e5b3cbee-56cd-402f-82fc-9479119acd4b",
   "metadata": {},
   "source": [
    "# Part 3: Arbitrary data structures"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57a210ca-3289-4d36-8617-63a562849a7a",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "So far, all the arrays we've dealt with have been rectangular (in $n$ dimensions; \"rectilinear\").\n",
    "\n",
    "<center>\n",
    "<img src=\"../img/8-layer_cube.jpg\" width=\"50%\">\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0b3d7bc-bd9f-4be3-8591-31438d04848b",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "What if we had data like this?\n",
    "\n",
    "```json\n",
    "[\n",
    "  [[1.84, 0.324]],\n",
    "  [[-1.609, -0.713, 0.005], [0.953, -0.993, 0.011, 0.718]],\n",
    "  [[0.459, -1.517, 1.545], [0.33, 0.292]],\n",
    "  [[-0.376, -1.46, -0.206], [0.65, 1.278]],\n",
    "  [[], [], [1.617]],\n",
    "  []\n",
    "]\n",
    "[\n",
    "  [[-0.106, 0.611]],\n",
    "  [[0.118, -1.788, 0.794, 0.658], [-0.105]]\n",
    "]\n",
    "[\n",
    "  [[-0.384], [0.697, -0.856]],\n",
    "  [[0.778, 0.023, -1.455, -2.289], [-0.67], [1.153, -1.669, 0.305, 1.517, -0.292]]\n",
    "]\n",
    "[\n",
    "  [[0.205, -0.355], [-0.265], [1.042]],\n",
    "  [[-0.004], [-1.167, -0.054, 0.726, 0.213]],\n",
    "  [[1.741, -0.199, 0.827]]\n",
    "]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d47dd75e-f5c7-4a86-946f-984b6cf7f3e6",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "What if we had data like this?\n",
    "\n",
    "```json\n",
    "[\n",
    "  {\"fill\": \"#b1b1b1\", \"stroke\": \"none\", \"points\": [{\"x\": 5.27453, \"y\": 1.03276},\n",
    "    {\"x\": -3.51280, \"y\": 1.74849}]},\n",
    "  {\"fill\": \"#b1b1b1\", \"stroke\": \"none\", \"points\": [{\"x\": 8.21630, \"y\": 4.07844},\n",
    "    {\"x\": -0.79157, \"y\": 3.49478}, {\"x\": 16.38932, \"y\": 5.29399},\n",
    "    {\"x\": 10.38641, \"y\": 0.10832}, {\"x\": -2.07070, \"y\": 14.07140},\n",
    "    {\"x\": 9.57021, \"y\": -0.94823}, {\"x\": 1.97332, \"y\": 3.62380},\n",
    "    {\"x\": 5.66760, \"y\": 11.38001}, {\"x\": 0.25497, \"y\": 3.39276},\n",
    "    {\"x\": 3.86585, \"y\": 6.22051}, {\"x\": -0.67393, \"y\": 2.20572}]},\n",
    "  {\"fill\": \"#d0d0ff\", \"stroke\": \"none\", \"points\": [{\"x\": 3.59528, \"y\": 7.37191},\n",
    "    {\"x\": 0.59192, \"y\": 2.91503}, {\"x\": 4.02932, \"y\": -1.13601},\n",
    "    {\"x\": -1.01593, \"y\": 1.95894}, {\"x\": 1.03666, \"y\": 0.05251}]},\n",
    "  {\"fill\": \"#d0d0ff\", \"stroke\": \"none\", \"points\": [{\"x\": -8.78510, \"y\": -0.00497},\n",
    "    {\"x\": -15.22688, \"y\": 3.90244}, {\"x\": 5.74593, \"y\": 4.12718}]},\n",
    "  {\"fill\": \"none\", \"stroke\": \"#000000\", \"points\": [{\"x\": 4.40625, \"y\": -6.953125},\n",
    "    {\"x\": 4.34375, \"y\": -7.09375}, {\"x\": 4.3125, \"y\": -7.140625},\n",
    "    {\"x\": 4.140625, \"y\": -7.140625}]},\n",
    "  {\"fill\": \"none\", \"stroke\": \"#808080\", \"points\": [{\"x\": 0.46875, \"y\": -0.09375},\n",
    "    {\"x\": 0.46875, \"y\": -0.078125}, {\"x\": 0.46875, \"y\": 0.53125}]}\n",
    "]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "640e3e81-1816-47b5-8729-db3f5a82b7c5",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "What if we had data like this?\n",
    "\n",
    "```json\n",
    "[\n",
    "  {\"movie\": \"Evil Dead\", \"year\": 1981, \"actors\":\n",
    "    [\"Bruce Campbell\", \"Ellen Sandweiss\", \"Richard DeManincor\", \"Betsy Baker\"]\n",
    "  },\n",
    "  {\"movie\": \"Darkman\", \"year\": 1900, \"actors\":\n",
    "    [\"Liam Neeson\", \"Frances McDormand\", \"Larry Drake\", \"Bruce Campbell\"]\n",
    "  },\n",
    "  {\"movie\": \"Army of Darkness\", \"year\": 1992, \"actors\":\n",
    "    [\"Bruce Campbell\", \"Embeth Davidtz\", \"Marcus Gilbert\", \"Bridget Fonda\",\n",
    "     \"Ted Raimi\", \"Patricia Tallman\"]\n",
    "  },\n",
    "  {\"movie\": \"A Simple Plan\", \"year\": 1998, \"actors\":\n",
    "    [\"Bill Paxton\", \"Billy Bob Thornton\", \"Bridget Fonda\", \"Brent Briscoe\"]\n",
    "  },\n",
    "  {\"movie\": \"Spider-Man 2\", \"year\": 2004, \"actors\":\n",
    "    [\"Tobey Maguire\", \"Kristen Dunst\", \"Alfred Molina\", \"James Franco\",\n",
    "     \"Rosemary Harris\", \"J.K. Simmons\", \"Stan Lee\", \"Bruce Campbell\"]\n",
    "  },\n",
    "  {\"movie\": \"Drag Me to Hell\", \"year\": 2009, \"actors\":\n",
    "    [\"Alison Lohman\", \"Justin Long\", \"Lorna Raver\", \"Dileep Rao\", \"David Paymer\"]\n",
    "  }\n",
    "]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c498ff3-82d3-4af3-84ae-85e089672d5d",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "It might be possible to turn these datasets into tabular form using surrogate keys and database normalization, but\n",
    "\n",
    " * they could be inconvenient or less efficient in that form, depending on what we want to do,\n",
    " * they were very likely _given_ in a ragged/untidy form. You can't ignore the data-cleaning step!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ade6b839-221b-4f59-a305-78d4bffd357a",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "<br>\n",
    "\n",
    "Dealing with these datasets as JSON or Python objects is inefficient for the same reason as for lists of numbers."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f27a1fc-1ff9-4ada-b6b0-1fab7ae89276",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "<br>\n",
    "\n",
    "We want arbitrary data structure with array-oriented interface and performance...\n",
    "\n",
    "<center>\n",
    "<img src=\"../img/awkward-motivation-venn-diagram.svg\" width=\"40%\">\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ccfa260f-3e77-4989-a667-0d5da0dbee1a",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Libraries for irregular arrays\n",
    "\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c4aeb59-6edf-4a5f-a00b-715e51f22faa",
   "metadata": {
    "tags": []
   },
   "source": [
    "<table>\n",
    "<tr style=\"background: white;\"><td width=\"35%\"><img src=\"../img/logo-arrow.svg\" width=\"100%\"></td><td style=\"padding-left: 50px;\">In-memory format and an ecosystem of tools, an \"exploded database\" (database functionality provided as interchangeable pieces). Strong focus on delivering data, zero-copy, between processes.</td></tr>\n",
    "<tr style=\"background: white; height: 30px;\"><td></td><td></td></tr>\n",
    "<tr style=\"background: white;\"><td width=\"35%\"><img src=\"../img/logo-awkward.svg\" width=\"100%\"></td><td style=\"padding-left: 50px;\">Library for array-oriented programming like NumPy, but for arbitrary data structures. Losslessly zero-copy convertible to/from Arrow and Parquet.</td></tr>\n",
    "<tr style=\"background: white; height: 30px;\"><td></td><td></td></tr>\n",
    "<tr style=\"background: white;\"><td width=\"35%\"><img src=\"../img/logo-parquet.svg\" width=\"100%\"></td><td style=\"padding-left: 50px;\">Disk format for storing large datasets and (selectively) retrieving them.</td></tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53fc76c9-290a-49cd-ab6c-0065e8c94573",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "<img src=\"../img/logo-arrow.svg\" width=\"30%\">\n",
    "\n",
    "<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18a43904-14bd-4edc-b6fd-cd9af40da40a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pyarrow as pa"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c6c737c-10b1-4259-aa11-7f0d4cdf86d8",
   "metadata": {},
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f45c5b81-625d-4680-987e-a1eee32d968c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "arrow_array = pa.array([\n",
    "    [{\"x\": 1.1, \"y\": [1]}, {\"x\": 2.2, \"y\": [1, 2]}, {\"x\": 3.3, \"y\": [1, 2, 3]}],\n",
    "    [],\n",
    "    [{\"x\": 4.4, \"y\": [1, 2, 3, 4]}, {\"x\": 5.5, \"y\": [1, 2, 3, 4, 5]}]\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de7dff00-f312-4011-b62f-b7d21a71aa8b",
   "metadata": {},
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4ba0e82-f1c1-4db1-bce3-098ff07e4994",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "arrow_array.type"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56c2eb4d-8ff8-4fd0-a897-91a1d5f7d929",
   "metadata": {},
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a4c9d541-860b-4162-8dea-9722aeda0795",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "arrow_array"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a57bc1f2-4db4-49d7-8928-f4fa8d26620f",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "<img src=\"../img/logo-awkward.svg\" width=\"30%\">\n",
    "\n",
    "<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74f36f06-325c-45d1-8d38-3f21c6d2a827",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import awkward as ak"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3274cf50-1870-468f-a240-49e8f2d625a8",
   "metadata": {},
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e79beb7b-8df8-410b-a103-551c1fa16ad1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "awkward_array = ak.from_arrow(arrow_array)\n",
    "awkward_array"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36241997-055b-4fff-974a-9f48bef5d7d4",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "<img src=\"../img/logo-parquet.svg\" width=\"30%\">\n",
    "\n",
    "<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b3087fd-f3ea-45e0-9fa6-d4202490c5ad",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ak.to_parquet(awkward_array, \"/tmp/file.parquet\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae99fd9d-45bd-46e3-adcc-1036b4d4a656",
   "metadata": {},
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f348a947-d770-4526-9bfd-f73635d5831b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ak.from_parquet(\"/tmp/file.parquet\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "685aad8a-f0eb-4e25-91d7-6ebc4bfe9d47",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Awkward Array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b3ad23dd-4165-4cb2-a3f4-361e4ddabbbd",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ragged = ak.Array([\n",
    "    [\n",
    "      [[1.84, 0.324]],\n",
    "      [[-1.609, -0.713, 0.005], [0.953, -0.993, 0.011, 0.718]],\n",
    "      [[0.459, -1.517, 1.545], [0.33, 0.292]],\n",
    "      [[-0.376, -1.46, -0.206], [0.65, 1.278]],\n",
    "      [[], [], [1.617]],\n",
    "      []\n",
    "    ],\n",
    "    [\n",
    "      [[-0.106, 0.611]],\n",
    "      [[0.118, -1.788, 0.794, 0.658], [-0.105]]\n",
    "    ],\n",
    "    [\n",
    "      [[-0.384], [0.697, -0.856]],\n",
    "      [[0.778, 0.023, -1.455, -2.289], [-0.67], [1.153, -1.669, 0.305, 1.517, -0.292]]\n",
    "    ],\n",
    "    [\n",
    "      [[0.205, -0.355], [-0.265], [1.042]],\n",
    "      [[-0.004], [-1.167, -0.054, 0.726, 0.213]],\n",
    "      [[1.741, -0.199, 0.827]]\n",
    "    ]\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c55509b2-735a-4bd7-99e8-1e3a0f4482da",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "**Multidimensional indexing**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "40e71258-eb17-4aa7-a088-cddefcc80efd",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ragged[3, 1, -1, 2]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c26d50ec-8341-400f-9315-bb6cc9514537",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "**Basic slicing**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c20087e7-629f-43e7-aaee-d9da62dba40c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ragged[3, 1:, -1, 1:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1818b9c3-b7ec-42e4-83fc-148175dd6328",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "**Advanced slicing**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a326d775-71b1-4db4-9324-833bec673e97",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ragged[[False, False, True, True], [0, -1, 0, -1], 0, -1]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89e93ac5-f284-423d-a058-07f8bd70f956",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "**Awkward slicing**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e8a09bc1-8417-426f-8b13-752c4ce33601",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ragged > 0"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4a2c710-47cf-4ff6-a2bd-9c0c4b66eca0",
   "metadata": {},
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c725196-44f2-4aa6-aaf8-c6cd93ea71ab",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ragged[ragged > 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5bd55c25-f4d1-4144-a0dd-0544db811f5e",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "**Reductions**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0ba071c4-4c24-4c5a-8619-1665fef31f2b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ak.sum(ragged)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47e84e26-643a-47c3-8686-4c5d31a028cc",
   "metadata": {},
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4238ba79-2b8b-491b-82a1-e073a0dc0679",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ak.sum(ragged, axis=-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25c4052e-a49e-4f34-a18f-6218b4e8681d",
   "metadata": {},
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5c72ab16-0ac5-4aa9-865f-bd7a975f43cb",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ak.sum(ragged, axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "578d3cab-92a0-44bf-a05e-3c3a5b1af8cb",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "How are reductions even defined for ragged arrays?\n",
    "\n",
    "<center>\n",
    "<img src=\"../img/example-reduction-sum.svg\" width=\"50%\">\n",
    "</center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4fb2dce-818f-4cae-b737-38a7b72d1e25",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "small_ragged = ak.Array([[   1, 2, 4],\n",
    "                         [          ],\n",
    "                         [None, 8,  ],\n",
    "                         [  16      ]])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60218a96-6e1f-4f5e-bcfe-601877b43a74",
   "metadata": {},
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5eb46960-bdae-431c-bc33-c3cfc52be9cd",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ak.sum(small_ragged, axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dbe25c13-98be-4496-a857-dab7658b092f",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "structured = ak.Array([\n",
    "  {\"fill\": \"#b1b1b1\", \"stroke\": \"none\", \"points\": [{\"x\": 5.27453, \"y\": 1.03276},\n",
    "    {\"x\": -3.51280, \"y\": 1.74849}]},\n",
    "  {\"fill\": \"#b1b1b1\", \"stroke\": \"none\", \"points\": [{\"x\": 8.21630, \"y\": 4.07844},\n",
    "    {\"x\": -0.79157, \"y\": 3.49478}, {\"x\": 16.38932, \"y\": 5.29399},\n",
    "    {\"x\": 10.38641, \"y\": 0.10832}, {\"x\": -2.07070, \"y\": 14.07140},\n",
    "    {\"x\": 9.57021, \"y\": -0.94823}, {\"x\": 1.97332, \"y\": 3.62380},\n",
    "    {\"x\": 5.66760, \"y\": 11.38001}, {\"x\": 0.25497, \"y\": 3.39276},\n",
    "    {\"x\": 3.86585, \"y\": 6.22051}, {\"x\": -0.67393, \"y\": 2.20572}]},\n",
    "  {\"fill\": \"#d0d0ff\", \"stroke\": \"none\", \"points\": [{\"x\": 3.59528, \"y\": 7.37191},\n",
    "    {\"x\": 0.59192, \"y\": 2.91503}, {\"x\": 4.02932, \"y\": -1.13601},\n",
    "    {\"x\": -1.01593, \"y\": 1.95894}, {\"x\": 1.03666, \"y\": 0.05251}]},\n",
    "  {\"fill\": \"#d0d0ff\", \"stroke\": \"none\", \"points\": [{\"x\": -8.78510, \"y\": -0.00497},\n",
    "    {\"x\": -15.22688, \"y\": 3.90244}, {\"x\": 5.74593, \"y\": 4.12718}]},\n",
    "  {\"fill\": \"none\", \"stroke\": \"#000000\", \"points\": [{\"x\": 4.40625, \"y\": -6.953125},\n",
    "    {\"x\": 4.34375, \"y\": -7.09375}, {\"x\": 4.3125, \"y\": -7.140625},\n",
    "    {\"x\": 4.140625, \"y\": -7.140625}]},\n",
    "  {\"fill\": \"none\", \"stroke\": \"#808080\", \"points\": [{\"x\": 0.46875, \"y\": -0.09375},\n",
    "    {\"x\": 0.46875, \"y\": -0.078125}, {\"x\": 0.46875, \"y\": 0.53125}]}\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35873b9d-963d-421e-b16b-b9cb8ea6e9c2",
   "metadata": {
    "tags": []
   },
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d1523ff-2e39-4ec0-a5b3-9f45591a0544",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d933bfd1-c886-41b3-ad53-688283440b20",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "**Elementwise formulas**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ea1e4c5d-9660-451f-9fdd-3536c005ed55",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "np.sqrt(structured[\"points\", \"x\"]**2 + structured[\"points\", \"y\"]**2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc20b9c0-94a6-4099-b791-b94ff342c780",
   "metadata": {},
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be00be97-c4d5-48ff-ae34-712b84c390b4",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "np.sqrt(structured.points.x**2 + structured.points.y**2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "edbc74f6-d431-46d0-b3b5-4c9e588acaf1",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "**Quizlet:** Given the following dataset,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d0c4b9bd-2fb0-4c51-b2f6-e06cf150f4dd",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "data = ak.Array([\n",
    "    {\"movie\": \"Evil Dead\", \"year\": 1981, \"actors\":\n",
    "        [\"Bruce Campbell\", \"Ellen Sandweiss\", \"Richard DeManincor\", \"Betsy Baker\"]\n",
    "    },\n",
    "    {\"movie\": \"Darkman\", \"year\": 1900, \"actors\":\n",
    "        [\"Liam Neeson\", \"Frances McDormand\", \"Larry Drake\", \"Bruce Campbell\"]\n",
    "    },\n",
    "    {\"movie\": \"Army of Darkness\", \"year\": 1992, \"actors\":\n",
    "        [\"Bruce Campbell\", \"Embeth Davidtz\", \"Marcus Gilbert\", \"Bridget Fonda\",\n",
    "         \"Ted Raimi\", \"Patricia Tallman\"]\n",
    "    },\n",
    "    {\"movie\": \"A Simple Plan\", \"year\": 1998, \"actors\":\n",
    "        [\"Bill Paxton\", \"Billy Bob Thornton\", \"Bridget Fonda\", \"Brent Briscoe\"]\n",
    "    },\n",
    "    {\"movie\": \"Spider-Man 2\", \"year\": 2004, \"actors\":\n",
    "        [\"Tobey Maguire\", \"Kristen Dunst\", \"Alfred Molina\", \"James Franco\",\n",
    "         \"Rosemary Harris\", \"J.K. Simmons\", \"Stan Lee\", \"Bruce Campbell\"]\n",
    "    },\n",
    "    {\"movie\": \"Drag Me to Hell\", \"year\": 2009, \"actors\":\n",
    "        [\"Alison Lohman\", \"Justin Long\", \"Lorna Raver\", \"Dileep Rao\", \"David Paymer\"]\n",
    "    }\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "769bdcb4-e2b3-4777-b141-f5584ac517dc",
   "metadata": {},
   "source": [
    "select movies that do _not_ contain `\"Bruce Campbell\"`. See [ak.all](https://awkward-array.org/doc/main/reference/generated/ak.all.html), [ak.any](https://awkward-array.org/doc/main/reference/generated/ak.any.html), [np.invert](https://numpy.org/doc/stable/reference/generated/numpy.invert.html), and [ak.num](https://awkward-array.org/doc/main/reference/generated/ak.num.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e96bab44-41f3-4efe-b322-ef8ad0f3e63d",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%html\n",
    "<!-- This will only work on the day of the live tutorial. -->\n",
    "<div style=\"overflow: hidden;\"><iframe src=\"https://app.sli.do/event/dFsiXggxZ3t2B9yhr5zmxZ\" width=\"100%\" height=\"280\" scrolling=\"no\" style=\"border: none;\"></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2053ebb7-e9a5-43ed-8543-5e165131e891",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "**Answer:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4d753591-488f-4012-968f-d4ed1bcdf939",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "is_bruce_campbell = data.actors == \"Bruce Campbell\"\n",
    "is_bruce_campbell"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "55be9deb-277f-49d4-b4d4-b60af957dfab",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "all_not_bruce_campbell = ak.all(~is_bruce_campbell, axis=1)\n",
    "all_not_bruce_campbell"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ab8a7ecc-7a32-4377-a769-895cef959490",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "data[all_not_bruce_campbell]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94536394-aa7d-41a9-8c22-d55e6faa5d06",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Combinatorics"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5624b269-5725-430e-8e6c-35df59f483ee",
   "metadata": {},
   "source": [
    "Some operations are more meaningful on irregular arrays than rectilinear ones:\n",
    "\n",
    "<table style=\"width: 60%\">\n",
    "<tr style=\"background: white; padding-top: 0px;\"><td width=\"50%\"><img src=\"../img/cartoon-cartesian.svg\" width=\"100%\"></td><td width=\"50%\"><img src=\"../img/cartoon-combinations.svg\" width=\"100%\"></td></tr>\n",
    "</table>\n",
    "\n",
    "[ak.cartesian](https://awkward-array.org/doc/main/reference/generated/ak.cartesian.html) takes a [Cartesian product](https://en.wikipedia.org/wiki/Cartesian_product) of lists from $n$ different arrays, making an array of lists of $n$-tuples.\n",
    "\n",
    "[ak.combinations](https://awkward-array.org/doc/main/reference/generated/ak.combinations.html) takes $n$ [samples without replacement](http://prob140.org/sp18/textbook/notebooks-md/5_04_Sampling_Without_Replacement.html) of lists from a single array, making an array of lists of $n$-tuples."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f287196-4044-4585-be8f-354634226451",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "<center>\n",
    "<img src=\"../img/cartoon-cartesian.svg\" width=\"30%\">\n",
    "</center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf80340c-0348-4641-9c76-34ac7585efcf",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "numbers = ak.Array([[1, 2, 3], [], [4]])\n",
    "letters = ak.Array([[\"a\", \"b\"], [\"c\"], [\"d\", \"e\"]])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c960d4e-b790-4e75-a6dd-8d27548fd8e1",
   "metadata": {},
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4fc5d7d9-6e1e-4721-aadf-4a0becb0b4e4",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ak.cartesian([numbers, letters])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98047723-2a46-49fa-a704-a3a7ef52625b",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "<center>\n",
    "<img src=\"../img/cartoon-combinations.svg\" width=\"30%\">\n",
    "</center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cfad842c-b3a2-46bd-b98e-e86a3dd42d9f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "values = ak.Array([[1.1, 2.2, 3.3, 4.4], [], [5.5, 6.6]])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47fb2d0b-20e1-49b4-b3ef-4e1cd69e310f",
   "metadata": {},
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fc1070a7-3c1a-4057-b129-36666b88f24d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ak.combinations(values, 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7cc08c9-8644-4a04-87cf-22911afb2f2c",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "**Example:** Which actors in the same movie have names that are the same length?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "36f13812-e11f-48ce-9564-c505d47de78e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "actor_pairs = ak.combinations(data.actors, 2, axis=1)\n",
    "actor_pairs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b3064f4-c34c-47c2-9f1d-ca14bdb1a22b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "actor_pairs[\"0\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "56b37960-5d63-4a5f-98fc-32e1732cbfe1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ak.num(actor_pairs[\"0\"], axis=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "53e0869f-6efc-4880-8c4c-eef8d44cae12",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "actor_pairs[(ak.num(actor_pairs[\"0\"], axis=2) == ak.num(actor_pairs[\"1\"], axis=2))]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2edc4640-4cb6-4025-95a3-a8885c83b693",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "**Go to the [Part 3 project](project.ipynb) now!**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "scipy2024-tutorial-thinking-in-arrays",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}