Add a notebook to demonstrate how to run notebooks from another noteb…

…ook (#127) * Add a notebook to demonstrate how to run notebooks from another notebook * fix typo * rename occurrences of execute/executing to run/running * add %run_shared example
singlestore-labs · Dec 17, 2024 · cf043dc · cf043dc
1 parent 500585e
commit cf043dc
Show file tree

Hide file tree

Showing 2 changed files with 298 additions and 0 deletions.
diff --git a/notebooks/running-notebooks-from-another-notebook-with-fusion-sql/meta.toml b/notebooks/running-notebooks-from-another-notebook-with-fusion-sql/meta.toml
@@ -0,0 +1,13 @@
+[meta]
+authors=["singlestore"]
+title="Running Notebooks from Another Notebook with Fusion SQL"
+description="""\
+    Learn how to run Notebooks from another Notebook
+    in SingleStoreDB Cloud using Fusion SQL.
+    """
+icon="files"
+difficulty="intermediate"
+tags=["notebooks", "jobs", "python", "fusion", "starter"]
+lesson_areas=["Python SDK"]
+destinations=["spaces"]
+minimum_tier="standard"
diff --git a/notebooks/running-notebooks-from-another-notebook-with-fusion-sql/notebook.ipynb b/notebooks/running-notebooks-from-another-notebook-with-fusion-sql/notebook.ipynb
@@ -0,0 +1,285 @@
+{
+  "cells": [
+    {
+      "id": "55bb0386",
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "<div id=\"singlestore-header\" style=\"display: flex; background-color: rgba(255, 224, 129, 0.25); padding: 5px;\">\n",
+        "    <div id=\"icon-image\" style=\"width: 90px; height: 90px;\">\n",
+        "        <img width=\"100%\" height=\"100%\" src=\"https://raw.githubusercontent.com/singlestore-labs/spaces-notebooks/master/common/images/header-icons/files.png\" />\n",
+        "    </div>\n",
+        "    <div id=\"text\" style=\"padding: 5px; margin-left: 10px;\">\n",
+        "        <div id=\"badge\" style=\"display: inline-block; background-color: rgba(0, 0, 0, 0.15); border-radius: 4px; padding: 4px 8px; align-items: center; margin-top: 6px; margin-bottom: -2px; font-size: 80%\">SingleStore Notebooks</div>\n",
+        "        <h1 style=\"font-weight: 500; margin: 8px 0 0 4px;\">Running Notebooks from Another Notebook with Fusion SQL</h1>\n",
+        "    </div>\n",
+        "</div>"
+      ]
+    },
+    {
+      "attachments": {},
+      "cell_type": "markdown",
+      "id": "18ed6021",
+      "metadata": {},
+      "source": [
+        "In this notebook, we demonstrate how to use **[Fusion SQL](https://www.singlestore.com/spaces/getting-started-with-fusion-sql/)** to run a notebook from another notebook, either within the **same session** (useful to avoid code duplication) or in a **new session** (useful for parallel execution)."
+      ]
+    },
+    {
+      "attachments": {},
+      "cell_type": "markdown",
+      "id": "54d88908",
+      "metadata": {},
+      "source": [
+        "## Creating the Sample Notebook\n",
+        "In this section, we will create a sample notebook to be used in our examples."
+      ]
+    },
+    {
+      "attachments": {},
+      "cell_type": "markdown",
+      "id": "a2c382f9",
+      "metadata": {},
+      "source": [
+        "Create the sample notebook in the local filesystem:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 1,
+      "id": "dd837f57",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "import nbformat as nbf\n",
+        "\n",
+        "nb = nbf.v4.new_notebook()\n",
+        "\n",
+        "cell = nbf.v4.new_code_cell(\"\"\"# This is a code cell\n",
+        "if 'sample_var' not in globals():\n",
+        "    sample_var = 'sample value'\n",
+        "print('Sample Notebook has been executed!')\"\"\")\n",
+        "\n",
+        "cell.metadata = {\n",
+        "    \"language\": \"python\"\n",
+        "}\n",
+        "\n",
+        "nb.cells.append(cell)\n",
+        "\n",
+        "# Save the notebook to a file\n",
+        "with open('sample_notebook.ipynb', 'w') as f:\n",
+        "    nbf.write(nb, f)\n",
+        "\n",
+        "print(\"Notebook 'sample_notebook.ipynb' created successfully in the local filesystem.\")"
+      ]
+    },
+    {
+      "attachments": {},
+      "cell_type": "markdown",
+      "id": "bb858912",
+      "metadata": {},
+      "source": [
+        "Upload it to the **Data Studio** shared files for later download:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 2,
+      "id": "d39b0564",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "import time\n",
+        "\n",
+        "sample_notebook_name='Sample Notebook {}.ipynb'.format(int(time.time() * 1_000_000))\n",
+        "\n",
+        "%sql UPLOAD SHARED FILE TO '{{ sample_notebook_name }}' FROM 'sample_notebook.ipynb';\n",
+        "\n",
+        "print(\"Notebook '{}' has been created in the Data Studio shared files.\".format(sample_notebook_name))"
+      ]
+    },
+    {
+      "attachments": {},
+      "cell_type": "markdown",
+      "id": "4f0cbad1",
+      "metadata": {},
+      "source": [
+        "## Running the Sample Notebook in the Current Session\n",
+        "\n",
+        "In this example, we will run the previously created sample notebook within the current session. This approach is useful for avoiding code duplication in your notebooks, such as environment setup or reusable functions."
+      ]
+    },
+    {
+      "attachments": {},
+      "cell_type": "markdown",
+      "id": "9c08ff46",
+      "metadata": {},
+      "source": [
+        "To run the notebook, we can use the `%run_shared` magic command. Let's run the notebook and confirm that the `sample_var` variable set in the sample notebook is accessible in the current session:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 3,
+      "id": "85ad1f06",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "if 'sample_var' in globals():\n",
+        "    del sample_var\n",
+        "\n",
+        "%run_shared {{ sample_notebook_name }}\n",
+        "\n",
+        "print(\"The value of 'sample_var' is '{}'.\\n\".format(sample_var))"
+      ]
+    },
+    {
+      "attachments": {},
+      "cell_type": "markdown",
+      "id": "1a059fcb",
+      "metadata": {},
+      "source": [
+        "## Running the Sample Notebook in a New Session\n",
+        "\n",
+        "Instead of running a notebook in the current session, we can run it in a new session \u2014 either synchronously or asynchronously \u2014 using jobs. This approach is useful for parallel execution or running code in a separate runtime environment.\n",
+        "\n",
+        "<div class=\"alert alert-block alert-warning\">\n",
+        "    <b class=\"fa fa-solid fa-exclamation-circle\"></b>\n",
+        "    <div>\n",
+        "        <p><b>Note</b></p>\n",
+        "        <p>SingleStore enforces <a href=\"https://docs.singlestore.com/cloud/developer-resources/notebooks/using-notebooks/#rate-limits\">limits</a> on the number of compute sessions that can be created, which can be increased upon request. Please contact <a href=\"https://support.singlestore.com/hc/en-us\">Support</a> for more information.</p>\n",
+        "    </div>\n",
+        "</div>"
+      ]
+    },
+    {
+      "attachments": {},
+      "cell_type": "markdown",
+      "id": "b5d35764",
+      "metadata": {},
+      "source": [
+        "Run **two jobs** in parallel and wait for their completion (each job will run on a separate session):"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 4,
+      "id": "2cdaea66",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "job_ids = []\n",
+        "for x in range(2):\n",
+        "    print(\"Running job for {}...\".format(x))\n",
+        "    job_res = %sql RUN JOB USING NOTEBOOK '{{ sample_notebook_name }}' WITH PARAMETERS {\"sample_var\": \"{{x}}\"}\n",
+        "    job_ids.append(job_res[0].JobID)\n",
+        "\n",
+        "print(f'Waiting for jobs to complete... {job_ids}')\n",
+        "success = %sql WAIT ON JOBS {{ job_ids }} WITH TIMEOUT 60 MINUTES\n",
+        "\n",
+        "print(f'All jobs completed with success: {bool(success[0].Success)}')"
+      ]
+    },
+    {
+      "attachments": {},
+      "cell_type": "markdown",
+      "id": "eaa1de71",
+      "metadata": {},
+      "source": [
+        "View the executions of the jobs we ran:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 5,
+      "id": "b81fa588",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "for job_id in job_ids:\n",
+        "  execs = %sql SHOW JOB EXECUTIONS FOR {{ job_id }} from 1 to 1\n",
+        "  print(execs)"
+      ]
+    },
+    {
+      "attachments": {},
+      "cell_type": "markdown",
+      "id": "f64a5020",
+      "metadata": {},
+      "source": [
+        "The jobs can now be dropped:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 6,
+      "id": "5f7b74a6",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "for id in job_ids:\n",
+        "    print(f\"Dropping job '{id}'...\")\n",
+        "    %sql DROP JOBS {{id}}"
+      ]
+    },
+    {
+      "attachments": {},
+      "cell_type": "markdown",
+      "id": "48372f6b",
+      "metadata": {},
+      "source": [
+        "## Cleanup"
+      ]
+    },
+    {
+      "attachments": {},
+      "cell_type": "markdown",
+      "id": "00b5f172",
+      "metadata": {},
+      "source": [
+        "Delete the sample notebook from the **Data Studio** shared files:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 7,
+      "id": "c368e8a6",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "%%sql\n",
+        "DROP SHARED FILE '{{ sample_notebook_name }}'"
+      ]
+    },
+    {
+      "id": "7259e797",
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "<div id=\"singlestore-footer\" style=\"background-color: rgba(194, 193, 199, 0.25); height:2px; margin-bottom:10px\"></div>\n",
+        "<div><img src=\"https://raw.githubusercontent.com/singlestore-labs/spaces-notebooks/master/common/images/singlestore-logo-grey.png\" style=\"padding: 0px; margin: 0px; height: 24px\"/></div>"
+      ]
+    }
+  ],
+  "metadata": {
+    "kernelspec": {
+      "display_name": "Python 3 (ipykernel)",
+      "language": "python",
+      "name": "python3"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.11.9"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 5
+}