From cf043dc1b7d560c491bc31f3a044381892d166be Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nuno=20Gon=C3=A7alves?= <93784204+nunogoncalves03@users.noreply.github.com> Date: Tue, 17 Dec 2024 19:28:32 +0000 Subject: [PATCH] Add a notebook to demonstrate how to run notebooks from another notebook (#127) * Add a notebook to demonstrate how to run notebooks from another notebook * fix typo * rename occurrences of execute/executing to run/running * add %run_shared example --- .../meta.toml | 13 + .../notebook.ipynb | 285 ++++++++++++++++++ 2 files changed, 298 insertions(+) create mode 100644 notebooks/running-notebooks-from-another-notebook-with-fusion-sql/meta.toml create mode 100644 notebooks/running-notebooks-from-another-notebook-with-fusion-sql/notebook.ipynb diff --git a/notebooks/running-notebooks-from-another-notebook-with-fusion-sql/meta.toml b/notebooks/running-notebooks-from-another-notebook-with-fusion-sql/meta.toml new file mode 100644 index 0000000..8feda32 --- /dev/null +++ b/notebooks/running-notebooks-from-another-notebook-with-fusion-sql/meta.toml @@ -0,0 +1,13 @@ +[meta] +authors=["singlestore"] +title="Running Notebooks from Another Notebook with Fusion SQL" +description="""\ + Learn how to run Notebooks from another Notebook + in SingleStoreDB Cloud using Fusion SQL. + """ +icon="files" +difficulty="intermediate" +tags=["notebooks", "jobs", "python", "fusion", "starter"] +lesson_areas=["Python SDK"] +destinations=["spaces"] +minimum_tier="standard" diff --git a/notebooks/running-notebooks-from-another-notebook-with-fusion-sql/notebook.ipynb b/notebooks/running-notebooks-from-another-notebook-with-fusion-sql/notebook.ipynb new file mode 100644 index 0000000..dc51591 --- /dev/null +++ b/notebooks/running-notebooks-from-another-notebook-with-fusion-sql/notebook.ipynb @@ -0,0 +1,285 @@ +{ + "cells": [ + { + "id": "55bb0386", + "cell_type": "markdown", + "metadata": {}, + "source": [ + "
\n", + "
\n", + " \n", + "
\n", + "
\n", + "
SingleStore Notebooks
\n", + "

Running Notebooks from Another Notebook with Fusion SQL

\n", + "
\n", + "
" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "id": "18ed6021", + "metadata": {}, + "source": [ + "In this notebook, we demonstrate how to use **[Fusion SQL](https://www.singlestore.com/spaces/getting-started-with-fusion-sql/)** to run a notebook from another notebook, either within the **same session** (useful to avoid code duplication) or in a **new session** (useful for parallel execution)." + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "id": "54d88908", + "metadata": {}, + "source": [ + "## Creating the Sample Notebook\n", + "In this section, we will create a sample notebook to be used in our examples." + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "id": "a2c382f9", + "metadata": {}, + "source": [ + "Create the sample notebook in the local filesystem:" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "dd837f57", + "metadata": {}, + "outputs": [], + "source": [ + "import nbformat as nbf\n", + "\n", + "nb = nbf.v4.new_notebook()\n", + "\n", + "cell = nbf.v4.new_code_cell(\"\"\"# This is a code cell\n", + "if 'sample_var' not in globals():\n", + " sample_var = 'sample value'\n", + "print('Sample Notebook has been executed!')\"\"\")\n", + "\n", + "cell.metadata = {\n", + " \"language\": \"python\"\n", + "}\n", + "\n", + "nb.cells.append(cell)\n", + "\n", + "# Save the notebook to a file\n", + "with open('sample_notebook.ipynb', 'w') as f:\n", + " nbf.write(nb, f)\n", + "\n", + "print(\"Notebook 'sample_notebook.ipynb' created successfully in the local filesystem.\")" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "id": "bb858912", + "metadata": {}, + "source": [ + "Upload it to the **Data Studio** shared files for later download:" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "d39b0564", + "metadata": {}, + "outputs": [], + "source": [ + "import time\n", + "\n", + "sample_notebook_name='Sample Notebook {}.ipynb'.format(int(time.time() * 1_000_000))\n", + "\n", + "%sql UPLOAD SHARED FILE TO '{{ sample_notebook_name }}' FROM 'sample_notebook.ipynb';\n", + "\n", + "print(\"Notebook '{}' has been created in the Data Studio shared files.\".format(sample_notebook_name))" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "id": "4f0cbad1", + "metadata": {}, + "source": [ + "## Running the Sample Notebook in the Current Session\n", + "\n", + "In this example, we will run the previously created sample notebook within the current session. This approach is useful for avoiding code duplication in your notebooks, such as environment setup or reusable functions." + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "id": "9c08ff46", + "metadata": {}, + "source": [ + "To run the notebook, we can use the `%run_shared` magic command. Let's run the notebook and confirm that the `sample_var` variable set in the sample notebook is accessible in the current session:" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "85ad1f06", + "metadata": {}, + "outputs": [], + "source": [ + "if 'sample_var' in globals():\n", + " del sample_var\n", + "\n", + "%run_shared {{ sample_notebook_name }}\n", + "\n", + "print(\"The value of 'sample_var' is '{}'.\\n\".format(sample_var))" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "id": "1a059fcb", + "metadata": {}, + "source": [ + "## Running the Sample Notebook in a New Session\n", + "\n", + "Instead of running a notebook in the current session, we can run it in a new session \u2014 either synchronously or asynchronously \u2014 using jobs. This approach is useful for parallel execution or running code in a separate runtime environment.\n", + "\n", + "
\n", + " \n", + "
\n", + "

Note

\n", + "

SingleStore enforces limits on the number of compute sessions that can be created, which can be increased upon request. Please contact Support for more information.

\n", + "
\n", + "
" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "id": "b5d35764", + "metadata": {}, + "source": [ + "Run **two jobs** in parallel and wait for their completion (each job will run on a separate session):" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "2cdaea66", + "metadata": {}, + "outputs": [], + "source": [ + "job_ids = []\n", + "for x in range(2):\n", + " print(\"Running job for {}...\".format(x))\n", + " job_res = %sql RUN JOB USING NOTEBOOK '{{ sample_notebook_name }}' WITH PARAMETERS {\"sample_var\": \"{{x}}\"}\n", + " job_ids.append(job_res[0].JobID)\n", + "\n", + "print(f'Waiting for jobs to complete... {job_ids}')\n", + "success = %sql WAIT ON JOBS {{ job_ids }} WITH TIMEOUT 60 MINUTES\n", + "\n", + "print(f'All jobs completed with success: {bool(success[0].Success)}')" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "id": "eaa1de71", + "metadata": {}, + "source": [ + "View the executions of the jobs we ran:" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "b81fa588", + "metadata": {}, + "outputs": [], + "source": [ + "for job_id in job_ids:\n", + " execs = %sql SHOW JOB EXECUTIONS FOR {{ job_id }} from 1 to 1\n", + " print(execs)" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "id": "f64a5020", + "metadata": {}, + "source": [ + "The jobs can now be dropped:" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "5f7b74a6", + "metadata": {}, + "outputs": [], + "source": [ + "for id in job_ids:\n", + " print(f\"Dropping job '{id}'...\")\n", + " %sql DROP JOBS {{id}}" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "id": "48372f6b", + "metadata": {}, + "source": [ + "## Cleanup" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "id": "00b5f172", + "metadata": {}, + "source": [ + "Delete the sample notebook from the **Data Studio** shared files:" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "c368e8a6", + "metadata": {}, + "outputs": [], + "source": [ + "%%sql\n", + "DROP SHARED FILE '{{ sample_notebook_name }}'" + ] + }, + { + "id": "7259e797", + "cell_type": "markdown", + "metadata": {}, + "source": [ + "
\n", + "
" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.9" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +}