refactor: content and organization for RADIUSS 2024

I am starting preliminary changes for RADIUSS 2024. So far: - the launcher is fixed to load the tutorials via buttons - I am moving "core" flux tutorial content separate from what we can consider external or plugins, like dyad. The reason is because these are experimental and change from year to year, and having it alongside core content makes a promise about consistency that I am not sure we can keep. I have not done this yet, but I am going to separate the files (images and notebooks) cleanly as well. - top level: everything is there for the tutorial user when the notebook opens, no need to navigate into many sub- directories. - terminal button: instead of "click this long list of annoying paths to open a terminal" I figured out how to make a button in the notebook directly. - marginal content changes: I am starting to tweak / update content, this will be a multi-step process. Signed-off-by: vsoch <[email protected]>
flux-framework · Jul 5, 2024 · 95798a0 · 95798a0
1 parent 44190e5
commit 95798a0
Show file tree

Hide file tree

Showing 33 changed files with 273 additions and 2,030 deletions.
diff --git a/.github/workflows/docker-builds.yaml b/.github/workflows/docker-builds.yaml
@@ -10,10 +10,13 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-# Tutorial is over - these builds are disabled
+# Tutorials are over - these builds are disabled
 #        test: [["2023-RADIUSS-AWS/JupyterNotebook", "docker/Dockerfile.hub", "ghcr.io/flux-framework/flux-jupyter-hub:2023"],
 #               ["2023-RADIUSS-AWS/JupyterNotebook", "docker/Dockerfile.init", "ghcr.io/flux-framework/flux-jupyter-init:2023"],
 #               ["2023-RADIUSS-AWS/JupyterNotebook", "docker/Dockerfile.spawn", "ghcr.io/flux-framework/flux-jupyter-spawn:2023"]]
+#               ["2024-RIKEN-AWS/JupyterNotebook", "docker/Dockerfile.hub", "ghcr.io/flux-framework/flux-jupyter-hub:riken-2024"],
+#               ["2024-RIKEN-AWS/JupyterNotebook", "docker/Dockerfile.init", "ghcr.io/flux-framework/flux-jupyter-init:riken-2024"],
+#               ["2024-RIKEN-AWS/JupyterNotebook", "docker/Dockerfile.spawn", "ghcr.io/flux-framework/flux-jupyter-spawn:riken-2024"]]
         test: [["2024-RADIUSS-AWS/JupyterNotebook", "docker/Dockerfile.hub", "ghcr.io/flux-framework/flux-jupyter-hub:radiuss-2024"],
                ["2024-RADIUSS-AWS/JupyterNotebook", "docker/Dockerfile.init", "ghcr.io/flux-framework/flux-jupyter-init:radiuss-2024"],
                ["2024-RADIUSS-AWS/JupyterNotebook", "docker/Dockerfile.spawn", "ghcr.io/flux-framework/flux-jupyter-spawn:radiuss-2024"]]

diff --git a/2024-RADIUSS-AWS/JupyterNotebook/README.md b/2024-RADIUSS-AWS/JupyterNotebook/README.md
@@ -3,8 +3,8 @@
 This set of tutorials provides:
 
  - [Building Base Images](#build-images)
- - [Deploy A Cluster to AWS or Google Cloud Using](#deploy-to-kubernetes) using Google Cloud or AWS
  - [Local Development or Usage](#local-usage)
+ - [Deploy A Cluster to AWS or Google Cloud Using](#deploy-to-kubernetes) using Google Cloud or AWS
 
 Pre-requisites:
 
@@ -19,6 +19,7 @@ For AWS Tutorial Day users:
 ## Build Images
 
 Let's build a set of images - one spawner and one hub, and an init. You can customize the tag to your liking.
+Remember that if you just want to test locally, you can jump to the [local usage](#local-usage) section.
 
 ```bash
 docker build -t ghcr.io/flux-framework/flux-jupyter-hub:radiuss-2024 -f docker/Dockerfile.hub .
@@ -30,12 +31,11 @@ Note that these are available under the flux-framework organization GitHub packa
 to build them unless you are developing or changing them. 
 
 If you do build (and use a different name) be sure to push your images to a public registry (or load them locally to your development cluster).
-Remember that if you just want to test locally, you can jump to the [local usage](#local-usage) section.
 
-## Local Deploy
 
-While the tutorial here is intended for deployment on AWS or Google Cloud, you can also give it a try on your local machine with a single container! You will need to [install Docker](https://docs.docker.com/engine/install/).
-When you have Docker available, you can build and run the tutorial with:
+## Local Usage
+
+While the tutorial here is intended for deployment on AWS or Google Cloud, you can also give it a try on your local machine with a single container! You will need to [install Docker](https://docs.docker.com/engine/install/). When you have Docker available, you can build and run the tutorial with:
 
 ```bash
 docker build -t flux-tutorial -f docker/Dockerfile.spawn .

diff --git a/2024-RADIUSS-AWS/JupyterNotebook/docker/Dockerfile.spawn b/2024-RADIUSS-AWS/JupyterNotebook/docker/Dockerfile.spawn
@@ -1,4 +1,4 @@
-FROM fluxrm/flux-sched:focal
+FROM fluxrm/flux-sched:jammy
 
 # Based off of https://github.com/jupyterhub/zero-to-jupyterhub-k8s/tree/main/images/singleuser-sample
 # Local usage
@@ -26,8 +26,8 @@ RUN apt-get update \
     ca-certificates \
     dnsutils \
     iputils-ping \
-    python3.9 \
-    python3.9-dev \
+    python3 \
+    python3-dev \
     python3-pip \
     python3-venv \
     openmpi-bin \
@@ -47,8 +47,6 @@ RUN python3 -m pip install -r requirements.txt && \
     python3 -m pip install ipython==7.34.0  && \
     python3 -m IPython kernel install
 
-COPY ./tutorial /home/jovyan/flux-tutorial-2024
-
 # This is code to install DYAD
 # This was added to the RADIUSS 2023 tutorials on AWS
 RUN git clone https://github.com/openucx/ucx.git \
@@ -76,22 +74,31 @@ RUN git clone https://github.com/flux-framework/dyad.git \
     && cd ../.. \
     && rm -rf dyad
 
-
 # This adds the flux-tree command, which is provided in flux-sched source
 # but not installed alongside production flux-core
 COPY ./flux-tree/* /usr/libexec/flux/cmd/
 RUN chmod +x /usr/libexec/flux/cmd/flux-tree*
 
+RUN apt-get update && apt-get install -y nodejs && apt-get clean && rm -rf /var/lib/apt/lists/*
+
+RUN wget https://nodejs.org/dist/v20.15.0/node-v20.15.0-linux-x64.tar.xz && \
+    apt-get update && apt-get install -y xz-utils && rm -rf /var/lib/apt/lists/* && \
+    xz -d -v node-v20.15.0-linux-x64.tar.xz && \
+    tar -C /usr/local --strip-components=1 -xvf node-v20.15.0-linux-x64.tar 
+
 # This customizes the launcher UI
 # https://jupyter-app-launcher.readthedocs.io/en/latest/usage.html
 RUN python3 -m pip install jupyter_app_launcher && \
     python3 -m pip install --upgrade jupyter-server && \
+    python3 -m pip install jupyter-launcher-shortcuts && \
     mkdir -p /usr/local/share/jupyter/lab/jupyter_app_launcher
-COPY ./docker/jupyter-launcher.yaml /usr/local/share/jupyter/lab/jupyter_app_launcher/config.yaml
-ENV JUPYTER_APP_LAUNCHER_PATH /usr/local/share/jupyter/lab/jupyter_app_launcher
+
+COPY ./tutorial /home/jovyan/
+COPY ./docker/jupyter-launcher.yaml /usr/local/share/jupyter/lab/jupyter_app_launcher/jp_app_launcher.yaml
+ENV JUPYTER_APP_LAUNCHER_PATH=/usr/local/share/jupyter/lab/jupyter_app_launcher/
 
 # Give jovyan user permissions to tutorial materials
-RUN chmod -R 777 ~/flux-tutorial-2024
+RUN chmod -R 777 ~/ /home/jovyan
 
 WORKDIR $HOME
 COPY ./docker/flux-icon.png $HOME/flux-icon.png

diff --git a/2024-RADIUSS-AWS/JupyterNotebook/docker/jupyter-launcher.yaml b/2024-RADIUSS-AWS/JupyterNotebook/docker/jupyter-launcher.yaml
@@ -1,19 +1,26 @@
-# These don't work, but we can try again next year.
-#- title: Flux Tutorial Notebook
-#  description: This is the main Flux Framework Tutorial
-#  source: /home/jovyan/flux-tutorial/notebook/flux.ipynb
-#  cwd: /home/jovyan/flux-tutorial/notebook/
-#  type: notebook
-#  catalog: Notebook
-#  icon: /home/jovyan/flux-icon.png
+- title: Flux Tutorial Notebook
+  description: This is the main Flux Framework Tutorial
+  type: jupyterlab-commands
+  icon: ./flux-icon.png
+  source:
+    - label: Flux Tutorial
+      id: 'filebrowser:open-path'
+      args:
+        path: 01_flux_tutorial.ipynb
+        icon: ./flux-icon.png
+  catalog: Notebook
 
-# - title: Dyad Notebook Tutorial
-#  description: This is a tutorial for using Dyad
-#  source: /home/jovyan/flux-tutorial/notebook/dyad.ipynb
-#  cwd: /home/jovyan/flux-tutorial/notebook/  
-#  type: notebook
-#  catalog: Notebook
-#  icon: /home/jovyan/flux-icon.png
+- title: Dyad Notebook Tutorial
+  description: This is a tutorial for using Dyad
+  type: jupyterlab-commands
+  icon: flux-icon.png
+  source:
+    - label: Dyad Tutorial
+      id: 'filebrowser:open-path'
+      args:
+        path: dyad_dlio.ipynb
+        icon: ./flux-icon.png
+  catalog: Notebook
 
 - title: Flux Framework Portal
   description: Flux Framework portal for projects, releases, and publication.

diff --git a/2024-RADIUSS-AWS/JupyterNotebook/flux-tree/flux-tree b/2024-RADIUSS-AWS/JupyterNotebook/flux-tree/flux-tree
@@ -212,7 +212,7 @@ jsonify() {
 
     if [[ "${avail}" = "yes" ]]
     then
-        avg=$(flux ion-resource stat | grep "Avg" | awk '{print $4}')
+        avg=$(flux ion-resource stats | grep "Avg" | awk '{print $4}')
         el_match=$(awk "BEGIN {print ${avg}*${njobs}*1000000.0}")
     fi