JHP-75, JHP-76, JHP-106, JHP-111: Update to JupyterHub and JupyterLab…

… 4.0 (#5)

JHP-75: Update to JupyterLab 4.0
JHP-76: Update to JupyterHub 4.0
JHP-106: Add check_allowed method to Authenticator 
JHP-111: Fix websocket 403 error in notebooks

.env

@@ -1,5 +1,5 @@
-JH_VERSION=3.0.0
-JH_DOCKERSPAWNER_VERSION=12.1.0
+JH_VERSION=4.0.2
+JH_DOCKERSPAWNER_VERSION=13.0.0
 JH_NETWORK=jupyterhub-network
 JH_START_TIMEOUT=180
 JH_XNAT_URL=http://172.17.0.1

.github/workflows/docker-image-ci.yml

@@ -61,6 +61,9 @@ jobs:
     runs-on: ubuntu-latest
 
     steps:
+      - name: Delete huge unnecessary tools folder
+        run: rm -rf /opt/hostedtoolcache
+
       - name: Checkout
         uses: actions/checkout@v4
 
@@ -107,6 +110,9 @@ jobs:
     runs-on: ubuntu-latest
 
     steps:
+      - name: Delete huge unnecessary tools folder
+        run: rm -rf /opt/hostedtoolcache
+
       - name: Checkout
         uses: actions/checkout@v4
 
@@ -144,6 +150,6 @@ jobs:
         with:
           context: dockerfiles/xnat-tensorflow-notebook
           push: true
-          platforms: linux/amd64
+          platforms: linux/amd64,linux/arm64
           tags: ${{ steps.meta.outputs.tags }}
           labels: ${{ steps.meta.outputs.labels }}

CHANGELOG.md

@@ -7,13 +7,23 @@ documented in the [xnat-jupyter-plugin](https://bitbucket.org/xnatx/xnat-jupyter
 
 ### Added
 
+- [JHP-75]: Update to JupyterLab 4.0.
+- [JHP-76]: Update to JupyterHub 4.0.
 - [JHP-96]: Add xnat/tensorflow-notebook image based on jupyter/tensorflow-notebook image. This image includes 
             TensorFlow and other helpful packages for working with XNAT data.
 - [JHP-101]: Add pyradiomics to xnat/datascience-notebook and xnat/tensorflow-notebook images. This package is useful for 
              extracting radiomic features from DICOM images.
 - [JHP-102]: Add highdicom to xnat/datascience-notebook. This package is useful for working with DICOM segmentation 
              objects and other DICOM objects.
 
+### Fixed
+
+- [JHP-106]: From the JupyterHub 4.0 upgrade, add check_allowed method to the Authenticator class. The default behavior
+             changes in version 5.0 so go ahead and add the method now instead of relying on the default behavior.
+- [JHP-111]: From the JuptyerHub 4.0 upgrade, fix websocket http 403 error in JupyterLab. This is fixed by adding the 
+             `JUPYTERHUB_SINGLEUSER_EXTENSION=0` environment variable to the JupyterHub deployment. Jupyter 5.0 has
+             fixed this issue, but we are not quite ready to upgrade to 5.0 yet.
+
 ## [1.2.0] - 2024-06-27
 
 ### Added
@@ -53,6 +63,8 @@ documented in the [xnat-jupyter-plugin](https://bitbucket.org/xnatx/xnat-jupyter
 
 [JHP-67]: https://radiologics.atlassian.net/jira/software/c/projects/JHP/issues/JHP-67
 [JHP-73]: https://radiologics.atlassian.net/jira/software/c/projects/JHP/issues/JHP-73
+[JHP-75]: https://radiologics.atlassian.net/jira/software/c/projects/JHP/issues/JHP-75
+[JHP-76]: https://radiologics.atlassian.net/jira/software/c/projects/JHP/issues/JHP-76
 [JHP-77]: https://radiologics.atlassian.net/jira/software/c/projects/JHP/issues/JHP-77
 [JHP-81]: https://radiologics.atlassian.net/jira/software/c/projects/JHP/issues/JHP-81
 [JHP-82]: https://radiologics.atlassian.net/jira/software/c/projects/JHP/issues/JHP-82
@@ -61,4 +73,6 @@ documented in the [xnat-jupyter-plugin](https://bitbucket.org/xnatx/xnat-jupyter
 [JHP-94]: https://radiologics.atlassian.net/jira/software/c/projects/JHP/issues/JHP-94
 [JHP-96]: https://radiologics.atlassian.net/jira/software/c/projects/JHP/issues/JHP-96
 [JHP-101]: https://radiologics.atlassian.net/jira/software/c/projects/JHP/issues/JHP-101
-[JHP-102]: https://radiologics.atlassian.net/jira/software/c/projects/JHP/issues/JHP-102
+[JHP-102]: https://radiologics.atlassian.net/jira/software/c/projects/JHP/issues/JHP-102
+[JHP-106]: https://radiologics.atlassian.net/jira/software/c/projects/JHP/issues/JHP-106
+[JHP-111]: https://radiologics.atlassian.net/jira/software/c/projects/JHP/issues/JHP-111

dockerfiles/jupyterhub/Dockerfile

@@ -1,7 +1,7 @@
-ARG JH_VERSION=3.0.0
+ARG JH_VERSION=4.0.2
 FROM jupyterhub/jupyterhub:$JH_VERSION
 LABEL maintainer="[email protected]"
-ARG JH_DOCKERSPAWNER_VERSION=12.1.0
+ARG JH_DOCKERSPAWNER_VERSION=13.0.0
 ENV JH_NETWORK="jupyterhub-network"
 ENV JH_XNAT_URL="http://172.17.0.1:80"
 ENV JH_XNAT_SERVICE_TOKEN="secret-token"

dockerfiles/jupyterhub/jupyterhub_config.py

@@ -55,6 +55,9 @@
         c.SwarmSpawner.environment.update(environment)
         c.SwarmSpawner.extra_container_spec = {'user': '0'}
 
+# Issue in JupyterHub 4.0 where cookie auth is not persisted from a token in url authenticated request
+c.SwarmSpawner.environment.update({'JUPYTERHUB_SINGLEUSER_EXTENSION': '0'})
+
 # Spawner config
 c.JupyterHub.spawner_class = 'dockerspawner.SwarmSpawner'
 c.Spawner.start_timeout = int(os.environ['JH_START_TIMEOUT'])
@@ -90,18 +93,41 @@ class XnatAuthenticator(Authenticator):
 
     async def authenticate(self, handler, data):
         xnat_url = f'{os.environ["JH_XNAT_URL"]}'
-        xnat_auth_api = f'{xnat_url}/data/services/auth'
-
-        logger.debug(f'User {data["username"]} is attempting to login.')
+        username, password = data["username"], data["password"]
 
-        response = requests.put(xnat_auth_api, data=f'username={data["username"]}&password={data["password"]}')
+        logger.info(f'User {username} is attempting to login.')
 
-        if response.status_code == 200:
-            logger.info(f'User {data["username"]} authenticated with XNAT.')
-            return {'name': data['username']}
-        else:
-            logger.info(f'Failed to authenticate user {data["username"]} with XNAT.')
+        # Authenticate user with XNAT
+        # If they can't access their own roles, they are not authenticated
+        roles_response = requests.get(f'{xnat_url}/xapi/users/{username}/roles', auth=(username, password))
+        if roles_response.status_code == 401:
+            logger.info(f'User {username} not authenticated with XNAT.')
             return None
+        elif not roles_response.ok:
+            logger.error(f'Failed to authenticate user {username} with XNAT. '
+                         f'Status code: {roles_response.status_code}. '
+                         f'Response: {roles_response.text}')
+            return None
+
+        logger.info(f'User {username} authenticated with XNAT.')
+
+        # Check if user has the jupyter role
+        if 'jupyter' in [role.lower() for role in roles_response.json()]:
+            logger.info(f'User {username} authorized to use Jupyter.')
+            return {'name': username, 'allowed': True}
+
+        # Check if allUsersCanStartJupyter preference is enabled
+        prefs_response = requests.get(f'{xnat_url}/xapi/jupyterhub/preferences/allUsersCanStartJupyter',
+                                      auth=(username, password))
+        if prefs_response.ok and prefs_response.json().get('allUsersCanStartJupyter'):
+            logger.info(f'User {username} authorized to use Jupyter.')
+            return {'name': username, 'allowed': True}
+
+        logger.info(f'User {username} not authorized to use Jupyter.')
+        return None
+
+    def check_allowed(self, username, authentication=None):
+        return authentication['allowed']
 
 
 c.JupyterHub.authenticator_class = XnatAuthenticator

dockerfiles/jupyterhub/templates/login.html

@@ -41,7 +41,7 @@ <h1>Sign in</h1>
       {{login_error}}
     </p>
     {% endif %}
-    <p id="" class="">Use your XNAT credentials to access Jupyter.</p>
+    <p id="" class="">Use your XNAT credentials to access JupyterHub. You must first be enabled by an XNAT administrator.</p>
     <input type="hidden" name="_xsrf" value="{{ xsrf }}"/>
     <label for="username_input" style="margin-top: 5px;">User</label>
     <input

dockerfiles/xnat-datascience-notebook/Dockerfile

@@ -1,4 +1,4 @@
-ARG JH_VERSION=3.0.0
+ARG JH_VERSION=4.0.2
 FROM jupyter/datascience-notebook:hub-$JH_VERSION
 
 USER root
@@ -13,7 +13,7 @@ USER ${NB_UID}
 COPY --chown=${NB_UID}:${NB_GID} requirements.txt /tmp/
 
 RUN python3 -m pip install --requirement /tmp/requirements.txt && \
-    jupyter lab build --minimize=False && \
+    jupyter lab build --dev-build=False --minimize=True && \
     fix-permissions "${CONDA_DIR}" && \
     fix-permissions "/home/${NB_USER}" && \
     rm /tmp/requirements.txt

dockerfiles/xnat-datascience-notebook/requirements.txt

@@ -2,14 +2,14 @@ pyxnat>=1.4
 xnat==0.4.3
 pydicom>=2.3.0
 papermill>=2.3.3
-jupyterlab-git<0.50.0
-voila<0.5.0
+jupyterlab-git>=0.50.0
+voila
 bqplot
 dash
 dash-bootstrap-components
 panel
 hvplot
-pyviz-comms<3.0.0
+pyviz-comms
 streamlit
 jupyter-server-proxy
 jhsingle-native-proxy
@@ -19,4 +19,5 @@ bokeh-root-cmd
 tciaclient
 ipydatagrid
 highdicom
-pyradiomics
+pyradiomics
+nbclassic<0.4

dockerfiles/xnat-tensorflow-notebook/Dockerfile

@@ -1,4 +1,4 @@
-ARG JH_VERSION=3.0.0
+ARG JH_VERSION=4.0.2
 FROM jupyter/tensorflow-notebook:hub-$JH_VERSION
 
 USER root
@@ -13,7 +13,7 @@ USER ${NB_UID}
 COPY --chown=${NB_UID}:${NB_GID} requirements.txt /tmp/
 
 RUN python3 -m pip install --requirement /tmp/requirements.txt && \
-    jupyter lab build --minimize=False && \
+    jupyter lab build --dev-build=False --minimize=True && \
     fix-permissions "${CONDA_DIR}" && \
     fix-permissions "/home/${NB_USER}" && \
     rm /tmp/requirements.txt

dockerfiles/xnat-tensorflow-notebook/requirements.txt

@@ -2,14 +2,14 @@ pyxnat>=1.4
 xnat==0.4.3
 pydicom>=2.3.0
 papermill>=2.3.3
-jupyterlab-git<0.50.0
-voila<0.5.0
+jupyterlab-git>=0.50.0
+voila
 bqplot
 dash
 dash-bootstrap-components
 panel
 hvplot
-pyviz-comms<3.0.0
+pyviz-comms
 streamlit
 jupyter-server-proxy
 jhsingle-native-proxy
@@ -19,4 +19,5 @@ bokeh-root-cmd
 tciaclient
 ipydatagrid
 highdicom
-pyradiomics
+pyradiomics
+nbclassic<0.4

xnat-jupyterhub-chart/Chart.yaml

@@ -11,6 +11,6 @@ dependencies:
   repository: "https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami"
   condition: postgresql.enabled
 - name: jupyterhub
-  version: "3.0.0" # jupyterhub app version 4.0.0
+  version: "3.2.1" # jupyterhub app version 4.0.2
   repository: "https://hub.jupyter.org/helm-chart/"
   condition: jupyterhub.enabled

xnat-jupyterhub-chart/files/pre_spawn_hook.py

@@ -64,6 +64,11 @@ def pre_spawn_hook(spawner):
                 'NB_USER': f'{spawner.user.name}'
             })
 
+            # Issue in JupyterHub 4.0 where cookie auth is not persisted from a token in url authenticated request
+            spawner.environment.update({
+                'JUPYTERHUB_SINGLEUSER_EXTENSION': '0'
+            })
+
             if 'mounts' in container_spec:
                 logger.debug(
                     f'Adding mounts to user {spawner.user.name} server {spawner.name} from XNAT. Mounts: {container_spec["mounts"]}')

xnat-jupyterhub-chart/files/xnat_authenticator.py

@@ -1,10 +1,8 @@
 import os
-import sys
 import requests
 import xnat_logger
 
 from jupyterhub.auth import Authenticator
-from requests.auth import HTTPBasicAuth
 
 # Logging config
 logger = xnat_logger.logger
@@ -19,15 +17,38 @@ class XnatAuthenticator(Authenticator):
 
     async def authenticate(self, handler, data):
         xnat_url = f'{os.environ["JH_XNAT_URL"]}'
-        xnat_auth_api = f'{xnat_url}/data/services/auth'
-
-        logger.debug(f'User {data["username"]} is attempting to login.')
-
-        response = requests.put(xnat_auth_api, data=f'username={data["username"]}&password={data["password"]}')
-
-        if response.status_code == 200:
-            logger.info(f'User {data["username"]} authenticated with XNAT.')
-            return {'name': data['username']}
-        else:
-            logger.info(f'Failed to authenticate user {data["username"]} with XNAT.')
-            return None
+        username, password = data["username"], data["password"]
+
+        logger.info(f'User {username} is attempting to login.')
+
+        # Authenticate user with XNAT
+        # If they can't access their own roles, they are not authenticated
+        roles_response = requests.get(f'{xnat_url}/xapi/users/{username}/roles', auth=(username, password))
+        if roles_response.status_code == 401:
+            logger.info(f'User {username} not authenticated with XNAT.')
+            return None
+        elif not roles_response.ok:
+            logger.error(f'Failed to authenticate user {username} with XNAT. '
+                         f'Status code: {roles_response.status_code}. '
+                         f'Response: {roles_response.text}')
+            return None
+
+        logger.info(f'User {username} authenticated with XNAT.')
+
+        # Check if user has the jupyter role
+        if 'jupyter' in [role.lower() for role in roles_response.json()]:
+            logger.info(f'User {username} authorized to use Jupyter.')
+            return {'name': username, 'allowed': True}
+
+        # Check if allUsersCanStartJupyter preference is enabled
+        prefs_response = requests.get(f'{xnat_url}/xapi/jupyterhub/preferences/allUsersCanStartJupyter',
+                                      auth=(username, password))
+        if prefs_response.ok and prefs_response.json().get('allUsersCanStartJupyter'):
+            logger.info(f'User {username} authorized to use Jupyter.')
+            return {'name': username, 'allowed': True}
+
+        logger.info(f'User {username} not authorized to use Jupyter.')
+        return None
+
+    def check_allowed(self, username, authentication=None):
+        return authentication['allowed']

xnat-jupyterhub-chart/values.yaml

@@ -22,7 +22,7 @@ jupyterhub:
 #          - admin
     image:
       name: "jupyterhub/k8s-hub"
-      tag: "3.0.0"
+      tag: "3.2.1"
       pullPolicy: Never
     extraEnv:
       # Some of these values are dependent on your XNAT deployment and may need to be changed