Merge pull request #40 from yambottle/DEV-530

README.md

@@ -1,15 +1,68 @@
-TODO - will make this a proper README.md later
-
-Pro
-- Break the chain of dependencies of self-managed base images
-  - Reduce image maintenance hell
-  - Use jupyter/minimal-notebook as base image instead
-- Compatible with previous supported feature as many as possible
-- Provide simpler way for mainteners to manage system dependencies and for users to install packages
-- Provide a way to config jupyter through environment variables
-- This can be used as jupyter lab, so we can archive djlab-docker and only focus on this repo
-- Add jupyterhub docker for single user image testing at local
-
-Con
-- Username if jovyan by default, consider this is a minor issue, since it's only visible in the terminal
-- Larger image size, this can be optimized when it becomes a problem
+# djlabhub-docker
+
+## Table of Contents
+  - [Introduction](#introduction)
+  - [Quick Start](#quick-start)
+  - [Build Jupyterhub Singleuser Image](#build-jupyterhub-singleuser-image)
+    - [Configuration](#configuration)
+    - [Build](#build)
+    - [Test in Local Jupyterhub](#test-in-local-jupyterhub)
+      - [Local Jupyterhub Configuration](#local-jupyterhub-configuration)
+      - [Start Local Jupyterhub](#start-local-jupyterhub)
+
+## Introduction
+
+This image is made for building customized jupyterhub profile/server option images. It was originally based on [datajoint/djlab](https://github.com/datajoint/djlab-docker) all the way up to [datajoint/djbase](https://github.com/datajoint/djbase-docker). To reduce the maintenance effort, we decided to use quay.io/jupyter/minimal-notebook as the base image. We also kept the previous implementation under `legacy` directory, just in case there are unforeseen issues with the new implementation.
+
+> **Note**: `datajoint/djlab` is deprecated since the `singleuser` part of this image overlaps with the `djlab` image, it'll run jupyter lab server by default.
+
+
+## Quick Start
+Directory explain:
+- `~/legacy` contains the old implementation of the image
+- `~/singleuser` is used to build the jupyterhub profile images
+- `~/hub` is a Docker based jupyterhub host server using DockerSpawner and Docker-in-Docker to launch jupyterhub singleuser server as a Docker container, in order to locally validate the singleuser images for development purpose.(**Don't recommend to use this in production, due to the security concern of Docker-in-Docker**)
+
+
+## Build Jupyterhub Singleuser Image
+
+### Configuration
+We put all the dependencies in the `~/singleuser/config` directory including:
+- `apt_install.sh` for system level dependencies
+- `pip_requirements.txt` for python packages
+- `before_start_hook.sh` to run before the jupyterhub singleuser server starts, [doc](https://jupyter-docker-stacks.readthedocs.io/en/latest/using/common.html#startup-hooks)
+- `jupyter**config.py` for jupyter related configurations
+  - Jupyter Server Config, [doc](https://jupyter-server.readthedocs.io/en/latest/other/full-config.html#other-full-config)
+  - Jupyter Notebook Server Config, [doc](https://jupyter-notebook.readthedocs.io/en/5.7.4/config.html)
+  - Jupyter Lab Server Config, [doc](https://jupyterlab-server.readthedocs.io/en/latest/api/app-config.html)
+  - Optionally, you can add more configurations for ipython kernel, etc.
+> **Note**: We added `jupyter**config.py` to extract configurations from the environment variables for some that are not supported to set by environment variables directly. The goal is to make the jupyterhub profile list clear and easy to maintain.
+
+### Build
+```
+# make the .env file from the example.env
+set -a && source .env && set +a
+docker compose build
+
+# To run this image as local jupyterlab server for testing
+# The default password is set by the `JUPYTER_SERVER_APP_PASSWORD` in the container
+docker compose up djlab
+```
+
+### Test in Local Jupyterhub
+#### Local Jupyterhub Configuration
+Similarly to the singleuser image, we the `jupyterhub_config.py` in the `~/hub/config` directory, you need to modify the `c.DockerSpawner.container_image` and `c.DockerSpawner.environment` to configure your singleuser server container.
+
+#### Start Local Jupyterhub
+> **Note**: Local Jupyterhub requires jupyterhub host container to mount `/var/run/docker.sock:/var/run/docker.sock` to enable Docker-in-Docker, [doc](https://devopscube.com/run-docker-in-docker/) 
+```
+# make the .env file from the example.env
+# OAUTH2 related configurations are not necessary
+# since the c.JupyterHub.authenticator_class = "jupyterhub.auth.DummyAuthenticator"
+set -a && source .env && set +a
+
+# It takes any random username and password
+docker compose up
+```
+
+

hub/config/jupyterhub_config.py

@@ -77,7 +77,7 @@
 # https://github.com/jupyterhub/jupyterhub/issues/2913#issuecomment-580535422
 c.Spawner.http_timeout = 60
 c.Spawner.start_timeout = 60
-c.DockerSpawner.container_image = "datajoint/djlabhub:singleuser-4.0.2-py3.10-qa"
+c.DockerSpawner.container_image = "datajoint/djlabhub:singleuser-4.0.2-py3.10"
 
 c.DockerSpawner.environment = {
     ## Jupyter Official Environment Variables

singleuser/Dockerfile

@@ -1,9 +1,14 @@
 ARG JUPYTERHUB_VERSION
 FROM quay.io/jupyter/minimal-notebook:hub-${JUPYTERHUB_VERSION}
 
-COPY ./config /tmp/config
+ARG PYTHON_VERSION
+RUN if [ "$(python -c 'import sys; print(".".join(map(str, sys.version_info[:2])))')" != "${PYTHON_VERSION}" ]; then \
+    echo "Installing python ${PYTHON_VERSION}.." && conda install --quiet --no-pin --yes python=${PYTHON_VERSION}; \
+    else echo "Python version matching"; \
+    fi
 
 USER root
+COPY ./config /tmp/config
 RUN \
     # Install dependencies: apt
     bash /tmp/config/apt_install.sh \
@@ -13,6 +18,7 @@ RUN \
     # Add jupyter*config*.py
     && cp /tmp/config/jupyter*config*.py /etc/jupyter/
 
+USER $NB_UID
 RUN \
     # remove default work directory
     [ -d "/home/jovyan/work" ] && rm -r /home/jovyan/work \

singleuser/docker-compose.yaml

@@ -1,12 +1,12 @@
-version: '3.7'
 services:
   singleuser:
     build:
       context: .
       dockerfile: Dockerfile
       args:
         - JUPYTERHUB_VERSION
-    image: datajoint/djlabhub:singleuser-${JUPYTERHUB_VERSION}
+        - PYTHON_VERSION
+    image: datajoint/djlabhub:singleuser-${JUPYTERHUB_VERSION}-py${PYTHON_VERSION}
     container_name: djlabhub-singleuser
     env_file: .env
     ports:
@@ -18,4 +18,4 @@ services:
   djlab:
     extends: singleuser
     container_name: djlab
-    image: datajoint/djlab:inherit-hub-${JUPYTERHUB_VERSION}
+    image: datajoint/djlab:inherit-hub-${JUPYTERHUB_VERSION}-py${PYTHON_VERSION}

singleuser/example.env

@@ -1,5 +1,5 @@
 ## Build Args
-PYTHON_VERSION=3.10
+PYTHON_VERSION=3.11
 JUPYTERHUB_VERSION=4.0.2
 
 ## Jupyter Official Environment Variables