Merge branch 'master' of https://github.com/bird-house/birdhouse-deploy…

… into stac-hooks

.bumpversion.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 1.33.5
+current_version = 1.35.2
 commit = True
 tag = False
 tag_name = {new_version}
@@ -30,11 +30,11 @@ search = {current_version}
 replace = {new_version}
 
 [bumpversion:file:RELEASE.txt]
-search = {current_version} 2023-10-02T15:13:02Z
+search = {current_version} 2023-10-24T21:05:12Z
 replace = {new_version} {utcnow:%Y-%m-%dT%H:%M:%SZ}
 
 [bumpversion:part:releaseTime]
-values = 2023-10-02T15:13:02Z
+values = 2023-10-24T21:05:12Z
 
 [bumpversion:file(version):birdhouse/config/canarie-api/docker_configuration.py.template]
 search = 'version': '{current_version}'

CHANGES.md

@@ -25,17 +25,61 @@
   - Checking same instance `source` path is necessary because `STAC` could refer to external assets, and we do not want
     to inject Magpie resource that are not part of the active instance where the hook is running.
 
+[1.35.2](https://github.com/bird-house/birdhouse-deploy/tree/1.35.2) (2023-10-24)
+------------------------------------------------------------------------------------------------------------------
+
+## Fixes
+
+- Fix warning from JupyterHub regarding DockerSpawner method never awaited.
+  - [`DockerSpawner.start`](
+    https://github.com/jupyterhub/dockerspawner/blob/a6bf72e7/dockerspawner/dockerspawner.py#L1246) is defined
+    as `async`. Therefore, `async def` and `await super().start()` where not properly invoked by `CustomDockerSpawner`
+    in [`jupyterhub_config.py.template`](./birdhouse/config/jupyterhub/jupyterhub_config.py.template).
+
+[1.35.1](https://github.com/bird-house/birdhouse-deploy/tree/1.35.1) (2023-10-18)
+------------------------------------------------------------------------------------------------------------------
+
+## Fixes
+
+- Jupyterhub cull interval setting must be an integer:
+  - Previously, the default `jupyter_idle_kernel_cull_interval` setting is calculated by dividing the 
+    `jupyter_idle_kernel_cull_timeout` setting by 2 using float division. This meant that the result was a float 
+    instead of the expected integer value. This caused and error when the jupyterlab server spawned.
+    In order to fix this, the value is cast to an integer after division.
+
+[1.35.0](https://github.com/bird-house/birdhouse-deploy/tree/1.35.0) (2023-10-16)
+------------------------------------------------------------------------------------------------------------------
+
+## Changes
+- Jupyterhub configurable idle server culling.
+  - Add optional variables `JUPYTER_IDLE_SERVER_CULL_TIMEOUT`, `JUPYTER_IDLE_KERNEL_CULL_TIMEOUT` and
+    `JUPYTER_IDLE_KERNEL_CULL_INTERVAL` that allows fined-grained configuration of user-kernel and server-wide
+    docker image culling when their activity status reached a certain idle timeout threshold.
+  - Enable idle kernel culling by default with a timeout of 1 day, and user server culling with timeout of 3 days.
+  - Avoids the need for custom `JUPYTERHUB_CONFIG_OVERRIDE` specifically for idle server culling.
+    If similar argument parameters should be defined using an older `JUPYTERHUB_CONFIG_OVERRIDE` definition,
+    the new configuration strategy can be skipped by setting `JUPYTER_IDLE_KERNEL_CULL_TIMEOUT=0`.
+
+[1.34.0](https://github.com/bird-house/birdhouse-deploy/tree/1.34.0) (2023-10-10)
+------------------------------------------------------------------------------------------------------------------
+
+## Changes
+- Allow users to submit a Weaver job requesting to store outputs to the public location instead of their user-workspace.
+- Update default Weaver version from [4.22.0](https://github.com/crim-ca/weaver/tree/4.22.0)
+  to [4.32.0](https://github.com/crim-ca/weaver/tree/4.32.0).
+- Add `COWBIRD_LOG_LEVEL` environment variable to allow control over logging level of Cowbird services.
+
 [1.33.5](https://github.com/bird-house/birdhouse-deploy/tree/1.33.5) (2023-10-02)
 ------------------------------------------------------------------------------------------------------------------
 
-## CHANGES
+## Changes
 
 - Adding a description for the STAC service that will be served at the `/services` endpoint
 
 [1.33.4](https://github.com/bird-house/birdhouse-deploy/tree/1.33.4) (2023-10-02)
 ------------------------------------------------------------------------------------------------------------------
 
-# Fixes
+## Fixes
 - Clean up: Make bind-mount locations more flexible
 
   Clean up unused variables and correct file paths from the changes made in 1.33.2

Makefile

@@ -1,7 +1,7 @@
 # Generic variables
 override SHELL       := bash
 override APP_NAME    := birdhouse-deploy
-override APP_VERSION := 1.33.5
+override APP_VERSION := 1.35.2
 
 # utility to remove comments after value of an option variable
 override clean_opt = $(shell echo "$(1)" | $(_SED) -r -e "s/[ '$'\t'']+$$//g")

README.rst

@@ -14,13 +14,13 @@ for a full-fledged production platform.
     * - releases
       - | |latest-version| |commits-since|
 
-.. |commits-since| image:: https://img.shields.io/github/commits-since/bird-house/birdhouse-deploy/1.33.5.svg
+.. |commits-since| image:: https://img.shields.io/github/commits-since/bird-house/birdhouse-deploy/1.35.2.svg
     :alt: Commits since latest release
-    :target: https://github.com/bird-house/birdhouse-deploy/compare/1.33.5...master
+    :target: https://github.com/bird-house/birdhouse-deploy/compare/1.35.2...master
 
-.. |latest-version| image:: https://img.shields.io/badge/tag-1.33.5-blue.svg?style=flat
+.. |latest-version| image:: https://img.shields.io/badge/tag-1.35.2-blue.svg?style=flat
     :alt: Latest Tag
-    :target: https://github.com/bird-house/birdhouse-deploy/tree/1.33.5
+    :target: https://github.com/bird-house/birdhouse-deploy/tree/1.35.2
 
 .. |readthedocs| image:: https://readthedocs.org/projects/birdhouse-deploy/badge/?version=latest
     :alt: ReadTheDocs Build Status (latest version)

RELEASE.txt

@@ -1 +1 @@
-1.33.5 2023-10-02T15:13:02Z
+1.35.2 2023-10-24T21:05:12Z

birdhouse/components/cowbird/config/cowbird/cowbird.ini.template

@@ -80,13 +80,13 @@ handlers = console
 formatter = generic
 
 [logger_cowbird]
-level = INFO
+level = ${COWBIRD_LOG_LEVEL}
 handlers =
 qualname = cowbird
 formatter = generic
 
 [logger_celery]
-level = INFO
+level = ${COWBIRD_LOG_LEVEL}
 handlers = console
 qualname = celery
 propagate = 0

birdhouse/components/cowbird/default.env

@@ -15,6 +15,7 @@ EXTRA_VARS='
     ${COWBIRD_MONGODB_VERSION}
     ${COWBIRD_MONGODB_HOST}
     ${COWBIRD_MONGODB_PORT}
+    ${COWBIRD_LOG_LEVEL}
     ${USER_WORKSPACES}
     ${PUBLIC_WORKSPACE_WPS_OUTPUTS_SUBDIR}
     ${SECURE_DATA_PROXY_NAME}
@@ -39,6 +40,12 @@ export COWBIRD_MONGODB_VERSION=5.0
 export COWBIRD_MONGODB_HOST=cowbird-mongodb
 export COWBIRD_MONGODB_PORT=27017
 
+# Logging level for Cowbird
+#   DEBUG:  logs detailed information about operations/settings (not for production, could leak sensitive data)
+#   INFO:   reports useful information, not leaking details about settings
+#   WARN:   only potential problems/unexpected results reported
+export COWBIRD_LOG_LEVEL=INFO
+
 # Subdirectory of DATA_PERSIST_SHARED_ROOT containing the user workspaces used by Cowbird
 export USER_WORKSPACES="user_workspaces"
 

birdhouse/components/weaver/.gitignore

@@ -1,4 +1,5 @@
 config/magpie/config.yml
+config/magpie/weaver_hooks.py
 config/weaver/data_sources.yml
 config/weaver/request_options.yml
 config/weaver/wps_processes.yml

birdhouse/components/weaver/config/magpie/weaver_hooks.py → birdhouse/components/weaver/config/magpie/weaver_hooks.py.template

@@ -30,7 +30,24 @@
 
     from magpie.adapter import HookContext
 
-LOGGER = get_logger("birdhouse-weaver-hooks")
+LOGGER = get_logger("magpie.weaver-hooks")
+
+# NOTE:
+#  This value should correspond to the directory defined for public data on WPS outputs.
+#  The output hierarchy would be:
+#
+#       ${WEAVER_WPS_OUTPUTS_DIR}/
+#           users/
+#               <user-id>/
+#                   <job-id>/
+#           ${PUBLIC_WORKSPACE_WPS_OUTPUTS_SUBDIR}/
+#               <job-id>/
+#
+PUBLIC_WORKSPACE_WPS_OUTPUTS_SUBDIR = "${PUBLIC_WORKSPACE_WPS_OUTPUTS_SUBDIR}" or "public"
+PUBLIC_USER_CONTEXTS = {
+    "public",  # shortcut notation to avoid instance-specific configuration
+    PUBLIC_WORKSPACE_WPS_OUTPUTS_SUBDIR,
+}
 
 
 def is_admin(request):
@@ -47,20 +64,16 @@ def add_x_wps_output_context(request):
     Apply the ``X-WPS-Output-Context`` for saving outputs in the user-context WPS-outputs directory.
     """
     header = get_header("X-WPS-Output-Context", request.headers)
-    # if explicitly provided, ensure it is permitted (admin allow any, otherwise self-user reference only)
-    if header is not None:
-        if request.user is None:
-            header = "public"
-        else:
-            if not is_admin(request):
-                # override disallowed writing to other location
-                # otherwise, up to admin to have written something sensible
-                header = "users/" + str(request.user.id)
-    else:
-        if request.user is None:
-            header = "public"
-        else:
-            header = "users/" + str(request.user.id)
+    # if no user logged in, ignore specified value to avoid writing in someone else's user-workspace
+    # if explicitly requested to make their job output public right away, whether logged in or not,
+    # use the public location instead of specific user-workspace
+    if request.user is None or header in PUBLIC_USER_CONTEXTS:
+        header = PUBLIC_WORKSPACE_WPS_OUTPUTS_SUBDIR
+    # if explicitly provided, ensure it is permitted, and if not provided, default to current user
+    # admin-level user is allowed to write anywhere (up to them to use a sensible location)
+    # otherwise, enforce the current user-workspace even if provided explicitly
+    elif not header or not is_admin(request):
+        header = "users/" + str(request.user.id)
     request.headers["X-WPS-Output-Context"] = header
     return request
 

birdhouse/components/weaver/default.env

@@ -47,7 +47,7 @@ OPTIONAL_VARS="
 export WEAVER_CONFIG=HYBRID
 
 # default release version that will be used to fetch docker images (API mananger & celery workers services)
-export WEAVER_VERSION=4.22.0
+export WEAVER_VERSION=4.32.0
 
 # default release of the MongoDB version employed by Weaver
 # NOTE:

birdhouse/config/canarie-api/docker_configuration.py.template

@@ -109,8 +109,8 @@ SERVICES = {
             # NOTE:
             #   Below version and release time auto-managed by 'make VERSION=x.y.z bump'.
             #   Do NOT modify it manually. See 'Tagging policy' in 'birdhouse/README.rst'.
-            'version': '1.33.5',
-            'releaseTime': '2023-10-02T15:13:02Z',
+            'version': '1.35.2',
+            'releaseTime': '2023-10-24T21:05:12Z',
             'institution': 'Ouranos',
             'researchSubject': 'Climatology',
             'supportEmail': '${SUPPORT_EMAIL}',
@@ -142,8 +142,8 @@ PLATFORMS = {
             # NOTE:
             #   Below version and release time auto-managed by 'make VERSION=x.y.z bump'.
             #   Do NOT modify it manually. See 'Tagging policy' in 'birdhouse/README.rst'.
-            'version': '1.33.5',
-            'releaseTime': '2023-10-02T15:13:02Z',
+            'version': '1.35.2',
+            'releaseTime': '2023-10-24T21:05:12Z',
             'institution': 'Ouranos',
             'researchSubject': 'Climatology',
             'supportEmail': '${SUPPORT_EMAIL}',

birdhouse/config/jupyterhub/default.env

@@ -43,6 +43,18 @@ export JUPYTER_LOGIN_BANNER_BOTTOM_SECTION=""
 # server for the change to take effect.
 export JUPYTERHUB_README=""
 
+# Timeout (in seconds, default: 3 days) to shut down the user server when no kernels or terminals
+# are running and there is no activity. If undefined or set to zero, the feature will not be enabled.
+export JUPYTER_IDLE_SERVER_CULL_TIMEOUT=259200
+# Timeout (in seconds, default: 1 day) after which individual
+# user kernels/terminals are considered idle and ready to be culled.
+export JUPYTER_IDLE_KERNEL_CULL_TIMEOUT=86400
+# Interval (in seconds) on which to check for idle kernels exceeding the cull timeout value.
+# Enabled only if 'JUPYTER_IDLE_KERNEL_CULL_TIMEOUT' is provided and greater than zero.
+# If this value is not provided, equal to zero, or is set higher than 'JUPYTER_IDLE_KERNEL_CULL_TIMEOUT',
+# it will be automatically reduced by half of the timeout value to ensure that it can be effective.
+export JUPYTER_IDLE_KERNEL_CULL_INTERVAL=0
+
 # Allow for adding new config or override existing config in
 # config/jupyterhub/jupyterhub_config.py.template.
 export JUPYTERHUB_CONFIG_OVERRIDE=""
@@ -68,6 +80,9 @@ OPTIONAL_VARS="
   \$JUPYTERHUB_CONFIG_OVERRIDE
   \$JUPYTERHUB_DOCKER
   \$JUPYTERHUB_VERSION
+  \$JUPYTER_IDLE_SERVER_CULL_TIMEOUT
+  \$JUPYTER_IDLE_KERNEL_CULL_TIMEOUT
+  \$JUPYTER_IDLE_KERNEL_CULL_INTERVAL
 "
 
 # add any component that this component requires to run

birdhouse/config/jupyterhub/jupyterhub_config.py.template

@@ -23,7 +23,7 @@ c.JupyterHub.db_url = '/persist/jupyterhub.sqlite'
 c.JupyterHub.template_paths = ['/custom_templates']
 
 class CustomDockerSpawner(DockerSpawner):
-    def start(self):
+    async def start(self):
         if(os.environ['MOUNT_IMAGE_SPECIFIC_NOTEBOOKS'] == 'true'):
             host_dir = join(os.environ['JUPYTERHUB_USER_DATA_DIR'], 'tutorial-notebooks-specific-images')
 
@@ -50,7 +50,7 @@ class CustomDockerSpawner(DockerSpawner):
                 "bind": "/notebook_dir/tutorial-notebooks",
                 "mode": "ro"
             }
-        return super().start()
+        return await super().start()
 
 c.JupyterHub.spawner_class = CustomDockerSpawner
 
@@ -187,4 +187,38 @@ blocked_users = {'authtest', '${CATALOG_USERNAME}', 'anonymous'}
 c.Authenticator.blacklist = blocked_users  # v0.9+
 c.Authenticator.blocked_users = blocked_users  # v1.2+
 
+
+# ------------------------------------------------------------------------------
+# Shutdown idle user server based on configured timeouts.
+# ------------------------------------------------------------------------------
+# Timeout (in seconds, default: 3 days) to shut down the user server when no kernels or terminals
+# are running and there is no activity. If undefined or set to zero, the feature will not be enabled.
+jupyter_idle_server_cull_timeout = int("${JUPYTER_IDLE_SERVER_CULL_TIMEOUT}" or 0)
+if jupyter_idle_server_cull_timeout:
+    c.Spawner.args.append('--NotebookApp.shutdown_no_activity_timeout={}'.format(jupyter_idle_server_cull_timeout))
+# Timeout (in seconds, default: 1 day) after which individual
+# user kernels/terminals are considered idle and ready to be culled.
+jupyter_idle_kernel_cull_timeout = int("${JUPYTER_IDLE_KERNEL_CULL_TIMEOUT}" or 0)
+# Interval (in seconds, default: half of timeout) on which to check for idle kernels exceeding the cull timeout value.
+jupyter_idle_kernel_cull_interval = int("${JUPYTER_IDLE_KERNEL_CULL_INTERVAL}" or 0)
+if jupyter_idle_kernel_cull_timeout:
+    if not jupyter_idle_kernel_cull_interval or jupyter_idle_kernel_cull_interval > jupyter_idle_kernel_cull_timeout:
+        jupyter_idle_kernel_cull_interval = max(1, int(jupyter_idle_kernel_cull_timeout / 2))
+    c.Spawner.args.extend([
+        '--MappingKernelManager.cull_idle_timeout={}'.format(jupyter_idle_kernel_cull_timeout),
+        '--MappingKernelManager.cull_interval={}'.format(jupyter_idle_kernel_cull_interval),
+        '--TerminalManager.cull_inactive_timeout={}'.format(jupyter_idle_kernel_cull_timeout),
+        '--TerminalManager.cull_interval={}'.format(jupyter_idle_kernel_cull_interval),
+    ])
+# Culling kernels which have one or more connections for idle but open notebooks and/or terminals.
+# Otherwise, browser tabs, notebooks and terminals all have to be closed for culling to work.
+if jupyter_idle_server_cull_timeout or jupyter_idle_kernel_cull_timeout:
+    c.Spawner.args.extend([
+        '--MappingKernelManager.cull_connected=True',
+        '--TerminalManager.cull_connected=True',
+    ])
+
+# ------------------------------------------------------------------------------
+# Configuration overrides
+# ------------------------------------------------------------------------------
 ${JUPYTERHUB_CONFIG_OVERRIDE}    # noqa