Follow-up #47

[ethho]

singleuser CD fails when attempting to checkout submodule, e.g. this CD job.

To fix, we need to run the checkout step with submodule: true, as we do with works-api CD: https://github.com/datajoint/djlabhub-docker
The value of SSH_PRIVATE_KEY is stored as ipython-datajoint-creds-updater djlabhub in LastPass.
I set the GitHub secret using:

gh secret set SSH_PRIVATE_KEY < /home/eho/.ssh/ipython-datajoint-creds-updater-djlabhub

[ethho]

Triggered successful build https://github.com/datajoint/djlabhub-docker/actions/runs/9893617659

[ethho]

Using image datajoint/djlabhub:singleuser-4.0.2-py3.10-ad87a29, pod initialization fails in QA (sciops-codebook.datajoint.io):

INFO::Datajoint Startup Hook
INFO::Changing Markdown Preview to preview as default
INFO::Cloning repo https://github.com/datajoint/datajoint-tutorials.git
fatal: destination path '/home/jovyan/datajoint-tutorials' already exists and is not an empty directory.
WARNING::Failed to clone https://github.com/datajoint/datajoint-tutorials.git. Continuing...
INFO::Switch to branch main
fatal: invalid reference: main
WARNING::Failed to checkout branch main. Continuing...
total 48K
drwsrws--- 1 jovyan users 4.0K Jul 11 18:48 .
drwxr-xr-x 1 root   root  4.0K Mar 19 10:13 ..
-rw-rw-r-- 1 jovyan users  220 Jan  6  2022 .bash_logout
-rw-rw-r-- 1 jovyan users 3.8K Mar 19 10:13 .bashrc
drwxr-sr-x 1 jovyan users 4.0K Jul 11 18:44 .cache
drwsrwsr-x 1 jovyan users 4.0K Mar 19 10:15 .conda
drwsrws--- 2 jovyan users 4.0K Mar 19 10:15 .jupyter
drwsrwsr-x 3 jovyan users 4.0K Mar 19 10:15 .npm
-rw-rw-r-- 1 jovyan users  807 Jan  6  2022 .profile
-rw-r--r-- 1 root   users  165 Jul 11 18:44 .wget-hsts
drwxr-sr-x 3 jovyan users 4.0K Jul 11 18:48 datajoint-tutorials
total 12K
drwxr-sr-x 3 jovyan users 4.0K Jul 11 18:48 .
drwsrws--- 1 jovyan users 4.0K Jul 11 18:48 ..
drwxr-sr-x 7 jovyan users 4.0K Jul 11 18:48 .git
INFO::Installing repo
Obtaining file:///home/jovyan/datajoint-tutorials
ERROR: file:///home/jovyan/datajoint-tutorials does not appear to be a Python project: neither 'setup.py' nor 'pyproject.toml' found.

Building (from job https://github.com/datajoint/djlabhub-docker/actions/runs/9897039148/job/27340619425_ the last working image from commit 0cd58c9: we get the same "failed start hook" error:

❯ kcl -n jhub-test logs jupyter-ioc-2duser -f             
Defaulted container "notebook" out of: notebook, block-cloud-metadata (init)
Entered start.sh with args: jupyterhub-singleuser --NotebookApp.tornado_settings={"headers":{"Content-Security-Policy": "frame-ancestors * self https://works-qa.datajoint.io https://fakeservices.datajoint.io", "Access-Control-Allow-Origin": "*"}}
Running hooks in: /usr/local/bin/start-notebook.d as uid: 1000 gid: 100
Done running hooks in: /usr/local/bin/start-notebook.d
Running hooks in: /usr/local/bin/before-notebook.d as uid: 1000 gid: 100
Sourcing shell script: /usr/local/bin/before-notebook.d/10activate-conda-env.sh
Sourcing shell script: /usr/local/bin/before-notebook.d/before_start_hook.sh
INFO::Datajoint Startup Hook
INFO::Changing Markdown Preview to preview as default
INFO::Cloning repo https://github.com/datajoint/datajoint-tutorials.git
fatal: destination path '/home/jovyan/datajoint-tutorials' already exists and is not an empty directory.
❯ kcl -n jhub-test describe pod jupyter-ioc-2duser 
Name:                 jupyter-ioc-2duser
Namespace:            jhub-test
Priority:             0
Priority Class Name:  jhub-test-default-priority
Service Account:      default
Node:                 i-0dc22c2272e76d718/172.20.45.254
Start Time:           Thu, 11 Jul 2024 14:06:16 -0500
Labels:               app=jupyterhub
                      chart=jupyterhub-3.2.1
                      component=singleuser-server
                      heritage=jupyterhub
                      hub.jupyter.org/network-access-hub=true
                      hub.jupyter.org/servername=
                      hub.jupyter.org/username=ioc-2duser
                      release=jhub-test
Annotations:          hub.jupyter.org/username: ioc-user
Status:               Running
IP:                   100.96.4.56
IPs:
  IP:  100.96.4.56
Init Containers:
  block-cloud-metadata:
    Container ID:  containerd://8fe2f471518ecd1f8ff7dda6e1d27405ce2962ba7430dc0064b97c23f556e240
    Image:         quay.io/jupyterhub/k8s-network-tools:3.2.1
    Image ID:      quay.io/jupyterhub/k8s-network-tools@sha256:e532a2dc4761685cedc87d4c0df864b8b1fad45f4dd54535b1b5dff23897d870
    Port:          <none>
    Host Port:     <none>
    Command:
      iptables
      --append
      OUTPUT
      --protocol
      tcp
      --destination
      169.254.169.254
      --destination-port
      80
      --jump
      DROP
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 11 Jul 2024 14:06:17 -0500
      Finished:     Thu, 11 Jul 2024 14:06:17 -0500
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:         <none>
Containers:
  notebook:
    Container ID:  containerd://77ad065a6de6da1c6736348c2ee2a857f3f123700dbda88af581cb1708c5ef60
    Image:         datajoint/djlabhub:singleuser-4.0.2-py3.10
    Image ID:      docker.io/datajoint/djlabhub@sha256:c2accf55bc6e7308af3de849472ceb132c2fa66eacd28ae01f7e62e9e9c02f78
    Port:          8888/TCP
    Host Port:     0/TCP
    Args:
      jupyterhub-singleuser
      --NotebookApp.tornado_settings={"headers":{"Content-Security-Policy": "frame-ancestors * self https://works-qa.datajoint.io https://fakeservices.datajoint.io", "Access-Control-Allow-Origin": "*"}}
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    128
      Started:      Thu, 11 Jul 2024 14:07:13 -0500
      Finished:     Thu, 11 Jul 2024 14:07:14 -0500
    Ready:          False
    Restart Count:  2
    Limits:
      cpu:     1
      memory:  1073741824
    Requests:
      cpu:     500m
      memory:  524288k
    Environment:
      CPU_GUARANTEE:                              0.5
      CPU_LIMIT:                                  1.0
      DJLABHUB_REPO:                              https://github.com/datajoint/datajoint-tutorials.git
      DJLABHUB_REPO_BRANCH:                       main
      DJLABHUB_REPO_INSTALL:                      TRUE
      JPY_API_TOKEN:                              db112c760e274bb6b0711296320a0227
      JUPYTERHUB_ACTIVITY_URL:                    http://hub:8081/hub/api/users/ioc-user/activity
      JUPYTERHUB_ADMIN_ACCESS:                    1
      JUPYTERHUB_API_TOKEN:                       db112c760e274bb6b0711296320a0227
      JUPYTERHUB_API_URL:                         http://hub:8081/hub/api
      JUPYTERHUB_BASE_URL:                        /
      JUPYTERHUB_CLIENT_ID:                       jupyterhub-user-ioc-user
      JUPYTERHUB_COOKIE_OPTIONS:                  {"SameSite": "None", "Secure": true}
      JUPYTERHUB_DEBUG:                           1
      JUPYTERHUB_DEFAULT_URL:                     /lab
      JUPYTERHUB_HOST:                            
      JUPYTERHUB_OAUTH_ACCESS_SCOPES:             ["access:servers!server=ioc-user/", "access:servers!user=ioc-user"]
      JUPYTERHUB_OAUTH_CALLBACK_URL:              /user/ioc-user/oauth_callback
      JUPYTERHUB_OAUTH_CLIENT_ALLOWED_SCOPES:     []
      JUPYTERHUB_OAUTH_SCOPES:                    ["access:servers!server=ioc-user/", "access:servers!user=ioc-user"]
      JUPYTERHUB_SERVER_NAME:                     
      JUPYTERHUB_SERVICE_PREFIX:                  /user/ioc-user/
      JUPYTERHUB_SERVICE_URL:                     http://0.0.0.0:8888/user/ioc-user/
      JUPYTERHUB_USER:                            ioc-user
      JUPYTER_FILE_CONTENTS_MANAGER_ROOT_DIR:     /home/jovyan
      JUPYTER_FILE_CONTENTS_MANAGER_SAVE_OUTPUT:  TRUE
      JUPYTER_IMAGE:                              datajoint/djlabhub:singleuser-4.0.2-py3.10
      JUPYTER_IMAGE_SPEC:                         datajoint/djlabhub:singleuser-4.0.2-py3.10
      JUPYTER_YDOCEXTENSION_DISABLE_RTC:          TRUE
      MEM_GUARANTEE:                              524288000
      MEM_LIMIT:                                  1073741824
    Mounts:                                       <none>
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:            <none>
QoS Class:          Burstable
Node-Selectors:     <none>
Tolerations:        hub.jupyter.org/dedicated=user:NoSchedule
                    node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                    node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason               Age                From                      Message
  ----     ------               ----               ----                      -------
  Normal   Scheduled            75s                jhub-test-user-scheduler  Successfully assigned jhub-test/jupyter-ioc-2duser to i-0dc22c2272e76d718
  Normal   Pulled               75s                kubelet                   Container image "quay.io/jupyterhub/k8s-network-tools:3.2.1" already present on machine
  Normal   Created              75s                kubelet                   Created container block-cloud-metadata
  Normal   Started              75s                kubelet                   Started container block-cloud-metadata
  Normal   Pulled               41s                kubelet                   Successfully pulled image "datajoint/djlabhub:singleuser-4.0.2-py3.10" in 34.306s (34.306s including waiting)
  Normal   Pulled               37s                kubelet                   Successfully pulled image "datajoint/djlabhub:singleuser-4.0.2-py3.10" in 211ms (211ms including waiting)
  Normal   Created              19s (x3 over 41s)  kubelet                   Created container notebook
  Normal   Started              19s (x3 over 41s)  kubelet                   Started container notebook
  Normal   Pulling              19s (x3 over 75s)  kubelet                   Pulling image "datajoint/djlabhub:singleuser-4.0.2-py3.10"
  Normal   Pulled               19s                kubelet                   Successfully pulled image "datajoint/djlabhub:singleuser-4.0.2-py3.10" in 198ms (198ms including waiting)
  Warning  FailedPostStartHook  18s (x3 over 40s)  kubelet                   PostStartHook failed
  Normal   Killing              18s (x3 over 40s)  kubelet                   FailedPostStartHook
  Warning  BackOff              17s (x4 over 35s)  kubelet                   Back-off restarting failed container notebook in pod jupyter-ioc-2duser_jhub-test(d5692d25-56c6-4b98-9b5e-d007a5950732)

[ethho]

Pod initialization was failing due to the presence of a postStart.command in the z2jh config:

https://github.com/datajoint-company/dj-gitops/blob/99a243002620e9aea70a44c451c5f77333ae6897/applications/k8s/test-deployments/codebook-sciops/deployment.yaml#L425-L432

Pod initialized successfully with the image built by https://github.com/datajoint/djlabhub-docker/actions/runs/9902160296 and testing on QA codebook on commit https://github.com/datajoint-company/dj-gitops/pull/343/commits/d3529072eb90d8fcc8bbe68fc7d09dfa789af7e1