Skip to content

Commit

Permalink
Document JupyterHub Storage Increase Process (#3240)
Browse files Browse the repository at this point in the history
  • Loading branch information
SorenSpicknall authored Jan 22, 2024
1 parent a56525c commit 368a2e3
Show file tree
Hide file tree
Showing 2 changed files with 27 additions and 10 deletions.
31 changes: 21 additions & 10 deletions docs/analytics_tools/jupyterhub.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,15 +12,16 @@ Analyses on JupyterHub are accomplished using notebooks, which allow users to mi

## Table of Contents

1. [Using JupyterHub](#using-jupyterhub)
2. [Logging in to JupyterHub](#logging-in-to-jupyterhub)
3. [Connecting to the Warehouse](#connecting-to-the-warehouse)
4. [Increasing the Query Limit](#increasing-the-query-limit)
5. [Querying with SQL in JupyterHub](querying-sql-jupyterhub)
6. [Saving Code to Github](saving-code-jupyter)
7. [Environment Variables](#environment-variables)
8. [Jupyter Notebook Best Practices](notebook-shortcuts)
9. [Developing warehouse models in Jupyter](jupyterhub-warehouse)
01. [Using JupyterHub](#using-jupyterhub)
02. [Logging in to JupyterHub](#logging-in-to-jupyterhub)
03. [Connecting to the Warehouse](#connecting-to-the-warehouse)
04. [Increasing the Query Limit](#increasing-the-query-limit)
05. [Increase the User Storage Limit](#increasing-the-storage-limit)
06. [Querying with SQL in JupyterHub](querying-sql-jupyterhub)
07. [Saving Code to Github](saving-code-jupyter)
08. [Environment Variables](#environment-variables)
09. [Jupyter Notebook Best Practices](notebook-shortcuts)
10. [Developing warehouse models in Jupyter](jupyterhub-warehouse)

## Using JupyterHub

Expand Down Expand Up @@ -77,6 +78,16 @@ tbls._init()

(querying-sql-jupyterhub)=

### Increasing the Storage Limit

By default, new JupyterHub instances are subject to a 10GB storage limit. This setting comes from the underlying infrastructure configuration, and requires some interaction with Kubernetes Engine in Google Cloud to modify.

If a JupyterHub user experiences an error indicating `no space left on device` or similar, their provisioned storage likely needs to be increased. This can be done from within the Storage section of [the Google Kubernetes Engine web UI](https://console.cloud.google.com/kubernetes/list/overview?project=cal-itp-data-infra). Click into the "claim-\[username\]" entry associated with the user (*not* the "pvc-\[abc123\]" persistent volume associated with that entry), navigate to the "YAML" tab, and change the storage resource request under `spec` and the storage capacity limit under `status`.

After making the configuration change in GKE, shut down and restart the user's JupyterHub instance. If the first restart attempt times out, try again once or twice - it can take a moment for the scaleup to complete and properly link the storage volume to the JupyterHub instance.

100GB should generally be more than enough for a given user - if somebody's storage has already been set to 100GB and they hit a space limit again, that may indicate a need to clean up past work rather than a need to increase available storage.

### Querying with SQL in JupyterHub

JupyterHub makes it easy to query SQL in the notebooks.
Expand Down Expand Up @@ -157,7 +168,7 @@ GITHUB_API_KEY = os.environ["GITHUB_API_KEY"]

External resources:

- [Cheat Sheet - Jupyter Notebook ](https://defkey.com/jupyter-notebook-shortcuts?pdf=true&modifiedDate=20200909T053706)
- [Cheat Sheet - Jupyter Notebook](https://defkey.com/jupyter-notebook-shortcuts?pdf=true&modifiedDate=20200909T053706)
- [Using Markdown in Jupyter Notebook](https://www.datacamp.com/community/tutorials/markdown-in-jupyter-notebook)

(jupyterhub-warehouse)=
Expand Down
6 changes: 6 additions & 0 deletions docs/requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,9 @@ jupyter-book==0.12.0
# https://github.com/pydata/pydata-sphinx-theme/issues/508
pydata-sphinx-theme==0.7.2
sphinxcontrib-mermaid==0.8.1
# Temporary pinning of older sphinxcontrib-[xyz] reqs until we use Sphinx 5.0.0 or higher
sphinxcontrib-applehelp==1.0.4
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==2.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.5

0 comments on commit 368a2e3

Please sign in to comment.