From 801578a46e7b1015ab0718ffb5a5c185cb5674d1 Mon Sep 17 00:00:00 2001 From: atheo89 Date: Tue, 24 Oct 2023 13:56:40 +0200 Subject: [PATCH] Update docs in favor the 2023b release --- docs/developer-guide.md | 12 +++--- docs/user-guide.md | 8 ++-- docs/workbench-imagestreams.md | 29 +++++++-------- docs/workbenches.md | 68 +++++++++++++++++++++------------- 4 files changed, 66 insertions(+), 51 deletions(-) diff --git a/docs/developer-guide.md b/docs/developer-guide.md index 26833d919..4eb5c5be4 100644 --- a/docs/developer-guide.md +++ b/docs/developer-guide.md @@ -8,7 +8,7 @@ These release branches follow a specific naming format: YYYYx, where "YYYY" repr The structure of the notebook's build chain is derived from the parent image. To better comprehend this concept, refer to the following graph.

- +

@@ -62,7 +62,7 @@ spec: ### **ImageStream definitions for the supported out-of-the-box images in ODH** -The ImageStream definitions of the out-of-the-box workbench images for ODH can be found [here](https://github.com/opendatahub-io/odh-manifests/tree/master/notebook-images). +The ImageStream definitions of the out-of-the-box workbench images for ODH can be found [here](https://github.com/opendatahub-io/notebooks/tree/main/manifests). ### **Example ImageStream object definition** @@ -140,19 +140,19 @@ tests: ## GitHub Actions This section provides an overview of the automation functionalities. -**Piplock Renewal** [[Link](https://github.com/opendatahub-io/notebooks/blob/main/.github/workflows/piplock-renewal-2023a.yml)] +### **Piplock Renewal** [[Link]](https://github.com/opendatahub-io/notebooks/blob/main/.github/workflows/piplock-renewal-2023a.yml) This GitHub action is configured to be triggered on a weekly basis, specifically every Monday at 22:00 PM UTC. Its main objective is to automatically update the Pipfile.lock files by fetching the most recent minor versions available. Additionally, it also updates the hashes for the downloaded files of Python dependencies, including any sub-dependencies. Once the updated files are pushed, the CI pipeline is triggered to generate new updated images based on these changes. -**Sync the downstream release branch with the upstream** [[Link](https://github.com/red-hat-data-services/notebooks/blob/main/.github/workflows/sync-release-branch-2023a.yml)] +### **Sync the downstream release branch with the upstream** [[Link]](https://github.com/red-hat-data-services/notebooks/blob/main/.github/workflows/sync-release-branch-2023a.yml) This GitHub action is configured to be triggered on a weekly basis, specifically every Tuesday at 08:00 AM UTC. Its main objective is to automatically update the downstream release branch with the upstream branch. -**Digest Updater workflow on the manifests** [[Link](https://github.com/opendatahub-io/odh-manifests/blob/master/.github/workflows/notebooks-digest-updater-upstream.yaml)] +### **Digest Updater workflow on the manifests** [[Link]](https://github.com/opendatahub-io/odh-manifests/blob/master/.github/workflows/notebooks-digest-updater-upstream.yaml) This GitHub action is designed to be triggered on a weekly basis, specifically every Friday at 12:00 AM UTC. Its primary purpose is to automate the process of updating the SHA digest of the notebooks. It achieves this by fetching the new SHA values from the quay.io registry and updating the [param.env](https://github.com/opendatahub-io/odh-manifests/blob/master/notebook-images/base/params.env) file, which is hosted on the odh-manifest repository. By automatically updating the SHA digest, this action ensures that the notebooks remain synchronized with the latest changes. -**Digest Updater workflow on the live-builder** [[Link](https://gitlab.cee.redhat.com/data-hub/rhods-live-builder/-/blob/main/.gitlab/notebook-sha-digest-updater.yml)] +### **Digest Updater workflow on the live-builder** [[Link]](https://gitlab.cee.redhat.com/data-hub/rhods-live-builder/-/blob/main/.gitlab/notebook-sha-digest-updater.yml) This GitHub action works with the same logic as the above and is designed to be triggered on a weekly basis, specifically every Friday. It is also update the SHA digest of the images into the [CSV](https://gitlab.cee.redhat.com/data-hub/rhods-live-builder/-/blob/main/rhods-operator-live/bundle/template/manifests/clusterserviceversion.yml.j2#L725) file on the live-builder repo. diff --git a/docs/user-guide.md b/docs/user-guide.md index aca5cd7da..b09a087b8 100644 --- a/docs/user-guide.md +++ b/docs/user-guide.md @@ -6,7 +6,7 @@ There are two options for launching a workbench image: either through the Enable In the ODH dashboard, you can navigate to Applications -> Enabled -> Launch Application from the Jupyter tile. The notebook server spawner page displays a list of available container images you can run as a single user."

- +

@@ -15,7 +15,7 @@ In the ODH dashboard, you can navigate to Applications -> Enabled -> Launch Appl A user can navigate to the Data Science Project menu and create a project like the following:

- +

## Updates @@ -26,12 +26,12 @@ This section provides an overview of the rebuilding plan. There are two types of 1. Patch updates - These updates will be carried out weekly and will focus on incorporating security updates to the notebook images. -**Scope and frequency of the updates** +### **Scope and frequency of the updates** When performing major updates to the notebook images, all components are reviewed for updates, including the base OS image, OS packages, Python version, and Python packages and libraries. A major update, which will be named “YYYYx” release (where YYYY is the year and x is an increased letter), will fix the set of components that are included, as well as their MAJOR and MINOR versions. During the release lifecycle, which is the period during which the update is supported, the only updates that will be made are patches to allow for security updates and fixes while maintaining compatibility. These weekly updates will only modify the PATCH version, leaving the major and minor versions unchanged. -**Support** +### **Support** Our goal is to ensure that notebook images are supported for a minimum of one year, meaning that typically two supported images will be available at any given time. This provides sufficient time for users to update their code to use components from the latest notebook images. We will continue to make older images available in the registry for users to add as custom notebook images, even if they are no longer supported. This way, users can still access the older images if needed. Example lifecycle (not actual dates): diff --git a/docs/workbench-imagestreams.md b/docs/workbench-imagestreams.md index e465c3ecd..b576d37f2 100644 --- a/docs/workbench-imagestreams.md +++ b/docs/workbench-imagestreams.md @@ -1,12 +1,12 @@ -# Workbench ImageStreams +## Workbench ImageStreams ODH supports multiple out-of-the-box pre-built workbench images ([provided in this repository](https://github.com/opendatahub-io/notebooks)). For each of those workbench images, there is a dedicated ImageStream object definition. This ImageStream object references the actual image tag(s) and contains additional metadata that describe the workbench image. - -## Annotations + +### **Annotations** Aside from the general ImageStream config values, there are additional annotations that can be provided in the workbench ImageStream definition. This additional data is leveraged further by the [odh-dashboard](https://github.com/opendatahub-io/odh-dashboard/). -### ImageStream-specific annotations +### **ImageStream-specific annotations** The following labels and annotations are specific to the particular workbench image. They are provided in their respective sections in the `metadata` section. ```yaml metadata: @@ -15,17 +15,16 @@ metadata: annotations: ... ``` -#### Available labels +### **Available labels** - **`opendatahub.io/notebook-image:`** - a flag that determines whether the ImageStream references a workbench image that is meant be shown in the UI -#### Available annotations +### **Available annotations** - **`opendatahub.io/notebook-image-url:`** - a URL reference to the source of the particular workbench image - **`opendatahub.io/notebook-image-name:`** - a desired display name string for the particular workbench image (used in the UI) - **`opendatahub.io/notebook-image-desc:`** - a desired description string of the of the particular workbench image (used in the UI) - **`opendatahub.io/notebook-image-order:`** - an index value for the particular workbench ImageStream (used by the UI to list available workbench images in a specific order) +- **`opendatahub.io/recommended-accelerators`** - a string that represents the list of recommended hardware accelerators for the particular workbench ImageStream (used in the UI) -- **`opendatahub.io/recommended-accelerators`** - a string that represents the list of recommended hardware accelerators for the particular workbench ImageStream. This is used by the ODH dashboard component to correctly assign accelerators and tolerations to the workbench. Ex: `'["nvidia.com/gpu", "habana.com/gen1"]'` -Note: The annotation is not required for non-accelerator recommended images i.e. you do not need to include an empty array. -### Tag-specific annotations +### **Tag-specific annotations** One ImageStream can reference multiple image tags. The following annotations are specific to a particular workbench image tag and are provided in its `annotations:` section. ```yaml spec: @@ -37,17 +36,17 @@ spec: name: image-repository/tag name: tag-name ``` -#### Available annotations +### **Available annotations** - **`opendatahub.io/notebook-software:`** - a string that represents the technology stack included within the workbench image. Each technology in the list is described by its name and the version used (e.g. `'[{"name":"CUDA","version":"11.8"},{"name":"Python","version":"v3.9"}]`') - **`opendatahub.io/notebook-python-dependencies:`** - a string that represents the list of Python libraries included within the workbench image. Each library is described by its name and currently used version (e.g. `'[{"name":"Numpy","version":"1.24"},{"name":"Pandas","version":"1.5"}]'`) - **`openshift.io/imported-from:`** - a reference to the image repository where the workbench image was obtained (e.g. `quay.io/repository/opendatahub/workbench-images`) - **`opendatahub.io/workbench-image-recommended:`** - a flag that allows the ImageStream tag to be marked as Recommended (used by the UI to distinguish which tags are recommended for use, e.g., when the workbench image offers multiple tags to choose from) -## ImageStream definitions for the supported out-of-the-box images in ODH +### **ImageStream definitions for the supported out-of-the-box images in ODH** -The ImageStream definitions of the out-of-the-box workbench images for ODH can be found [here](https://github.com/opendatahub-io/odh-manifests/tree/master/notebook-images). +The ImageStream definitions of the out-of-the-box workbench images for ODH can be found [here](https://github.com/opendatahub-io/notebooks/tree/main/manifests). -## Example ImageStream object definition +### **Example ImageStream object definition** An exemplary, non-functioning ImageStream object definition that uses all the aforementioned annotations is provided below. @@ -62,14 +61,14 @@ metadata: opendatahub.io/notebook-image-name: "Example Jupyter Notebook" opendatahub.io/notebook-image-desc: "Exemplary Jupyter notebook image just for demonstrative purposes" opendatahub.io/notebook-image-order: "1" - opendatahub.io/recommended-accelerators: '["nvidia.com/gpu"]' + opendatahub.io/recommended-accelerators: '["nvidia.com/gpu", "habana.com/gen1"]' name: example-jupyter-notebook spec: lookupPolicy: local: true tags: - annotations: - opendatahub.io/notebook-software: '[{"name":"CUDA","version":"11.8"},{"name":"Python","version":"v3.9"}]' + opendatahub.io/notebook-software: '[{"name":"Python","version":"v3.9"}]' opendatahub.io/notebook-python-dependencies: '[{"name":"Boto3","version":"1.26"},{"name":"Kafka-Python","version":"2.0"},{"name":"Kfp-tekton","version":"1.5"},{"name":"Matplotlib","version":"3.6"},{"name":"Numpy","version":"1.24"},{"name":"Pandas","version":"1.5"},{"name":"Scikit-learn","version":"1.2"},{"name":"Scipy","version":"1.10"}]' openshift.io/imported-from: quay.io/opendatahub/workbench-images opendatahub.io/workbench-image-recommended: 'true' diff --git a/docs/workbenches.md b/docs/workbenches.md index 5205d3fda..eb51cc2fe 100644 --- a/docs/workbenches.md +++ b/docs/workbenches.md @@ -3,63 +3,79 @@ Workbench images are supported for a minimum of one year. Major updates to pre-c Open Data Hub contains the following workbench images with different variations: -| Workbenches | ODH | RHODS | OS | GPU | Runtimes | -|----------------------|-----|-------|--------|-----|----------| -| Jupyter Minimal | V | V | UBI8/9 | - | - | -| CUDA | V | V | UBI8/9 | V | - | -| Jupyter Data Science | V | V | UBI8/9 | - | V | -| Jupyter Tensorflow | V | V | UBI8/9 | V | V | -| Jupyter PyTorch | V | V | UBI8/9 | V | V | -| Jupyter TrustyAI | V | V | UBI9 | - | - | -| Code Server | V | - | C9S | - | - | -| R Studio | V | - | C9S | V | - | - -These notebooks are incorporated to be used in conjunction with Open Data Hub, specifically utilizing the ODH Notebook Controller as the launching platform. +| Workbenches | ODH | RHODS | OS | GPU | Runtimes | +|----------------------|-------------|-------------|----------|-------------|-------------| +| Jupyter Minimal | ☑ | ☑ | UBI8/9 | ☒| ☒| +| CUDA | ☑ | ☑ | UBI8/9 | ☑ | ☒| +| HabanaAI | ☑ | ☒| UBI8/9 | ☑ | ☒| +| Jupyter Data Science | ☑ | ☑ | UBI8/9 | ☒| ☑ | +| Jupyter Tensorflow | ☑ | ☑ | UBI8/9 | ☑ | ☑ | +| Jupyter PyTorch | ☑ | ☑ | UBI8/9 | ☑ | ☑ | +| Jupyter TrustyAI | ☑ | ☑ | UBI9 | ☒| ☒| +| Code Server | ☑ | ☒| C9S | ☒| ☒| +| R Studio | ☑ | ☒| C9S | ☑ | ☒| + +These notebooks are incorporated to be used in conjunction with Open Data Hub, specifically utilizing the ODH Notebook Controller as the launching platform. The table above provides insights into the characteristics of each notebook, including their availability in both ODH and RHODS environments, GPU support, and whether they are offered as runtimes ie without the JupyterLab UI. + +All the notebooks are available on the[ Quay.io registry](https://quay.io/repository/opendatahub/workbench-images?tab=tags&tag=latest); please filter the results by using the tag "2023b" for the latest release and "2023a" for the n-1. ## Jupyter Minimal - +Jupyter Minimal provides a browser-based integrated development environment where you can write, edit, and debug code using the familiar interface and features of JupyterLab. If you do not require advanced machine learning features or additional resources for compute-intensive data science work, you can use the Minimal Python image to develop your models. -[Installed Packages](https://github.com/opendatahub-io/notebooks/blob/main/jupyter/minimal/ubi9-python-3.9/Pipfile) +[2023b Packages](https://github.com/opendatahub-io/notebooks/blob/2023b/jupyter/minimal/ubi9-python-3.9/Pipfile) || [2023a Packages](https://github.com/opendatahub-io/notebooks/blob/2023a/jupyter/minimal/ubi9-python-3.9/Pipfile) ## CUDA -If you are working with compute-intensive data science models that require GPU support, use the Compute Unified Device Architecture (CUDA) notebook image to gain access to the NVIDIA CUDA Toolkit. Using this toolkit, you can optimize your work using GPU-accelerated libraries and optimization tools. +CUDA provides a browser-based integrated development environment where you can write, edit, and debug code using the familiar interface and features of JupyterLab. If you are working with compute-intensive data science models that require GPU support, use the Compute Unified Device Architecture (CUDA) notebook image to gain access to the NVIDIA CUDA Toolkit. You can optimize your work using GPU-accelerated libraries and optimization tools using this toolkit. + +[2023b Packages](https://github.com/opendatahub-io/notebooks/blob/2023b/jupyter/minimal/ubi9-python-3.9/Pipfile) || [2023a Packages](https://github.com/opendatahub-io/notebooks/blob/2023a/jupyter/minimal/ubi9-python-3.9/Pipfile) + +## HabanaAI -[Installed Packages](https://github.com/opendatahub-io/notebooks/blob/main/jupyter/minimal/ubi9-python-3.9/Pipfile) +HabanaAI provides a browser-based integrated development environment where you can write, edit, and debug code using the familiar interface and features of JupyterLab. The HabanaAI notebook image optimizes high-performance deep learning (DL) with Habana Gaudi devices. Habana Gaudi devices accelerate DL training workloads and maximize training throughput and efficiency. + +[2023b Packages](https://github.com/opendatahub-io/notebooks/blob/2023b/habana/1.11.0/ubi8-python-3.8/Pipfile) || [2023a Packages](https://github.com/opendatahub-io/notebooks/blob/2023a/habana/1.10.0/ubi8-python-3.8/Pipfile) ## Jupyter Data Science -Use the Standard Data Science notebook image for models that do not require TensorFlow or PyTorch. This image contains commonly used libraries to assist you in developing your machine-learning models. +Standard Data Science provides a browser-based integrated development environment where you can write, edit, and debug code using the familiar interface and features of JupyterLab. Use the Standard Data Science notebook image for models that do not require TensorFlow or PyTorch. +This image contains commonly used libraries to assist you in developing your machine-learning models. Furthermore, we have integrated several useful libraries and applications. Notably, we've included **Mesa-libgl**, an additional library designed for OpenCV tasks. We've also introduced **Git-lfs**, which provides an efficient solution for handling large files, such as audio samples, videos, datasets, and graphics. The integration of **unixODBC** offers a standardized API for accessing data sources, including SQL Servers and other data sources with ODBC drivers. Lastly, the addition of **Libsndfile** makes it easier to read and write files containing sampled audio data. Additionally, this notebook comes equipped with standard **database clients** for MySQL, PostgreSQL, MSSQL, and MongoDB. + +**NOTE:** All notebooks derived from the Jupyter Data Science Notebook inherit these libraries and applications, with the exception of the minimal and CUDA variants. -[Installed Packages](https://github.com/opendatahub-io/notebooks/blob/main/jupyter/datascience/ubi9-python-3.9/Pipfile) +[2023b Packages](https://github.com/opendatahub-io/notebooks/blob/2023b/jupyter/datascience/ubi9-python-3.9/Pipfile) || [2023a Packages](https://github.com/opendatahub-io/notebooks/blob/2023a/jupyter/datascience/ubi9-python-3.9/Pipfile) ## Jupyter Tensorflow -TensorFlow is an open-source platform for machine learning. With TensorFlow, you can build, train and deploy your machine learning models. TensorFlow contains advanced data visualization features, such as computational graph visualizations. It also allows you to easily monitor and track the progress of your models. +TensorFlow is an open-source platform for machine learning. It provides a browser-based integrated development environment where you can write, edit, and debug code using the familiar interface and features of JupyterLab. With TensorFlow, you can build, train and deploy your machine learning models. TensorFlow contains advanced data visualization features, such as computational graph visualizations. It also allows you to easily monitor and track the progress of your models. -[Installed Packages](https://github.com/opendatahub-io/notebooks/blob/main/jupyter/tensorflow/ubi9-python-3.9/Pipfile) +[2023b Packages](https://github.com/opendatahub-io/notebooks/blob/2023b/jupyter/tensorflow/ubi9-python-3.9/Pipfile) || [2023a Packages](https://github.com/opendatahub-io/notebooks/blob/2023a/jupyter/tensorflow/ubi9-python-3.9/Pipfile) ## Jupyter PyTorch -PyTorch is an open-source machine learning library optimized for deep learning. If you are working with computer vision or natural language processing models, use the Pytorch notebook image. +PyTorch is an open-source machine learning library optimized for deep learning. If you are working with computer vision or natural language processing models, use the Pytorch notebook image. It provides a browser-based integrated development environment where you can write, edit, and debug code using the familiar interface and features of JupyterLab. -[Installed Packages](https://github.com/opendatahub-io/notebooks/blob/main/jupyter/pytorch/ubi9-python-3.9/Pipfile) +[2023b Packages](https://github.com/opendatahub-io/notebooks/blob/2023b/jupyter/pytorch/ubi9-python-3.9/Pipfile) || [2023a Packages](https://github.com/opendatahub-io/notebooks/blob/2023a/jupyter/pytorch/ubi9-python-3.9/Pipfile) ## Jupyter TrustyAI -Use the TrustyAI notebook image to leverage your data science work with model explainability, tracing and accountability, and runtime monitoring. +Use the TrustyAI notebook image to leverage your data science work with model explainability, tracing and accountability, and runtime monitoring. It provides a browser-based integrated development environment where you can write, edit, and debug code using the familiar interface and features of JupyterLab. -[Installed Packages](https://github.com/opendatahub-io/notebooks/blob/main/jupyter/trustyai/ubi9-python-3.9/Pipfile) +[2023b Packages](https://github.com/opendatahub-io/notebooks/blob/2023b/jupyter/trustyai/ubi9-python-3.9/Pipfile) || [2023a Packages](https://github.com/opendatahub-io/notebooks/blob/2023a/jupyter/trustyai/ubi9-python-3.9/Pipfile) ## Code Server -Code Server (VS Code) provides a browser-based integrated development environment (IDE) where you can write, edit, and debug code using the familiar interface and features of VS Code. It is particularly useful for collaborating with team members, as everyone can access the same development environment from their own devices. In order to unlock the code server notebook is required to enable the additional overlay layer on the kfdef. +Code Server (VS Code) provides a browser-based integrated development environment (IDE) where you can write, edit, and debug code using the familiar interface and features of VS Code. It is particularly useful for collaborating with team members, as everyone can access the same development environment from their own devices. + + ## R Studio -It provides a powerful integrated development environment specifically designed for R programming. By integrating R Studio IDE into ODH, you equip data analysts with a dedicated environment for exploring and manipulating data, building models, and generating insightful visualizations. Moreover, If you are working with compute-intensive data science models that require GPU support, use the CUDA R Studio notebook image to gain access to the NVIDIA CUDA Toolkit. In order to unlock the R Studio notebook is required to enable the additional overlay layer on the kfdef. +It provides a powerful integrated development environment specifically designed for R programming. By integrating R Studio IDE into ODH, you equip data analysts with a dedicated environment for exploring and manipulating data, building models, and generating insightful visualizations. Moreover, If you are working with compute-intensive data science models that require GPU support, use the CUDA R Studio notebook image to gain access to the NVIDIA CUDA Toolkit. + +