Fix and improve CI unit test workaround #271

atheo89 · 2023-10-18T13:21:03Z

Related to: #269
Improvement of: #176

Fix and improve CI unit test recipe. This can be a temporary solution till figure out the Possible other improvements that the the issue is mentioning.

Description

This PR improves the code quality of the recipe by utilizing functions and avoiding repetitive code, moreover, when a test_notebook.ipynb is broken or doesn't fulfill the unit tests it returns 0 code and stops the execution of the e2e test with an ERROR message.

How Has This Been Tested?

Below are the logs on how the test behaves in healthy and problematic notebooks:

Problematic notebook:
make test-jupyter-trustyai-ubi9-python-3.9

# Running tests for jupyter-trustyai-ubi9-python-3-9 notebook...
# Test common notebook logic
bin/kubectl wait --for=condition=ready pod -l app=jupyter-trustyai-ubi9-python-3-9 --timeout=600s
pod/jupyter-trustyai-ubi9-python-3-9-notebook-0 condition met
bin/kubectl port-forward svc/jupyter-trustyai-ubi9-python-3-9-notebook 8888:8888 & curl --retry 5 --retry-delay 5 --retry-connrefused http://localhost:8888/notebook/opendatahub/jovyan/api ; EXIT_CODE=$?; echo && pkill --full "^bin/kubectl.*port-forward.*"
curl: (7) Failed to connect to localhost port 8888 after 0 ms: Connection refused
Warning: Problem : connection refused. Will retry in 5 seconds. 5 retries 
Warning: left.
Forwarding from 127.0.0.1:8888 -> 8888
Forwarding from [::1]:8888 -> 8888
Handling connection for 8888
{"version": "2.1.0"}
# Test specific notebooks
if echo "jupyter-trustyai-ubi9-python-3-9-notebook-0" | grep -q "minimal-ubi9"; then \
                bin/kubectl exec jupyter-trustyai-ubi9-python-3-9-notebook-0 -- /bin/sh -c "wget https://raw.githubusercontent.com/opendatahub-io/notebooks/main/jupyter/minimal/ubi9-python-3.9/test/test_notebook.ipynb -O test_notebook.ipynb && python3 -m papermill test_notebook.ipynb minimal_ubi9_output.ipynb --kernel python3 --stderr-file minimal_ubi9_error.txt && cat minimal_ubi9_error.txt | grep FAILED > /dev/null" ; if [ $? -eq 0 ]; then echo "ERROR: The minimal ubi9 notebook encountered a failure. To investigate the issue, you can review the logs located in the ocp-ci cluster on 'artifacts/notebooks-e2e-tests/jupyter-minimal-ubi9-python-3.9-test-e2e' directory or run 'cat minimal_ubi9_error.txt' within your container. The make process has been aborted." ; exit 1 ; fi \
elif echo "jupyter-trustyai-ubi9-python-3-9-notebook-0" | grep -q "datascience-ubi9"; then \
        make validate-ubi9-datascience -e FULL_NOTEBOOK_NAME=jupyter-trustyai-ubi9-python-3-9-notebook-0; \
elif echo "jupyter-trustyai-ubi9-python-3-9-notebook-0" | grep -q "pytorch-ubi9"; then \
        make validate-ubi9-datascience -e FULL_NOTEBOOK_NAME=jupyter-trustyai-ubi9-python-3-9-notebook-0; \
                bin/kubectl exec jupyter-trustyai-ubi9-python-3-9-notebook-0 -- /bin/sh -c "wget https://raw.githubusercontent.com/opendatahub-io/notebooks/main/jupyter/pytorch/ubi9-python-3.9/test/test_notebook.ipynb -O test_notebook.ipynb && python3 -m papermill test_notebook.ipynb pytorch_ubi9_output.ipynb --kernel python3 --stderr-file pytorch_ubi9_error.txt && cat pytorch_ubi9_error.txt | grep FAILED > /dev/null" ; if [ $? -eq 0 ]; then echo "ERROR: The pytorch ubi9 notebook encountered a failure. To investigate the issue, you can review the logs located in the ocp-ci cluster on 'artifacts/notebooks-e2e-tests/jupyter-pytorch-ubi9-python-3.9-test-e2e' directory or run 'cat pytorch_ubi9_error.txt' within your container. The make process has been aborted." ; exit 1 ; fi \
elif echo "jupyter-trustyai-ubi9-python-3-9-notebook-0" | grep -q "tensorflow-ubi9"; then \
                bin/kubectl exec jupyter-trustyai-ubi9-python-3-9-notebook-0 -- /bin/sh -c "wget https://raw.githubusercontent.com/opendatahub-io/notebooks/main/jupyter/tensorflow/ubi9-python-3.9/test/test_notebook.ipynb -O test_notebook.ipynb && python3 -m papermill test_notebook.ipynb tensorflow_ubi9_output.ipynb --kernel python3 --stderr-file tensorflow_ubi9_error.txt && cat tensorflow_ubi9_error.txt | grep FAILED > /dev/null" ; if [ $? -eq 0 ]; then echo "ERROR: The tensorflow ubi9 notebook encountered a failure. To investigate the issue, you can review the logs located in the ocp-ci cluster on 'artifacts/notebooks-e2e-tests/jupyter-tensorflow-ubi9-python-3.9-test-e2e' directory or run 'cat tensorflow_ubi9_error.txt' within your container. The make process has been aborted." ; exit 1 ; fi \
elif echo "jupyter-trustyai-ubi9-python-3-9-notebook-0" | grep -q "trustyai-ubi9"; then \
        make validate-ubi9-datascience -e FULL_NOTEBOOK_NAME=jupyter-trustyai-ubi9-python-3-9-notebook-0; \
                bin/kubectl exec jupyter-trustyai-ubi9-python-3-9-notebook-0 -- /bin/sh -c "wget https://raw.githubusercontent.com/opendatahub-io/notebooks/main/jupyter/trustyai/ubi9-python-3.9/test/test_notebook.ipynb -O test_notebook.ipynb && python3 -m papermill test_notebook.ipynb trustyai_ubi9_output.ipynb --kernel python3 --stderr-file trustyai_ubi9_error.txt && cat trustyai_ubi9_error.txt | grep FAILED > /dev/null" ; if [ $? -eq 0 ]; then echo "ERROR: The trustyai ubi9 notebook encountered a failure. To investigate the issue, you can review the logs located in the ocp-ci cluster on 'artifacts/notebooks-e2e-tests/jupyter-trustyai-ubi9-python-3.9-test-e2e' directory or run 'cat trustyai_ubi9_error.txt' within your container. The make process has been aborted." ; exit 1 ; fi \
elif echo "jupyter-trustyai-ubi9-python-3-9-notebook-0" | grep -q "minimal-ubi8"; then \
                bin/kubectl exec jupyter-trustyai-ubi9-python-3-9-notebook-0 -- /bin/sh -c "wget https://raw.githubusercontent.com/opendatahub-io/notebooks/main/jupyter/minimal/ubi8-python-3.8/test/test_notebook.ipynb -O test_notebook.ipynb && python3 -m papermill test_notebook.ipynb minimal_ubi8_output.ipynb --kernel python3 --stderr-file minimal_ubi8_error.txt && cat minimal_ubi8_error.txt | grep FAILED > /dev/null" ; if [ $? -eq 0 ]; then echo "ERROR: The minimal ubi8 notebook encountered a failure. To investigate the issue, you can review the logs located in the ocp-ci cluster on 'artifacts/notebooks-e2e-tests/jupyter-minimal-ubi8-python-3.8-test-e2e' directory or run 'cat minimal_ubi8_error.txt' within your container. The make process has been aborted." ; exit 1 ; fi \
elif echo "jupyter-trustyai-ubi9-python-3-9-notebook-0" | grep -q "datascience-ubi8"; then \
                bin/kubectl exec jupyter-trustyai-ubi9-python-3-9-notebook-0 -- /bin/sh -c "wget https://raw.githubusercontent.com/opendatahub-io/notebooks/main/jupyter/datascience/ubi8-python-3.8/test/test_notebook.ipynb -O test_notebook.ipynb && python3 -m papermill test_notebook.ipynb datascience_ubi8_output.ipynb --kernel python3 --stderr-file datascience_ubi8_error.txt && cat datascience_ubi8_error.txt | grep FAILED > /dev/null" ; if [ $? -eq 0 ]; then echo "ERROR: The datascience ubi8 notebook encountered a failure. To investigate the issue, you can review the logs located in the ocp-ci cluster on 'artifacts/notebooks-e2e-tests/jupyter-datascience-ubi8-python-3.8-test-e2e' directory or run 'cat datascience_ubi8_error.txt' within your container. The make process has been aborted." ; exit 1 ; fi \
elif echo "jupyter-trustyai-ubi9-python-3-9-notebook-0" | grep -q "pytorch-ubi8"; then \
                bin/kubectl exec jupyter-trustyai-ubi9-python-3-9-notebook-0 -- /bin/sh -c "wget https://raw.githubusercontent.com/opendatahub-io/notebooks/main/jupyter/pytorch/ubi8-python-3.8/test/test_notebook.ipynb -O test_notebook.ipynb && python3 -m papermill test_notebook.ipynb pytorch_ubi8_output.ipynb --kernel python3 --stderr-file pytorch_ubi8_error.txt && cat pytorch_ubi8_error.txt | grep FAILED > /dev/null" ; if [ $? -eq 0 ]; then echo "ERROR: The pytorch ubi8 notebook encountered a failure. To investigate the issue, you can review the logs located in the ocp-ci cluster on 'artifacts/notebooks-e2e-tests/jupyter-pytorch-ubi8-python-3.8-test-e2e' directory or run 'cat pytorch_ubi8_error.txt' within your container. The make process has been aborted." ; exit 1 ; fi \
elif echo "jupyter-trustyai-ubi9-python-3-9-notebook-0" | grep -q "tensorflow-ubi8"; then \
                bin/kubectl exec jupyter-trustyai-ubi9-python-3-9-notebook-0 -- /bin/sh -c "wget https://raw.githubusercontent.com/opendatahub-io/notebooks/main/jupyter/tensorflow/ubi8-python-3.8/test/test_notebook.ipynb -O test_notebook.ipynb && python3 -m papermill test_notebook.ipynb tensorflow_ubi8_output.ipynb --kernel python3 --stderr-file tensorflow_ubi8_error.txt && cat tensorflow_ubi8_error.txt | grep FAILED > /dev/null" ; if [ $? -eq 0 ]; then echo "ERROR: The tensorflow ubi8 notebook encountered a failure. To investigate the issue, you can review the logs located in the ocp-ci cluster on 'artifacts/notebooks-e2e-tests/jupyter-tensorflow-ubi8-python-3.8-test-e2e' directory or run 'cat tensorflow_ubi8_error.txt' within your container. The make process has been aborted." ; exit 1 ; fi \
elif echo "jupyter-trustyai-ubi9-python-3-9-notebook-0" | grep -q "trustyai-ubi8"; then \
                bin/kubectl exec jupyter-trustyai-ubi9-python-3-9-notebook-0 -- /bin/sh -c "wget https://raw.githubusercontent.com/opendatahub-io/notebooks/main/jupyter/trustyai/ubi8-python-3.8/test/test_notebook.ipynb -O test_notebook.ipynb && python3 -m papermill test_notebook.ipynb trustyai_ubi8_output.ipynb --kernel python3 --stderr-file trustyai_ubi8_error.txt && cat trustyai_ubi8_error.txt | grep FAILED > /dev/null" ; if [ $? -eq 0 ]; then echo "ERROR: The trustyai ubi8 notebook encountered a failure. To investigate the issue, you can review the logs located in the ocp-ci cluster on 'artifacts/notebooks-e2e-tests/jupyter-trustyai-ubi8-python-3.8-test-e2e' directory or run 'cat trustyai_ubi8_error.txt' within your container. The make process has been aborted." ; exit 1 ; fi \
elif echo "jupyter-trustyai-ubi9-python-3-9-notebook-0" | grep -q "anaconda"; then \
        echo "There is no test notebook implemented yet for Anaconda Notebook...." \
else \
        echo "No matching condition found for jupyter-trustyai-ubi9-python-3-9-notebook-0." ; \
fi
make[1]: Entering directory '/home/atheodor/Documents/REPOS/notebooks'
bin/kubectl exec jupyter-trustyai-ubi9-python-3-9-notebook-0 -- /bin/sh -c "wget https://raw.githubusercontent.com/opendatahub-io/notebooks/main/jupyter/minimal/ubi9-python-3.9/test/test_notebook.ipynb -O test_notebook.ipynb && python3 -m papermill test_notebook.ipynb minimal_ubi9_output.ipynb --kernel python3 --stderr-file minimal_ubi9_error.txt && cat minimal_ubi9_error.txt | grep FAILED > /dev/null" ; if [ $? -eq 0 ]; then echo "ERROR: The minimal ubi9 notebook encountered a failure. To investigate the issue, you can review the logs located in the ocp-ci cluster on 'artifacts/notebooks-e2e-tests/jupyter-minimal-ubi9-python-3.9-test-e2e' directory or run 'cat minimal_ubi9_error.txt' within your container. The make process has been aborted." ; exit 1 ; fi
Will not apply HSTS. The HSTS database must be a regular and non-world-writable file.
ERROR: could not open HSTS store at '/opt/app-root/src/.wget-hsts'. HSTS will be disabled.
--2023-10-18 12:41:50--  https://raw.githubusercontent.com/opendatahub-io/notebooks/main/jupyter/minimal/ubi9-python-3.9/test/test_notebook.ipynb
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1822 (1.8K) [text/plain]
Saving to: ‘test_notebook.ipynb’

     0K .                                                     100% 38.0M=0s

2023-10-18 12:41:51 (38.0 MB/s) - ‘test_notebook.ipynb’ saved [1822/1822]

Input Notebook:  test_notebook.ipynb
Output Notebook: minimal_ubi9_output.ipynb
Notebook JSON is invalid: Additional properties are not allowed ('id' was unexpected)

Failed validating 'additionalProperties' in code_cell:

On instance['cells'][0]:
{'cell_type': 'code',
 'execution_count': None,
 'id': '2d972b6b-1211-4c21-a7e8-a1683b72a62c',
 'metadata': {},
 'outputs': ['...0 outputs...'],
 'source': 'import unittest\n'
           'import jupyterlab as jp\n'
           'from platform import pyt...'}
Executing:   0%|          | 0/1 [00:00<?, ?cell/s]Executing notebook with kernel: python3
Executing: 100%|██████████| 1/1 [00:03<00:00,  3.20s/cell]
command terminated with exit code 1
bin/kubectl exec jupyter-trustyai-ubi9-python-3-9-notebook-0 -- /bin/sh -c "wget https://raw.githubusercontent.com/opendatahub-io/notebooks/main/jupyter/datascience/ubi9-python-3.9/test/test_notebook.ipynb -O test_notebook.ipynb && python3 -m papermill test_notebook.ipynb datascience_ubi9_output.ipynb --kernel python3 --stderr-file datascience_ubi9_error.txt && cat datascience_ubi9_error.txt | grep FAILED > /dev/null" ; if [ $? -eq 0 ]; then echo "ERROR: The datascience ubi9 notebook encountered a failure. To investigate the issue, you can review the logs located in the ocp-ci cluster on 'artifacts/notebooks-e2e-tests/jupyter-datascience-ubi9-python-3.9-test-e2e' directory or run 'cat datascience_ubi9_error.txt' within your container. The make process has been aborted." ; exit 1 ; fi
Will not apply HSTS. The HSTS database must be a regular and non-world-writable file.
ERROR: could not open HSTS store at '/opt/app-root/src/.wget-hsts'. HSTS will be disabled.
--2023-10-18 12:41:58--  https://raw.githubusercontent.com/opendatahub-io/notebooks/main/jupyter/datascience/ubi9-python-3.9/test/test_notebook.ipynb
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 9909 (9.7K) [text/plain]
Saving to: ‘test_notebook.ipynb’

     0K .........                                             100% 6.59M=0.001s

2023-10-18 12:41:58 (6.59 MB/s) - ‘test_notebook.ipynb’ saved [9909/9909]

Input Notebook:  test_notebook.ipynb
Output Notebook: datascience_ubi9_output.ipynb
Notebook JSON is invalid: Additional properties are not allowed ('id' was unexpected)

Failed validating 'additionalProperties' in code_cell:

On instance['cells'][0]:
{'cell_type': 'code',
 'execution_count': None,
 'id': '0c234518-7728-4cf7-95cc-32a53dd78853',
 'metadata': {},
 'outputs': ['...0 outputs...'],
 'source': 'import unittest\n'
           'from unittest import mock\n'
           'from platform import p...'}
Executing:   0%|          | 0/1 [00:00<?, ?cell/s]Executing notebook with kernel: python3
Executing: 100%|██████████| 1/1 [00:06<00:00,  6.64s/cell]
command terminated with exit code 1
make[1]: Leaving directory '/home/atheodor/Documents/REPOS/notebooks'
Will not apply HSTS. The HSTS database must be a regular and non-world-writable file.
ERROR: could not open HSTS store at '/opt/app-root/src/.wget-hsts'. HSTS will be disabled.
--2023-10-18 12:42:09--  https://raw.githubusercontent.com/opendatahub-io/notebooks/main/jupyter/trustyai/ubi9-python-3.9/test/test_notebook.ipynb
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4328 (4.2K) [text/plain]
Saving to: ‘test_notebook.ipynb’

     0K ....                                                  100% 59.7M=0s

2023-10-18 12:42:09 (59.7 MB/s) - ‘test_notebook.ipynb’ saved [4328/4328]

Input Notebook:  test_notebook.ipynb
Output Notebook: trustyai_ubi9_output.ipynb
Executing:   0%|          | 0/1 [00:00<?, ?cell/s]Executing notebook with kernel: python3
Executing: 100%|██████████| 1/1 [00:13<00:00, 13.04s/cell]
...F
======================================================================
FAIL: test_trustyai_version (__main__.TestTrustyaiNotebook)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/ipykernel_310/2273224652.py", line 19, in test_trustyai_version
    self.assertEqual(actual_major_minor, expected_major_minor, "incorrect version")
AssertionError: '0.3' != '0.2'
- 0.3
?   ^
+ 0.2
?   ^
 : incorrect version

----------------------------------------------------------------------
Ran 4 tests in 6.672s

FAILED (failures=1)
ERROR: The trustyai ubi9 notebook encountered a failure. To investigate the issue, you can review the logs located in the ocp-ci cluster on 'artifacts/notebooks-e2e-tests/jupyter-trustyai-ubi9-python-3.9-test-e2e' directory or run 'cat trustyai_ubi9_error.txt' within your container. The make process has been aborted.
make: *** [Makefile:351: test-jupyter-trustyai-ubi9-python-3.9] Error 1

Healthy Notebook:
make test-jupyter-datascience-ubi9-python-3.9

# Running tests for jupyter-datascience-ubi9-python-3-9 notebook...
# Verify the notebook's readiness by pinging the /api endpoint
bin/kubectl wait --for=condition=ready pod -l app=jupyter-datascience-ubi9-python-3-9 --timeout=600s
pod/jupyter-datascience-ubi9-python-3-9-notebook-0 condition met
bin/kubectl port-forward svc/jupyter-datascience-ubi9-python-3-9-notebook 8888:8888 & curl --retry 5 --retry-delay 5 --retry-connrefused http://localhost:8888/notebook/opendatahub/jovyan/api ; EXIT_CODE=$?; echo && pkill --full "^bin/kubectl.*port-forward.*"
curl: (7) Failed to connect to localhost port 8888 after 0 ms: Connection refused
Warning: Problem : connection refused. Will retry in 5 seconds. 5 retries 
Warning: left.
Forwarding from 127.0.0.1:8888 -> 8888
Forwarding from [::1]:8888 -> 8888
Handling connection for 8888
{"version": "2.1.0"}
# Tests notebook's functionalities 
if echo "jupyter-datascience-ubi9-python-3-9-notebook-0" | grep -q "minimal-ubi9"; then \
                bin/kubectl exec jupyter-datascience-ubi9-python-3-9-notebook-0 -- /bin/sh -c "wget https://raw.githubusercontent.com/opendatahub-io/notebooks/main/jupyter/minimal/ubi9-python-3.9/test/test_notebook.ipynb -O test_notebook.ipynb && python3 -m papermill test_notebook.ipynb minimal_ubi9_output.ipynb --kernel python3 --stderr-file minimal_ubi9_error.txt && cat minimal_ubi9_error.txt | grep FAILED > /dev/null" ; if [ $? -eq 0 ]; then echo "ERROR: The minimal ubi9 notebook encountered a failure. To investigate the issue, you can review the logs located in the ocp-ci cluster on 'artifacts/notebooks-e2e-tests/jupyter-minimal-ubi9-python-3.9-test-e2e' directory or run 'cat minimal_ubi9_error.txt' within your container. The make process has been aborted." ; exit 1 ; fi \
elif echo "jupyter-datascience-ubi9-python-3-9-notebook-0" | grep -q "datascience-ubi9"; then \
        make validate-ubi9-datascience -e FULL_NOTEBOOK_NAME=jupyter-datascience-ubi9-python-3-9-notebook-0; \
elif echo "jupyter-datascience-ubi9-python-3-9-notebook-0" | grep -q "pytorch-ubi9"; then \
        make validate-ubi9-datascience -e FULL_NOTEBOOK_NAME=jupyter-datascience-ubi9-python-3-9-notebook-0; \
                bin/kubectl exec jupyter-datascience-ubi9-python-3-9-notebook-0 -- /bin/sh -c "wget https://raw.githubusercontent.com/opendatahub-io/notebooks/main/jupyter/pytorch/ubi9-python-3.9/test/test_notebook.ipynb -O test_notebook.ipynb && python3 -m papermill test_notebook.ipynb pytorch_ubi9_output.ipynb --kernel python3 --stderr-file pytorch_ubi9_error.txt && cat pytorch_ubi9_error.txt | grep FAILED > /dev/null" ; if [ $? -eq 0 ]; then echo "ERROR: The pytorch ubi9 notebook encountered a failure. To investigate the issue, you can review the logs located in the ocp-ci cluster on 'artifacts/notebooks-e2e-tests/jupyter-pytorch-ubi9-python-3.9-test-e2e' directory or run 'cat pytorch_ubi9_error.txt' within your container. The make process has been aborted." ; exit 1 ; fi \
elif echo "jupyter-datascience-ubi9-python-3-9-notebook-0" | grep -q "tensorflow-ubi9"; then \
        make validate-ubi9-datascience -e FULL_NOTEBOOK_NAME=jupyter-datascience-ubi9-python-3-9-notebook-0; \
                bin/kubectl exec jupyter-datascience-ubi9-python-3-9-notebook-0 -- /bin/sh -c "wget https://raw.githubusercontent.com/opendatahub-io/notebooks/main/jupyter/tensorflow/ubi9-python-3.9/test/test_notebook.ipynb -O test_notebook.ipynb && python3 -m papermill test_notebook.ipynb tensorflow_ubi9_output.ipynb --kernel python3 --stderr-file tensorflow_ubi9_error.txt && cat tensorflow_ubi9_error.txt | grep FAILED > /dev/null" ; if [ $? -eq 0 ]; then echo "ERROR: The tensorflow ubi9 notebook encountered a failure. To investigate the issue, you can review the logs located in the ocp-ci cluster on 'artifacts/notebooks-e2e-tests/jupyter-tensorflow-ubi9-python-3.9-test-e2e' directory or run 'cat tensorflow_ubi9_error.txt' within your container. The make process has been aborted." ; exit 1 ; fi \
elif echo "jupyter-datascience-ubi9-python-3-9-notebook-0" | grep -q "trustyai-ubi9"; then \
        make validate-ubi9-datascience -e FULL_NOTEBOOK_NAME=jupyter-datascience-ubi9-python-3-9-notebook-0; \
                bin/kubectl exec jupyter-datascience-ubi9-python-3-9-notebook-0 -- /bin/sh -c "wget https://raw.githubusercontent.com/opendatahub-io/notebooks/main/jupyter/trustyai/ubi9-python-3.9/test/test_notebook.ipynb -O test_notebook.ipynb && python3 -m papermill test_notebook.ipynb trustyai_ubi9_output.ipynb --kernel python3 --stderr-file trustyai_ubi9_error.txt && cat trustyai_ubi9_error.txt | grep FAILED > /dev/null" ; if [ $? -eq 0 ]; then echo "ERROR: The trustyai ubi9 notebook encountered a failure. To investigate the issue, you can review the logs located in the ocp-ci cluster on 'artifacts/notebooks-e2e-tests/jupyter-trustyai-ubi9-python-3.9-test-e2e' directory or run 'cat trustyai_ubi9_error.txt' within your container. The make process has been aborted." ; exit 1 ; fi \
elif echo "jupyter-datascience-ubi9-python-3-9-notebook-0" | grep -q "minimal-ubi8"; then \
                bin/kubectl exec jupyter-datascience-ubi9-python-3-9-notebook-0 -- /bin/sh -c "wget https://raw.githubusercontent.com/opendatahub-io/notebooks/main/jupyter/minimal/ubi8-python-3.8/test/test_notebook.ipynb -O test_notebook.ipynb && python3 -m papermill test_notebook.ipynb minimal_ubi8_output.ipynb --kernel python3 --stderr-file minimal_ubi8_error.txt && cat minimal_ubi8_error.txt | grep FAILED > /dev/null" ; if [ $? -eq 0 ]; then echo "ERROR: The minimal ubi8 notebook encountered a failure. To investigate the issue, you can review the logs located in the ocp-ci cluster on 'artifacts/notebooks-e2e-tests/jupyter-minimal-ubi8-python-3.8-test-e2e' directory or run 'cat minimal_ubi8_error.txt' within your container. The make process has been aborted." ; exit 1 ; fi \
elif echo "jupyter-datascience-ubi9-python-3-9-notebook-0" | grep -q "datascience-ubi8"; then \
        make validate-ubi8-datascience -e FULL_NOTEBOOK_NAME=jupyter-datascience-ubi9-python-3-9-notebook-0; \
elif echo "jupyter-datascience-ubi9-python-3-9-notebook-0" | grep -q "pytorch-ubi8"; then \
        make validate-ubi8-datascience -e FULL_NOTEBOOK_NAME=jupyter-datascience-ubi9-python-3-9-notebook-0; \
                bin/kubectl exec jupyter-datascience-ubi9-python-3-9-notebook-0 -- /bin/sh -c "wget https://raw.githubusercontent.com/opendatahub-io/notebooks/main/jupyter/pytorch/ubi8-python-3.8/test/test_notebook.ipynb -O test_notebook.ipynb && python3 -m papermill test_notebook.ipynb pytorch_ubi8_output.ipynb --kernel python3 --stderr-file pytorch_ubi8_error.txt && cat pytorch_ubi8_error.txt | grep FAILED > /dev/null" ; if [ $? -eq 0 ]; then echo "ERROR: The pytorch ubi8 notebook encountered a failure. To investigate the issue, you can review the logs located in the ocp-ci cluster on 'artifacts/notebooks-e2e-tests/jupyter-pytorch-ubi8-python-3.8-test-e2e' directory or run 'cat pytorch_ubi8_error.txt' within your container. The make process has been aborted." ; exit 1 ; fi \
elif echo "jupyter-datascience-ubi9-python-3-9-notebook-0" | grep -q "tensorflow-ubi8"; then \
        make validate-ubi8-datascience -e FULL_NOTEBOOK_NAME=jupyter-datascience-ubi9-python-3-9-notebook-0; \
                bin/kubectl exec jupyter-datascience-ubi9-python-3-9-notebook-0 -- /bin/sh -c "wget https://raw.githubusercontent.com/opendatahub-io/notebooks/main/jupyter/tensorflow/ubi8-python-3.8/test/test_notebook.ipynb -O test_notebook.ipynb && python3 -m papermill test_notebook.ipynb tensorflow_ubi8_output.ipynb --kernel python3 --stderr-file tensorflow_ubi8_error.txt && cat tensorflow_ubi8_error.txt | grep FAILED > /dev/null" ; if [ $? -eq 0 ]; then echo "ERROR: The tensorflow ubi8 notebook encountered a failure. To investigate the issue, you can review the logs located in the ocp-ci cluster on 'artifacts/notebooks-e2e-tests/jupyter-tensorflow-ubi8-python-3.8-test-e2e' directory or run 'cat tensorflow_ubi8_error.txt' within your container. The make process has been aborted." ; exit 1 ; fi \
elif echo "jupyter-datascience-ubi9-python-3-9-notebook-0" | grep -q "trustyai-ubi8"; then \
        make validate-ubi8-datascience -e FULL_NOTEBOOK_NAME=jupyter-datascience-ubi9-python-3-9-notebook-0; \
                bin/kubectl exec jupyter-datascience-ubi9-python-3-9-notebook-0 -- /bin/sh -c "wget https://raw.githubusercontent.com/opendatahub-io/notebooks/main/jupyter/trustyai/ubi8-python-3.8/test/test_notebook.ipynb -O test_notebook.ipynb && python3 -m papermill test_notebook.ipynb trustyai_ubi8_output.ipynb --kernel python3 --stderr-file trustyai_ubi8_error.txt && cat trustyai_ubi8_error.txt | grep FAILED > /dev/null" ; if [ $? -eq 0 ]; then echo "ERROR: The trustyai ubi8 notebook encountered a failure. To investigate the issue, you can review the logs located in the ocp-ci cluster on 'artifacts/notebooks-e2e-tests/jupyter-trustyai-ubi8-python-3.8-test-e2e' directory or run 'cat trustyai_ubi8_error.txt' within your container. The make process has been aborted." ; exit 1 ; fi \
elif echo "jupyter-datascience-ubi9-python-3-9-notebook-0" | grep -q "anaconda"; then \
        echo "There is no test notebook implemented yet for Anaconda Notebook...." \
else \
        echo "No matching condition found for jupyter-datascience-ubi9-python-3-9-notebook-0." ; \
fi
make[1]: Entering directory '/home/atheodor/Documents/REPOS/notebooks'
bin/kubectl exec jupyter-datascience-ubi9-python-3-9-notebook-0 -- /bin/sh -c "wget https://raw.githubusercontent.com/opendatahub-io/notebooks/main/jupyter/minimal/ubi9-python-3.9/test/test_notebook.ipynb -O test_notebook.ipynb && python3 -m papermill test_notebook.ipynb minimal_ubi9_output.ipynb --kernel python3 --stderr-file minimal_ubi9_error.txt && cat minimal_ubi9_error.txt | grep FAILED > /dev/null" ; if [ $? -eq 0 ]; then echo "ERROR: The minimal ubi9 notebook encountered a failure. To investigate the issue, you can review the logs located in the ocp-ci cluster on 'artifacts/notebooks-e2e-tests/jupyter-minimal-ubi9-python-3.9-test-e2e' directory or run 'cat minimal_ubi9_error.txt' within your container. The make process has been aborted." ; exit 1 ; fi
--2023-10-18 12:49:55--  https://raw.githubusercontent.com/opendatahub-io/notebooks/main/jupyter/minimal/ubi9-python-3.9/test/test_notebook.ipynb
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.109.133, 185.199.110.133, 185.199.111.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.109.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1822 (1.8K) [text/plain]
Saving to: ‘test_notebook.ipynb’

     0K .                                                     100% 24.7M=0s

2023-10-18 12:49:55 (24.7 MB/s) - ‘test_notebook.ipynb’ saved [1822/1822]

Input Notebook:  test_notebook.ipynb
Output Notebook: minimal_ubi9_output.ipynb
Notebook JSON is invalid: Additional properties are not allowed ('id' was unexpected)

Failed validating 'additionalProperties' in code_cell:

On instance['cells'][0]:
{'cell_type': 'code',
 'execution_count': None,
 'id': '2d972b6b-1211-4c21-a7e8-a1683b72a62c',
 'metadata': {},
 'outputs': ['...0 outputs...'],
 'source': 'import unittest\n'
           'import jupyterlab as jp\n'
           'from platform import pyt...'}
Executing:   0%|          | 0/1 [00:00<?, ?cell/s]Executing notebook with kernel: python3
Executing: 100%|██████████| 1/1 [00:03<00:00,  3.17s/cell]
command terminated with exit code 1
bin/kubectl exec jupyter-datascience-ubi9-python-3-9-notebook-0 -- /bin/sh -c "wget https://raw.githubusercontent.com/opendatahub-io/notebooks/main/jupyter/datascience/ubi9-python-3.9/test/test_notebook.ipynb -O test_notebook.ipynb && python3 -m papermill test_notebook.ipynb datascience_ubi9_output.ipynb --kernel python3 --stderr-file datascience_ubi9_error.txt && cat datascience_ubi9_error.txt | grep FAILED > /dev/null" ; if [ $? -eq 0 ]; then echo "ERROR: The datascience ubi9 notebook encountered a failure. To investigate the issue, you can review the logs located in the ocp-ci cluster on 'artifacts/notebooks-e2e-tests/jupyter-datascience-ubi9-python-3.9-test-e2e' directory or run 'cat datascience_ubi9_error.txt' within your container. The make process has been aborted." ; exit 1 ; fi
Will not apply HSTS. The HSTS database must be a regular and non-world-writable file.
ERROR: could not open HSTS store at '/opt/app-root/src/.wget-hsts'. HSTS will be disabled.
--2023-10-18 12:50:02--  https://raw.githubusercontent.com/opendatahub-io/notebooks/main/jupyter/datascience/ubi9-python-3.9/test/test_notebook.ipynb
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.109.133, 185.199.110.133, 185.199.111.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.109.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 9909 (9.7K) [text/plain]
Saving to: ‘test_notebook.ipynb’

     0K .........                                             100% 13.0M=0.001s

2023-10-18 12:50:02 (13.0 MB/s) - ‘test_notebook.ipynb’ saved [9909/9909]

Input Notebook:  test_notebook.ipynb
Output Notebook: datascience_ubi9_output.ipynb
Notebook JSON is invalid: Additional properties are not allowed ('id' was unexpected)

Failed validating 'additionalProperties' in code_cell:

On instance['cells'][0]:
{'cell_type': 'code',
 'execution_count': None,
 'id': '0c234518-7728-4cf7-95cc-32a53dd78853',
 'metadata': {},
 'outputs': ['...0 outputs...'],
 'source': 'import unittest\n'
           'from unittest import mock\n'
           'from platform import p...'}
Executing:   0%|          | 0/1 [00:00<?, ?cell/s]Executing notebook with kernel: python3
Executing: 100%|██████████| 1/1 [00:07<00:00,  7.10s/cell]
command terminated with exit code 1
make[1]: Leaving directory '/home/atheodor/Documents/REPOS/notebooks'

Merge criteria:

The commits are squashed in a cohesive manner and have meaningful messages.
Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
The developer has manually tested the changes and verified that the changes work

Makefile

jstourac · 2023-10-18T15:44:36Z

Failing CI is a good sign :) Also, having content of the error file right in the run log may be nice thing, since I think that the necessary file isn't saved among the artifacts at the moment, right?

atheo89 · 2023-10-18T16:14:47Z

Failing CI is a good sign :)

Yes, only for the Trusty notebook. 🥳

having content of the error file right in the run log may be nice thing

Fixed it!

…book is broken or doesn't fulfill the unitests

atheo89 · 2023-10-19T10:47:24Z

/test notebooks-e2e-tests

atheo89 · 2023-10-19T11:11:03Z

/retest

jstourac

LGTM, thank you! 🥳

atheo89 · 2023-10-19T11:37:25Z

/test notebooks-e2e-tests

atheo89 · 2023-10-19T11:43:18Z

Adding here the notebook-e2e-tests job before the ocp-ci went down.
https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/opendatahub-io_notebooks/271/pull-ci-opendatahub-io-notebooks-main-notebooks-e2e-tests/1714770387037851648
On the print screen below shows that the test cached successfully the unhealthy notebook.

/override ci/prow/notebooks-e2e-tests
/approve

Thank you for reviewing this! 🙂

openshift-ci · 2023-10-19T11:43:25Z

@atheo89: Overrode contexts on behalf of atheo89: ci/prow/notebooks-e2e-tests

In response to this:

Adding here the notebook-e2e-tests job before the ocp-ci went down.
https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/opendatahub-io_notebooks/271/pull-ci-opendatahub-io-notebooks-main-notebooks-e2e-tests/1714770387037851648
On the print screen below shows that the test cached successfully the unhealthy notebook.

/override ci/prow/notebooks-e2e-tests
/approve

Thank you for reviewing this! 🙂

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2023-10-19T11:43:30Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: atheo89, jstourac

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [atheo89]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

…st-updater-9520338673 [Digest Updater Action] Update Runtime Images

openshift-ci bot requested review from dibryant and rkpattnaik780 October 18, 2023 13:21

atheo89 requested a review from jstourac October 18, 2023 13:25

jstourac reviewed Oct 18, 2023

View reviewed changes

Makefile Outdated Show resolved Hide resolved

jstourac reviewed Oct 18, 2023

View reviewed changes

Makefile Outdated Show resolved Hide resolved

jstourac reviewed Oct 18, 2023

View reviewed changes

Makefile Show resolved Hide resolved

atheo89 force-pushed the test-test branch 3 times, most recently from 3e05e54 to 435e3f7 Compare October 18, 2023 16:13

atheo89 force-pushed the test-test branch 4 times, most recently from 1edc072 to 5ea9e76 Compare October 18, 2023 22:27

Fix test structure recipe as well as return an ERROR when a test_note…

47b7c24

…book is broken or doesn't fulfill the unitests

atheo89 force-pushed the test-test branch from 5ea9e76 to 47b7c24 Compare October 19, 2023 09:07

jstourac approved these changes Oct 19, 2023

View reviewed changes

openshift-ci bot assigned jstourac Oct 19, 2023

openshift-ci bot added the lgtm label Oct 19, 2023

openshift-ci bot added the approved label Oct 19, 2023

openshift-ci bot merged commit 19e93cf into opendatahub-io:main Oct 19, 2023

atheo89 mentioned this pull request Oct 20, 2023

Sprint release 1.35 #273

Closed

jstourac mentioned this pull request Oct 27, 2023

[Epic] Fix and improve CI unit testing #269

Open

1 task

harshad16 mentioned this pull request Nov 17, 2023

Sprint release 1.36 #329

Closed

atheo89 pushed a commit to atheo89/notebooks that referenced this pull request Jun 19, 2024

Merge pull request opendatahub-io#271 from red-hat-data-services/dige…

9c69ee8

…st-updater-9520338673 [Digest Updater Action] Update Runtime Images

atheo89 deleted the test-test branch October 23, 2024 08:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix and improve CI unit test workaround #271

Fix and improve CI unit test workaround #271

atheo89 commented Oct 18, 2023 •

edited

Loading

jstourac commented Oct 18, 2023

atheo89 commented Oct 18, 2023

atheo89 commented Oct 19, 2023

atheo89 commented Oct 19, 2023

jstourac left a comment

atheo89 commented Oct 19, 2023

atheo89 commented Oct 19, 2023

openshift-ci bot commented Oct 19, 2023

openshift-ci bot commented Oct 19, 2023

Fix and improve CI unit test workaround #271

Fix and improve CI unit test workaround #271

Conversation

atheo89 commented Oct 18, 2023 • edited Loading

Description

How Has This Been Tested?

Merge criteria:

jstourac commented Oct 18, 2023

atheo89 commented Oct 18, 2023

atheo89 commented Oct 19, 2023

atheo89 commented Oct 19, 2023

jstourac left a comment

Choose a reason for hiding this comment

atheo89 commented Oct 19, 2023

atheo89 commented Oct 19, 2023

openshift-ci bot commented Oct 19, 2023

openshift-ci bot commented Oct 19, 2023

atheo89 commented Oct 18, 2023 •

edited

Loading