feat: Validate individual tasks in GA CI #1447

tisutisu · 2024-09-19T06:58:15Z

This PR adds a Github Actions CI to validate the task tests (if present inside the task directory) and few example tests for git-clone task for reference.

Story: https://issues.redhat.com/browse/STONEBLD-2910

lcarva · 2024-11-13T08:23:27Z

task/git-clone/0.1/tests/run.yaml

@@ -0,0 +1,30 @@
+---


Have you considering using a Pipeline to encapsulate the Task execution and the required verification? The release folks took this approach, example: https://github.com/konflux-ci/release-service-catalog/blob/development/tasks/check-data-keys/tests/test-check-data-keys-fail-missing-cdn-key.yaml

@lcarva could you please help with more context around where these tests are running what approach they are following... any doc or CI pipeline script?

Here we should verify that the task really cloned the repo as expected (e.g. contains the expected files, is really at revision 1.0.0, ...). And this test should set the example for how to verify that the task works as expected.

That will probably require wrapping the Task in a Pipeline as Lui suggested

makes sense now, this is another approach to validate the tasks (we were not aware of this while investigating), the approach taken release-service-catalog repo has more flexibility on what to verify in the task results. With current approach taken in this PR, we are just validating task results are non-empty. If we want to choose the approach taken in release-service-catalog repo, the current PR needs to be rewritten almost.

As a contributor to the release-service-catalog, I just want to say that I like the approach they're using. Once I got the hang of writing mocks.sh for my tests there, it became easy to add new functional tests for my tasks including both positive and negative cases.

They have nice features, like the ability to assert that the task should fail hard and exit with a non-zero status code on certain inputs, or to assert that the task should succeed, but have certain properties in its output.

For local development / inner-loop, I can stand up kind locally, install upstream tekton, and then run export TEST_ITEMS=path/to/the/task/Im/working/on and then run the .github/scripts/test_tekton_tasks.sh from their repo in a loop while I hack on my task or test.

@ralphbean thanks for the feedback, looks like the approach used in release-service-catalog is better, will move to it.

took this approach in this PR and closed the current one

mmorhun

Can we check task results / logs / created images, etc. ?

mmorhun · 2024-11-12T11:43:53Z

.github/workflows/run-task-tests.yaml

+            task/**
+            test/**
+
+      - name: Free Disk Space


Why do we need this?

took it from here, don't aware of the exact reason, but I think might be helping to create more disk space in the github action env. Don't find any issue to keep it our CI as well.

How long does this take to run? According to the README, around 3 minutes https://github.com/jlumbroso/free-disk-space?tab=readme-ov-file
Which doesn't seem worth it unless really needed

Removed it now. Please check here.

mmorhun · 2024-11-13T10:49:30Z

README.md

+
+For example refer: Git Clone task [tests](./task/git-clone/0.1/tests/run.yaml)
+
+CI also validates the task yamls using [tektor](https://github.com/lcarva/tektor)


@lcarva is there any plans to move it to some public org from personal account?

Filed https://issues.redhat.com/browse/EC-1019

hack/run-test.sh

mmorhun · 2024-11-13T14:26:44Z

test/common.sh

+    local reason=''
+    local maxloop=60 # 10 minutes max
+
+    set +x


Why do we need set +x here?

to avoid unnecessary logs being printed in the console, while we looping through to check task run status, reason and results, if something fails show_failure() will dump the test resources in the console.

mmorhun · 2024-11-13T14:44:38Z

test/common.sh

+    done
+}
+
+function test_resource_creation() {


The function name and what id does differs

updated it to test_resource_creation_and_validation, hope the name fits with its functionality now

mmorhun · 2024-11-13T14:47:12Z

test/common.sh

+    echo ${CHANGED_FILES} |grep -o 'task/[^\/]*/[^\/]*/*[^/]*.yaml'|xargs -I {} dirname {}|awk '{print $1"/tests"}'
+}
+
+function get_new_changed_tasks() {


function name is not clear

changed it to print_changed_tasks_to_console, hope it better fit its functionality.

mmorhun · 2024-11-13T14:48:07Z

test/common.sh

+    echo ${CHANGED_FILES} |grep "test/" || true
+}
+
+function detect_new_changed_resources() {


function name is unclear

changed it to print_changed_resources_to_console now

mmorhun · 2024-11-13T14:49:44Z

test/run-test.sh

@@ -0,0 +1,47 @@
+#!/bin/bash


Isn't it duplicates hack/run-test.sh?

forgot to remove the old one hack/run-test.sh, this test/run-test.sh is the correct one.

chmeliik · 2024-11-13T13:07:27Z

.github/workflows/run-task-tests.yaml

+      - name: Checkout konflux-ci/konflux-ci Repository
+        if: steps.changed-files.outputs.any_changed == 'true'
+        uses: actions/checkout@v3
+        with:
+          repository: 'konflux-ci/konflux-ci'
+          path: konflux-ci


Should we pin konflux-ci/konflux-ci to a commit (and configure renovate to update it)? To avoid CI breaking overnight due to changes in the repo

updated to use specific commit from konflux-ci/konflux-ci while checking out. But for configuring the renovate to update the SHA, does only renovate config is sufficient or do we need to run any script on push at konflux-ci/konflux-ci repo?

should be doable with a Renovate regex manager, roughly similar to https://docs.renovatebot.com/modules/manager/regex/#advanced-capture

Addressed it in the new PR

.github/workflows/run-task-tests.yaml

chmeliik · 2024-11-13T15:00:41Z

README.md

+A helper script called `run-test.sh` is present inside `test` directory to help the developer run the test. Just specify resource type as first argument, resource name as second argument and the version as third argument i.e:
+
+```
+./test/run-test.sh task git-clone 0.1


type as first argument, resource name as second argument and the version as third argument

This seems unnecessary, passing the actual directory as the argument would work much better with tab completion :)

we could also test stepactions in similar ways, that functionality is not added fully in the current PR, which we can extend to in the future, so kept the type (task or stepactions) as first argument in the run-test.sh script.

Would ./test/run-test.sh stepactions/eaas-copy-secrets-to-ephemeral-cluster/0.1 not work?

yes that should work.... updated it now

test/common.sh

chmeliik · 2024-11-18T15:31:58Z

test/common.sh

+    echo ${CHANGED_FILES} | grep -o 'task/[^\/]*/[^\/]*/tests/[^/]*'|xargs -I {} dirname {}|sed 's/\(tests\).*/\1/g'
+    # detect for changes in the task manifest
+    echo ${CHANGED_FILES} | grep -o 'task/[^\/]*/[^\/]*/*[^/]*.yaml'|xargs -I {} dirname {}|awk '{print $1"/tests"}'


Suggested change

echo ${CHANGED_FILES} | grep -o 'task/[^\/]*/[^\/]*/tests/[^/]*'|xargs -I {} dirname {}|sed 's/$tests$.*/\1/g'

# detect for changes in the task manifest

echo ${CHANGED_FILES} | grep -o 'task/[^\/]*/[^\/]*/*[^/]*.yaml'|xargs -I {} dirname {}|awk '{print $1"/tests"}'

echo "${CHANGED_FILES}" | grep -o 'task/[^\/]*/[^\/]*/tests/[^/]*'|xargs -I {} dirname {}|sed 's/$tests$.*/\1/g'

# detect for changes in the task manifest

echo "${CHANGED_FILES}" | grep -o 'task/[^\/]*/[^\/]*/*[^/]*.yaml'|xargs -I {} dirname {}|awk '{print $1"/tests"}'

without the quotes, echo will print all the changed files on a single line and the grep probably won't work right

chmeliik · 2024-11-18T15:35:28Z

test/common.sh

+        done
+
+        # don't test the tasks which are deprecated
+        cat ${runtest} | grep 'tekton.dev/deprecated: \"true\"' && skipit=True


We use the build.appstudio.redhat.com/expires-on label (but its value can be in the future)

chmeliik · 2024-11-18T15:36:24Z

test/common.sh

+# Set the return code that the test script will return.
+# Parameters: $1 - return code (0-255)
+function set_test_return_code() {
+  # kubetest teardown might fail and thus incorrectly report failure of the
+  # script, even if the tests pass.
+  # We store the real test result to return it later, ignoring any teardown
+  # failure in kubetest.
+  # TODO(adrcunha): Get rid of this workaround.
+  echo -n "$1"> ${TEST_RESULT_FILE}


Is this relevant for us?

chmeliik · 2024-11-18T15:39:39Z

test/test-tasks.sh

+# Configure the number of parallel tests running at the same time, start from 0
+MAX_NUMBERS_OF_PARALLEL_TASKS=2 # => 8


What does the start from 0 and # => 8 mean here?

tisutisu · 2024-11-27T08:14:22Z

Closing in favour of #1644

tisutisu force-pushed the test-tasks branch 2 times, most recently from 17aa0a1 to 0cd56c4 Compare October 21, 2024 05:44

tisutisu force-pushed the test-tasks branch 2 times, most recently from 923122a to 59f79e3 Compare November 4, 2024 12:12

tisutisu force-pushed the test-tasks branch 2 times, most recently from ccafb0b to f93dad2 Compare November 12, 2024 04:37

tisutisu changed the title ~~WIP: Validate individual tasks/stepactions~~ feat: Validate individual tasks in GA CI Nov 12, 2024

tisutisu marked this pull request as ready for review November 12, 2024 06:24

openshift-ci bot requested review from brunoapimentel and tnevrlka November 12, 2024 06:24

tisutisu requested review from mmorhun, chmeliik and lcarva November 12, 2024 07:16

lcarva reviewed Nov 13, 2024

View reviewed changes

mmorhun reviewed Nov 13, 2024

View reviewed changes

chmeliik reviewed Nov 13, 2024

View reviewed changes

tisutisu force-pushed the test-tasks branch from 1237e1d to 6c9cafd Compare November 15, 2024 07:24

tisutisu added 2 commits November 15, 2024 15:24

tests for inidividual tasks

4a28349

Address review comments

9dd1446

tisutisu force-pushed the test-tasks branch from 88cc5ab to 9dd1446 Compare November 15, 2024 09:54

chmeliik reviewed Nov 18, 2024

View reviewed changes

tisutisu closed this Nov 27, 2024


		For example refer: Git Clone task [tests](./task/git-clone/0.1/tests/run.yaml)

		CI also validates the task yamls using [tektor](https://github.com/lcarva/tektor)

		# Configure the number of parallel tests running at the same time, start from 0
		MAX_NUMBERS_OF_PARALLEL_TASKS=2 # => 8

feat: Validate individual tasks in GA CI #1447

feat: Validate individual tasks in GA CI #1447

Conversation

tisutisu commented Sep 19, 2024 • edited Loading

Choose a reason for hiding this comment

tisutisu Nov 15, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tisutisu Nov 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mmorhun left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tisutisu Nov 15, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chmeliik Nov 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tisutisu commented Nov 27, 2024

tisutisu commented Sep 19, 2024 •

edited

Loading

tisutisu Nov 15, 2024 •

edited

Loading

tisutisu Nov 19, 2024 •

edited

Loading

tisutisu Nov 15, 2024 •

edited

Loading

chmeliik Nov 18, 2024 •

edited

Loading