Merge pull request #666 from opendevstack/feature/plain-tasks

Provide rendered tasks

CHANGELOG.md

@@ -10,6 +10,10 @@ listed in the changelog.
 
 ## [Unreleased]
 
+### Added
+
+- Rendered task versions are available under `tasks/` now. These can be referenced directly from pipeline runs through [remote resolution](https://tekton.dev/docs/pipelines/pipelines/#specifying-remote-tasks). In future versions, tasks may be removed from the Helm chart and only be accessible via Git. See [#665](https://github.com/opendevstack/ods-pipeline/issues/665).
+
 ### Changed
 
 - Update Tekton to v.041.1 (matching OpenShift Pipelines operator 1.9). Unfortunately the `package-image` task of v0.9.0 breaks in OpenShift Pipelines operator 1.9, so v0.9.0 is not compatible with 1.9, and v0.10.0 will not compatible with 1.6. For details of the change, see [#663](https://github.com/opendevstack/ods-pipeline/pull/663).

Makefile

@@ -38,7 +38,11 @@ lint-shell: ## Run shellcheck.
 
 ##@ Building
 
-docs: ## Render documentation for tasks.
+tasks: ## Render tasks.
+	go run cmd/tasks/main.go
+.PHONY: tasks
+
+docs: tasks ## Render documentation for tasks.
 	go run cmd/docs/main.go
 .PHONY: docs
 

cmd/docs/main.go

@@ -11,6 +11,7 @@ import (
 func main() {
 	err := docs.RenderTasks(
 		filepath.Join(projectpath.Root, "deploy/ods-pipeline/charts/tasks"),
+		filepath.Join(projectpath.Root, "docs/tasks/descriptions"),
 		filepath.Join(projectpath.Root, "docs/tasks"),
 	)
 	if err != nil {

cmd/tasks/main.go

@@ -0,0 +1,19 @@
+package main
+
+import (
+	"log"
+	"path/filepath"
+
+	"github.com/opendevstack/pipeline/internal/projectpath"
+	"github.com/opendevstack/pipeline/internal/tasks"
+)
+
+func main() {
+	err := tasks.Render(
+		filepath.Join(projectpath.Root, "deploy/ods-pipeline/charts/tasks"),
+		filepath.Join(projectpath.Root, "tasks"),
+	)
+	if err != nil {
+		log.Fatal(err)
+	}
+}

deploy/ods-pipeline/charts/tasks/templates/task-ods-build-go.yaml

@@ -9,51 +9,7 @@ spec:
   description: |
     Builds Go (module) applications.
 
-    The exact build recipe can be found at
-    link:https://github.com/opendevstack/ods-pipeline/blob/master/build/package/scripts/build-go.sh[build/package/scripts/build-go.sh].
-
-    The following provides an overview of the performed steps:
-
-    - Source files are checked to be formatted with `gofmt`.
-    - The go module cache is configured to be on the cache location of the PVC by setting environment variable `GOMODCACHE` to `.ods-cache/deps/gomod` (see link:https://go.dev/ref/mod#module-cache[go module cache]).
-    - `golanci-lint` is run. The linter can be configured via a
-      config file as described in the
-      link:https://golangci-lint.run/usage/configuration/[configuration documentation].
-    - Tests are executed. A potential `vendor` directory is excluded. Test
-      results are converted into xUnit format.
-    - Application binary (named `app`) is built and placed into the directory
-      specified by `output-dir`.
-
-    Finally, the application source code is scanned by SonarQube.
-    Default SonarQube project properties are provided unless `sonar-project.properties`
-    is present.
-    When `sonar-quality-gate` is set to `true`, the task will fail if the quality gate
-    is not passed. If SonarQube is not desired, it can be disabled via `sonar-skip`.
-    The SonarQube scan will include parameters to perform a pull request analysis if
-    there is an open pull request for the branch being built. If the
-    link:https://docs.sonarqube.org/latest/analysis/bitbucket-integration/[ALM integration]
-    is setup properly, pull request decoration in Bitbucket is done automatically.
-
-    The following artifacts are generated by the build task and placed into `.ods/artifacts/`
-
-    * `code-coverage/`
-      ** `coverage.out`
-    * `lint-reports/`
-      ** `report.txt`
-    * `sonarqube-analysis/`
-      ** `analysis-report.md`
-      ** `issues-report.csv`
-      ** `quality-gate.json`
-    * `xunit-reports/`
-      ** `report.xml`
-
-    Instead of the built-in script, one can also specify a build script located
-    in the Git repository using the `build-script` task parameter. This allows
-    full control of building and testing, including any generation of artifacts.
-    Note that some other task parameters have no effect when a custom build
-    script is used, unless they are handled properly in the script. At a
-    minimum, the custom script should place its outputs in the directory
-    identified by `output-dir`.
+    See https://github.com/opendevstack/ods-pipeline/blob/v{{.Chart.AppVersion}}/docs/tasks/ods-build-go.adoc
   params:
     - name: working-dir
       description: |

deploy/ods-pipeline/charts/tasks/templates/task-ods-build-gradle.yaml

@@ -8,107 +8,8 @@ metadata:
 spec:
   description: |
     Builds Gradle applications.
-    
-    The gradle build by default caches the downloaded version of the gradle wrapper and dependencies in the cache location of
-    the PVC by setting the environment variable `GRADLE_USER_HOME` to `/workspace/source/.ods-cache/deps/gradle`. 
 
-    The following steps are executed:
-
-    - build gradle application, using `gradlew clean build`, which includes tests execution and coverage report generation
-    - SonarQube quality scan
-
-    Notes:
-
-    - tests exclude the vendor directory.
-    - test results are converted into xUnit format.
-
-    Available environment variables:
-
-    - `ODS_OUTPUT_DIR`: this environment variable points to the folder
-    that this build expects generated application artifacts to be copied to.
-    The gradle script should read it and copy there the generated artifacts.
-    - `NEXUS_*` env vars: `NEXUS_URL`, `NEXUS_USERNAME` and `NEXUS_PASSWORD`
-    are available and should be read by the gradle script.
-
-    To enable the gradle script to copy the generated application artifacts script follow these steps:
-
-    - read the environment variable `ODS_OUTPUT_DIR` in the buildscript section of the gradle script:
-    ```
-    buildscript {
-        ext {
-          outputDir = System.getenv('ODS_OUTPUT_DIR')
-        }
-    }
-    ```
-    - customize the jar tasks to set the destination directory
-    ```
-    jar {
-        println("Set application jar name to 'app'")
-        archiveBaseName = 'app'
-        if (outputDir != null) {
-            println("Set destinationDirectory to '${projectDir}/${outputDir}'")
-            destinationDirectory = file("${projectDir}/${outputDir}")
-        }
-    }
-    ```
-
-    To create a coverage report be sure that you add to `gradle.properties` the required
-    configuration. For example to enable Jacoco coverage repot you will need to:
-
-    - add `jacoco` plugin:
-    ```
-    plugins {
-        id 'application'
-        id 'jacoco'
-    }
-    ```
-    - add task `jacocoTestReport`:
-    ```
-    jacocoTestReport {
-        reports {
-            xml.required = true
-        }
-    }
-    ```
-    - add `finalizedBy jacocoTestReport` to the task `test`:
-    ```
-    tasks.named('test') {
-        useJUnitPlatform()
-        finalizedBy jacocoTestReport
-    }
-    ```
-
-    The exact build recipe can be found at
-    link:https://github.com/opendevstack/ods-pipeline/blob/master/build/package/scripts/build-gradle.sh[build/package/scripts/build-gradle.sh].
-
-    After tests ran successfully, the application source code is scanned by SonarQube.
-    Default SonarQube project properties are provided unless `sonar-project.properties`
-    is present.
-    When `sonar-quality-gate` is set to `true`, the task will fail if the quality gate
-    is not passed. If SonarQube is not desired, it can be disabled via `sonar-skip`.
-    The SonarQube scan will include parameters to perform a pull request analysis if
-    there is an open pull request for the branch being built. If the
-    link:https://docs.sonarqube.org/latest/analysis/bitbucket-integration/[ALM integration]
-    is setup properly, pull request decoration in Bitbucket is done automatically.
-
-    The following artifacts are generated by the build task and placed into `.ods/artifacts/`
-
-    * `code-coverage/`
-      ** `coverage.xml`
-    * `sonarqube-analysis/`
-      ** `analysis-report.md`
-      ** `issues-report.csv`
-      ** `quality-gate.json`
-    * `xunit-reports/`
-      ** `report.xml`
-
-    Instead of the built-in script, one can also specify a build script located
-    in the Git repository using the `build-script` task parameter. This allows
-    full control of building and testing, including any generation of artifacts.
-    Note that some other task parameters have no effect when a custom build
-    script is used, unless they are handled properly in the script. At a
-    minimum, the custom script should place its outputs in the directory
-    identified by `output-dir`.
+    See https://github.com/opendevstack/ods-pipeline/blob/v{{.Chart.AppVersion}}/docs/tasks/ods-build-gradle.adoc
   params:
     - name: working-dir
       description: |

deploy/ods-pipeline/charts/tasks/templates/task-ods-build-npm.yaml

@@ -9,64 +9,7 @@ spec:
   description: |
     Builds Node.js applications using npm.
 
-    The built-in script executes the following steps:
-
-    - check that package.json and package-lock.json exist to require best practice of using lock files. See also link:https://github.com/opendevstack/ods-pipeline/discussions/411[discussion 411]
-    - linting using `npm run lint`
-    - build application, using `npm run build`
-    - test execution, using `npm run test`
-    - SonarQube quality scan
-
-    For linting to work there needs to be a `lint` task in the `package.json` file,
-    for example `npx eslint src --format compact`, together with a config file
-    (`eslintrc.json` or similar) at the root of the working directory. This can
-    be done by running `eslint --init` or by following the
-    link:https://eslint.org/docs/user-guide/getting-started[official documentation].
-
-    The exact build recipe can be found at
-    link:https://github.com/opendevstack/ods-pipeline/blob/master/build/package/scripts/build-npm.sh[build/package/scripts/build-npm.sh].
-    In particular, `npm run build` is expected to place outputs into `dist` and
-    `npm run test` is expected to create `build/test-results/test/report.xml`
-    and `build/coverage/{clover.xml,coverage-final.json,lcov.info}`.
-
-    An example configuration for the test script is:
-
-    ```
-    JEST_JUNIT_OUTPUT_DIR='build/test-results/test' JEST_JUNIT_OUTPUT_NAME='report.xml' npx jest --reporters=default --reporters=jest-junit --coverage --coverageDirectory=build/coverage --forceExit ./dist
-    ```
-
-    After tests ran successfully, the application source code is scanned by SonarQube.
-    Default SonarQube project properties are provided unless `sonar-project.properties`
-    is present.
-    When `sonar-quality-gate` is set to `true`, the task will fail if the quality gate
-    is not passed. If SonarQube is not desired, it can be disabled via `sonar-skip`.
-    The SonarQube scan will include parameters to perform a pull request analysis if
-    there is an open pull request for the branch being built. If the
-    link:https://docs.sonarqube.org/latest/analysis/bitbucket-integration/[ALM integration]
-    is setup properly, pull request decoration in Bitbucket is done automatically.
-
-    The following artifacts are generated by the build task and placed into `.ods/artifacts/`
-
-    * `code-coverage/`
-      ** `clover.xml`
-      ** `coverage-final.json`
-      ** `lcov.info`
-    * `lint-reports`
-      ** `report.txt`
-    * `sonarqube-analysis/`
-      ** `analysis-report.md`
-      ** `issues-report.csv`
-      ** `quality-gate.json`
-    * `xunit-reports/`
-      ** `report.xml`
-
-    Instead of the built-in script, one can also specify a build script located
-    in the Git repository using the `build-script` task parameter. This allows
-    full control of building and testing, including any generation of artifacts.
-    Note that some other task parameters have no effect when a custom build
-    script is used, unless they are handled properly in the script. At a
-    minimum, the custom script should place its outputs in the directory
-    identified by `output-dir`.
+    See https://github.com/opendevstack/ods-pipeline/blob/v{{.Chart.AppVersion}}/docs/tasks/ods-build-npm.adoc
   params:
     - name: working-dir
       description: |

deploy/ods-pipeline/charts/tasks/templates/task-ods-build-python.yaml

@@ -9,38 +9,7 @@ spec:
   description: |
     Builds Python applications.
 
-    The exact build recipe can be found at
-    link:https://github.com/opendevstack/ods-pipeline/blob/master/build/package/scripts/build-python.sh[build/package/scripts/build-python.sh].
-    In particular, the Python source files are expected to be located in `src`.
-
-    After tests ran successfully, the application source code is scanned by SonarQube.
-    Default SonarQube project properties are provided unless `sonar-project.properties`
-    is present.
-    When `sonar-quality-gate` is set to `true`, the task will fail if the quality gate
-    is not passed. If SonarQube is not desired, it can be disabled via `sonar-skip`.
-    The SonarQube scan will include parameters to perform a pull request analysis if
-    there is an open pull request for the branch being built. If the
-    link:https://docs.sonarqube.org/latest/analysis/bitbucket-integration/[ALM integration]
-    is setup properly, pull request decoration in Bitbucket is done automatically.
-
-    The following artifacts are generated by the build task and placed into `.ods/artifacts/`
-
-    * `code-coverage/`
-      ** `coverage.xml`
-    * `sonarqube-analysis/`
-      ** `analysis-report.md`
-      ** `issues-report.csv`
-      ** `quality-gate.json`
-    * `xunit-reports/`
-      ** `report.xml`
-
-    Instead of the built-in script, one can also specify a build script located
-    in the Git repository using the `build-script` task parameter. This allows
-    full control of building and testing, including any generation of artifacts.
-    Note that some other task parameters have no effect when a custom build
-    script is used, unless they are handled properly in the script. At a
-    minimum, the custom script should place its outputs in the directory
-    identified by `output-dir`.
+    See https://github.com/opendevstack/ods-pipeline/blob/v{{.Chart.AppVersion}}/docs/tasks/ods-build-python.adoc
   params:
     - name: working-dir
       description: |

deploy/ods-pipeline/charts/tasks/templates/task-ods-deploy-helm.yaml

@@ -9,63 +9,7 @@ spec:
   description: |
     Deploy Helm charts.
 
-    This tasks will install / upgrade a Helm chart into your Kubernetes /
-    OpenShift cluster using Helm.
-
-    Helm has the plugins `helm-diff` and `helm-secrets` installed. A diff is
-    performed before an upgrade is attempted. `helm-secrets` can be used to
-    encrypt sensitive values in the underlying Git repository using
-    https://age-encryption.org[age]. Secrets are decrypted on the fly if the
-    secret identified by the `age-key-secret` parameter exists and contains an
-    age secret key which corresponding public key was used as one of the
-    recipients to encrypt.
-
-    Based on the target environment, some values files are added automatically
-    to the invocation of the `helm` command if they are present in the chart
-    directory:
-
-    - `values.yaml`: the values file (automatically considered by Helm).
-    - `secrets.yaml`: a secrets file.
-    - `values.<STAGE>.yaml`: a values file named after the stage (`dev`, `qa` or `prod`) of the target environment.
-    - `secrets.<STAGE>.yaml`: a secrets file named after the stage (`dev`, `qa` or `prod`) of the target environment.
-    - `values.<ENVIRONMENT>.yaml`: a values file named after the name of the target environment.
-    - `secrets.<ENVIRONMENT>.yaml`: a secrets file named after the name of the target environment.
-
-    Further, the task automatically sets the `image.tag` value on the CLI which
-    equals the Git commit SHA being built. This value can be used in your Helm
-    templates to refer to images built via `ods-package-image`.
-
-    Before the Helm chart is applied, it is packaged, setting the `appVersion`
-    to the Git commit SHA and the `version` to the externally provided version,
-    if any. If `version` is not given, the value in `Chart.yaml` is used as-is.
-
-    If the pipeline runs for a repository defining subrepos in its `ods.y(a)ml`
-    file, then any charts in those subrepos are packaged as well, and added as
-    dependencies to the top-most chart under `charts/`. Note that values and
-    secrets files are only collected from the repository for which the pipeline
-    runs. Therefore, if you use an umbrella repository to promote an
-    application from a `dev` stage to `qa` and `prod`, the umbrella repository
-    needs to define the stage/environment specific values for the subcomponents
-    for `qa`/`prod` instead of having those files in the subrepo.
-
-    In order to produce correct `image.tag` values for subcomponents, the task
-    automatically sets `<subcomponent>.image.tag` equal to the Git commit SHA of
-    the subcomponent. Further, if no release name is explicitly configured, the
-    task also sets `<subcomponent>.fullnameOverride` equal to the respective
-    subcomponent to avoid resources being prefixed with the umbrella repository
-    component name (assuming your resources are named using the `chart.fullname`
-    helper).
-
-    If you do not have an existing Helm chart yet, you can use the provided
-    link:https://github.com/opendevstack/ods-pipeline/tree/sample-helm-chart[sample chart]
-    as a starting point. It is setup in a way that works with this task out of
-    the box.
-
-    The following artifacts are generated by the task and placed into `.ods/artifacts/`
-
-    * `deployments/`
-      ** `diff-<env>.txt`
-      ** `release-<env>.txt`
+    See https://github.com/opendevstack/ods-pipeline/blob/v{{.Chart.AppVersion}}/docs/tasks/ods-deploy-helm.adoc
   params:
     - name: chart-dir
       description: Helm chart directory that will be deployed