OPCT-226: cmd/report UX enhancements for CLI and Web report

This change introduce several improvements in the UX while reviewing the report by: - creationg an intuitive HTML report allowing users to quickly see issues and navigate to the logs for each test failure - introduce several gates/SLO/checks to be used as post-processor and get better visibility in the results, based in existing knowledge base/CI data or external ssytems See the PR with details of improvements: #76 ---- fix sippy query in the flake filter create a rank of tags, adding percentage of counters create html report saving data to json improve report html add log links and rank by error count update camgi tab add waiting interval support status: improve message field with plugin phase report: generating filter.html file with failures table renameing vfs path for html templates report html - update rank and output as json using report as data source of CLI report; apply docs add timers to the runtime report update report collecting more env information report: update html templates to improve filters adding suite errors menu Update suite error menu using native js create a rank of error counters adding an option to extract errors from must-gather extracting must-gather event filters adding a tab to the report adding the structure of report checks introducing dynamic checks moving report to summary Adding checks with acceptance values from baseline adding alert pop to Checks menu parallelism processing must-gather, loweing 3 times proc time adding support to all plugins in report http file server to serve report review: documenting and linking check rules reorg mkdocs/todo report: embeding etcd error parser to json report report/must-gather: consolidating etd parser report: supporting etcd, network check and embed data remove codegen script / not used distributing report to packages parsing meta/run.log extracting runtime information introducing new parsers: opct and meta config renaming sippy to its package renaming packages and cleanup review report html and threasholds review timers for plugins add etcd checks ; cleanup review thresholds bumping plugins image review documentation and doc strings update check platform type fix tests in tags fix dev docs dev/report: adding initial flow of report Fix Containerfile for CI when releasing Fix windows build update makefile to fix windows build isolating status watch interval from #87 reverting unrelated docs changes improving unit tests adding metaconfig unit tests and test data adding tests for meta-run.log parser adding parsers and tests for config maps fixes in the opct metrics report gen metrics report adding sample document to generate batch reports Review checks to support attributes review report to improve checks increasing the error pattern information for etcd taking notes for report dev doc adding plugin log extraction and link to the plugin name collecting node info extracting install-config review report html and metrics with adm parse cmd adding charts with ploty remove comments when plotting fix log save opct adm parse-etcd-logs to quickly access parsed logs (#4) increasing documentation coverage bump to use new quay.io org create target directory before extractors rename title of checks on report cli fixes in the report check prevent empty data fix/report/cli: show other plugin than k8s report/check: 012 - check plugin failure fixes after rebase rename namespace Supporting PDB to opct server add new gen openshift-tests plugin based in go allow custom openshift-tests image to devel w/ kind working version with entrypoint for plugin plugin manifests fixes to support remote entrypoint to tests image update plugin manifests to new plugin version refact CLI UI tables to enhance results intro yamllint and fixing yaml assets/manifests plugins working version the default flow enhance pre-run checks for missing config create local log file for all levels updatem plugins according to the latest goplugin version add collector image to the manifest template review packages / moving to report package using ETL-like cleanup must-gather/extractor packages refact/must-gather: isolate and tune leaky bucket processor cmd/run: fix flag to use full image path to allow CI presubmits review the result filter pipeline OPCT-292: cmd/publish - add experimental publish used in CI Add a experimental command to allow Prow CI to publish results in the opct-storage (S3) without needing to install dependencies in the test step, preventing failures. OPCT-292: cmd/publish - add metadata option when publishing artifact doc: add review/support documentations feat: upload baseline artifact on s3 report/feat: opct adm baseline (get|list|indexer|publish) review baseline reitrieving summary/baseline from service/API feat: add replay to filter pipeline feat: introduce adm setup-ndoe to helper in tests review plugin filter order fix checks to good tresholds
redhat-openshift-ecosystem · Aug 6, 2024 · c3c98c3 · c3c98c3
1 parent ed7ee37
commit c3c98c3
Show file tree

Hide file tree

Showing 83 changed files with 9,420 additions and 2,223 deletions.
diff --git a/.github/workflows/go.yaml b/.github/workflows/go.yaml
@@ -1,3 +1,4 @@
+---
 name: "OPCT"
 
 on:
@@ -49,8 +50,8 @@ jobs:
     name: go-test
     runs-on: ubuntu-latest
     needs:
-    - go-lint
-    - go-static
+      - go-lint
+      - go-static
     steps:
       - uses: actions/checkout@v4
       - uses: actions/setup-go@v5
@@ -63,8 +64,8 @@ jobs:
     name: "go-vet"
     runs-on: ubuntu-latest
     needs:
-    - go-lint
-    - go-static
+      - go-lint
+      - go-static
     steps:
       - uses: actions/checkout@v4
       - uses: actions/setup-go@v5
@@ -77,8 +78,8 @@ jobs:
     name: "build-artifact"
     runs-on: ubuntu-latest
     needs:
-    - go-test
-    - go-vet
+      - go-test
+      - go-vet
     steps:
       - uses: actions/checkout@v4
       - uses: actions/setup-go@v5
@@ -121,8 +122,7 @@ jobs:
   e2e-cmd_report:
     name: "e2e-cmd_report"
     runs-on: ubuntu-latest
-    needs:
-    - build
+    needs: [build]
     steps:
       - name: Download artifacts
         uses: actions/download-artifact@v4
@@ -132,24 +132,30 @@ jobs:
 
       - name: Running report
         env:
-          RESULT_ARTIFACT_URL: "https://openshift-provider-certification.s3.us-west-2.amazonaws.com"
-          RESULT_ARTIFACT_VERSION: "v0.4.0/default/4.15.0-20240228-HighlyAvailable-vsphere-None.tar.gz"
-          CUSTOM_BUILD_PATH: /tmp/build/opct-linux-amd64
+          BUCKET: openshift-provider-certification
+          REGION: us-west-2
+          OPCT_MODE: v0.4.0/default
+          EXEC_MODE: default
+          ARTIFACT: 4.15.0-20240228-HighlyAvailable-vsphere-None.tar.gz
+          OPCT: /tmp/build/opct-linux-amd64
         run: |
-          echo "> Downloading sample artifact: ${RESULT_ARTIFACT_URL}/${RESULT_ARTIFACT_VERSION}"
-          wget -qO /tmp/result.tar.gz "${RESULT_ARTIFACT_URL}/${RESULT_ARTIFACT_VERSION}"
+          URI=${OPCT_MODE}/${ARTIFACT}
+          URL=https://${BUCKET}.s3.${REGION}.amazonaws.com/${URI}
+
+          echo "> Downloading sample artifact: ${URL}"
+          wget -qO /tmp/result.tar.gz "${URL}"
 
           echo "> Setting run permissions to OPCT:"
-          chmod u+x ${CUSTOM_BUILD_PATH}
+          chmod u+x ${OPCT}
 
           echo "> Running OPCT report:"
-          ${CUSTOM_BUILD_PATH} report /tmp/result.tar.gz
+          ${OPCT} report /tmp/result.tar.gz
 
   e2e-cmd_adm-parse-etcd-logs:
     name: "e2e-cmd_adm-parse-etcd-logs"
     runs-on: ubuntu-latest
     needs:
-    - build
+      - build
     steps:
       - name: Download artifacts
         uses: actions/download-artifact@v4
@@ -159,14 +165,16 @@ jobs:
 
       - name: Preparing testdata
         env:
-          RESULT_ARTIFACT_URL: "https://openshift-provider-certification.s3.us-west-2.amazonaws.com"
-          RESULT_ARTIFACT_VERSION: "testdata/must-gather-etcd-logs.tar.gz"
+          BUCKET: openshift-provider-certification
+          REGION: us-west-2
+          VERSION: "testdata/must-gather-etcd-logs.tar.gz"
           CUSTOM_BUILD_PATH: /tmp/build/opct-linux-amd64
           LOCAL_TEST_DATA: /tmp/must-gather.tar.gz
           LOCAL_TEST_DATA_DIR: /tmp/must-gather
         run: |
-          echo "> Downloading sample artifact: ${RESULT_ARTIFACT_URL}/${RESULT_ARTIFACT_VERSION}"
-          wget -qO ${LOCAL_TEST_DATA} "${RESULT_ARTIFACT_URL}/${RESULT_ARTIFACT_VERSION}"
+          URL=https://${BUCKET}.s3.${REGION}.amazonaws.com
+          echo "> Downloading sample artifact: ${URL}/${VERSION}"
+          wget -qO ${LOCAL_TEST_DATA} "${URL}/${VERSION}"
 
           echo "> Setting run permissions to OPCT:"
           chmod u+x ${CUSTOM_BUILD_PATH}
@@ -178,37 +186,43 @@ jobs:
       - name: "e2e stdin reader: cat <etcd pod logs> | opct adm parse-etcd-logs"
         env:
           CUSTOM_BUILD_PATH: /tmp/build/opct-linux-amd64
-          LOCAL_TEST_DATA_DIR: /tmp/must-gather
+          TEST_DATA_DIR: /tmp/must-gather
+          NS_PATH_ETCD: namespaces/openshift-etcd/pods
+          LOG_PATH: etcd/etcd/logs
         run: |
-          cat ${LOCAL_TEST_DATA_DIR}/*/*/namespaces/openshift-etcd/pods/*/etcd/etcd/logs/*.log | \
+          cat ${TEST_DATA_DIR}/*/*/${NS_PATH_ETCD}/*/${LOG_PATH}/*.log | \
             ${CUSTOM_BUILD_PATH} adm parse-etcd-logs
 
       - name: "e2e must-gather dir: opct adm parse-etcd-logs <must-gather-dir>"
         env:
           CUSTOM_BUILD_PATH: /tmp/build/opct-linux-amd64
-          LOCAL_TEST_DATA_DIR: /tmp/must-gather
+          TEST_DATA_DIR: /tmp/must-gather
         run: |
-          ${CUSTOM_BUILD_PATH} adm parse-etcd-logs ${LOCAL_TEST_DATA_DIR}
+          ${CUSTOM_BUILD_PATH} adm parse-etcd-logs ${TEST_DATA_DIR}
 
-      - name: "e2e aggregate by hour: opct adm parse-etcd-logs --aggregator hour <must-gather-dir>"
+      - name: |
+          e2e aggregate by hour: opct adm parse-etcd-logs
+            --aggregator hour <must-gather-dir>
         env:
-          CUSTOM_BUILD_PATH: /tmp/build/opct-linux-amd64
-          LOCAL_TEST_DATA_DIR: /tmp/must-gather
+          OPCT: /tmp/build/opct-linux-amd64
+          TEST_DATA_DIR: /tmp/must-gather
         run: |
-          ${CUSTOM_BUILD_PATH} adm parse-etcd-logs --aggregator hour ${LOCAL_TEST_DATA_DIR}
+          ${OPCT} adm parse-etcd-logs --aggregator hour ${TEST_DATA_DIR}
 
-      - name: "e2e ignore error counters: opct adm parse-etcd-logs --skip-error-counter=true <must-gather-dir>"
+      - name: |
+          e2e ignore error counters: opct adm parse-etcd-logs
+            --skip-error-counter=true <must-gather-dir>
         env:
-          CUSTOM_BUILD_PATH: /tmp/build/opct-linux-amd64
-          LOCAL_TEST_DATA_DIR: /tmp/must-gather
+          OPCT: /tmp/build/opct-linux-amd64
+          TEST_DATA_DIR: /tmp/must-gather
         run: |
-          ${CUSTOM_BUILD_PATH} adm parse-etcd-logs --skip-error-counter=true ${LOCAL_TEST_DATA_DIR}
+          ${OPCT} adm parse-etcd-logs \
+            --skip-error-counter=true ${TEST_DATA_DIR}
 
   e2e-cmd_adm-parse-metrics:
     name: "e2e-cmd_adm-parse-metrics"
     runs-on: ubuntu-latest
-    needs:
-    - build
+    needs: [build]
     steps:
       - name: Download artifacts
         uses: actions/download-artifact@v4
@@ -218,21 +232,60 @@ jobs:
 
       - name: Preparing testdata
         env:
-          RESULT_ARTIFACT_URL: "https://openshift-provider-certification.s3.us-west-2.amazonaws.com"
-          RESULT_ARTIFACT_VERSION: "testdata/ci-external-aws-ccm_1757495441294888960-artifacts_must-gather-metrics.tar.xz"
+          BUCKET: openshift-provider-certification
+          REGION: us-west-2
+          PREFIX: testdata/ci-external-aws-ccm_
+          VERSION: 1757495441294888960-artifacts_must-gather-metrics.tar.xz
           CUSTOM_BUILD_PATH: /tmp/build/opct-linux-amd64
           LOCAL_TEST_DATA: /tmp/opct-metrics.tar.xz
         run: |
-          echo "> Downloading sample artifact: ${RESULT_ARTIFACT_URL}/${RESULT_ARTIFACT_VERSION}"
-          wget -qO ${LOCAL_TEST_DATA} "${RESULT_ARTIFACT_URL}/${RESULT_ARTIFACT_VERSION}"
+          DATA_VERSION=${PREFIX}${VERSION}
+          URL=https://${BUCKET}.s3.${REGION}.amazonaws.com
+          echo "> Downloading sample artifact: ${URL}/${DATA_VERSION}"
+          wget -qO ${LOCAL_TEST_DATA} "${URL}/${DATA_VERSION}"
 
-          echo "> Setting run permissions to OPCT:"
+          echo "> Setting exec permissions to OPCT:"
           chmod u+x ${CUSTOM_BUILD_PATH}
 
       - name: "e2e parse metrics: opct adm parse-etcd-logs <must-gather-dir>"
         env:
           CUSTOM_BUILD_PATH: /tmp/build/opct-linux-amd64
           LOCAL_TEST_DATA: /tmp/opct-metrics.tar.xz
         run: |
-          ${CUSTOM_BUILD_PATH} adm parse-metrics --input ${LOCAL_TEST_DATA} --output /tmp/metrics
+          ${CUSTOM_BUILD_PATH} adm parse-metrics \
+            --input ${LOCAL_TEST_DATA} --output /tmp/metrics
           tree /tmp/metrics
+
+  e2e-cmd_adm-baseline:
+    name: "e2e-cmd_adm-baseline"
+    runs-on: ubuntu-latest
+    needs: [build]
+    steps:
+      - name: Download artifacts
+        uses: actions/download-artifact@v4
+        with:
+          name: opct-linux-amd64
+          path: /tmp/build/
+
+      - name: Preparing testdata
+        env:
+          OPCT: /tmp/build/opct-linux-amd64
+        run: |
+          echo "> Setting exec permissions to OPCT:"
+          chmod u+x ${OPCT}
+
+      - name: "e2e adm baseline: opct adm baseline (list|get)"
+        env:
+          OPCT: /tmp/build/opct-linux-amd64
+        run: |
+          echo -e "\n\t#>> List latest baseline results"
+          ${OPCT} adm baseline list
+
+          echo -e "\n\t#>> List all baseline results"
+          ${OPCT} adm baseline list --all
+
+          echo -e "\n\t#>> Retrieve a baseline result by name"
+          ${OPCT} adm baseline get --name 4.16_None_latest --dump
+
+          echo -e "\n\t#>> Retrieve a baseline result by release and platform"
+          ${OPCT} adm baseline get --release 4.15 --platform None
diff --git a/.github/workflows/lint-yaml.yaml b/.github/workflows/lint-yaml.yaml
@@ -0,0 +1,28 @@
+---
+name: Lint YAML
+
+on:
+  push:
+    branches:
+      - main
+      - release-*
+  pull_request:
+    branches:
+      - main
+      - release-*
+
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v2
+
+      - name: Install yamllint
+        run: pip install yamllint
+
+      - name: Lint YAML files
+        run: |
+          echo ">> Running YAML lint on data/templates/plugins/"
+          yamllint data/templates/plugins/
diff --git a/.github/workflows/release.yaml b/.github/workflows/release.yaml
@@ -1,3 +1,4 @@
+---
 name: OPCT Release
 
 on:
@@ -48,7 +49,9 @@ jobs:
           podman login -u="${QUAY_USER}" -p="${QUAY_PASS}" quay.io
 
           echo "> Build container image:"
-          podman build -t ${CONTAINER_REPO}:${RELEASE_TAG} -f hack/Containerfile.ci .
+          podman build \
+            -t ${CONTAINER_REPO}:${RELEASE_TAG} \
+            -f hack/Containerfile.ci .
 
           echo "> Publish container image:"
           podman push ${CONTAINER_REPO}:${RELEASE_TAG}
@@ -83,6 +86,6 @@ jobs:
           body: |
             ## Changelog
             ${{steps.github_release.outputs.changelog}}
-            
+
             ## Container Images
             - [${{ env.CONTAINER_REPO }}:${{ env.RELEASE_TAG }}](https://quay.io/repository/ocp-cert/opct?tab=tags)
diff --git a/.gitignore b/.gitignore
@@ -5,6 +5,7 @@ kubeconfig
 
 # build files
 dist/
+build/
 
 # changelog is generated automaticaly by hack/generate-changelog.sh
 # available only in the rendered webpage (built by mkdocs).

diff --git a/Makefile b/Makefile
@@ -57,15 +57,30 @@ build-darwin-arm64: build
 linux-amd64-container: build-linux-amd64
 	podman build -t $(IMG):latest -f hack/Containerfile --build-arg=RELEASE_TAG=$(RELEASE_TAG) .
 
-.PHONY: image-mirror-sonobuoy
-image-mirror-sonobuoy:
-	./hack/image-mirror-sonobuoy/mirror.sh
-
-# Utils dev
-.PHONY: update-go
-update-go:
-	go get -u
-	go mod tidy
+# Publish devel binaries (non-official). Must be used only for troubleshooting in development/support.
+.PHONY: publish-amd64-devel
+publish-amd64-devel: build-linux-amd64
+	aws s3 cp $(BUILD_DIR)/opct-linux-amd64 s3://openshift-provider-certification/bin/opct-linux-amd64-devel
+
+.PHONY: publish-darwin-arm64-devel
+publish-darwin-arm64-devel: build-darwin-arm64
+	aws s3 cp $(BUILD_DIR)/opct-darwin-arm64 s3://openshift-provider-certification/bin/opct-darwin-arm64-devel
+
+.PHONY: publish-devel
+publish-devel: publish-amd64-devel
+publish-devel: publish-darwin-arm64-devel
+
+#
+# Test
+#
+
+.PHONY: test-lint
+test-lint:
+	@echo "Running linting tools"
+	# Download https://github.com/golangci/golangci-lint/releases/tag/v1.59.1
+	golangci-lint run --timeout=10m
+	# yamllint: pip install yamllint
+	yamllint .github/workflows/*.yaml
 
 .PHONY: test
 test:
@@ -90,3 +105,13 @@ build-changelog:
 .PHONY: build-docs
 build-docs: build-changelog
 	mkdocs build --site-dir ./site
+
+.PHONY: image-mirror-sonobuoy
+image-mirror-sonobuoy:
+	./hack/image-mirror-sonobuoy/mirror.sh
+
+# Utils dev
+.PHONY: update-go
+update-go:
+	go get -u
+	go mod tidy
-Original file line number
+Diff line change
@@ Expand Up / @@ -5,6 +5,7 @@ kubeconfig @@
     # build files
     dist/
+    build/
     # changelog is generated automaticaly by hack/generate-changelog.sh
     # available only in the rendered webpage (built by mkdocs).
@@ Expand Down @@