OPCT-226: opct adm parsers for etcd and metrics (#102)

Introducing the following commands to consume subset of features introduced on `opct report` (v0.5+, #76): - `opct adm parse-etcd-logs`: parse logs from etcd pod logs from must-gather providing a summary w/ timers from the cluster opration - `opct adm parse-metrics`: parse metrics read from Prometheus collected by OPCT and generate charts in HTML files The commands are a subset of improvements in UI (CLI and Web) of `opct report` that is planning to be available in the stable of v0.5. This subset of changes were extracted from under development OPCT report PR: #76
redhat-openshift-ecosystem · Jul 26, 2024 · 48eb4cf · 48eb4cf
1 parent 11d9e8e
commit 48eb4cf
Show file tree

Hide file tree

Showing 31 changed files with 3,059 additions and 74 deletions.
diff --git a/.github/workflows/go.yaml b/.github/workflows/go.yaml
@@ -9,35 +9,40 @@ on:
     tags:
       - '*'
 
+env:
+  GO_VERSION: 1.22
+  GOLANGCI_LINT_VERSION: v1.59
+  STATIC_CHECK_VERSION: 2023.1.7
+
 jobs:
   go-lint:
     name: go-lint
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v3
-      - uses: actions/setup-go@v4
+      - uses: actions/checkout@v4
+      - uses: actions/setup-go@v5
         with:
-          go-version: '1.22'
+          go-version: ${{ env.GO_VERSION }}
           cache: false
       # https://github.com/golangci/golangci-lint-action
       - name: golangci-lint
         uses: golangci/golangci-lint-action@v3
         with:
-          version: v1.55.2
+          version: ${{ env.GOLANGCI_LINT_VERSION }}
           args: --timeout=10m
 
   go-static:
     name: "go-staticcheck"
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
       - uses: WillAbides/[email protected]
         with:
-          go-version: '1.22'
+          go-version: ${{ env.GO_VERSION }}
       - name: Run static code analysis
         uses: dominikh/[email protected]
         with:
-          version: "2023.1.7"
+          version: "${{ env.STATIC_CHECK_VERSION }}"
           install-go: false
 
   go-test:
@@ -47,10 +52,10 @@ jobs:
     - go-lint
     - go-static
     steps:
-      - uses: actions/checkout@v3
-      - uses: actions/setup-go@v4
+      - uses: actions/checkout@v4
+      - uses: actions/setup-go@v5
         with:
-          go-version: '1.22'
+          go-version: ${{ env.GO_VERSION }}
       - name: Run unit tests
         run: make test
 
@@ -61,10 +66,10 @@ jobs:
     - go-lint
     - go-static
     steps:
-      - uses: actions/checkout@v3
-      - uses: actions/setup-go@v4
+      - uses: actions/checkout@v4
+      - uses: actions/setup-go@v5
         with:
-          go-version: '1.22'
+          go-version: ${{ env.GO_VERSION }}
       - name: Run go vet
         run: make vet
 
@@ -75,13 +80,10 @@ jobs:
     - go-test
     - go-vet
     steps:
-      - name: Checkout
-        uses: actions/checkout@v3
-
-      - name: Set up Go
-        uses: actions/setup-go@v4
+      - uses: actions/checkout@v4
+      - uses: actions/setup-go@v5
         with:
-          go-version: 1.22
+          go-version: ${{ env.GO_VERSION }}
 
       - name: Install dependencies
         run: |
@@ -97,7 +99,7 @@ jobs:
           make build-${OS_ARCH}
 
       - name: Save artifacts (OS=linux-amd64)
-        uses: actions/upload-artifact@v3
+        uses: actions/upload-artifact@v4
         with:
           name: opct-linux-amd64
           path: |
@@ -111,19 +113,19 @@ jobs:
           make build-${OS_ARCH}
 
       - name: Save artifacts (OS=darwin-arm64)
-        uses: actions/upload-artifact@v3
+        uses: actions/upload-artifact@v4
         with:
           name: opct-darwin-arm64
           path: |
             build/opct-*
-  cmd-report:
-    name: "run-report"
+  e2e-cmd_report:
+    name: "e2e-cmd_report"
     runs-on: ubuntu-latest
     needs:
     - build
     steps:
       - name: Download artifacts
-        uses: actions/download-artifact@v3
+        uses: actions/download-artifact@v4
         with:
           name: opct-linux-amd64
           path: /tmp/build/
@@ -142,3 +144,95 @@ jobs:
 
           echo "> Running OPCT report:"
           ${CUSTOM_BUILD_PATH} report /tmp/result.tar.gz
+
+  e2e-cmd_adm-parse-etcd-logs:
+    name: "e2e-cmd_adm-parse-etcd-logs"
+    runs-on: ubuntu-latest
+    needs:
+    - build
+    steps:
+      - name: Download artifacts
+        uses: actions/download-artifact@v4
+        with:
+          name: opct-linux-amd64
+          path: /tmp/build/
+
+      - name: Preparing testdata
+        env:
+          RESULT_ARTIFACT_URL: "https://openshift-provider-certification.s3.us-west-2.amazonaws.com"
+          RESULT_ARTIFACT_VERSION: "testdata/must-gather-etcd-logs.tar.gz"
+          CUSTOM_BUILD_PATH: /tmp/build/opct-linux-amd64
+          LOCAL_TEST_DATA: /tmp/must-gather.tar.gz
+          LOCAL_TEST_DATA_DIR: /tmp/must-gather
+        run: |
+          echo "> Downloading sample artifact: ${RESULT_ARTIFACT_URL}/${RESULT_ARTIFACT_VERSION}"
+          wget -qO ${LOCAL_TEST_DATA} "${RESULT_ARTIFACT_URL}/${RESULT_ARTIFACT_VERSION}"
+
+          echo "> Setting run permissions to OPCT:"
+          chmod u+x ${CUSTOM_BUILD_PATH}
+
+          echo "> Extracting testdata:"
+          mkdir ${LOCAL_TEST_DATA_DIR}
+          tar xfz ${LOCAL_TEST_DATA} -C ${LOCAL_TEST_DATA_DIR}
+
+      - name: "e2e stdin reader: cat <etcd pod logs> | opct adm parse-etcd-logs"
+        env:
+          CUSTOM_BUILD_PATH: /tmp/build/opct-linux-amd64
+          LOCAL_TEST_DATA_DIR: /tmp/must-gather
+        run: |
+          cat ${LOCAL_TEST_DATA_DIR}/*/*/namespaces/openshift-etcd/pods/*/etcd/etcd/logs/*.log | \
+            ${CUSTOM_BUILD_PATH} adm parse-etcd-logs
+
+      - name: "e2e must-gather dir: opct adm parse-etcd-logs <must-gather-dir>"
+        env:
+          CUSTOM_BUILD_PATH: /tmp/build/opct-linux-amd64
+          LOCAL_TEST_DATA_DIR: /tmp/must-gather
+        run: |
+          ${CUSTOM_BUILD_PATH} adm parse-etcd-logs ${LOCAL_TEST_DATA_DIR}
+
+      - name: "e2e aggregate by hour: opct adm parse-etcd-logs --aggregator hour <must-gather-dir>"
+        env:
+          CUSTOM_BUILD_PATH: /tmp/build/opct-linux-amd64
+          LOCAL_TEST_DATA_DIR: /tmp/must-gather
+        run: |
+          ${CUSTOM_BUILD_PATH} adm parse-etcd-logs --aggregator hour ${LOCAL_TEST_DATA_DIR}
+
+      - name: "e2e ignore error counters: opct adm parse-etcd-logs --skip-error-counter=true <must-gather-dir>"
+        env:
+          CUSTOM_BUILD_PATH: /tmp/build/opct-linux-amd64
+          LOCAL_TEST_DATA_DIR: /tmp/must-gather
+        run: |
+          ${CUSTOM_BUILD_PATH} adm parse-etcd-logs --skip-error-counter=true ${LOCAL_TEST_DATA_DIR}
+
+  e2e-cmd_adm-parse-metrics:
+    name: "e2e-cmd_adm-parse-metrics"
+    runs-on: ubuntu-latest
+    needs:
+    - build
+    steps:
+      - name: Download artifacts
+        uses: actions/download-artifact@v4
+        with:
+          name: opct-linux-amd64
+          path: /tmp/build/
+
+      - name: Preparing testdata
+        env:
+          RESULT_ARTIFACT_URL: "https://openshift-provider-certification.s3.us-west-2.amazonaws.com"
+          RESULT_ARTIFACT_VERSION: "testdata/ci-external-aws-ccm_1757495441294888960-artifacts_must-gather-metrics.tar.xz"
+          CUSTOM_BUILD_PATH: /tmp/build/opct-linux-amd64
+          LOCAL_TEST_DATA: /tmp/opct-metrics.tar.xz
+        run: |
+          echo "> Downloading sample artifact: ${RESULT_ARTIFACT_URL}/${RESULT_ARTIFACT_VERSION}"
+          wget -qO ${LOCAL_TEST_DATA} "${RESULT_ARTIFACT_URL}/${RESULT_ARTIFACT_VERSION}"
+
+          echo "> Setting run permissions to OPCT:"
+          chmod u+x ${CUSTOM_BUILD_PATH}
+
+      - name: "e2e parse metrics: opct adm parse-etcd-logs <must-gather-dir>"
+        env:
+          CUSTOM_BUILD_PATH: /tmp/build/opct-linux-amd64
+          LOCAL_TEST_DATA: /tmp/opct-metrics.tar.xz
+        run: |
+          ${CUSTOM_BUILD_PATH} adm parse-metrics --input ${LOCAL_TEST_DATA} --output /tmp/metrics
+          tree /tmp/metrics
diff --git a/cmd/root.go b/cmd/root.go
@@ -10,6 +10,7 @@ import (
 	"github.com/spf13/viper"
 	"github.com/vmware-tanzu/sonobuoy/cmd/sonobuoy/app"
 
+	"github.com/redhat-openshift-ecosystem/provider-certification-tool/pkg/cmd/adm"
 	"github.com/redhat-openshift-ecosystem/provider-certification-tool/pkg/cmd/get"
 	"github.com/redhat-openshift-ecosystem/provider-certification-tool/pkg/destroy"
 	"github.com/redhat-openshift-ecosystem/provider-certification-tool/pkg/report"
@@ -74,6 +75,7 @@ func init() {
 	rootCmd.AddCommand(version.NewCmdVersion())
 	rootCmd.AddCommand(report.NewCmdReport())
 	rootCmd.AddCommand(get.NewCmdGet())
+	rootCmd.AddCommand(adm.NewCmdAdm())
 
 	// Link in child commands direct from Sonobuoy
 	rootCmd.AddCommand(app.NewSonobuoyCommand())

diff --git a/docs/opct/adm/parse-etcd-logs.md b/docs/opct/adm/parse-etcd-logs.md
@@ -0,0 +1,46 @@
+# opct adm parse-etcd-logs
+
+Extract information from etcd logs from pods collected by must-gather.
+
+## Usage
+
+- Added to OPCT in release: v0.5.0-alpha.3
+- Command: `opct adm parse-etcd-logs options args`
+
+Options:
+
+- `--aggregator`: choose aggregator (all, day, hour, minute). Default: hour
+- `--skip-error-counter`: flag to skip the error counter calculatio to a faster report. Default: false
+
+Args:
+
+- `path/to/must-gather/directory` (optional)
+
+## Examples
+
+- Read from stdin:
+
+```bash
+export MUST_GATHER_PATH=./must-gather
+tar xfz must-gather.tar.gz -C ${MUST_GATHER_PATH}
+cat ${MUST_GATHER_PATH}/*/*/namespaces/openshift-etcd/pods/*/etcd/etcd/logs/*.log |\
+    opct adm parse-etcd-logs
+```
+
+- Parse a directory with must-gather:
+
+```bash
+opct adm parse-etcd-logs ${MUST_GATHER_PATH}
+```
+
+- Aggregate by day:
+
+```bash
+opct adm parse-etcd-logs --aggregator day ${MUST_GATHER_PATH}
+```
+
+- Ignore error counters:
+
+```bash
+opct adm parse-etcd-logs --skip-error-counter true ${MUST_GATHER_PATH} 
+```
diff --git a/docs/opct/adm/parse-metrics.md b/docs/opct/adm/parse-metrics.md
@@ -0,0 +1,82 @@
+# opct adm parse-metrics
+
+Process Prometheus metrics plotting HTML charts.
+
+## Usage
+
+- Added to OPCT in release: v0.5.0-alpha.3
+- Feature status: beta
+- Command: `opct adm parse-metrics options`
+
+Options:
+
+- `--input`: Input metrics file. Example: metrics.tar.xz
+- `--output`: Output directory. Example: /tmp/metrics
+
+## Metrics collector
+
+The metrics can be collected into two different ways:
+
+- OPCT archive (version v0.5.3-alha.3+)
+- must-gather-monitoring utility
+
+
+## Examples
+
+### Plot the metrics charts collected by OPCT
+
+1. Extract the must-gather-monitoring from OPCT archive
+
+```bash
+tar xfz archive.tar.gz plugins/99-openshift-artifacts-collector/results/global/artifacts_must-gather-metrics.tar.xz
+```
+
+2. Process the metrics generating charts
+
+```bash
+./opct adm parse-metrics \
+    --input plugins/99-openshift-artifacts-collector/results/global/artifacts_must-gather-metrics.tar.xz \
+    --output ./metrics
+```
+
+3. Open the metrics directory from your file to explore the charts.
+
+- `metrics.html`: charts plotted with [go-echarts](https://github.com/go-echarts)
+- `index.html`: charts plotted with [Plotly](https://plotly.com/javascript/)
+
+To explore the full javascript features, use a HTTP file server to view the charts:
+
+```bash
+cd ./metrics && python -m http.server 9090
+```
+
+### Plot the metrics charts collected by must-gather-monitoring
+
+1. Run must-gather-monitoring to collect the metrics
+
+```bash
+oc adm must-gather --image=quay.io/opct/must-gather-monitoring:v0.1.0 &&\
+tar xfJ must-gather-metrics.tar.xz
+```
+
+2. Process the metrics generating charts
+
+```bash
+./opct adm parse-metrics \
+    --input must-gather-metrics.tar.xz \
+    --output ./metrics
+```
+
+### Plot the metrics natively from OPCT report
+
+`opct report` command generates charts automatically when
+the metrics is available and the report HTML is enabled.
+
+- Generate the report:
+```bash
+./opct report archive.tar.gz --save-to ./report
+```
+
+- Open the HTML report in your browser at http://localhost:9090/metrics
+
+Read more about `opct report` in the [documentation](../report.md).