Skip to content

Commit

Permalink
Fix the error of vineyardctl and add the related test.
Browse files Browse the repository at this point in the history
Signed-off-by: Ye Cao <[email protected]>
  • Loading branch information
dashanji committed Aug 22, 2024
1 parent f1d9c37 commit 77262e2
Show file tree
Hide file tree
Showing 4 changed files with 89 additions and 0 deletions.
6 changes: 6 additions & 0 deletions .github/workflows/vineyard-operator.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@ jobs:
- e2e-tests-workflow
- e2e-tests-mars-examples
- e2e-tests-vineyardctl
- e2e-tests-vineyardctl-python-api
- e2e-tests-deploy-raw-backup-and-recover
- e2e-tests-three-etcd-nodes-failover
- e2e-tests-five-etcd-nodes-failover
Expand Down Expand Up @@ -278,6 +279,11 @@ jobs:
if: ${{ matrix.job == 'e2e-tests-vineyardctl' }}
run: |
make -C k8s/test/e2e e2e-tests-vineyardctl
- name: e2e-tests-vineyardctl-python-api
if: ${{ matrix.job == 'e2e-tests-vineyardctl-python-api' }}
run: |
make -C k8s/test/e2e e2e-tests-vineyardctl-python-api
- name: e2e-tests-deploy-raw-backup-and-recover
if: ${{ matrix.job == 'e2e-tests-deploy-raw-backup-and-recover' }}
Expand Down
3 changes: 3 additions & 0 deletions k8s/cmd/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,9 @@ func init() {

flags.ApplyGlobalFlags(cmd)

if cmd.Flags().Lookup("log-flush-frequency") != nil {
cmd.Flags().MarkHidden("log-flush-frequency")

Check failure on line 67 in k8s/cmd/main.go

View workflow job for this annotation

GitHub Actions / ci (e2e-tests-workflow)

Error return value of `(*github.com/spf13/pflag.FlagSet).MarkHidden` is not checked (errcheck)

Check failure on line 67 in k8s/cmd/main.go

View workflow job for this annotation

GitHub Actions / ci (e2e-tests-schedule-workflow-without-crd)

Error return value of `(*github.com/spf13/pflag.FlagSet).MarkHidden` is not checked (errcheck)

Check failure on line 67 in k8s/cmd/main.go

View workflow job for this annotation

GitHub Actions / ci (e2e-tests-deploy-raw-backup-and-recover)

Error return value of `(*github.com/spf13/pflag.FlagSet).MarkHidden` is not checked (errcheck)

Check failure on line 67 in k8s/cmd/main.go

View workflow job for this annotation

GitHub Actions / ci (e2e-tests-serialize)

Error return value of `(*github.com/spf13/pflag.FlagSet).MarkHidden` is not checked (errcheck)

Check failure on line 67 in k8s/cmd/main.go

View workflow job for this annotation

GitHub Actions / ci (e2e-tests-five-etcd-nodes-failover)

Error return value of `(*github.com/spf13/pflag.FlagSet).MarkHidden` is not checked (errcheck)

Check failure on line 67 in k8s/cmd/main.go

View workflow job for this annotation

GitHub Actions / ci (e2e-tests-autogenerated-helm-chart)

Error return value of `(*github.com/spf13/pflag.FlagSet).MarkHidden` is not checked (errcheck)

Check failure on line 67 in k8s/cmd/main.go

View workflow job for this annotation

GitHub Actions / ci (e2e-tests-schedule-workflow)

Error return value of `(*github.com/spf13/pflag.FlagSet).MarkHidden` is not checked (errcheck)

Check failure on line 67 in k8s/cmd/main.go

View workflow job for this annotation

GitHub Actions / ci (e2e-tests-three-etcd-nodes-failover)

Error return value of `(*github.com/spf13/pflag.FlagSet).MarkHidden` is not checked (errcheck)

Check failure on line 67 in k8s/cmd/main.go

View workflow job for this annotation

GitHub Actions / ci (e2e-tests-airflow-integration)

Error return value of `(*github.com/spf13/pflag.FlagSet).MarkHidden` is not checked (errcheck)

Check failure on line 67 in k8s/cmd/main.go

View workflow job for this annotation

GitHub Actions / ci (unit-tests)

Error return value of `(*github.com/spf13/pflag.FlagSet).MarkHidden` is not checked (errcheck)

Check failure on line 67 in k8s/cmd/main.go

View workflow job for this annotation

GitHub Actions / ci (e2e-tests-vineyardctl-python-api)

Error return value of `(*github.com/spf13/pflag.FlagSet).MarkHidden` is not checked (errcheck)

Check failure on line 67 in k8s/cmd/main.go

View workflow job for this annotation

GitHub Actions / ci (e2e-tests-assembly-distributed)

Error return value of `(*github.com/spf13/pflag.FlagSet).MarkHidden` is not checked (errcheck)

Check failure on line 67 in k8s/cmd/main.go

View workflow job for this annotation

GitHub Actions / ci (e2e-tests-failover)

Error return value of `(*github.com/spf13/pflag.FlagSet).MarkHidden` is not checked (errcheck)

Check failure on line 67 in k8s/cmd/main.go

View workflow job for this annotation

GitHub Actions / ci (e2e-tests-sidecar)

Error return value of `(*github.com/spf13/pflag.FlagSet).MarkHidden` is not checked (errcheck)

Check failure on line 67 in k8s/cmd/main.go

View workflow job for this annotation

GitHub Actions / ci (e2e-tests-repartition-dask)

Error return value of `(*github.com/spf13/pflag.FlagSet).MarkHidden` is not checked (errcheck)

Check failure on line 67 in k8s/cmd/main.go

View workflow job for this annotation

GitHub Actions / ci (release)

Error return value of `(*github.com/spf13/pflag.FlagSet).MarkHidden` is not checked (errcheck)

Check failure on line 67 in k8s/cmd/main.go

View workflow job for this annotation

GitHub Actions / ci (e2e-tests-vineyardctl)

Error return value of `(*github.com/spf13/pflag.FlagSet).MarkHidden` is not checked (errcheck)

Check failure on line 67 in k8s/cmd/main.go

View workflow job for this annotation

GitHub Actions / ci (e2e-tests-assembly-local)

Error return value of `(*github.com/spf13/pflag.FlagSet).MarkHidden` is not checked (errcheck)

Check failure on line 67 in k8s/cmd/main.go

View workflow job for this annotation

GitHub Actions / ci (e2e-tests-spill)

Error return value of `(*github.com/spf13/pflag.FlagSet).MarkHidden` is not checked (errcheck)

Check failure on line 67 in k8s/cmd/main.go

View workflow job for this annotation

GitHub Actions / ci (e2e-tests-schedule-workload)

Error return value of `(*github.com/spf13/pflag.FlagSet).MarkHidden` is not checked (errcheck)

Check failure on line 67 in k8s/cmd/main.go

View workflow job for this annotation

GitHub Actions / ci (e2e-tests-mars-examples)

Error return value of `(*github.com/spf13/pflag.FlagSet).MarkHidden` is not checked (errcheck)
}
// disable completion command
cmd.CompletionOptions.DisableDefaultCmd = true

Expand Down
10 changes: 10 additions & 0 deletions k8s/test/e2e/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -384,6 +384,16 @@ e2e-tests-vineyardctl: prepare-e2e-test build-local-cluster load-vineyardd-image
@make delete-local-cluster
.PHONY: e2e-tests-vineyardctl

############# vineyardctl python api testing ################################################

e2e-tests-vineyardctl-python-api: prepare-e2e-test build-local-cluster load-vineyardd-image load-vineyard-python-dev-image
@make -C ${E2E_DIR} install-vineyard-operator
@echo "Running vineyardctl python api e2e test..."
@cd ${ROOT_DIR} && ${GOBIN}/e2e run --config=${E2E_DIR}/vineyardctl-python-api/e2e.yaml
@echo "vineyardctl python api e2e test passed."
@make delete-local-cluster
.PHONY: e2e-tests-vineyardctl-python-api

############# backup and recover testing ################################################

e2e-tests-deploy-raw-backup-and-recover: prepare-e2e-test build-local-cluster load-vineyardd-image
Expand Down
70 changes: 70 additions & 0 deletions k8s/test/e2e/vineyardctl-python-api/e2e.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
# Copyright 2020-2023 Alibaba Group Holding Limited.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

setup:
env: kind
kubeconfig: /tmp/e2e-k8s.config
steps:
- name: download all workflow images into kind cluster
command: |
make -C k8s/test/e2e publish-workflow-images REGISTRY=localhost:5001
- name: install vineyardd
command: |
docker run --net=host -v /tmp/e2e-k8s.config:/root/.kube/config:ro localhost:5001/vineyard-python-dev:latest \
"python -c \"import vineyard; vineyard.deploy.vineyardctl.deploy.vineyardd(vineyardd_image='localhost:5001/vineyardd:latest', kubeconfig='/root/.kube/config')\""
- name: install job1
command: |
kubectl create namespace vineyard-job
kubectl apply -f k8s/test/e2e/workflow-demo/workflow-job1.yaml
wait:
- namespace: vineyard-job
resource: job/v6d-workflow-demo-job1
for: condition=Complete
- name: install job2
command: |
kubectl apply -f k8s/test/e2e/workflow-demo/workflow-job2.yaml
wait:
- namespace: vineyard-job
resource: job/v6d-workflow-demo-job2
for: condition=Complete
timeout: 20m

cleanup:
# always never success failure
on: never

verify:
# verify with retry strategy
retry:
# max retry count
count: 10
# the interval between two attempts, e.g. 10s, 1m.
interval: 10s
cases:
- query: |
kubectl get pod -l app=v6d-workflow-demo-job2 -n vineyard-job -oname | \
awk -F '/' '{print $2}' | \
head -n 1 | \
xargs kubectl logs -n vineyard-job | \
yq e '{"sum": .}' - | \
yq e 'to_entries' -
expected: ../verify/values.yaml
# test the job can only be scheduled on the nodes with the vineyardd
- query: |
export job1_nodes=$(kubectl get pod -l app=v6d-workflow-demo-job1 -nvineyard-job -o=custom-columns=NODE:.spec.nodeName | awk 'NR != 1' | sort | tr '\n' ' ')
export job2_nodes=$(kubectl get pod -l app=v6d-workflow-demo-job2 -nvineyard-job -o=custom-columns=NODE:.spec.nodeName | awk 'NR != 1' | sort | tr '\n' ' ')
export vineyardd_nodes=$(kubectl get pod -l app.kubernetes.io/instance=vineyardd -n vineyard-system -o=custom-columns=NODE:.spec.nodeName | awk 'NR != 1' | sort | tr '\n' ' ')
if [[ $job1_nodes = $vineyardd_nodes ]] && [[ $job2_nodes = $vineyardd_nodes ]]; then echo '{"AllJobInVineyarddNodes":"true"}' | yq e 'to_entries' -; fi
expected: ../verify/nodes.yaml

0 comments on commit 77262e2

Please sign in to comment.