Releases: redhat-best-practices-for-k8s/certsuite
v4.1.2 Dep updates, timeout increases, bugfixes
What's Changed
- Bump github.com/test-network-function/privileged-daemonset from 0.0.4 to 0.0.5 by @dependabot in #671
- Update certification files by @github-actions in #675
- Fix minor typos in bootparams by @sebrandon1 in #669
- Removed unused scaling test case identifier. by @greyerof in #678
- Use the right variable for bad statefulsets. by @greyerof in #679
- Fix networking tests that fail with istio-proxy related issues by @sebrandon1 in #673
- Switch context.Background() to context.TODO() for consistency by @sebrandon1 in #681
- Update certification files by @github-actions in #682
- Use '%v' when referencing an error by @sebrandon1 in #680
- Zeroing iptables chains counters before comparing by @edcdavid in #684
- Update certification files by @github-actions in #685
- Remove fetch tool to use OCT instead by @jmontesi in #677
- Make prefixes list for the port names check to be configurable by @shimritproj in #683
- Add missed parameters to an error message. by @rdavid in #686
- Update RHCOS to OCP version map by @github-actions in #687
- Simplify log output by @rdavid in #688
- Recording all operators in claims file by @edcdavid in #674
- Update RHCOS to OCP version map by @github-actions in #690
- Run lifecycle-pod-recreation TC in intrusive mode only. by @greyerof in #689
- Bump github.com/onsi/ginkgo/v2 from 2.5.0 to 2.5.1 by @dependabot in #692
- Make the offline DB location configurable by @jmontesi in #691
- Print more error data in case of a failure by @rdavid in #693
- Optimize and clean the TNF image by @jmontesi in #694
- Update RHCOS to OCP version map by @github-actions in #697
- access-control: change detection of SSH daemons by @jmontesi in #696
- copy missing oc file and add it to env variable by @edcdavid in #698
- update path env again by @edcdavid in #700
- Stop if oc doesn't exist by @rdavid in #699
- Update RHCOS to OCP version map by @github-actions in #703
- upgrading some test cases to normative by @edcdavid in #702
- Get rid of 'oc' binary dependency. by @greyerof in #701
- Tweaking result types for select testcases by @edcdavid in #706
- Update RHCOS to OCP version map by @github-actions in #707
- Allow Guaranteed Pods to be tested by lifecycle-pod-scheduling TC. by @greyerof in #705
- internal/api: refactor the api with the certification DB by @jmontesi in #704
- Remove oc related check by @rdavid in #710
- Fix certdb unit tests by @jmontesi in #713
- Remove certtool by @jmontesi in #712
- Fix shellcheck errors, logic is not changed by @rdavid in #714
- Bump github.com/operator-framework/api from 0.17.1 to 0.17.2 by @dependabot in #715
- Fix minor typos by @sebrandon1 in #716
- Update RHCOS to OCP version map by @github-actions in #717
- Update Go to v1.19.4 by @sebrandon1 in #718
- Bump timeout to 5 minutes for scaling tests by @sebrandon1 in #719
- Update version to v4.1.2 by @sebrandon1 in #720
Full Changelog: v4.1.1...v4.1.2
v4.1.1 - Patch Update; dep updates
Patch Release v4.1.1
Notable changes:
- #607 added test for security context from the v1.4 requirements document
- #661 fixes panic for SYS_PTRACE
- #635 added the ability to skip certain deployments and statefulsets via the TNF config to exclude them from the scaling tests
What's Changed
- Fix broken link by @sebrandon1 in #641
- Update certification files by @github-actions in #642
- Move more operator-related items by @sebrandon1 in #638
- Fix script execution by @bnshr in #643
- Add ability to skip scaling tests via config by @sebrandon1 in #635
- Update certification files by @github-actions in #649
- Move 4.7 to 'end of life' section by @sebrandon1 in #645
- Bump github.com/onsi/ginkgo/v2 from 2.4.0 to 2.5.0 by @dependabot in #651
- Add FailNow() if config marshal fails by @sebrandon1 in #648
- Created merged kube config file in the clientsholder. by @greyerof in #644
- Update checkout to v3; missed in previous updates by @sebrandon1 in #652
- Add tests for testhelper pkg by @sebrandon1 in #646
- Manually update Ginkgo to v2.5.0 by @sebrandon1 in #654
- Update certification files by @github-actions in #653
- Remove WaitForDebugDaemonset funcs by @sebrandon1 in #655
- Update RHCOS to OCP version map by @github-actions in #659
- Update certification files by @github-actions in #658
- add new test for the sec context by @aabughosh in #607
- Bump helm.sh/helm/v3 from 3.10.1 to 3.10.2 by @dependabot in #662
- Change StringInSlice to work with any string based type by @edcdavid in #657
- Bump k8s.io/kubectl from 0.25.3 to 0.25.4 by @dependabot in #666
- Update RHCOS to OCP version map by @github-actions in #668
- Add tests to certtool package by @sebrandon1 in #656
- Update certification files by @github-actions in #667
- Fix for SYS_PTRACE panic and logic by @edcdavid in #661
- Fetch the CNF DB using OCT by @jmontesi in #650
- Removed duplication of deployment scaling by @shaior in #660
- Bump partner to v4.1.1 by @sebrandon1 in #670
Full Changelog: v4.1.0...v4.1.1
v4.1.0 - Best practices v1.4 support
Document v1.4 support!
Our latest release has a large number of changes pertaining to the new version v1.4 best practices document.
What's Changed
- use Goclient to dynamically deploy the partner repo daemonset by @shimritproj in #309
- Skip pod-high-availability and pod-scheduling for SNO by @rdavid in #385
- networking: new test to check the usage of OCP reserved ports by @jmontesi in #384
- Add test for requests and limits for containers by @sebrandon1 in #372
- Fix reverse order of suites with trailing space by @rdavid in #389
- Update Go to 1.18.5 by @sebrandon1 in #391
- Update golangci-lint to v1.47.3; remove staticcheck workaround by @sebrandon1 in #392
- Add namespace resource quota test by @sebrandon1 in #386
- networking: refactor some utility functions by @jmontesi in #396
- Send output to the claim file for sys_nice by @sebrandon1 in #399
- Fix for platform-alteration-hugepages-config test case. by @greyerof in #395
- Update the 4.9 OCP compatibility dates by @sebrandon1 in #400
- Add toleration check test by @sebrandon1 in #365
- lifecycle: add new test case to check for containers startup probes by @jmontesi in #398
- Remove containerd replace statement in go.mod by @sebrandon1 in #403
- Fix capitalization in the catalog by @sebrandon1 in #401
- Print to claim file instead of logrus by @sebrandon1 in #406
- Change daemonset image to use partner repo if provided by @ramperher in #402
- Fix toleration err message by @sebrandon1 in #409
- Add test for persistent volume reclaim policy by @sebrandon1 in #390
- Removed err return from GetMcKernelArguments as its always nil by @shaior in #412
- Fix capitalization in 'OpenShift' by @sebrandon1 in #413
- Bump github.com/operator-framework/api from 0.15.0 to 0.16.0 by @dependabot in #414
- Do not fail if memory-pressure toleration is present and qosClass is different than BestEffort by @ramperher in #417
- Bump helm.sh/helm/v3 from 3.9.2 to 3.9.3 by @dependabot in #421
- observability: new test to check the validity of Pod Disruption Budgets by @jmontesi in #416
- Update compatibility pkg for 4.11 release by @sebrandon1 in #422
- Upgrade GolangCI-lint to v1.48.0 by @sebrandon1 in #427
- observability: add new failure condition to the PDB test case by @jmontesi in #430
- Update -claim to v1.0.6 by @sebrandon1 in #431
- Adding test to verify that all services are either single stack ipv6 or dual-stack by @edcdavid in #424
- access-control: add test to verify that no SSH daemons are run in a pod by @jmontesi in #433
- Bump k8s.io/client-go from 0.24.3 to 0.24.4 by @dependabot in #438
- Bump github.com/test-network-function/test-network-function-claim from 1.0.6 to 1.0.7 by @dependabot in #446
- Add test tags to allow the selection of a particular set of test cases by @jmontesi in #444
- Bump k8s.io/client-go from 0.24.4 to 0.25.0 by @dependabot in #450
- Delete schemas folder by @sebrandon1 in #447
- Bump helm.sh/helm/v3 from 3.9.3 to 3.9.4 by @dependabot in #454
- Typo in README by @rdavid in #455
- CNFCERT-322:add test for checking image tag exists] by @aabughosh in #441
- Adding iptables and nftables testcases by @edcdavid in #435
- Upgrade GolangCI-lint to v1.49.0 by @sebrandon1 in #456
- Bump github.com/operator-framework/api from 0.16.0 to 0.17.0 by @dependabot in #462
- Bump github.com/onsi/ginkgo/v2 from 2.1.4 to 2.1.5 by @dependabot in #463
- Ignoring nftable and iptables rules set by machine-config operator by @edcdavid in #460
- Remove deprecated linters by @sebrandon1 in #466
- Manually bump Ginkgo to v2.1.5 by @sebrandon1 in #467
- Bump github.com/onsi/ginkgo/v2 from 2.1.5 to 2.1.6 by @dependabot in #469
- Helper for adding test cases in the catalog by @edcdavid in #452
- add ipv6 whitelist for ip6tables by @edcdavid in #468
- Add test for deny-all network policy by @sebrandon1 in #423
- Add fix for the sysctlCommand in sysctl-config tc by @shaior in #476
- autodiscover: print error messages by @jmontesi in #475
- Ginkgo v2.1.6 manual update by @sebrandon1 in #477
- Remove unused func by @sebrandon1 in #479
- Add test for onlinecheck package by @sebrandon1 in #480
- Move DenyAllIdentifier to new scheme by @sebrandon1 in #481
- bugfix: adding missing suite name as a parameter by @edcdavid in #484
- Add networking test key by @sebrandon1 in #488
- Fetch tool improvement to minimize changes in operator pages. by @greyerof in #485
- Add test for LabelsMatch by @sebrandon1 in #478
- Add skip for reclaim policy test by @sebrandon1 in #487
- pkg/tnf/identifier: remove unused package by @jmontesi in #490
- Fix test cases placement by @jmontesi in #491
- Add CPU isolation test by @sebrandon1 in #434
- Update Go to 1.18.6 by @sebrandon1 in #492
- Exclude guaranteed pods from node selector check by @sebrandon1 in #498
- run-tnf-container.sh: add the -l option to allow using labels by @jmontesi in #496
- Consolidate into deployment/statefulset structs by @sebrandon1 in #501
- Add extended test for UID 1337 by @sebrandon1 in #495
- fix bad image name in run-tnf-container.sh by @edcdavid in #504
- Switch Pod and Container to nest the k8s str...
v4.0.2 New tests, dependency updates, bug fixes
v4.0.2 is a patch update to introduce a number of bug fixes and dependency updates as well as the addition of a few new tests that still fall under the v1.3 best practices document.
Notable Changes
- #380 PTrace test
- #310 OCP Node Operating System compatibility check
- #355 SYS_NICE realtime kernel test
Full Changelog
- Add netcommons tests by @sebrandon1 in #285
- Remove gradetool by @sebrandon1 in #292
- Use upload-artifact v3 by @sebrandon1 in #296
- Update codeql github actions by @sebrandon1 in #298
- Move RemoveEmptyStrings to stringhelper by @sebrandon1 in #297
- Copy chaostesting files in via Dockerfile by @sebrandon1 in #272
- Bump github.com/stretchr/testify from 1.7.4 to 1.7.5 by @dependabot in #301
- Bump github.com/hashicorp/go-version from 1.5.0 to 1.6.0 by @dependabot in #304
- Use set for target folders in platform tests by @rdavid in #305
- Remove duplicated code by @jmontesi in #307
- Bump github.com/stretchr/testify from 1.7.5 to 1.8.0 by @dependabot in #308
- Print more details about containers that fail the fsdiff test by @sebrandon1 in #311
- Bump github.com/mittwald/go-helm-client from 0.11.1 to 0.11.2 by @dependabot in #313
- Temporarily disable litmus by @sebrandon1 in #316
- Remove unused jsonschema by @sebrandon1 in #314
- Add node version check by @sebrandon1 in #310
- Fix the CI; OCP test fails because it runs Ubuntu by @sebrandon1 in #319
- Remove cmdrunner package; comment fixes by @sebrandon1 in #318
- Fix RHCOS automation by @sebrandon1 in #323
- Add some more context to errors in OCP node test by @sebrandon1 in #324
- Fix provider tests by @sebrandon1 in #325
- Fix offline certification unit tests by @sebrandon1 in #329
- Copy the RHCOS mapping file to image by @sebrandon1 in #328
- Cleanup chaostesting; add tests by @sebrandon1 in #330
- Fix RHCOS mapping file location by @sebrandon1 in #334
- Chaos testing pod-delete test case disabled and removed from catalog. by @greyerof in #333
- Fix rhcos_version_map copy by @sebrandon1 in #337
- run-cnf-suites.sh: avoid error when listing the specs by @jmontesi in #336
- Fix projectpath to handle proper workingdir by @sebrandon1 in #341
- Bump helm.sh/helm/v3 from 3.9.0 to 3.9.1 by @dependabot in #342
- Update Go to 1.18.4 by @sebrandon1 in #340
- Switch to relative paths by @sebrandon1 in #345
- Adjust where the provider is looking for the rhcos map by @sebrandon1 in #350
- Remove 'cnf-certification-test/' from path to RHCOS map file by @sebrandon1 in #351
- Fix node OCP version test failure logic by @sebrandon1 in #354
- Bump k8s.io/client-go from 0.24.2 to 0.24.3 by @dependabot in #348
- Add SYS_NICE realtime kernel check by @sebrandon1 in #355
- Add logging around compatibility by @sebrandon1 in #357
- Upgrade GolangCI-lint to v1.47.0 by @sebrandon1 in #358
- Remove extra dependabot yaml by @sebrandon1 in #359
- Bump github.com/sirupsen/logrus from 1.8.1 to 1.9.0 by @dependabot in #364
- Skip 'version-not-found' RHCOS nodes to avoid false failures by @sebrandon1 in #363
- Improve the catalog with exception processes by @sebrandon1 in #349
- Bump github.com/mittwald/go-helm-client from 0.11.2 to 0.11.3 by @dependabot in #368
- Bump helm.sh/helm/v3 from 3.9.1 to 3.9.2 by @dependabot in #371
- Improvements for the offline db fetch tool. by @greyerof in #362
- GolangCI-lint v1.47.1; fix lint problems by @sebrandon1 in #369
- Address more lint problems by @sebrandon1 in #377
- access-control: add test to check for SYS_PTRACE capability by @jmontesi in #380
- Update partner tag to v4.0.2 by @sebrandon1 in #381
New Contributors
Full Changelog: v4.0.1...v4.0.2
v4.0.1 More tests, stability fixes
We are excited to release v4.0.1 that contains a large number of updates, fixes, and improvements as well as newly added test cases. This release also contains an up-to-date copy of our offline operator certification database as of the day of this release.
Notable new test cases added:
- #259 contains a new test for checking if your OCP version has reached end of life.
- #228 contains a new test for Service Mesh availability and will flag any labeled/tested pods as failed if they are not using an available service mesh.
- #146 added the new
chaostesting
suite of tests. Although, at this time it is not recommended to enable via your tnf_config. #272 is working to fix this known issue.
What's Changed
- Bump helm.sh/helm/v3 from 3.8.2 to 3.9.0 by @dependabot in #216
- adding the chaos testing suite to cnf repo-added a pod delete test by @aabughosh in #146
- Bump github.com/hashicorp/go-version from 1.4.0 to 1.5.0 by @dependabot in #222
- Remove references to oc client by @sebrandon1 in #220
- Fix for observability-container-logging TC. by @greyerof in #225
- Manual certified db update. by @greyerof in #226
- Bump github.com/operator-framework/api from 0.14.0 to 0.15.0 by @dependabot in #230
- Bump k8s.io/client-go from 0.24.0 to 0.24.1 by @dependabot in #234
- Bump k8s.io/api from 0.24.0 to 0.24.1 by @dependabot in #236
- Remove references to mockgen by @sebrandon1 in #233
- Bump github.com/mittwald/go-helm-client from 0.11.0 to 0.11.1 by @dependabot in #238
- Update docker/login-action to v2 by @sebrandon1 in #239
- Fix for platform-alteration-base-image test case. by @greyerof in #242
- Upgrade golangci-lint to v1.46.2 by @sebrandon1 in #227
- Revert "Remove references to oc client" by @sebrandon1 in #250
- Avoid nil panic when debug pod missing by @sebrandon1 in #247
- Update test case catalog with Best Practices v1.3 references by @shimritproj in #244
- Fixed WaitDebugPodsReady to not return if DesiredNumberSchedued is not valid. by @greyerof in #253
- Fix verbiage by @sebrandon1 in #248
- Upgrade YAML package to v3 by @sebrandon1 in #246
- Optimize around strings.Builder by @sebrandon1 in #245
- Remove unused items by @sebrandon1 in #240
- Bump github.com/stretchr/testify from 1.7.1 to 1.7.2 by @dependabot in #256
- Update Go to 1.18.3 by @sebrandon1 in #254
- initial commit to normalize offline and online code by @hamadise in #232
- Remove gomega; add more context to failures by @sebrandon1 in #260
- Fix typos by @sebrandon1 in #262
- Shudtown typo by @sebrandon1 in #265
- Add nil check in testhelper by @sebrandon1 in #263
- Remove more contractions by @sebrandon1 in #264
- Remove unused diagnostic and common key by @sebrandon1 in #267
- Add safeguard for SNO clusters by @sebrandon1 in #266
- Fix operator certification url by @hamadise in #271
- More typos found by @sebrandon1 in #273
- Add nil check for deployment object by @sebrandon1 in #269
- Temporarily fix CVE-2022-31030 by @sebrandon1 in #276
- ServiceMesh by @shimritproj in #228
- Use CSV name to search in online catalog. by @greyerof in #275
- Increased timeout for waiting for the debug pods to be up. by @greyerof in #278
- Add some more catalog tests by @sebrandon1 in #279
- Add compatibility matrix for testing OCP versions by @sebrandon1 in #259
- Bump k8s.io/apimachinery from 0.24.1 to 0.24.2 by @dependabot in #284
- Bump k8s.io/api from 0.24.1 to 0.24.2 by @dependabot in #283
- Bump k8s.io/client-go from 0.24.1 to 0.24.2 by @dependabot in #282
- Bump github.com/stretchr/testify from 1.7.2 to 1.7.3 by @dependabot in #287
- Bump github.com/stretchr/testify from 1.7.3 to 1.7.4 by @dependabot in #288
- Bump github.com/spf13/cobra from 1.4.0 to 1.5.0 by @dependabot in #290
- updated service mesh by @shimritproj in #291
Full Changelog: v4.0.0...v4.0.1
v4.0.0 Initial Release
Initial release of the new go-client based architecture.
New Test Cases
- access-control-one-process-per-container
- observability-termination-policy
Non Backward Compatibility issues
- The debug daemonset is now deployed without node-selector field, so one debug pod will be deployed on each node of the cluster. Here, it's also important to know that, when running the container TNF image for v4.0.0, the "run-tnf-container.sh" script from this repo should be used. Otherwise, debug pods might not be correctly deployed.
- Environment variable LOG_LEVEL to control the tnf's output log level has been renamed to TNF_LOG_LEVEL.
- The file "testconfigure.yaml" does not apply any more. All the "combo" TCs will run for all the target pods/containers/operators. This affects to access-control and operators TCs, where these new TC names have been created:
- access-control-security-context-capabilities-check
- access-control-security-context-non-root-user-check
- access-control-security-context-privilege-escalation
- access-control-container-host-port
- access-control-pod-host-network
- access-control-pod-host-path
- accces-control-pod-host-ipc
- access-control-pod-host-pid
- operator-install-status-succeeded
- operator-install-status-no-privileges
- Claim file:
- Removed duplicate test suite name at the beginning of the test name:
platform-alteration-platform-alteration-isredhat-release -> platform-alteration-isredhat-releas - Now, CSI and CNI information will be shown per node, instead of a just one master and worker node. Also, all the node's HW information is shown in json format.
- Removed duplicate test suite name at the beginning of the test name: