Skip to content

Latest commit

 

History

History
259 lines (216 loc) · 27 KB

DOCUMENT.md

File metadata and controls

259 lines (216 loc) · 27 KB
Raw

Testing Document

This document describes how to define your tests with YAML. For the plugin cli usage, please check README

Test Suite

A test suite is a collection of tests with the same purpose and scope defined in one single file. The root structure of a test suite is like below:

suite: test deploy and service
values:
  - overallValues.yaml
set:
  image.pullPolicy: Always
templates:
  - templates/deployment.yaml
  - templates/web/service.yaml
release:
  name: my-release
  namespace: my-namespace
  revision: 1
  upgrade: true
capabilities:
  majorVersion: 1
  minorVersion: 10
  apiVersions:
    - br.dev.local/v2
chart:
  version: 1.0.0
  appVersion: 1.0.0
kubernetesProvider:
  scheme:
    "v1/Pod":
      gvr:
        version:  "v1"
        resource: "pods"
      namespaced: true
  objects:
    - kind: Pod
      apiVersion: v1
      metadata:
        name: unittest
        namespace: default
tests:
  - it: should test something
    ...

  • suite: string, optional. The suite name to show on test result output.

  • values: array of string, optional. The values files used to renders the chart, think it as the -f, --values options of helm install. The file path should be the relative path from the test suite file itself.

  • set: object of any, optional. Set the values directly in suite file. The key is the value path with the format just like --set option of helm install, for example image.pullPolicy. The value is anything you want to set to the path specified by the key, which can be even an array or an object. This set will override values which are already set in the values file.

  • templates: array of string, recommended. The template files scope to test in this suite. Only the selected files will be rendered. Template files that are put in a templates sub-folder can be addressed with a linux path separator. Also the templates/ can be omitted. Using wildcards it is possible to test multiple templates without listing them one-by-one. Partial templates (which are prefixed with and _ or have the .tpl extension) are added automatically even if it is in a templates sub-folder, you don't need to add them.

  • release: object, optional. Define the {{ .Release }} object.

    • name: string, optional. The release name, default to "RELEASE-NAME".
    • namespace: string, optional. The namespace which release be installed to, default to "NAMESPACE".
    • revision: int, optional. The revision of current build, default to 0.
    • upgrade: bool, optional. Whether the build is an upgrade, default to false.

  • capabilities: object, optional. Define the {{ .Capabilities }} object.

    • majorVersion: int, optional. The kubernetes major version, default to the major version which is set by helm.
    • minorVersion: int, optional. The kubernetes minor version, default to the minor version which is set by helm.
    • apiVersions: array of string, optional. A set of versions, default to the version set used by the defined kubernetes version. Defaults to the capabilities from system's Helm installation itself.

  • chart: object, optional. Define the {{ .Chart }} object.

    • version: string, optional. The semantic version of the chart, default to the version set in the Chart.
    • appVersion: string, optional. The app-version of the chart, default to the app-version set in the Chart.

  • kubernetesProvider: object, optional. Define Kubernetes resources to fake.

    • scheme: object. Define the Kubernetes schema to fake
    • objects: array of objects. Define the Kubernetes objects to fake

  • tests: array of test job, required. Where you define your test jobs to run, check Test Job.

Test Job

The test job is the base unit of testing. Your chart is rendered each time a test job run, and validated with assertions defined in the test. You can setup your values used to render the chart in the test job with external values files or directly in the test job definition. Below is a test job example with all of its options defined:

...
tests:
  - it: should pass
    values:
      - ./values/staging.yaml
    set:
      image.pullPolicy: Always
      resources:
        limits:
          memory: 128Mi
    template: deployment.yaml
    documentIndex: 0
    documentSelector:
      path: metadata.name
      value: my-service-name
    release:
      name: my-release
      namespace:
      revision: 9
      upgrade: true
    capabilities:
      majorVersion: 1
      minorVersion: 12
      apiVersions:
        - custom.api/v1
    chart:
      version: 1.0.0
      appVersion: 1.0.0
    asserts:
      - equal:
          path: metadata.name
          value: my-deploy

  • it: string, recommended. Define the name of the test with TDD style or any message you like.

  • values: array of string, optional. The values files used to renders the chart, think it as the -f, --values options of helm install. The file path should be the relative path from the test suite file itself. This file will override existing values set in the suite values.

  • set: object of any, optional. Set the values directly in suite file. The key is the value path with the format just like --set option of helm install, for example image.pullPolicy. The value is anything you want to set to the path specified by the key, which can be even an array or an object. This set will override values which are already set in the values file.

  • template: string, optional.
    templates: array of string, optional. The template file(s) which render the manifest to be tested, default to the list of template file defined in templates of suite file, unless template is defined in the assertion(s) (check Assertion).

  • documentIndex: int, optional. The index of rendered documents (divided by ---) to be tested, default to -1, which results in asserting all documents (see Assertion). Generally you can ignored this field if the template file render only one document.

  • documentSelector: DocumentSelector, optional. The path of the key to be match and the match value to assert. Using this information, helm-unittest will automatically discover the documents for asserting. Generally you can ignore this field if the template file render only one document.

    • path: string. The documentSelector path to assert.
    • value: string|object. The expected value.
    • matchMany: bool, optional. Set to true to allow matching multiple documents. Defaults to false which means selector has to match single document across all templates.
    • skipEmptyTemplates: bool, optional. Set to true to skip asserting templates which didn't render any matching documents. Defaults to false which means selector have to find at least one document in every template.

  • release: object, optional. Define the {{ .Release }} object.

    • name: string, optional. The release name, default to "RELEASE-NAME".
    • namespace: string, optional. The namespace which release be installed to, default to "NAMESPACE".
    • revision: int, optional. The revision of current build, default to 0.
    • upgrade: bool, optional. Whether the build is an upgrade, default to false.

  • capabilities: object, optional. Define the {{ .Capabilities }} object.

    • majorVersion: int, optional. The kubernetes major version, default to the major version which is set by helm.
    • minorVersion: int, optional. The kubernetes minor version, default to the minor version which is set by helm.
    • apiVersions: array of string, empty, optional. A set of versions, default to the version set used by the defined kubernetes version. Defaults to system's Helm installation itself or suite level when provided.

  • chart: object, optional. Define the {{ .Chart }} object.

    • version: string, optional. The semantic version of the chart, default to the version set in the Chart.
    • appVersion: string, optional. The app-version of the chart, default to the app-version set in the Chart.

  • kubernetesProvider: object, optional. Define Kubernetes resources to fake.

    • scheme: object, optional. Define the Kubernetes schema to fake
    • objects: array of objects. Define the Kubernetes objects to fake

  • asserts: array of assertion, required. The assertions to validate the rendered chart, check Assertion.

Assertion

Define assertions in the test job to validate the manifests rendered with values provided. The example below tests the instances' name with 2 equal assertion.

templates:
  - deployment.yaml
  - web/service.yaml
tests:
  - it: should pass
    asserts:
      - equal:
          path: metadata.name
          value: my-deploy
        documentSelector:
          path: metadata.name
          value: my-service-name
      - equal:
          path: metadata.name
          value: your-service
        not: true
        template: web/service.yaml
        documentIndex: 0

The assertion is defined with the assertion type as the key and its parameters as value, there can be only one assertion type key exists in assertion definition object. And there are four more options can be set at root of assertion definition:

  • not: bool, optional. Set to true to assert contrarily, default to false. The second assertion in the example above asserts that the service name is NOT your-service.

  • template: string, optional. The template file which render the manifest to be asserted, default to the list of template file defined in templates of suite file, unless the template is in the testjob (see TestJob). For example the first assertion above with no template specified asserts for both deployment.yaml and service.yaml by default. If no template file specified in neither suite, testjob and assertion, the assertion returns an error and fail the test.

  • documentIndex: int, optional. The index of rendered documents (divided by ---) to be tested, default to -1, which results in asserting all documents (see Assertion). Generally you can ignored this field if the template file render only one document.

  • documentSelector: DocumentSelector, optional. The path of the key to be match and the match value to assert. Using this information, helm-unittest will automatically discover the documents for asserting. Generally you can ignore this field if the template file render only one document.

    • path: string. The documentSelector path to assert.
    • value: string|object. The expected value.
    • matchMany: bool, optional. Set to true to allow matching multiple documents. Defaults to false which means selector has to match single document across all templates.
    • skipEmptyTemplates: bool, optional. Set to true to skip asserting templates which didn't render any matching documents. Defaults to false which means selector have to find at least one document in every template.

Map keys in path containing periods (.) are supported with the use of a jsonPath syntax: For more detail on the jsonPath syntax.

- equal:
    path: metadata.annotations["kubernetes.io/ingress.class"]
    value: nginx

Assertion Types

Available assertion types are listed below:

Assertion Type Parameters Description Example
containsDocument kind: string. Expected kind of manifest.
apiVersion: string. Expected apiVersion of manifest.
name: string, optional. The value of the metadata.name.
namespace: string, optional. The value of the metadata.namespace.
any: bool, optional. ignores any other documents.		 Asserts the documents rendered by the kind and apiVersion specified. 
containsDocument:
  kind: Deployment
  apiVersion: apps/v1
  name: foo
  namespace: bar
contains path: string. The set path to assert, the value must be an array.
content: any. The content to be contained.
count: int, optional. The count of content to be contained.
any: bool, optional. ignores any other values within the found content.		 Assert the array as the value of specified path contains the content. 
contains:
  path: spec.ports
  content:
    name: web
    port: 80
    targetPort: 80
    protocol:TCP

contains:
  path: spec.ports
  content:
    name: web
  count: 1
  any: true
notContains path: string. The set path to assert, the value must be an array.
content: any. The content NOT to be contained.
any: bool, optional. ignores any other values within the found content.		 Assert the array as the value of specified path NOT contains the content. 
notContains:
  path: spec.ports
  content:
    name: server
    port: 80
    targetPort: 80
    protocol: TCP

notContains:
  path: spec.ports
  content:
    name: web
  any: true
equal path: string. The set path to assert.
value: any. The expected value.
decodeBase64: bool, optional. Decode the base64 before checking		 Assert the value of specified path equal to the value. 
equal:
  path: metadata.name
  value: my-deploy
notEqual path: string. The set path to assert.
value: any. The value expected not to be.
decodeBase64: bool, optional. Decode the base64 before checking		 Assert the value of specified path NOT equal to the value. 
notEqual:
  path: metadata.name
  value: my-deploy
equalRaw
value: string. Assert the expected value in a NOTES.txt file.		 Assert equal to the value. 
equalRaw:
  value: my-deploy
notEqualRaw
value: string. Assert the expected value in a NOTES.txt file not to be.		 Assert equal NOT to the value. 
notEqualRaw:
  value: my-deploy
exists
(deprecates isNotNull)		 path: string. The set path to assert. Assert if the specified path exists. 
exists:
  path: spec.strategy
notExists
(deprecates isNull)		 path: string. The set path to assert. Assert if the specified path NOT exists. 
notExists:
  path: spec.strategy
failedTemplate errorMessage: string. The (human readable) errorMessage that should occur.
errorPattern: string. The regex syntax pattern to match the error		 Assert the value of errorMessage is the same as the human readable template rendering error. errorPattern allows to match an error that would happen before template execution (ex: validation of values against schema) 
failedTemplate:
 errorMessage: Required value
or 
failedTemplate: {}
or 
failedTemplate:
 errorPattern: "value"
notFailedTemplate Assert that no failure occurs while templating. 
notFailedTemplate: {}
greaterOrEqual path: string. The set path to assert.
value: int, float, string.		 Assert the value of specified path is greater or equal to the value. 
greaterOrEqual:
  path: resources.requests.cpu
  value: 2
notGreaterOrEqual path: string. The set path to assert.
value: int, float, string.		 Assert the value of specified path is NOT greater or equal to the value. 
notGreaterOrEqual:
  path: resources.requests.cpu
  value: 2
hasDocuments count: int. Expected count of documents rendered. Assert the documents count rendered by the template specified. The documentIndex or documentSelector option is ignored here. 
hasDocuments:
  count: 2
lessOrEqual path: string. The set path to assert.
value: int, float, string.		 Assert the value of specified path is less or equal to the value. 
lessOrEqual:
  path: spec.runAsUser
  value: 2000
notLessOrEqual path: string. The set path to assert.
value: int, float, string.		 Assert the value of specified path is NOT less or equal to the value. 
notLessOrEqual:
  path: spec.runAsUser
  value: 2000
isAPIVersion of: string. Expected apiVersion of manifest. Assert the apiVersion value of manifest, is equilevant to:
equal:
  path: apiVersion
  value: ...
 
isAPIVersion:
  of: v2
isKind of: String. Expected kind of manifest. Assert the kind value of manifest, is equilevant to:
equal:
  path: kind
  value: ...
 
isKind:
  of: Deployment
isNullOrEmpty
isEmpty		 path: string. The set path to assert. Assert the value of specified path is null or empty (null, "", 0, [], {}). 
isNullOrEmpty:
  path: spec.tls
isNotNullOrEmpty
isNotEmpty		 path: string. The set path to assert. Assert the value of specified path is NOT null or empty (null, "", 0, [], {}). 
isNotNullOrEmpty:
  path: spec.selector
isSubset path: string. The set path to assert, the value must be an object.
content: any. The content to be contained.		 Assert the object as the value of specified path that contains the content. 
isSubset:
  path: spec.template
  content:
    metadata: 
      labels: 
        app: basic
        release: MY-RELEASE
isNotSubset path: string. The set path to assert, the value must be an object.
content: any. The content NOT to be contained.		 Assert the object as the value of specified path that NOT contains the content. 
isSubset:
  path: spec.template
  content:
    metadata: 
      labels: 
        app: basic
        release: MY-RELEASE
isType path: string. The set path to assert.
type: string. The expected type of the value.		 Assert the type of the object is equal to the value of the specified path 
isType:
  path: metadata.name
  type: string
isNotType path: string. The set path to assert.
type: string. The expected type of the value.		 Assert the type of the object is NOT equal to the value of the specified path 
isNotType:
  path: metadata.name
  type: string
lengthEqual path: string, optional. The set path to assert the count of array values.
paths: string, optional. The set array of paths to assert the count validation of the founded arrays.
count: int, optional. The count of the values in the array.		 Assert the count of the path or paths to be equal. 
lengthEqual:
  path: spec.tls
  count: 1
notLengthEqual path: string, optional. The set path to assert the count of array values.
paths: string, optional. The set array of paths to assert the count validation of the founded arrays.
count: int, optional. The count of the values in the array.		 Assert the count of the path or paths NOT to be equal. 
notLengthEqual:
  path: spec.tls
  count: 1
matchRegex path: string. The set path to assert, the value must be a string.
pattern: string. The regex syntax pattern to match (without quoting /).
decodeBase64: bool, optional. Decode the base64 before checking		 Assert the value of specified path match pattern. 
matchRegex:
  path: metadata.name
  pattern: -my-chart$
notMatchRegex path: string. The set path to assert, the value must be a string.
pattern: string. The regex syntax pattern NOT to match (without quoting /).
decodeBase64: bool, optional. Decode the base64 before checking		 Assert the value of specified path NOT match pattern. 
notMatchRegex:
  path: metadata.name
  pattern: -my-chat$
matchRegexRaw pattern: string. The regex syntax pattern to match (without quoting /) in a NOTES.txt file. Assert the value match pattern. 
matchRegexRaw:
  pattern: -my-notes$
notMatchRegexRaw pattern: string. The regex syntax pattern NOT to match (without quoting /) in a NOTES.txt file. Assert the value NOT match pattern. 
notMatchRegexRaw:
  pattern: -my-notes$
matchSnapshot path: string. The set path for snapshot. Assert the value of path is the same as snapshotted last time. Check doc below. 
matchSnapshot:
  path: spec
matchSnapshotRaw Assert the value in the NOTES.txt is the same as snapshotted last time. Check doc below. 
matchSnapshotRaw: {}

Antonym and not

Notice that there are some antonym assertions, the following two assertions actually have same effect:

- equal:
    path: kind
    value: Pod
  not: true
# works the same as
- notEqual:
    path: kind
    value: Pod