Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Promote JUnit output 'terraform test' feature from experimental status #36228

Closed
wants to merge 5 commits into from
Closed

Promote JUnit output 'terraform test' feature from experimental status #36228

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
33c3051
Promote JUnit output 'terraform test' feature from experimental status
SarahFrench Dec 16, 2024
4a21385
Update `test` docs to include the `-junit-xml` flag
SarahFrench Jan 6, 2025
60b01d1
Add output format section to the `test` docs
SarahFrench Jan 6, 2025
269fe75
Refactor how data about XML elements is presented, small formatting c…
SarahFrench Jan 6, 2025
a7bbdec
Add details about `<skipped>`
SarahFrench Jan 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 0 additions & 17 deletions internal/command/test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,23 +103,6 @@ func (c *TestCommand) Run(rawArgs []string) int {
view := views.NewTest(args.ViewType, c.View)
var junitXMLView *views.TestJUnitXMLFile
if args.JUnitXMLFile != "" {
// JUnit XML output is currently experimental, so that we can gather
// feedback on exactly how we should map the test results to this
// JUnit-oriented format before anyone starts depending on it for real.
if !c.AllowExperimentalFeatures {
diags = diags.Append(tfdiags.Sourceless(
tfdiags.Error,
"JUnit XML output is not available",
"The -junit-xml option is currently experimental and therefore available only in alpha releases of Terraform CLI.",
))
view.Diagnostics(nil, nil, diags)
return 1
}
diags = diags.Append(tfdiags.Sourceless(
tfdiags.Warning,
"JUnit XML output is experimental",
"The -junit-xml option is currently experimental and therefore subject to breaking changes or removal, even in patch releases.",
))
junitXMLView = views.NewTestJUnitXMLFile(args.JUnitXMLFile)
view = views.TestMulti{
view,
Expand Down
151 changes: 151 additions & 0 deletions website/docs/cli/commands/test.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@ The following options apply to the Terraform `test` command:

* `-json` - Displays machine-readable JSON output for your testing results.

* `-junit-xml=<output file path>` - Saves testing results output in JUnit XML format to the specified file. The file path must be relative or absolute.

* `-test-directory=<relative directory>` - Overrides the directory that Terraform looks into for test files. Note that Terraform always loads testing files within the main configuration directory. The default testing directory is `tests`.

* `-verbose` - Prints out the plan or state for each `run` block within a test file, based on the `command` attribute of each `run` block.
Expand Down Expand Up @@ -132,3 +134,152 @@ The testing directory must be beneath the main configuration directory, but it c
> Note: Test files within the root configuration directory are always loaded, regardless of the `-test-directory` value.

We do not recommend changing the default test directory. The option for customization is included for configuration authors who may have included a `tests` submodule in their configuration before the `terraform test` command was released. In general, the default test directory of `tests` should always be used.

## Example: Test Output Format Options

Below is a contrived example of Terraform testing that makes assertions about the values of local variables `true` and `false`.
There are two test files: one contains a passing test, and one contains a failing test.

```hcl
# main.tf

locals {
true = "true"
false = "true" # incorrect, should be "false"!
}
```

The assertion that local.true == "true" in example_1.tftest.hcl will pass:

```hcl
# example_1.tftest.hcl

run "true_is_true" {
assert {
condition = local.true == "true"
error_message = "local.true did not match expected value"
}
}
```

The assertion that local.false == "false" in example_2.tftest.hcl will fail:

```hcl
# example_2.tftest.hcl

run "false_is_false" {
assert {
condition = local.false == "false"
error_message = "local.false did not match expected value"
}
}
```

### General features of test output

The `test` command will print output to the terminal:
* To show progress running tests across all files.
* To display any error messages resulting from failing test assertions.
* Finally, to summarize the results across all files.

Below is the output of `terraform test` when testing the example files above. When no additional flags are supplied, the command's output defaults to a human-readable format:

```shell
example_1.tftest.hcl... in progress
run "true_is_true"... pass
example_1.tftest.hcl... tearing down
example_1.tftest.hcl... pass
example_2.tftest.hcl... in progress
run "false_is_false"... fail
╷
│ Error: Test assertion failed
│
│ on example_2.tftest.hcl line 21, in run "false_is_false":
│ 21: condition = local.false == "false"
│ ├────────────────
│ │ local.false is "true"
│
│ local.false did not match expected value
╵
example_2.tftest.hcl... tearing down
example_2.tftest.hcl... fail

Failure! 1 passed, 1 failed.
```

### Test output in JSON format

Below is the output of the `terraform test -json` command using the same example files. The -json flag changes terminal output to be in a machine-readable format. The content of the output is otherwise unchanged:

```shell
{"@level":"info","@message":"Terraform 1.11.0-dev","@module":"terraform.ui","@timestamp":"2025-01-06T12:19:01.903603Z","terraform":"1.11.0-dev","type":"version","ui":"1.2"}
{"@level":"info","@message":"Found 2 files and 2 run blocks","@module":"terraform.ui","@timestamp":"2025-01-06T12:19:02.229367Z","test_abstract":{"example_1.tftest.hcl":["true_is_true"],"example_2.tftest.hcl":["false_is_false"]},"type":"test_abstract"}
{"@level":"info","@message":"example_1.tftest.hcl... in progress","@module":"terraform.ui","@testfile":"example_1.tftest.hcl","@timestamp":"2025-01-06T12:19:02.229432Z","test_file":{"path":"example_1.tftest.hcl","progress":"starting"},"type":"test_file"}
{"@level":"info","@message":" \"true_is_true\"... in progress","@module":"terraform.ui","@testfile":"example_1.tftest.hcl","@testrun":"true_is_true","@timestamp":"2025-01-06T12:19:02.229458Z","test_run":{"path":"example_1.tftest.hcl","run":"true_is_true","progress":"starting","elapsed":0},"type":"test_run"}
{"@level":"info","@message":" \"true_is_true\"... pass","@module":"terraform.ui","@testfile":"example_1.tftest.hcl","@testrun":"true_is_true","@timestamp":"2025-01-06T12:19:02.230577Z","test_run":{"path":"example_1.tftest.hcl","run":"true_is_true","progress":"complete","status":"pass"},"type":"test_run"}
{"@level":"info","@message":"example_1.tftest.hcl... tearing down","@module":"terraform.ui","@testfile":"example_1.tftest.hcl","@timestamp":"2025-01-06T12:19:02.230590Z","test_file":{"path":"example_1.tftest.hcl","progress":"teardown"},"type":"test_file"}
{"@level":"info","@message":"example_1.tftest.hcl... pass","@module":"terraform.ui","@testfile":"example_1.tftest.hcl","@timestamp":"2025-01-06T12:19:02.230598Z","test_file":{"path":"example_1.tftest.hcl","progress":"complete","status":"pass"},"type":"test_file"}
{"@level":"info","@message":"example_2.tftest.hcl... in progress","@module":"terraform.ui","@testfile":"example_2.tftest.hcl","@timestamp":"2025-01-06T12:19:02.230605Z","test_file":{"path":"example_2.tftest.hcl","progress":"starting"},"type":"test_file"}
{"@level":"info","@message":" \"false_is_false\"... in progress","@module":"terraform.ui","@testfile":"example_2.tftest.hcl","@testrun":"false_is_false","@timestamp":"2025-01-06T12:19:02.230642Z","test_run":{"path":"example_2.tftest.hcl","run":"false_is_false","progress":"starting","elapsed":0},"type":"test_run"}
{"@level":"info","@message":" \"false_is_false\"... fail","@module":"terraform.ui","@testfile":"example_2.tftest.hcl","@testrun":"false_is_false","@timestamp":"2025-01-06T12:19:02.231331Z","test_run":{"path":"example_2.tftest.hcl","run":"false_is_false","progress":"complete","status":"fail"},"type":"test_run"}
{"@level":"error","@message":"Error: Test assertion failed","@module":"terraform.ui","@testfile":"example_2.tftest.hcl","@testrun":"false_is_false","@timestamp":"2025-01-06T12:19:02.231485Z","diagnostic":{"severity":"error","summary":"Test assertion failed","detail":"local.false did not match expected value","range":{"filename":"example_2.tftest.hcl","start":{"line":21,"column":21,"byte":334},"end":{"line":21,"column":43,"byte":356}},"snippet":{"context":"run \"false_is_false\"","code":" condition = local.false == \"false\"","start_line":21,"highlight_start_offset":20,"highlight_end_offset":42,"values":[{"traversal":"local.false","statement":"is \"true\""}]}},"type":"diagnostic"}
{"@level":"info","@message":"example_2.tftest.hcl... tearing down","@module":"terraform.ui","@testfile":"example_2.tftest.hcl","@timestamp":"2025-01-06T12:19:02.231552Z","test_file":{"path":"example_2.tftest.hcl","progress":"teardown"},"type":"test_file"}
{"@level":"info","@message":"example_2.tftest.hcl... fail","@module":"terraform.ui","@testfile":"example_2.tftest.hcl","@timestamp":"2025-01-06T12:19:02.231557Z","test_file":{"path":"example_2.tftest.hcl","progress":"complete","status":"fail"},"type":"test_file"}
{"@level":"info","@message":"Failure! 1 passed, 1 failed.","@module":"terraform.ui","@timestamp":"2025-01-06T12:19:02.231581Z","test_summary":{"status":"fail","passed":1,"failed":1,"errored":0,"skipped":0},"type":"test_summary"}
```

### Test output in JUnit XML format, saved to file

Below is the output of the `terraform test -junit-xml=./output.xml` command using the same example files.
The test output is:

* Printed to the terminal in the default, human-readable format, as described above.
* Also saved in JUnit XML format in the specified file.

Below is the contents of the resulting `output.xml` file:

```xml
<?xml version="1.0" encoding="UTF-8"?><testsuites>
<testsuite name="example_1.tftest.hcl" tests="1" skipped="0" failures="0" errors="0">
<testcase name="true_is_true" classname="example_1.tftest.hcl" time="0.001064666"></testcase>
</testsuite>
<testsuite name="example_2.tftest.hcl" tests="1" skipped="0" failures="1" errors="0">
<testcase name="false_is_false" classname="example_2.tftest.hcl" time="0.000870083">
<failure message="Test run failed"></failure>
<system-err><![CDATA[
Error: Test assertion failed

on example_2.tftest.hcl line 21:
(source code not available)

local.false did not match expected value
]]></system-err>
Comment on lines +248 to +256
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is likely to change after refactoring prior to promoting the feature

</testcase>
</testsuite>
</testsuites>
```

The file maps Terraform test command concepts to JUnit XML format according to the table below:

| Terraform test concept | Element in JUnit XML output |
| -----------------------| ----------------------------|
| Test directory | `<testsuites>` |
| Run block | `<testcase>` |
| Test failure error | `<failure>` |
| ??? | `<system-err>` |

A JUnit format XML file will contain a single `<testsuites>` element that wraps all other elements in the XML output, as `terraform test` only processes a single test directory. Inside `<testsuites>` there will be as many `<testsuite>` elements as there are test files. Test files can contain many run blocks, so `<testsuite>` elements will contain a `<testcase>` element per run block within that test file. If a run block's assertions fail then that `<testcase>` element will contain a `<failure>` element describing the test failure. Similarly, a test may contain a nested `<skipped>` element either if the test is skipped as a result of previous errors encountered in a given test file or if an interrupt is received by Terraform while tests are still running. Any other issues that arise while executing that run block will be reported in a nested `<system-err>` element.

Some elements have attributes that provide addition information about the testing output data. These are summarized below:

| JUnit XML Element | Attribute | Meaning |
| -----------------------| ----------------------------|-------------------------------------------------------|
| `<testsuite>` | `name` | The test file name |
| `<testsuite>` | `tests` | Count of total tests (run blocks) in the file |
| `<testsuite>` | `skipped` | Count of skipped tests (run blocs) count in the file |
| `<testsuite>` | `failures` | Count of failed tests (run blocks) count in the file |
| `<testsuite>` | `errored` | Count of errored tests (run blocks) count in the file |
| `<testcase>` | `name` | The run block's name |
| `<testcase>` | `classname` | The name of the file containing the run block |
| `<testcase>` | `time` | The duration of executing the run block |
| `<failure>` | `message` | A message describing the test failure |
Loading