Skip to content

Commit

Permalink
Explicitly exit with the correct status
Browse files Browse the repository at this point in the history
Why:

* When running tests on a directory, the exit status was 0, regardless of the
  actual exit status of the runner
* CI builds depend on status codes for each step in the build process to
  determine whether the build passed or failed. Since `exit()` is being used
  here, test suites that experience runtime errors will still exit with a
  0 status code.

This change addresses the need by:

* Exiting with 1 when there is a `RuntimeError`
* After the rake task runs, exit with the exit status of the runner
* Creates a method to assert output when the underlying process failed, instead
  of assuming that it will always be successful
  • Loading branch information
davidcpell authored and zamith committed May 6, 2016
1 parent 14f4563 commit 1783276
Show file tree
Hide file tree
Showing 5 changed files with 48 additions and 3 deletions.
7 changes: 7 additions & 0 deletions lib/error_tests/error_test.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
require_relative "../../test/test_helper.rb"

class ErrorTest < MTest
def test_purposeful_error
raise RuntimeError
end
end
4 changes: 2 additions & 2 deletions lib/m/parser.rb
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,9 @@ def parse
begin
Rake::Task['m_custom'].invoke
rescue RuntimeError
exit
exit(1)
ensure
exit
exit($?.exitstatus)
end
else
return testable
Expand Down
2 changes: 1 addition & 1 deletion test/everything_test.rb
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ def test_running_tests_within_a_subdirectory

def test_running_tests_with_failures_within_a_subdirectory
output = m('examples/subdir_with_failures')
assert_output(/1 tests, 1 assertions, 1 failures/, output)
assert_output_for_failed_execution(/1 tests, 1 assertions, 1 failures/, output)
end

def test_blank_file_is_quieter
Expand Down
33 changes: 33 additions & 0 deletions test/exit_codes_test.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
require "test_helper"

class ExitCodesTest < MTest
def test_failing_test_returns_1
m("examples/subdir_with_failures/a_test")
refute $?.success?, "expected exit code to be 1 but it was #{$?.exitstatus}"
end

def test_test_with_error_returns_1
m("../lib/error_tests/error_test")
refute $?.success?, "expected exit code to be 1 but it was #{$?.exitstatus}"
end

def test_dir_with_failure_returns_1
m("examples/subdir_with_failures")
refute $?.success?, "expected exit code to be 1 but it was #{$?.exitstatus}"
end

def test_dir_with_error_returns_1
m("../lib/error_tests")
refute $?.success?, "expected exit code to be 1 but it was #{$?.exitstatus}"
end

def test_without_errors_or_failures_returns_0
m("examples/subdir/a_test")
assert $?.success?, "expected exit code to be 0 but it was #{$?.exitstatus}"
end

def test_dir_without_errors_or_failures_returns_0
m("examples/subdir")
assert $?.success?, "expected exit code to be 0 but it was #{$?.exitstatus}"
end
end
5 changes: 5 additions & 0 deletions test/test_helper.rb
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,11 @@ def assert_output(regexp, output)
assert $?.success?, "Execution failed, output:\n\n#{output}"
assert_match regexp, output
end

def assert_output_for_failed_execution(regexp, output)
refute $?.success?, "Execution did not fail, but it should"
assert_match regexp, output
end
end

require 'm'
Expand Down

1 comment on commit 1783276

@davidcpell
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zamith this looks great to me

Please sign in to comment.