Create quarantine strategy

[lordofthejars]

Issue Overview

If you have non-deterministic tests keep them in a different test suite to your healthy tests. That way you'll you can continue to pay attention to what's going on with your healthy tests and get good feedback from them.

Then the question is what to do with the quarantined test suites. They are useless as regression tests, but they do have a future as work items for cleaning up. You should not abandon such tests, since any tests you have in quarantine are not helping you with your regression coverage.

A danger here is that tests keep getting thrown into quarantine and forgotten, which means your bug detection system is eroding. As a result it's worthwhile to have a mechanism that ensures that tests don't stay in quarantine too long. I've come across various ways to do this. One is a simple numeric limit: e.g. only allow 8 tests in quarantine. Once you hit the limit you must spend time to clear all the tests out. This has the advantage of batching up your test-cleaning if that's how you like to do things. Another route is to put a time limit on how long a test may be in quarantine, su

So far my idea is to provide this in smart testing. So for example tests that are annotated with @Quarantine or tests that has failed 5 times (configurable value) in a raw (this information would come from hub), then are put in quarantine.

Putting in quarantine means that they are dropped from list of tests to execute. Then if you reach the max number of test into quarantine we can either put them as a part of the build so the build fails, log the problem and just continue skipping quarantine tests or abort the execution.

What I see is that this should not be a strategy but a mode. So you can do something like: mvn clean test -Dsmart.testing.mode=ordering, quarantine which means that it takes into consideration the quarantine tests.

WDYT?

Reference: https://martinfowler.com/articles/nonDeterminism.html