Skip to content

Commit

Permalink
[CI] Enable Develocity test retry selectively (#21479)
Browse files Browse the repository at this point in the history
The `sbt-develocity` plugin offers the ability to retry failing tests,
and to detect flaky ones. A flaky test is marked as flaky in the
Develocity test report, which will help us track them across many CI
executions. See for example the flaky test chart in the [Pekko
dashboard](https://ge.apache.org/scans/tests?search.relativeStartTime=P90D&search.rootProjectNames=Pekko&search.tags=not:CI&search.timeZoneId=Europe%2FZurich).

This PR contains the following configuration:
- retry each failing test once (to be adjusted if needed)
- don't retry if more than 10 tests fail in the current test run (to be
adjusted if needed)
- fail the build if a test is flaky
- disable test retry in a predefined set of test classes: the
compilation test classes. In those classes, we don't have enough
granularity for the test retry to be meaningful, because each test is
responsible for compiling many independent files.

@dotta @c00ler @Duhemm @lrytz
  • Loading branch information
adpi2 authored Aug 28, 2024
2 parents e9e046d + b4dfab9 commit c7c5f97
Showing 1 changed file with 19 additions and 0 deletions.
19 changes: 19 additions & 0 deletions project/Build.scala
Original file line number Diff line number Diff line change
Expand Up @@ -276,6 +276,18 @@ object Build {
val config = develocityConfiguration.value
val buildScan = config.buildScan
val buildCache = config.buildCache
// disable test retry on compilation test classes
val noRetryTestClasses = Set(
"dotty.tools.dotc.BestEffortOptionsTests",
"dotty.tools.dotc.CompilationTests",
"dotty.tools.dotc.FromTastyTests",
"dotty.tools.dotc.IdempotencyTests",
"dotty.tools.dotc.ScalaJSCompilationTests",
"dotty.tools.dotc.TastyBootstrapTests",
"dotty.tools.dotc.coverage.CoverageTests",
"dotty.tools.dotc.transform.PatmatExhaustivityTest",
"dotty.tools.repl.ScriptedTests"
)
config
.withProjectId(ProjectId("scala3"))
.withServer(config.server.withUrl(Some(url("https://develocity.scala-lang.org"))))
Expand All @@ -293,6 +305,13 @@ object Build {
.withLocal(buildCache.local.withEnabled(false))
.withRemote(buildCache.remote.withEnabled(false))
)
.withTestRetryConfiguration(
config.testRetryConfiguration
.withFlakyTestPolicy(FlakyTestPolicy.Fail)
.withMaxRetries(1)
.withMaxFailures(10)
.withClassesFilter((className, _) => !noRetryTestClasses.contains(className))
)
}
)

Expand Down

0 comments on commit c7c5f97

Please sign in to comment.