akka-uct intermittent fail run by fastdebug openjdk11

[sendaoYan]

test command:

for i in `seq 10` ; do time ./openjdk-11.0.19-internal-fastdebug/bin/java -jar renaissance-gpl-0.14.2.jar akka-uct &> test-openjdk11-$i.log ; done

result:

fail rate: 9/10

fail message:

[WARN] [03/24/2023 14:57:12.998] [main] [EventStream(akka://UCT)] Logger log1-Logging$DefaultLogger did not respond within Timeout(5000 milliseconds) to InitializeLogger(bus)
error while starting up loggers
akka.ConfigurationException: Logger specified in config can't be loaded [akka.event.Logging$DefaultLogger] due to [akka.event.Logging$LoggerInitializationException: Logger log1-Logging$DefaultLogger did not respond with LoggerInitialized, sent instead [TIMEOUT]]
	at akka.event.LoggingBus$$anonfun$$nestedInanonfun$startDefaultLoggers$3$1.applyOrElse(Logging.scala:136)
	at akka.event.LoggingBus$$anonfun$$nestedInanonfun$startDefaultLoggers$3$1.applyOrElse(Logging.scala:131)
	at scala.util.Failure.recover(Try.scala:233)
	at akka.event.LoggingBus.$anonfun$startDefaultLoggers$3(Logging.scala:131)
	at scala.collection.Iterator$$anon$9.next(Iterator.scala:577)
	at scala.collection.mutable.Growable.addAll(Growable.scala:62)
	at scala.collection.mutable.Growable.addAll$(Growable.scala:57)
	at scala.collection.immutable.VectorBuilder.addAll(Vector.scala:1565)
	at scala.collection.immutable.Vector$.from(Vector.scala:1349)
	at scala.collection.immutable.Vector$.from(Vector.scala:34)
	at scala.collection.IterableOps$WithFilter.map(Iterable.scala:891)
	at akka.event.LoggingBus.startDefaultLoggers(Logging.scala:123)
	at akka.event.LoggingBus.startDefaultLoggers$(Logging.scala:108)
	at akka.event.EventStream.startDefaultLoggers(EventStream.scala:26)
	at akka.actor.LocalActorRefProvider.init(ActorRefProvider.scala:609)
	at akka.actor.ActorSystemImpl.liftedTree2$1(ActorSystem.scala:1026)
	at akka.actor.ActorSystemImpl._start$lzycompute(ActorSystem.scala:1022)
	at akka.actor.ActorSystemImpl._start(ActorSystem.scala:1022)
	at akka.actor.ActorSystemImpl.start(ActorSystem.scala:1045)
	at akka.actor.ActorSystem$.apply(ActorSystem.scala:272)
	at akka.actor.ActorSystem$.apply(ActorSystem.scala:316)
	at akka.actor.ActorSystem$.apply(ActorSystem.scala:290)
	at edu.rice.habanero.actors.AkkaActorState$.newActorSystem(AkkaActor.scala:196)
	at edu.rice.habanero.benchmarks.uct.UctAkkaActorBenchmark$UctAkkaActorBenchmark.runIteration(UctAkkaActorBenchmark.scala:25)
	at org.renaissance.actors.AkkaUct.$anonfun$run$1(AkkaUct.scala:46)
	at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:190)
	at org.renaissance.actors.AkkaUct.run(AkkaUct.scala:45)
	at org.renaissance.harness.ExecutionDriver.executeOperation(ExecutionDriver.java:137)
	at org.renaissance.harness.ExecutionDriver.executeBenchmark(ExecutionDriver.java:93)
	at org.renaissance.harness.RenaissanceSuite$.$anonfun$runBenchmarks$1(RenaissanceSuite.scala:163)
	at org.renaissance.harness.RenaissanceSuite$.$anonfun$runBenchmarks$1$adapted(RenaissanceSuite.scala:159)
	at scala.collection.immutable.List.foreach(List.scala:333)
	at org.renaissance.harness.RenaissanceSuite$.runBenchmarks(RenaissanceSuite.scala:159)
	at org.renaissance.harness.RenaissanceSuite$.main(RenaissanceSuite.scala:132)
	at org.renaissance.harness.RenaissanceSuite.main(RenaissanceSuite.scala)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.renaissance.core.Launcher.loadAndInvokeHarnessClass(Launcher.java:129)
	at org.renaissance.core.Launcher.launchHarnessClass(Launcher.java:78)
	at org.renaissance.core.Launcher.main(Launcher.java:43)
Caused by: akka.event.Logging$LoggerInitializationException: Logger log1-Logging$DefaultLogger did not respond with LoggerInitialized, sent instead [TIMEOUT]
	at akka.event.LoggingBus.addLogger(Logging.scala:217)
	at akka.event.LoggingBus.$anonfun$startDefaultLoggers$4(Logging.scala:129)
	at scala.util.Success.map(Try.scala:262)
	at akka.event.LoggingBus.$anonfun$startDefaultLoggers$3(Logging.scala:128)
	... 38 more
====== akka-uct (concurrency) [default], iteration 0 failed (ConfigurationException) ======
Benchmark 'akka-uct' failed with exception:
akka.ConfigurationException: Could not start logger due to [akka.ConfigurationException: Logger specified in config can't be loaded [akka.event.Logging$DefaultLogger] due to [akka.event.Logging$LoggerInitializationException: Logger log1-Logging$DefaultLogger did not respond with LoggerInitialized, sent instead [TIMEOUT]]]
	at akka.event.LoggingBus.startDefaultLoggers(Logging.scala:165)
	at akka.event.LoggingBus.startDefaultLoggers$(Logging.scala:108)
	at akka.event.EventStream.startDefaultLoggers(EventStream.scala:26)
	at akka.actor.LocalActorRefProvider.init(ActorRefProvider.scala:609)
	at akka.actor.ActorSystemImpl.liftedTree2$1(ActorSystem.scala:1026)
	at akka.actor.ActorSystemImpl._start$lzycompute(ActorSystem.scala:1022)
	at akka.actor.ActorSystemImpl._start(ActorSystem.scala:1022)
	at akka.actor.ActorSystemImpl.start(ActorSystem.scala:1045)
	at akka.actor.ActorSystem$.apply(ActorSystem.scala:272)
	at akka.actor.ActorSystem$.apply(ActorSystem.scala:316)
	at akka.actor.ActorSystem$.apply(ActorSystem.scala:290)
	at edu.rice.habanero.actors.AkkaActorState$.newActorSystem(AkkaActor.scala:196)
	at edu.rice.habanero.benchmarks.uct.UctAkkaActorBenchmark$UctAkkaActorBenchmark.runIteration(UctAkkaActorBenchmark.scala:25)
	at org.renaissance.actors.AkkaUct.$anonfun$run$1(AkkaUct.scala:46)
	at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:190)
	at org.renaissance.actors.AkkaUct.run(AkkaUct.scala:45)
	at org.renaissance.harness.ExecutionDriver.executeOperation(ExecutionDriver.java:137)
	at org.renaissance.harness.ExecutionDriver.executeBenchmark(ExecutionDriver.java:93)
	at org.renaissance.harness.RenaissanceSuite$.$anonfun$runBenchmarks$1(RenaissanceSuite.scala:163)
	at org.renaissance.harness.RenaissanceSuite$.$anonfun$runBenchmarks$1$adapted(RenaissanceSuite.scala:159)
	at scala.collection.immutable.List.foreach(List.scala:333)
	at org.renaissance.harness.RenaissanceSuite$.runBenchmarks(RenaissanceSuite.scala:159)
	at org.renaissance.harness.RenaissanceSuite$.main(RenaissanceSuite.scala:132)
	at org.renaissance.harness.RenaissanceSuite.main(RenaissanceSuite.scala)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.renaissance.core.Launcher.loadAndInvokeHarnessClass(Launcher.java:129)
	at org.renaissance.core.Launcher.launchHarnessClass(Launcher.java:78)
	at org.renaissance.core.Launcher.main(Launcher.java:43)
The following benchmarks failed: akka-uct
[WARN] [03/24/2023 14:57:34.623] [UCT-shutdown-hook-1] [CoordinatedShutdown(akka://UCT)] CoordinatedShutdown from JVM shutdown failed: Future timed out after [10000 milliseconds]

test enviroment:

[lbulej]

Looks like some internal race in Akka or the logging libraries. I can't reproduce it on a 16-core (32-thread) machine on Fedora 37 with OpenJDK 11.0.18. Does version 0.14.1 fail too?

[sendaoYan]

Looks like some internal race in Akka or the logging libraries. I can't reproduce it on a 16-core (32-thread) machine on Fedora 37 with OpenJDK 11.0.18. Does version 0.14.1 fail too?

version 0.14.1 also fail on the same machine by the same openjdk binary, fail rate: 7/10:

test-openjdk11-9.log
test-openjdk11-8.log
test-openjdk11-7.log
test-openjdk11-6.log
test-openjdk11-5.log
test-openjdk11-4.log
test-openjdk11-3.log
test-openjdk11-2.log
test-openjdk11-1.log
test-openjdk11-10.log

On the other hand, this fail only occur by fastdebug. release binary always run success.
Mine fastdebug openjdk configure is:

bash configure --with-jobs=`nproc` --prefix=$PWD/install --with-debug-level=fastdebug