Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] org.opensearch.gateway.remote.RemoteClusterStateCleanupManagerIT.testRemoteCleanupDeleteStale is flaky #13968

Closed
reta opened this issue Jun 4, 2024 · 3 comments · Fixed by #13995 or #14230
Closed

[BUG] org.opensearch.gateway.remote.RemoteClusterStateCleanupManagerIT.testRemoteCleanupDeleteStale is flaky #13968

reta opened this issue Jun 4, 2024 · 3 comments · Fixed by #13995 or #14230
Labels
bug Something isn't working ClusterManager:RemoteState flaky-test Random test failure that succeeds on second run untriaged

Comments

@reta
Copy link
Collaborator

reta commented Jun 4, 2024

Describe the bug

The test case org.opensearch.gateway.remote.RemoteClusterStateCleanupManagerIT.testRemoteCleanupDeleteStale is flaky:

java.lang.AssertionError: Current number of manifest files: 30
	at __randomizedtesting.SeedInfo.seed([AF3CFB99BA9F2DEE:D90361535B7B574A]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.assertTrue(Assert.java:42)
	at org.opensearch.gateway.remote.RemoteClusterStateCleanupManagerIT.lambda$testRemoteCleanupDeleteStale$0(RemoteClusterStateCleanupManagerIT.java:127)
	at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1128)
	at org.opensearch.gateway.remote.RemoteClusterStateCleanupManagerIT.testRemoteCleanupDeleteStale(RemoteClusterStateCleanupManagerIT.java:121)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at org.opensearch.test.OpenSearchTestClusterRule$1.evaluate(OpenSearchTestClusterRule.java:369)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:1583)
	Suppressed: java.lang.AssertionError: Current number of manifest files: 30
		at org.junit.Assert.fail(Assert.java:89)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.opensearch.gateway.remote.RemoteClusterStateCleanupManagerIT.lambda$testRemoteCleanupDeleteStale$0(RemoteClusterStateCleanupManagerIT.java:127)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 40 more
	Suppressed: java.lang.AssertionError: Current number of manifest files: 30
		at org.junit.Assert.fail(Assert.java:89)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.opensearch.gateway.remote.RemoteClusterStateCleanupManagerIT.lambda$testRemoteCleanupDeleteStale$0(RemoteClusterStateCleanupManagerIT.java:127)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 40 more
	Suppressed: java.lang.AssertionError: Current number of manifest files: 30
		at org.junit.Assert.fail(Assert.java:89)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.opensearch.gateway.remote.RemoteClusterStateCleanupManagerIT.lambda$testRemoteCleanupDeleteStale$0(RemoteClusterStateCleanupManagerIT.java:127)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 40 more
	Suppressed: java.lang.AssertionError: Current number of manifest files: 30
		at org.junit.Assert.fail(Assert.java:89)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.opensearch.gateway.remote.RemoteClusterStateCleanupManagerIT.lambda$testRemoteCleanupDeleteStale$0(RemoteClusterStateCleanupManagerIT.java:127)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 40 more
	Suppressed: java.lang.AssertionError: Current number of manifest files: 30
		at org.junit.Assert.fail(Assert.java:89)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.opensearch.gateway.remote.RemoteClusterStateCleanupManagerIT.lambda$testRemoteCleanupDeleteStale$0(RemoteClusterStateCleanupManagerIT.java:127)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 40 more
	Suppressed: java.lang.AssertionError: Current number of manifest files: 30
		at org.junit.Assert.fail(Assert.java:89)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.opensearch.gateway.remote.RemoteClusterStateCleanupManagerIT.lambda$testRemoteCleanupDeleteStale$0(RemoteClusterStateCleanupManagerIT.java:127)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 40 more
	Suppressed: java.lang.AssertionError: Current number of manifest files: 30
		at org.junit.Assert.fail(Assert.java:89)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.opensearch.gateway.remote.RemoteClusterStateCleanupManagerIT.lambda$testRemoteCleanupDeleteStale$0(RemoteClusterStateCleanupManagerIT.java:127)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 40 more
	Suppressed: java.lang.AssertionError: Current number of manifest files: 30
		at org.junit.Assert.fail(Assert.java:89)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.opensearch.gateway.remote.RemoteClusterStateCleanupManagerIT.lambda$testRemoteCleanupDeleteStale$0(RemoteClusterStateCleanupManagerIT.java:127)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 40 more
	Suppressed: java.lang.AssertionError: Current number of manifest files: 30
		at org.junit.Assert.fail(Assert.java:89)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.opensearch.gateway.remote.RemoteClusterStateCleanupManagerIT.lambda$testRemoteCleanupDeleteStale$0(RemoteClusterStateCleanupManagerIT.java:127)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 40 more

Related component

Storage:Remote

To Reproduce

./gradlew ':server:internalClusterTest' --tests "org.opensearch.gateway.remote.RemoteClusterStateCleanupManagerIT.testRemoteCleanupDeleteStale" -Dtests.seed=AF3CFB99BA9F2DEE

Expected behavior

The test must always pass

Additional Details

Plugins
Standard

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

  • CI

Additional context
@reta reta added bug Something isn't working untriaged flaky-test Random test failure that succeeds on second run labels Jun 4, 2024
@reta reta mentioned this issue Jun 4, 2024
Merged
9 tasks
@reta reta mentioned this issue Jun 4, 2024
Merged
9 tasks
@shiv0408
Copy link
Member

shiv0408 commented Jun 4, 2024

After running multiple times, was able to reproduce this is in local. Seems like delete task was still scheduled when we closed the test. Seems like we might be sharing the same single threaded pool for delete task and assert statement. As when the assert call is causing the thread to sleep, deletion task is still waiting to be executed when the test is closed.

com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=312, name=opensearch[node_t4][scheduler][T#1], state=RUNNABLE, group=TGRP-RemoteClusterStateCleanupManagerIT]
	at __randomizedtesting.SeedInfo.seed([AF3CFB99BA9F2DEE:D252E09652838F84]:0)
Caused by: org.opensearch.core.concurrency.OpenSearchRejectedExecutionException: rejected execution of java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@4f38345e[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@1ef204c9[Wrapped task = org.opensearch.gateway.remote.RemoteClusterStateCleanupManager$AsyncStaleFileDeletion@5e6633c0]] on org.opensearch.threadpool.Scheduler$SafeScheduledThreadPoolExecutor@738ca017[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 38]
	at __randomizedtesting.SeedInfo.seed([AF3CFB99BA9F2DEE]:0)
	at app//org.opensearch.common.util.concurrent.OpenSearchAbortPolicy.rejectedExecution(OpenSearchAbortPolicy.java:67)
	at [email protected]/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:841)
	at [email protected]/java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:340)
	at [email protected]/java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:562)
	at app//org.opensearch.threadpool.ThreadPool.schedule(ThreadPool.java:469)
	at app//org.opensearch.common.util.concurrent.AbstractAsyncTask.rescheduleIfNecessary(AbstractAsyncTask.java:109)
	at app//org.opensearch.common.util.concurrent.AbstractAsyncTask.run(AbstractAsyncTask.java:174)
	at app//org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:854)
	at [email protected]/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
	at [email protected]/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at [email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at [email protected]/java.lang.Thread.run(Thread.java:1583)

This was referenced Jun 4, 2024
Closed
Merged
@reta
Copy link
Collaborator Author

reta commented Jun 5, 2024

Closing, since #13969 was reverted

@reta reta closed this as completed Jun 5, 2024
@github-project-automation github-project-automation bot moved this from 🆕 New to ✅ Done in Storage Project Board Jun 5, 2024
@reta
Copy link
Collaborator Author

reta commented Jun 12, 2024 •
edited
Loading 

./gradlew ':server:internalClusterTest' --tests "org.opensearch.gateway.remote.RemoteClusterStateCleanupManagerIT.testRemoteCleanupDeleteStale" -Dtests.seed=321594D2A7776D01

java.lang.AssertionError: Current number of manifest files: 66
	at __randomizedtesting.SeedInfo.seed([321594D2A7776D01:442A0E18469317A5]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.assertTrue(Assert.java:42)
	at org.opensearch.gateway.remote.RemoteClusterStateCleanupManagerIT.lambda$testRemoteCleanupDeleteStale$0(RemoteClusterStateCleanupManagerIT.java:127)
	at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1128)
	at org.opensearch.gateway.remote.RemoteClusterStateCleanupManagerIT.testRemoteCleanupDeleteStale(RemoteClusterStateCleanupManagerIT.java:121)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at org.opensearch.test.OpenSearchTestClusterRule$1.evaluate(OpenSearchTestClusterRule.java:369)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:1583)
	Suppressed: java.lang.AssertionError: Current number of manifest files: 70
		at org.junit.Assert.fail(Assert.java:89)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.opensearch.gateway.remote.RemoteClusterStateCleanupManagerIT.lambda$testRemoteCleanupDeleteStale$0(RemoteClusterStateCleanupManagerIT.java:127)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 40 more
	Suppressed: java.lang.AssertionError: Current number of manifest files: 70
		at org.junit.Assert.fail(Assert.java:89)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.opensearch.gateway.remote.RemoteClusterStateCleanupManagerIT.lambda$testRemoteCleanupDeleteStale$0(RemoteClusterStateCleanupManagerIT.java:127)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 40 more
	Suppressed: java.lang.AssertionError: Current number of manifest files: 70
		at org.junit.Assert.fail(Assert.java:89)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.opensearch.gateway.remote.RemoteClusterStateCleanupManagerIT.lambda$testRemoteCleanupDeleteStale$0(RemoteClusterStateCleanupManagerIT.java:127)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 40 more
	Suppressed: java.lang.AssertionError: Current number of manifest files: 70
		at org.junit.Assert.fail(Assert.java:89)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.opensearch.gateway.remote.RemoteClusterStateCleanupManagerIT.lambda$testRemoteCleanupDeleteStale$0(RemoteClusterStateCleanupManagerIT.java:127)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 40 more
	Suppressed: java.lang.AssertionError: Current number of manifest files: 70
		at org.junit.Assert.fail(Assert.java:89)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.opensearch.gateway.remote.RemoteClusterStateCleanupManagerIT.lambda$testRemoteCleanupDeleteStale$0(RemoteClusterStateCleanupManagerIT.java:127)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 40 more
	Suppressed: java.lang.AssertionError: Current number of manifest files: 70
		at org.junit.Assert.fail(Assert.java:89)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.opensearch.gateway.remote.RemoteClusterStateCleanupManagerIT.lambda$testRemoteCleanupDeleteStale$0(RemoteClusterStateCleanupManagerIT.java:127)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 40 more
	Suppressed: java.lang.AssertionError: Current number of manifest files: 70
		at org.junit.Assert.fail(Assert.java:89)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.opensearch.gateway.remote.RemoteClusterStateCleanupManagerIT.lambda$testRemoteCleanupDeleteStale$0(RemoteClusterStateCleanupManagerIT.java:127)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 40 more
	Suppressed: java.lang.AssertionError: Current number of manifest files: 70
		at org.junit.Assert.fail(Assert.java:89)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.opensearch.gateway.remote.RemoteClusterStateCleanupManagerIT.lambda$testRemoteCleanupDeleteStale$0(RemoteClusterStateCleanupManagerIT.java:127)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1116)
		... 40 more
	Suppressed: java.lang.AssertionError: Current number of manifest files: 70
		at org.junit.Assert.fail(Assert.java:89)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.opensearch.gateway.remote.RemoteClusterStateCleanupManagerIT.lambda$testRemoteCleanupDeleteStale$0(RemoteClusterStateCleanupManagerIT.java:127)
		at org.opensearch.te

@reta reta reopened this Jun 12, 2024
@github-project-automation github-project-automation bot moved this from ✅ Done to 🏗 In progress in Storage Project Board Jun 12, 2024
@reta reta mentioned this issue Jun 12, 2024
Closed
@bowenlan-amzn bowenlan-amzn mentioned this issue Jun 12, 2024
Merged
9 tasks
@reta reta mentioned this issue Jun 13, 2024
Merged
@shiv0408 shiv0408 mentioned this issue Jun 13, 2024
Merged
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working ClusterManager:RemoteState flaky-test Random test failure that succeeds on second run untriaged
Projects
None yet
Milestone
No milestone
3 participants
@reta @shiv0408 @shourya035