[BUG]org.opensearch.search.aggregations.bucket.TermsFixedDocCountErrorIT.testSliceLevelDocCountErrorMultiShard {p0={"search.concurrent_segment_search.enabled":"true"}} flaky

[dblock]

Describe the bug

java.lang.AssertionError: expected:<2> but was:<3>

Related component

Storage:Durability

To Reproduce

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.search.aggregations.bucket.TermsFixedDocCountErrorIT" -Dtests.method="testSliceLevelDocCountErrorMultiShard {p0={"search.concurrent_segment_search.enabled":"true"}}" -Dtests.seed=41B9C6A508DBF95D -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=es-MX -Dtests.timezone=Australia/North -Druntime.java=21
NOTE: leaving temporary files on disk at: /var/jenkins/workspace/gradle-check/search/server/build/testrun/internalClusterTest/temp/org.opensearch.search.aggregations.bucket.TermsFixedDocCountErrorIT_41B9C6A508DBF95D-001
NOTE: test params are: codec=Asserting(Lucene99): {s_value=Lucene99, index_uuid=Lucene99, _id=Lucene99, type=PostingsFormat(name=Asserting)}, docValues:{s_value=DocValuesFormat(name=Lucene90), _seq_no=DocValuesFormat(name=Asserting), _primary_term=DocValuesFormat(name=Lucene90), _version=DocValuesFormat(name=Lucene90)}, maxPointsInLeafNode=1593, maxMBSortInHeap=6.303042065496907, sim=Asserting(RandomSimilarity(queryNorm=false): {}), locale=es-MX, timezone=Australia/North
NOTE: Linux 5.15.0-1048-aws amd64/Eclipse Adoptium 21.0.1 (64-bit)/cpus=32,threads=1,free=460063632,total=933232640
NOTE: All tests run in this JVM: [RejectionActionIT, CloneIndexIT, GetIndexIT, ActiveShardsObserverIT, MinimumClusterManagerNodesIT, VotingConfigurationIT, ClusterSettingsIT, AliasedIndexDocumentActionsIT, HiddenIndexIT, MultiFieldsIntegrationIT, IndexingMemoryControllerIT, IndexPrimaryRelocationIT, UpdateNumberOfReplicasIT, SimpleNodesInfoIT, RemoteIndexRecoveryIT, RemoteStoreMultipartFileCorruptionIT, AggregationsIntegrationIT, GlobalIT, TermsFixedDocCountErrorIT]

Expected behavior

Test to pass.

Additional Details

https://build.ci.opensearch.org/job/gradle-check/32347/testReport/junit/org.opensearch.search.aggregations.bucket/TermsFixedDocCountErrorIT/testSliceLevelDocCountErrorMultiShard__p0___search_concurrent_segment_search_enabled___true___/

[jed326]

This is a new test I added in #11680, will take a look.

[jed326]

OpenSearch/server/src/internalClusterTest/java/org/opensearch/search/aggregations/bucket/TermsFixedDocCountErrorIT.java

Line 307 in c267c5a

assertEquals(2, segmentResponse.getIndices().get("idx_mshard_2").getShards().get(0).getShards()[0].getSegments().size());

Failure is coming from checking the segment count of one of the indices and in the shared output we are getting 3 segments instead of 2. We need the segment count to be stable and docs indexed in a certain way in order for the doc count errors to be accurate here so will need to think more on how we can accomplish that.