[Test Failures] Test failures for 2.5 branch

[kotwanikunal]

The test org.opensearch.action.support.replication.TransportReplicationActionTests.testClosedIndexOnReroute has been failing for post merge gradle checks on 2.5.

Compiled list of test failures for 2.5 -

Commit Link	Test Link	Failure
dcafb10	https://build.ci.opensearch.org/job/gradle-check/9198	https://build.ci.opensearch.org/job/gradle-check/9198/testReport/junit/org.opensearch.action.support.replication/TransportReplicationActionTests/testClosedIndexOnReroute/
9f4e169	https://build.ci.opensearch.org/job/gradle-check/9194/console	https://build.ci.opensearch.org/job/gradle-check/9194/testReport/junit/org.opensearch.action.support.replication/TransportReplicationActionTests/testClosedIndexOnReroute/
a3d57c9	https://build.ci.opensearch.org/job/gradle-check/9138/	https://build.ci.opensearch.org/job/gradle-check/9138/testReport/junit/org.opensearch.action.support.replication/TransportReplicationActionTests/testClosedIndexOnReroute/

First commit since failure was observed: a3d57c9

---

BwC Test failures

Failure	From Version	To Version	How to reproduce?
org.opensearch.backwards.IndexingIT.testUpdateSnapshotStatus	2.6	2.5	./gradlew ':qa:mixed-cluster:v2.5.0#mixedClusterTest' --tests "org.opensearch.backwards.IndexingIT.testUpdateSnapshotStatus" -Dtests.seed=BE95B870ADDCC4FF -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=en-ZA -Dtests.timezone=Etc/GMT-7
org.opensearch.backwards.IndexingIT.testSeqNoCheckpoints	2.6	2.5	./gradlew ':qa:mixed-cluster:v2.5.0#mixedClusterTest' --tests "org.opensearch.backwards.IndexingIT.testSeqNoCheckpoints" -Dtests.seed=BE95B870ADDCC4FF -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=en-ZA -Dtests.timezone=Etc/GMT-7
org.opensearch.upgrades.RecoveryIT.testRelocationWithConcurrentIndexing	2.6	2.5	./gradlew ':qa:rolling-upgrade:v2.5.0#oneThirdUpgradedTest' --tests "org.opensearch.upgrades.RecoveryIT.testRelocationWithConcurrentIndexing" -Dtests.seed=BE95B870ADDCC4FF -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=hu-HU -Dtests.timezone=Africa/Libreville

[andrross]

The "Add support of default replica count change" commit seems to be the culprit here. If I revert that change the test seems to pass consistently.

[andrross]

Looks like this was fixed with #5737 but never backported. Doing the backports now.

[kotwanikunal]

For the BwC tests: @dreamer-89 RCAd the issue here: #5766 (comment)
The fix is out on #5808

[kotwanikunal]

Verified all the tests listed on this issue pass post the fixes merged in.

[peternied]

Looks like this was fixed with #5737 but never backported. Doing the backports now.

@andrross @kotwanikunal any thoughts on we can catch these issues better, a lot of FUD for a small miss