review: test: Fixed 4 non-deterministic flaky tests

[nikunjagarwal321]

What changes were proposed in this pull request?

Sorting the order of imports list variable and enums list variable in 4 flaky tests in order to make them deterministic. The assertions are written with an assumption that the list will always be in a particular predefined order, but that may not be the case. The PR contains fix for the below test cases to make them deterministic:

• spoon.test.imports.ImportTest#staticImports_ofNestedTypes_shouldBeRecorded
• spoon.test.imports.ImportTest#staticTypeAndMethodImport_importShouldAppearOnlyOnceIfTheirSimpleNamesAreEqual
• spoon.test.position.PositionTest#testPositionOfCtImport
• spoon.test.template.SubstitutionTest#testCreateTypeFromTemplate

Why are the changes needed?

Multiple flaky tests were detected while trying to run the tests using the nondex tool. NonDex is a tool for detecting and debugging wrong assumptions on under-determined Java APIs.

Sample ERROR logs:

Reason for Failure

As mentioned above, the assertions assume that the list will always be in a particular predefined order, but that may not be the case. For eg, in test case : staticImports_ofNestedTypes_shouldBeRecorded, we have the following test case

List<CtImport> imports = mainType.getPosition().getCompilationUnit().getImports();
assertThat(imports, hasSize(2));

CtImport import0 = imports.get(0);
assertThat(import0.getReference().getSimpleName(), is("InnerClass"));

The above code assumes that the first element of the list will be InnerClass which may not be true. This is because the imports is being set by JDTImportBuilder.java class where the imports variable is implemented as HashSet. As per the javadocs, the order of elements is not maintained in HashSet.

Hence, we sort the fields before asserting based on certain fields to make the order deterministic.

Steps to reproduce flakiness using nondex -

mvn install -DskipTests

mvn edu.illinois:nondex-maven-plugin:2.1.7:nondex -Dtest=spoon.test.position.PositionTest#testPositionOfCtImport
mvn edu.illinois:nondex-maven-plugin:2.1.7:nondex -Dtest=spoon.test.template.SubstitutionTest#testCreateTypeFromTemplate
mvn edu.illinois:nondex-maven-plugin:2.1.7:nondex -Dtest=spoon.test.imports.ImportTest#staticTypeAndMethodImport_importShouldAppearOnlyOnceIfTheirSimpleNamesAreEqual
mvn edu.illinois:nondex-maven-plugin:2.1.7:nondex -Dtest=spoon.test.imports.ImportTest#staticImports_ofNestedTypes_shouldBeRecorded

Please let me know if you have any questions or would like to discuss this further for any clarificaitions.