Installation script isn't working on the Apache Pulsar project

[devinbost]

When I run the installation script for Apache Pulsar, nothing happens.

To reproduce:

# clone iFixFlakies repo
git clone https://github.com/TestingResearchIllinois/iFixFlakies.git
# clone Apache Pulsar repo
git clone https://github.com/apache/pulsar.git
# run installer script
bash iFixFlakies/pom-modify/modify-project.sh pulsar/

Pulsar's POM files are unmodified. Verify as follows:

cd pulsar
git status

Also, if I try manually adding the plugin to the top-level POM file and running mvn testrunner:testplugin, I get this error:

[ERROR] No plugin found for prefix 'testrunner' in the current project and in the plugin groups [org.apache.maven.plugins, org.codehaus.mojo] available from the repositories [local (/Users/dbost/.m2/repository), apache.snapshots (https://repository.apache.org/snapshots), central (https://repo.maven.apache.org/maven2)] -> [Help 1]

[august782]

Hi,

Thanks for trying out iFixFlakies.

When you run the installer script (bash iFixFlakies/pom-modify/modify-project.sh pulsar/), do you get anything printed out? For example, on my end, when I run the script as you specified, it prints out all the paths to modules within the project that get modified. After navigating into pulsar/, I indeed see that the various pom.xml files have been modified.

When you added the plugin manually to the top-level pom.xml, did it stop building at the first module and skip the rest? It seems that the first module, buildtools, does not declare the top-level module as a parent and therefore does not inherit the plugin from it. Then, the overall build would stop at the first buildtools module and skip the rest. You would need to add the plugin to this module as well (or ultimately it probably is best to add to all modules, as the script is trying to do).

[devinbost]

When I run the script, I don't get anything printed out.
I'm running MacOS 10.15.7. What operating system are you using?

When I tried adding the plugin manually to the top-level pom.xml, it stopped very early in the process. I've attached the complete console output.

build_output_ifixflakies.txt

[august782]

I run the script in a Linux environment (Ubuntu 20.04). And you are saying you do not see anything printed out, not even any error message? The script itself compiles the PomFile.java and calls it on a pom.xml. Could you try first compiling the PomFile.java (javac PomFile.java) and then running "java PomFile ifixflakies 1.0.0-SNAPSHOT edu.illinois.cs.dt.tools.fixer.CleanerFixerPlugin " to see if it would modify that pom.xml file?

From your provided output, it seems to be the case of what I described earlier, where it stops early at that buildtools module that did not inherit the plugin from the top-level pom.xml you modified. You could try modifying the pom.xml under buildtools/ to include the plugin and try running again, though in general you probably just want to modify the pom.xml of all modules in the project just to be safe.

[devinbost]

And you are saying you do not see anything printed out, not even any error message?

That is correct.

I successfully compiled PomFile.java.
However, how do I point it to the pom.xml file?
When I run:
$ java PomFile ifixflakies 1.0.0-SNAPSHOT edu.illinois.cs.dt.tools.fixer.CleanerFixerPlugin
from the directory pom-modify, I get:

Exception in thread "main" java.lang.NullPointerException
at PomFile.(PomFile.java:44)
at PomFile.main(PomFile.java:265)

If I try to point to it from a higher-level directory, I get this:

$ java ./iFixFlakies/pom-modify/PomFile ifixflakies 1.0.0-SNAPSHOT edu.illinois.cs.dt.tools.fixer.CleanerFixerPlugin
Error: Could not find or load main class ..iFixFlakies.pom-modify.PomFile

[august782]

For the java command, you need to pass it the path to a pom.xml you want to modify as the last input parameter. You should only run it from within the directory where the compiled PomFile.class is (otherwise you would need to include some classpath so java would know where the class is).

[devinbost]

Here's what I get:

$ java PomFile ifixflakies 1.0.0-SNAPSHOT edu.illinois.cs.dt.tools.fixer.CleanerFixerPlugin ../../pulsar/pom.xml
Usage: java PomFile <artifact_id> <artifact_version> <configuration_class_name>

Exception in thread "main" java.lang.NullPointerException
	at PomFile.<init>(PomFile.java:44)
	at PomFile.main(PomFile.java:265)

Am I passing that path to the pom.xml file correctly?

[devinbost]

I get the same result if I pass an absolute path, like this:

$ java PomFile ifixflakies 1.0.0-SNAPSHOT edu.illinois.cs.dt.tools.fixer.CleanerFixerPlugin /Users/dbost/src/apache-pulsar/pulsar/pom.xml

[august782]

Sorry, my mistake, you do not pass in the path to the pom.xml as an argument but rather pipe into it as an input. In other words, could you try running this command:

echo "/Users/dbost/src/apache-pulsar/pulsar/pom.xml" | java ifixflakies 1.0.0-SNAPSHOT edu.illinois.cs.dt.tools.fixer.CleanerFixerPlugin

[devinbost]

Here are my results:

$ echo "/Users/dbost/src/apache-pulsar/pulsar/pom.xml" | java PomFile ifixflakies 1.0.0-SNAPSHOT edu.illinois.cs.dt.tools.fixer.CleanerFixerPlugin
/Users/dbost/src/apache-pulsar/pulsar

It looks like it only modified the top-level pom.xml file.

[august782]

Yes, that is expected, that command should only modify the passed in pom.xml. The modify-project.sh script actually searches for all the pom.xml files via the find command and pipes those in as inputs to the java command as a means to modify all those pom.xml files.

I'm not sure why that combination of find piped to this java command does not work in your environment. You could try some loop where you find all the pom.xml files and echo each one one-by-one into this command to modify all of them.

[devinbost]

I was able to run the command on all of the POM files in my project via:
$ find /Users/dbost/src/apache-pulsar/pulsar -name "pom.xml" | xargs -n 1 sh -c 'echo $0 | java PomFile ifixflakies 1.0.0-SNAPSHOT edu.illinois.cs.dt.tools.fixer.CleanerFixerPlugin'

[devinbost]

I confirmed that all of the POM files in the project have been updated.

Now, when I run mvn testrunner:testplugin, I get:

[ERROR] Failed to execute goal edu.illinois.cs:testrunner-maven-plugin:1.0:testplugin (default-cli) on project buildtools: Execution default-cli of goal edu.illinois.cs:testrunner-maven-plugin:1.0:testplugin failed: Plugin edu.illinois.cs:testrunner-maven-plugin:1.0 or one of its dependencies could not be resolved: Could not find artifact edu.illinois.cs:ifixflakies:jar:1.0.0-SNAPSHOT in apache.snapshots (https://repository.apache.org/snapshots) -> [Help 1]

(It fails again at the Pulsar Build Tools module.)

[august782]

You will need to "mvn install" the iFixFlakies project first.

Also, seeing as you are moving forward in trying to run iFixFlakies on modules in this project, I think you will need to have some detected order-dependent flaky tests first. You would need to run iDFlakies (https://github.com/idflakies/iDFlakies) on the project first.

[devinbost]

I cloned iDFlakies and ran:

cd iDFlakies
cd pom-modify
javac PomFile.java
find /Users/dbost/src/apache-pulsar/pulsar -name "pom.xml" | xargs -n 1 sh -c 'echo $0 | java PomFile idflakies 1.1.0 edu.illinois.cs.dt.tools.detection.DetectorPlugin'

After that, I inspected the POM files and noticed that the headings were all chopped off...

(They're all like this.)

[devinbost]

I'm not sure if that happened after running the first java file or the second one.

[august782]

Could you try resetting all the pom.xml files in the project first before running the command to see if it would still cut off parts of the pom.xml file?

[devinbost]

I reset the pom.xml files and ran:
find /Users/dbost/src/apache-pulsar/pulsar -name "pom.xml" | xargs -n 1 sh -c 'echo $0 | java PomFile idflakies 1.1.0 edu.illinois.cs.dt.tools.detection.DetectorPlugin'
The pom.xml files looked okay after that.
I then ran:
find /Users/dbost/src/apache-pulsar/pulsar -name "pom.xml" | xargs -n 1 sh -c 'echo $0 | java PomFile ifixflakies 1.0.0-SNAPSHOT edu.illinois.cs.dt.tools.fixer.CleanerFixerPlugin'

The pom.xml files look like they should... So, maybe I deleted parts of the pom.xml files on accident while constructing the find command.

[devinbost]

Oh, I see what happened... I ran a mvn command to fix the license headers, and it stripped off the top line that the PomFile tool concatenated.

[devinbost]

Here's what the top of the POM file looks like before running the PomFile tool:

After running the tool, you can see that the top part has been concatenated onto the same line as the end comment mark for the license header:

So, when I format the license, it cuts off that line because it thinks it's part of the license header (which it replaces.)

If that could please be fixed in the PomFile tool (so it adds a newline at the top if it comes after a comment), that would help a lot. (I can't run mvn install for this project without first running mvn initialize license:format or the build won't proceed.)

[devinbost]

I'm going to try a bulk find/replace to insert a newline character between the end of the comment and the start of <project

[august782]

Thanks for pointing that out about PomFile changes up the existing formatting. Currently, PomFile uses built-in XML parsing and XML writing from the Java standard library, and some of of that writing inserts its own formatting rules. There is probably some option that allows us to specify a better formatting, but we'll have to look at it some more to find any such options.

For now, in your specific case with pulsar, I think you can add the option -Dlicense.skip to skip the license checking.

[devinbost]

Thanks for the explanation.
I was able to use a bulk find/replace to prevent the license format step from deleting parts of my pom.xml files, but that's a helpful tip about using -Dlicense.skip.
I ran mvn clean install -DskipTests -Pcore-modules, and it just completed.
I then ran:
mvn testrunner:testplugin -Ddetector.detector_type=random-class-method -Ddt.randomize.rounds=10 -Ddt.detector.original_order.all_must_pass=false
and got this on the Pulsar Build Tools module:

[ERROR] Failed to execute goal edu.illinois.cs:testrunner-maven-plugin:1.0:testplugin (default-cli) on project buildtools: Execution default-cli of goal edu.illinois.cs:testrunner-maven-plugin:1.0:testplugin failed: Plugin edu.illinois.cs:testrunner-maven-plugin:1.0 or one of its dependencies could not be resolved: Failure to find edu.illinois.cs:ifixflakies:jar:1.0.0-SNAPSHOT in https://repository.apache.org/snapshots was cached in the local repository, resolution will not be reattempted until the update interval of apache.snapshots has elapsed or updates are forced -> [Help 1]

[august782]

Did you run both scripts to add both iDFlakies and iFixFlakies to the project? Since both work off of the same testrunner-maven-plugin, you should not run both scripts on the same file without resetting in between.

In other words, in this case, you should first reset and run the commands to add the iDFlakies to the project before running the iDFlakies command to detect flaky tests.

Also, I suspect you may still have not built the iFixFlakies project since it seems to be unable to find the 1.0.0-SNAPSHOT version of iFixFlakies.

[devinbost]

So, you're saying that I should reset the pom.xml files, run the command to insert iDFlakies into the pom.xml files, and then run mvn clean install -DskipTests -Pcore-modules -Dlicense.skip and then mvn testrunner:testplugin -Ddetector.detector_type=random-class-method -Ddt.randomize.rounds=10 -Ddt.detector.original_order.all_must_pass=false
to run iDFlakies.

Then, after that, I should reset the pom.xml files again and repeat the steps to run the iFixFlakies module.

Is that right?

[august782]

Yes, except afterwards when you run the iFixFlakies part you should not need to mvn install again.

[devinbost]

Okay. It's building now. I'll let you know what happens.

[devinbost]

Thanks for all of the help. I really appreciate it!

[devinbost]

Most of the modules gave output like this:

ERROR (WRITE_ERROR_STDOUT_STRING): org.apache.pulsar:pulsar-client-api:2.8.0-SNAPSHOT
Message was:
Module is not using a supported test framework (probably not JUnit), or there is no test.
[INFO] Accessing cachePath:
[INFO] Accessing cachePath:
TRY_COPY_ALL_FAILING_TEST_OUTPUT

Most of the tests in Pulsar are using the TestNG framework instead of JUnit.
Also, a lot of tests use Mockito and some use PowerMockito (for mocking classes with private methods.)

It ran on the python-functions-utils module, but it didn't find any flaky tests. (To be fair, I don't know of any flaky tests in that module. It's one of the simplest modules in the project.)

Does iDFlakies currently only support JUnit tests?

I'd really love to see iDFlakies find some flaky tests in Apache Pulsar. I'm a contributor to Pulsar, and flaky tests are a major problem. I think it would be a great Apache project for flaky test research because the Pulsar architecture makes it very difficult to make some tests not flaky. For example, testing timeout behavior in asynchronous messaging (which is a key aspect of Pulsar architecture) causes lots of ASYNC WAIT flaky tests.

[august782]

Unfortunately, currently iDFlakies only supports JUnit tests (mainly JUnit 4 and recently JUnit 5 tests as well). We don't have immediate plans to support TestNG, but if there is enough demand we can look into how to run TestNG tests through the testrunner and iDFlakies framework.

I will note that iDFlakies classifies flaky tests detected as order-dependent or non-order-dependent due to how it runs tests in different orders. The ASYNC WAIT flaky tests you mentioned would likely be under the non-order-dependent category, and they would mainly be discovered due to numerous reruns as part of the iDFlakies process.

[devinbost]

Thanks for the feedback.

One other detail about the flaky tests in Apache Pulsar is that they usually don't fail when the tests are run locally. It's when they're sent to the CI (which runs them with minimal resources) that we see massive failures.

Some of the flaky tests have been documented here: https://github.com/apache/pulsar/issues?q=is%3Aissue+flaky+
Some of those issues actually document groups of flaky tests. The flaky tests are such a problem that the community is discussing how to unblock the CI builds (which often require many re-runs before all the tests pass, sometimes spanning days of re-runs.) (Discussion is here: apache/pulsar#10148)
There are currently 28 test classes in the "flaky" test group. I counted 367 tests in that group marked as flaky.
That number is pretty large compared to the other Apache projects evaluated as part of Luo, Q., Hariri, F., Eloussi, L., & Marinov, D. (2014, November). An empirical analysis of flaky tests. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (pp. 643-653).
(I'm guessing you're familiar with that study.)

Also, the popularity of Pulsar is greatly increasing. (It's definitely the most advanced open-source messaging framework available, and I believe that it's one of the most significant Apache projects overall.)

So, this could be a great opportunity for your research to really make a difference and get some publicity. :)

[august782]

Thank you for the feedback, and thank you very much for all the links and resources on Pulsar flaky tests, those are all very interesting! It looks like something I want to look more into in detail to understand what kind of flakiness has been documented and how people are addressing them.

[devinbost]

Sounds good. Please let me know if I can be of any help.