jf mvn wrapper alters mvn behavior by not honoring -Dmaven.test.failure.ignore=true

[jfsarang]

Describe the bug

JFrog CLI doesn't ignore the failed tests, which led the build became "unstable" and build failed.

Current behavior

When passing "-Dmaven.test.failure.ignore=true" in jf mvn deploy command, some test fail and user is getting these errors:

2024-06-25T21:18:57.7016614Z [ThreadedStreamConsumer] ERROR org.apache.maven.plugin.surefire.SurefirePlugin - Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.763 s <<< FAILURE! - in com.roche.upath.schedulers.unmark.service.impl.CaseUnmarkServiceImplTest
2024-06-25T21:18:57.7027306Z [ThreadedStreamConsumer] ERROR org.apache.maven.plugin.surefire.SurefirePlugin - testUnmarkBookmarkedCase  Time elapsed: 1.715 s  <<< FAILURE!

This makes the build becomes “unstable”

[main] WARNING org.jfrog.build.extractor.maven.BuildDeploymentHelper - Artifactory Build Info Recorder: unstable build, artifacts will not be deployed...

and leads to

{
"status": "failure",
"totals": {
"success": 0,
"failure": 19
 },
"files": []
}

It looked like JFrog CLI would search for the failed tests under target/surefire-reports directory. If there are failed tests, the jars are not published.

Reproduction steps

I cloned the repo https://github.com/jfrog/project-examples/tree/master/maven-examples/maven-example and modified this line in /multi1/src/test/java/artifactory/test/AppTest.java

    public void testApp()
    {
        assertTrue( false );
    }

Add to pom.xml:

<dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.9</version>
            <scope>test</scope>
        </dependency>
           <dependency>
            <groupId>org.testng</groupId>
            <artifactId>testng</artifactId>
            <version>6.9.8</version>
            <scope>test</scope>
            </dependency>
    </dependencies>
.
.
.
         <plugin>
              <artifactId>maven-surefire-plugin</artifactId>
              <version>3.2.2</version>
              <configuration>
                  <includes>
                      <include>AppTest.java</include>
                  </includes>
                  <testFailureIgnore>true</testFailureIgnore>
              </configuration>
          </plugin>

Then execute

jf mvn deploy -Dsurefire.skipAfterFailureCount=1 --build-name=test-mvn-deploy --build-number=2.0701 -Dmaven.test.failure.ignore=true

the test is failing

[main] ERROR org.apache.maven.plugin.surefire.SurefirePlugin - Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
.
.
.
{
  "status": "failure",
  "totals": {
    "success": 0,
    "failure": 9
  }
}

The only way to overcome is setting -Dartifactory.publish.unstable=true .

Expected behavior

It should ignore all failed tests and complete the build without any failures.

JFrog CLI version

2.52.10.

Operating system type and version

macOS - 14.4.1 - 23E224

JFrog Artifactory version

No response

JFrog Xray version

No response

[bedge]

Any progress on this?
It's forcing CI changes across all of our repos that are using jf mvn.
This is a big down-side to using jFrog tool wrappers.
It would be a different story if the confirmed bugs were addressed in a timely fashion.

[jfsarang]

Will reopen this as a feature request