You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Bug #570 was a real blocker and an eye opener that we need integration tests.
I wanted to wait till we have implemented more of the required core features before we work on integration test but now that I saw how easily we have green tests for a totally broken product, I think it is time to introduce integration tests.
IMHO with minimal invest, we can introduce very simple integration tests.
Already by calling ide help or ideasy help we could have revealed this error before the merge.
Some guidance for writing integration tests with Java and Maven:
First of all to understand the context: Unlike a regular JUnit test, here in this context we mean with integration test that our product is already build (mvn package) and after that we run tests using the resulting package that has been build. Regular JUnit tests are executed by surefire-plugin before the package is build and if they fail, the build is aborted and no package will be build at all.
in general if we want to do something with JUnit on an integration test level, failsafe-plugin is the way to go. It will run JUnits using IT instead of Test as suffix.
as a minimal test, we could even use maven-exec-plugin to run IDEasy in a non-mocked productive way and trigger some trivial usecases like ideasy help or ideasy create demoproject. For the letter we need to prepare a real IDEasy installation and an IDE_ROOT variable in target somehow. This way we would only verify that this is running with exit code 0 and otherwise break the build. But for making real tests with assertions this is not the way to go. It just allows with minimal effort to have something that is better than nothing.
as a maximum test, we would include the native image build and build the full package, install it and then run tests on that. In devonfw-ide we have setup tests like but with pure bash-scripting technology. We do not want to do the same but still it might help to inspire and give a vision of what we want to archive here...
By default (mvn install) these new integration tests shall not be executed. They may also turn red if something external breaks but there might be no bug in our product directly. Also they will require more time and resources as well as external network activity (downloads, etc.). Therefore we should define a maven profile to activate this that is triggered in our nightly build.
If somebody can create a PR for the minimal approach in a day or less, that would be a nice first step.
Apart from that, I would expect that we create real integration tests (failsafe plugin) and create some boostrapping step before the tests are run or as setup method for each of these tests that "installs" our product somewhere in the target folder (${project.build.directory}) and can then run the test on top of this.
JUnit is not really great for testing bash and CMD executions so we would need some test-infrastructure to invoke ide commands in a given shell of that pre-setup installation. Still we can discuss if we want to write integration tests in JUnit (or bash or whatever).
Please note that here we do not want to assert lists of files to be present like software/java/lib/security/blocked.certs. Instead a test could look like this:
ide create maven-example-project -
cd maven-example-project
ide set-version java 21.0.4
ide install mvn 3.9.7
mvn -v
now we assert that the output of that last comment contains the correct Java and Maven version
The text was updated successfully, but these errors were encountered:
For the record:
There is also https://testcontainers.com/
At least for linux there are free images of e.g. ubunut. We could run automated integration tests on linux that would work on any operating system with docker installed.
Just an idea... Still not sure if we want to do this...
Bug #570 was a real blocker and an eye opener that we need integration tests.
I wanted to wait till we have implemented more of the required core features before we work on integration test but now that I saw how easily we have green tests for a totally broken product, I think it is time to introduce integration tests.
IMHO with minimal invest, we can introduce very simple integration tests.
Already by calling
ide help
orideasy help
we could have revealed this error before the merge.Some guidance for writing integration tests with Java and Maven:
mvn package
) and after that we run tests using the resulting package that has been build. Regular JUnit tests are executed by surefire-plugin before the package is build and if they fail, the build is aborted and no package will be build at all.IT
instead ofTest
as suffix.ideasy help
orideasy create demoproject
. For the letter we need to prepare a real IDEasy installation and an IDE_ROOT variable in target somehow. This way we would only verify that this is running with exit code 0 and otherwise break the build. But for making real tests with assertions this is not the way to go. It just allows with minimal effort to have something that is better than nothing.mvn install
) these new integration tests shall not be executed. They may also turn red if something external breaks but there might be no bug in our product directly. Also they will require more time and resources as well as external network activity (downloads, etc.). Therefore we should define a maven profile to activate this that is triggered in our nightly build.If somebody can create a PR for the minimal approach in a day or less, that would be a nice first step.
Apart from that, I would expect that we create real integration tests (failsafe plugin) and create some boostrapping step before the tests are run or as
setup
method for each of these tests that "installs" our product somewhere in thetarget
folder (${project.build.directory}
) and can then run the test on top of this.JUnit is not really great for testing bash and CMD executions so we would need some test-infrastructure to invoke
ide
commands in a given shell of that pre-setup installation. Still we can discuss if we want to write integration tests in JUnit (or bash or whatever).Please note that here we do not want to assert lists of files to be present like
software/java/lib/security/blocked.certs
. Instead a test could look like this:ide create maven-example-project -
cd maven-example-project
ide set-version java 21.0.4
ide install mvn 3.9.7
mvn -v
The text was updated successfully, but these errors were encountered: