Testing WaterBear
Why do we test?
In short, we test so we can find bugs before they are released in to production. It tells us if what we intended to happen, actually happens. For a project that tries to tackle as much as Waterbear does it is important that the developers get the code right. If we do not, we may be teaching users how to code badly or turn them away from programming, which is the opposite of what we want.
In order to test Waterbear one must first install node and then install all of the dependencies with node using the command:
npm install
once all of the dev dependency packages are installed continue to the next section.
In order to create graphical tests for Waterbear, download and install Firefox if you do not already have it. Once you have a working Firefox, open up Firefox and go to Selenium Downloads page and download the Selenium IDE Firefox plugin. This should prompt for a Firefox plugin installation that needs to be done. Once the Selenium IDE is installed, you can now create Test cases that will mimic your movements if recorded.
WaterBear uses mocha as a JS testing framework. Mocha is simple and easy to use. Mocha is a javascript test framework that can be used for synchronous and asynchronous testing. All mocha test cases are run serially and in conjunction with Mocha we use assertion libraries like chai and sinon for stubs and spies.
A small mocha test file example using the chai assertion library would be:
describe("File Test Suite", function(){
it("Create File", function(){
// some code to attempt and test file creation
assert.isNotNull(file, 'error message');
});
it("Delete File", function() {
// some code to attempt test file deletion.
assert.isNull(file, 'error message');
});
});
Karma is the test runner that is used for Waterbear. Karma allows a developer to run multiple instances of test files across multiple browsers allowing us to test Waterbear across all major browsers. This way you can find bugs that only show on certain platforms. Mobile devices can even connect to an instance of Karma by accessing the same IP address and port number that the Karma server is using.
Karma is the program that will load all files that you want to test using the karma.conf.js. It is in the karma.conf.js that all the parameters of testing is defined. It will load the frameworks like mocha, chai, and sinon, it will define how to report the test results, level of logging, the port number the server will use, the browsers to load, and which files to watch changes for. Whenever a watched file is changed (and the feature is not disabled) the tests will be ran again. More information on karma.conf.js can be found here.
To create a test file for a Waterbear file, the file to be tested should be set to load when Karma starts. So add the file to be tested to the files array in the karma.conf.js
e.g. files: ['random_runtime.js', 'vector_runtime.js', 'some_file_to_be_tested.js']
then create file.spec.js. Any JS file that is in the specs folder will be loaded in to the browser when Karma is started.
A spec.js file is generally a JS file filled with the tests to be run using mocha, chai, and/or sinon. One example is vector.spec.js
describe("Vector Test Suite", function(){
it("Test vectors", function(){
var vector = new Vector(1,2);
assert.strictEqual(vector.x, 1, 'Expected vector.x to be 1');
assert.strictEqual(vector.y, 2, 'Expected vector.y to be 2');
});
});
Once you create the spec.js file and have it placed in the waterbear/framework/specs/, start running the tests! To learn how to run the tests move on to the next section.
Running the tests are easy. You must call Karma with the config file
karma karma.conf.js
or you can use
make test
inside the framework directory, which should work too.
Using the Selenium IDE to test the graphical user interface requires for the user to begin recording their session for a test case, by clicking the red button on the top right as shown below.
Once you have a test case recorded you can click on the file tab and click save as and save the test case in the framework/Selenium_tests directory with other test cases.
To open a saved test case you can click on the "file" tab and go to the "open" option and select a test case from its respective directory.
To run a test case you can click the green triangle button with a single dash to its right.
The purpose of testing is to find bugs. If the tests reveal a bug, look at the given information and file the bug is coming from and attempt to deduce why it is happening and fix it. If the fix is not obvious, ask for help on the irc. Every one is happy to help. Happy Testing! Please feel free to add any tests that are deemed necessary for finding bugs.
- If you find that *_runtime.js depends on a value generated at runtime, from the window object, another file create stubs for that needed value.
- Only load browsers that you can use
- This framework can currently only tests pure JS files. In the future tests for the GUI may be added.