Being advocates of continuous delivery and firm believers of TDD, we understand the power that automated testing brings. Automated tests are core to the development process of Go, and we strive to ensure that all production code is covered by automated tests. Tests are written at multiple levels i.e. unit, integration and functional. The rule of thumb is :
- unit test any code that you write. This includes tests at all levels, ie. UI, controller, service, repository, domain, javascript etc.
- write an integration test if the code being modified involves interaction of more than one class or layer.
- functional tests cover mostly the happy path of any given feature. Since, execution of these test is much slower as compared to an unit or an integration test, try to cover all alternate paths using an integration test, unless the scenario absolutely requires to be end-to-end.
The tool that you use to write/run the test depends on what exactly needs to be tested. Rails code is tested using RSpec, java using JUnit, javascript using JsUnit, and functional tests using Twist.
Most of the Go UI is implemented using Rails. We use RSpec to test all of rails views, controllers, models and any other ruby code that we write. So far, rspec tests are mostly written at unit level, i.e. for a particular view, or controller etc. In a unit test, all interactions with other layers are mocked.
You can find them under server/webapp/WEB-INF/rails/spec folder.
These tests can be executed from command prompt as well as individually from your editor. To run all rspec tests from commandline, execute the following:
./b compile cruise:rails:spec ratchet=no test=no
To run a subset, a pattern could be specified, for instance:
./b compile cruise:rails:spec ratchet=no test=no spec_module=controllers/admin
OR
./b compile cruise:rails:spec ratchet=no test=no spec_module=views/[a-z]*
Alternately, you could have a spec server running while you code, and you could run individual test as and when you make changes to any of the rails code.
server/webapp/WEB-INF/rails/script/spec -X <file path> -l <line number> with server/webapp/WEB-INF/rails as the working directory.
Intellij users could configure to run an external tool with the below mentioned settings. Intellij would take care of passing in the spec file path and line number for the execution.
Program: $ModuleFileDir$/webapp/WEB-INF/rails/script/spec
Parameters: -X $FilePath$ -l $LineNumber$
Working directory: $ModuleFileDir$/webapp/WEB-INF/rails
Please bear in mind that any changes to rails config files or java code would require a restart of spec server for the changes to take effect.
You would see a test folder under each module, these usually have a unit and integration folder wherein the respective tests for the module are written.
These tests use mocks and stubs to achieve isolation, they do not access the db or other layers and have minimal interaction with other classes.
These tests on the other hand cuts across all layers including read/write to db and file-system. Database used by integration tests is the same as the one DevelopmentServer uses, hence make sure you shutdown DevelopmentServer when running integration tests.
To run these from command line, cd to the module directory and run the following command:
mvn -Dtest=<TestClassName> test
Any format for test specification that is supported by the Maven surefire plugin can be used in the above command.
Standalone version of jsunit, placed under server/jsunit, is used to test javascripts.
Run all tests:
./b cruise:misc:jsunit
Run individual test:
You could launch a test server using the below command. The url for the same would be printed on the console. Hit this url from a browser window after updating the value of testPage param to the desired value and run test.
./b cruise:misc:jsunit_server
Functional tests are end-to-end tests which interacts with the web application. Running these require Twist to be installed.
These could be broken down further into three categories - Smoke, Acceptance, and Regression.
-
Smoke: covers basic functionality such as if server and agent come up after an upgrade, and if one can create a pipeline etc. Such tests are marked with Twist tag 'smoke'.
-
Acceptance: covers happy paths of all major features. These are tagged as 'stage1'.
-
Regression: covers happy paths of all minor features and alternate paths of major features. Tests which are neither marked as 'smoke' nor 'stage1' would be executed as a part of Regression test run which are scheduled to run once in a day as they take atleast a few hours to run.
We use Yourkit java profiler to profile Go. The test involves a setup of 100 materials of each supported SCM type, with 500 pipelines and 100 agents.
To simulate a very active system, continuous commits are made to each of the different materials which are configured. This would cause each of the pipelines to run utilizing all available agents. In the meanwhile, multiple ab requests are made to different pages of the application to simulate user interaction. During this entire activity, Yourkit profiler is started to perform memory, CPU and Sql profiling. This goes on for about 30 minutes after which yourkit snapshot is saved and examined.
By default, each job in the build pipeline would run tests from a given module. However, few modules like server and common have a lot of tests. Running them in a serial fashion would take along time. Quick feedback through build is very important, however as the number of tests increase the longer the test execution time would be. In order to tackle this issue, we use a Test load balancer to perform test parallelization. TLB creates 'n' number of partitions of tests. There are multiple jobs created in the build pipeline each with a allocated TLB_PARTITION_NUMBER. Based on the TLB_PARTITION_NUMBER, TLB would assign the tests in a given partition to the job.