-
Notifications
You must be signed in to change notification settings - Fork 61
Testing
This wiki page provides you with our testing processes, quick start and setup, and our best practices.
Our guiding principle is:
ALL functionality will be fully unit tested. Period. Integration tests are optional, except as noted in this wiki.
Why?
Testing serves multiple purposes including:
- It ensures a release candidate and release is stable. Hint: That's a guiding principle.
- It ensures that each piece of the code is performing as designed, intended, and expected.
- Testing flushes out vulnerabilities, wonky behavior, side-effects, and assumptions. Seriously, I can't tell you how many times I've designed something, ran the tests, and realized I had an issue I hadn't considered.
- Tests are a great way to discover how some piece of code works in the codebase. I usually start with the test suite to understand a new framework, app, or tool. It quickly shows me how to implement the functionality and how it should behave.
- When we make changes (and we will), tests assure us that our change works and doesn't break another area in Beans.
- If it worked before you made the change, a failing test helps you to sniff out the root cause.
Okay, did I convince you? I hope so. I'll be building testing labs on Know the Code to walk you through the entire process A-to-Z.
Testing has multiple levels to it, each of which serves a different purpose and adds a layer of assurance to the project's stability and code coverage.
- Single unit tests (without WordPress)
- Integration tests (WordPress runs)
- Live single website testing on your local machine
- Live multisite testing on your local machine
- Alpha testing
- Beta testing
There are other forms of testing, including Usability Tests. We'll focus on 1-4 above.
Reach out on Slack in the
#contributegroup. We are here to help you contribute to Beans! And don't worry. ALL QUESTIONS should be asked. If you're wondering, someone else is too.
A core principle of testing is to test in isolation. Have you heard that phrase before? It means that you are testing a single unit and not the framework, language, application, etc. Huh? You are testing the functionality within a function or method, i.e. one single function.
I can hear your doubts and questions. Stick with me.
WordPress gives us a lot of functionality that we use throughout Beans. Right? Things like add_action, add_filter, get_the_ID(), and so on. We need these functions.
However, when we're testing our code, we expect these functions to behave and give us some value or do something for us. We are not testing WordPress Core. Nope. The Core team and contributors build tests for Core. Not us.
In our tests, we assume and expect that when we call get_the_ID(), we get the ID.
Using monkey patching, which is also called to mock a dependency, we can jump around code that is not in Beans, such as WordPress Core functions.
What happens is: WordPress never runs when unit testing. Instead, using the command line, we run our tests by typing: phpunit. Everything we need loads and runs, test-by-test.
How do we mock or monkey patch? We have standardized on Brain Monkey by Giuseppe Mazzapica. Brain Monkey gives us the means to mock any functionality we want.
What happens? WordPress never runs. Our unit tests test only our code. That's it. We can then focus our energies on the quality, performance, and functionality of our stuff.
They are almost always needed across the board. Rarely though is a codebase 100% tested and has full code coverage. We will strive for 90% and keep pushing for full coverage.
A rule of thumb: Test it and submit your tests with your PR.
An integration test means that we are integrating a third-party dependency(ies) to run with our code in order to test the application's functionality. In other words, it tests how does everything run together in a testing environment.
To do this, you would need to install and set up the WordPress Automated Testing Suite. Instructions are found here for PHP and here for JavaScript.
If you've contributed to WordPress Core or some theme/plugin that uses these automated tests, then you are likely already set up and ready to go.
Integration tests are handy when you need to test the sequence and interactivity between all the dependencies, but you want to do it in an automated, controlled fashion.
You might be thinking: "Hey, I can just run the website on my local machine and manually test." (You were thinking that, weren't you? LOL). Well, there's a problem. We are not good at sniffing out every single interaction, received response, and so on. We want to run tests at the granular level where we can get a report that tells us the details.
Some examples of integration testing areas are:
- Authentication
- Templates
- The Loop
- Menu structures
- Commenting
- etc.
In other words, those areas where our code is heavily dependent upon WordPress doing its thang and then calling us at the right time.
Bottomline: We first do our single unit test per function. Then we build the needed integration tests to test timing and interactivity.
We all manually test in the wild by running the website and seeing what happens. Right? This technique is valid and extremely helpful once we've completed the single unit tests and integration tests. Now it's time to run the website for real and try to break it.
In the "single" website local tests, you have a standalone website, i.e. not multisite. It's running on your machine. You have the latest version of a Beans' branch that you want to test.
- Turn on Debug mode.
- Run the site and try to break it.
- Monitor the web server logs (depending upon which localhost app you're using, those logs could be in different places).
- If you find something weird, stop and analyze. Try to recreate it. Look at the contributing factors. Try to find the root cause. If it requires a change in the code, then adjust the unit and integration tests. Repeat.
Beans can run on both standalone and multisite sites. Using your localhost web server app, create 2 multisites: (1) for a subdirectory mu and (2) the other for subdomain mu.
Just like with the single standalone live test above, repeat the process for each of the sub-sites. Try to break it. If you do, figure out why, what's the root cause, and then fix the problem. Repeat.
I love writing unit tests. I do. I'm an avid TDD (Test-Driven-Development) workflow engineer.
.......more to come here.......