-
Notifications
You must be signed in to change notification settings - Fork 265
Automated Browser Testing
Automated Tests for the Persona sign-in service fall into three groups.
- Front End:Tests which exercise code that runs on client browsers
- API: Tests which exercise the HTTP APIs that Front end code interacts with.
- Browser: Tests which simulate user interaction via selenium
This document proposes an organization for automated browser-based tests of the Persona sign-in service. The goals of the structure are to focus efforts at building out automated test coverage so highest value tests are authored first. To organize tests by product feature, so that it possible to quickly understand the nature and intent of individual tests. To make it clear where new tests should be categorized, and to provide a rough at-a-glance idea of current test coverage. Finally, we must define evenly sized groups of tests whose collective status gives a clear picture of the health of the service.
Automated browser testing is designed with the following goals:
- A way to robustly test changes before merging into our development branch
- A way to continuously perform rigorous integration testing of our development branch
- Tools to perform holistic high frequency health checks of our production service
- Tools to perform rigorous low frequency audits our production service
- Re-focus human QA to perform pro-active performance analysis, tool creation, and security auditing, to do the things that the computers cannot.
We have settled on a group of tools to build these tests:
- Selenium Webdriver - A cross browser, cross-OS platform for the development of automated browser based tests.
- Saucelabs - A service that allows for the remote execution of selenium tests on a wide array of VMs running different OS/browser combinations
- Jenkins - The world's favorite periodic execution tool for testing, and it has a dashboard.
The language and libraries we use to write tests is at time of writing python leveraging selenium webdriver bindings. Whether this is the right choice for Persona is a less important question that is out of the scope of this document.
Authoring a test and getting it to succeed once on a single platform is incredibly easy. Ensuring that this same test runs without a false positive across a grid of 7 or more browser/operating over 30 or more iterations is incredibly hard. At the same time, tests which yield false positives more than once a month are useless.
If we run high level automated tests every 20 minutes to test the Persona service in production, that amounts to 2160 iterations per month. Tests run at this frequency are what is described above as "holistic health checks". Automated browser testing that can run at this volume with an acceptable false positive rate is no small achievement. These tests need only run in one or two browser environment to provide their core value.
Similarly, if tests that are more specific that high level health checks yield false positives more than once in 30 executions, this leads to wasted time, and renders the tests useless.
Because Persona is a service built on top of email, it is necessary to be able to check email in an automated fashion in order to be able to build meaningful end to end tests.
Two services exist to help:
- restmail.net - provides email addresses that can be checked without passwords via an api. This allows us to Test all elements of UI that trigger emails, but to receive email programmatically to verify addresses.
- personatestuser.org - automatically creates ephemeral test accounts that can be used during testing. Leveraging this service allows us to eliminate redundancy in individual tests and keep test implementation more concise and focused on test goals.
This remainder of this document defines test groups. Each test group targets a feature or small class of product features. For each group a proposed "short name" is provided (useful for naming directories), as well as a bit of prose describing the feature that the test group targets.
The test groups are described in priority order with the hope that this will also be the order of implementation.
These tests answer the question "is the service functioning". They test, at a high level, the basic new user sign up using both email addresses from providers who support Persona ("primary" email addresses), and addreses which have no such support ("secondary" email addresses).
These tests ensure that the service is usable, and run at a high frequency in production (20 minutes) in 2 or three browser environments.
These tests excercise the new user sign up process in much more depth than
health tests. They verify the new user experience works via both demo
sites (which use different APIs to interact with persona), that the new
user experience works via the persona.org site as well, and that either
secondary or primary addresses can be used successfully. We also test
proper behavior when verifying an email address in a browser other than
the initiator of the email.
Finally, we test both the cases where a new user signs up with a "primary" email address with an existing authenticated session to their provider, and without.
These tests focus on the password reset flow. They verify that upon forgetting a password, a user can reset their password via either persona.org or in context as they are logging into a site. The test that resetting a password on an account with more than one email address behaves properly, allowing the user to "re-verify" email accounts as they are used to log into sites after a user resets their password.
These tests verify that an existing Persona user can sign into sites using persona. It tests the "email picker screen". It verifies that when logging into a site you've recently visited on a particular device, the last used email is bold and pre-selected. Verifies that when you have not visited a particular site that there is no default selected email address.
These tests verify that users can add emails to their existing persona account in context and that the software behaves properly. Included tests are adding a "primary" email to an account with only a "secondary" email. Adding a "secondary" address to an account with only a "primary" (during which the user must select a password). Adding a "secondary" email to an account with a "secondary", and adding a "primary" email to an exisiting account with a "primary" address. Finally, verification of email addition should be tested from within the same browser, as well as from a different browser (the codepath where a user verifies an added secondary address, chooses their password, and then verifies in a separate browser is especially annoying to manually test and is a great thing to automate).
Persona attempts to improve user's security at public terminals by varying the duration of an authenticated session. This works by logging the user in "ephemerally" (with a shorter duration session and key material) during the first usage of an account on a browser than has not been used before. In a subsequent login with persona seperated by more than 60s from the first, the user will be asked "Is this your computer". A positive response will extend the duration of the session, while a negative response will reset the process.
Hint: the implementor of these tests may choose to directly modify localStorage in the browser to allow the test to execute without waiting.
When a user changes their password, in addition to a new password they are logged out of all other browsers that they are currently authenticated to persona on. This includes being logged out of websites (upon the subsequent visit). Tests in this category excercise these intended behaviors.
Removing an email has obvious, and non-obvious affects. The basic tests in this group will verify that the persona.org management features to remove an email work properly. A non-obvious affect that email_removal should have is to log a user out of websites that the user is authenticated to with the email address being removed.
A test to verify that account cancelation works, and causes all email addresses associated with the account being canceled to be removed. Additionally, these tests verify that cancelation causes all sites one is currently logged into to log you out.
Persona has features to display links to legal documents to users when they sign into websites. One set of legal documents is from the persona service itself. Additionally, websites using persona may themselves provide ToS and EULA documents.
This test group verifies that regardless of the user flow through the dialog, the are ToS and EULA documents for persona and the site are visible when required.
Websites using persona may provide a human readable short name, and/or a site logo that will be displayed inside the persona dialog to afford the user an easier time recognizing the site they're logging into. These tests verify that across all browsers names and logos are displayed correctly.
With persona, there are several different relevant sessions that co-exist. The first session is an authenticated session with the persona servers themselves. Other sessions are authenticated sessions to websites which use persona for authentication.
Logout tests verify that logging out from one site is persisent (upon page reload the user is still logged out), while a different sites remains unaffected (upon page reload the user is still logged in).
Finally, logout tests that logging out of the persona service terminates all other sessions where websites support it (123done is an example website that supports this feature).
Persona can automatically issue an assertion to a site that a user visits the the user has recently signed into. This test verifies the proper functioning of this feature across all browsers.