Experiments

We have A/B testing in the extension to perform staged rollouts of new functionality. If a user belongs to an experiment group, they will see some special features which are visible only to the users of that group. This way we can test a new feature on a subset of users and making sure it's stable, before releasing it to all users. Check out Wikipedia for more details on A/B experiments.

We initially implemented our own A/B testing framework, however it turns out there is a common A/B testing framework used by other teams, for example the Docker extension. Now that we've integrated it (see implementation), we're deprecating our custom implementation.

Common key specs include but are not limited to:

• Experiment groups are logged in the beginning when the extension loads
• If the user is in an experiment, ensure that experiments are used in the first session itself
• If the user opted out of telemetry, then they are also opted out of A/B testing
• Conditional behavior depending on whether an experiment is enabled or not: for example if the user is in the “NativeNotebookEditor” experiment, then the extension uses the new VS Code Native Notebook editor to view/edit notebooks (*.ipynb files).

Opt into/out from experiments

• Add the experiment name to the "jupyter.experiments.optInto" and "jupyter.experiments.optOutFrom" keys in package.json:

"jupyter.experiments.optInto": {
    "type": "array",
    "default": [],
    "items": {
        "enum": [
            "NativeNotebookEditor",