pytest-run-parallel

A simple pytest plugin to run tests concurrently

---

This pytest plugin takes a set of tests that would be normally be run serially and execute them in parallel.

The main goal of pytest-run-parallel is to discover thread-safety issues that could exist when using C libraries, this is of vital importance after PEP703, which provides a path for a CPython implementation without depending on the Global Interpreter Lock (GIL), thus allowing for proper parallelism in programs that make use of the CPython interpreter.

For more information about C thread-safety issues, please visit the free-threaded community guide at https://py-free-threading.github.io/

How it works

This plugin is not an alternative to pytest-xdist and does not run all of the tests in a test suite simultaneously in a thread pool. Instead, it runs many instances of the same test in a thread pool. It is only useful as a tool to do multithreaded stress tests using an existing test suite and is not useful to speed up the execution of a test suite via multithreaded parallelism.

Given an existing test taking arguments *args and keyword arguments **kwargs, this plugin creates a new test that is equivalent to the following Python code:

import threading
from concurrent.futures import ThreadPoolExecutor

def run_test(b, *args, **kwargs):
    for _ in range(num_iterations):
        b.wait()
        execute_pytest_test(*args, **kwargs)


with ThreadPoolExecutor(max_workers=num_parallel_threads) as tpe:
    b = threading.Barrer(num_parallel_threads)
    for _ in range(num_parallel_threads):
        tpe.submit(run_test, b, *args, **kwargs)

The execute_pytest_test function hides some magic to ensure errors and failures get propagated correctly to the main testing thread. Using this plugin avoids the boilerplate of rewriting existing tests to run in parallel in a thread pool. Note that args and kwargs might include pytest marks and fixtures, and the way this plugin is currently written, those fixtures are shared between threads.

Features

• Two global CLI flags:

  • --parallel-threads to run a test suite in parallel
  • --iterations to run multiple times in each thread

• Three corresponding markers:

  • pytest.mark.parallel_threads(n) to mark a single test to run in parallel in n threads
  • pytest.mark.thread_unsafe to mark a single test to run in a single thread. It is equivalent to using pytest.mark.parallel_threads(1)
  • pytest.mark.iterations(n) to mark a single test to run n times in each thread

• And the corresponding fixtures:

  • num_parallel_threads: The number of threads the test will run in
  • num_iterations: The number of iterations the test will run in each thread

Requirements

pytest-run-parallel depends exclusively on pytest.

Installation

You can install "pytest-run-parallel" via pip from PyPI:

$ pip install pytest-run-parallel

Caveats

Pytest itself is not thread-safe and it is not safe to share stateful pytest fixtures or marks between threads. Existing tests relying on setting up mutable state via a fixture will see the state shared between threads. Tests that dynamically set marks or share marks will also likely not be thread-safe. See the pytest documentation for more detail and the community-maintained free threaded Python porting guide for more detail about using pytest in a multithreaded context on the free-threaded build of Python.

We suggest marking tests that are incompatible with this plugin's current design with @pytest.mark.thread_unsafe.

Usage

This plugin has two modes of operation, one via the --parallel-threads and --iterations pytest CLI flags, which allows a whole test suite to be run in parallel:

pytest --parallel-threads=10 --iterations=10 tests

By default, the value for both flags will be 1, thus not modifying the usual behaviour of pytest except when the flag is set.

Note that using pytest-xdist and setting iterations to a number greater than one might cause tests to run even more times than intended.

The other mode of operation occurs at the individual test level, via the pytest.mark.parallel_threads and pytest.mark.iterations markers:

# test_file.py
import pytest

@pytest.fixture
def my_fixture():
    ...

@pytest.mark.parallel_threads(2)
@pytest.mark.iterations(10)
def test_something_1():
    # This test will be run in parallel using two concurrent threads
    # and 10 times in each thread
    ...

@pytest.mark.parametrize('arg', [1, 2, 3])
@pytest.mark.parallel_threads(3)
def test_fixture(my_fixture, arg):
    # pytest markers and fixtures are supported as well
    ...

Both modes of operations are supported simultaneously, i.e.,

# test_something_1 and test_fixture will be run using their set number of
# threads; other tests will be run using 5 threads.
pytest -x -v --parallel-threads=5 test_file.py

Additionally, pytest-run-parallel exposes the num_parallel_threads and num_iterations fixtures which enable a test to be aware of the number of threads that are being spawned and the number of iterations each test will run:

# test_file.py
import pytest

def test_skip_if_parallel(num_parallel_threads):
    if num_parallel_threads > 1:
        pytest.skip(reason='does not work in parallel')
    ...

Finally, the thread_comp fixture allows for parallel test debugging, by providing an instance of ThreadComparator, whose __call__ method allows to check if all the values produced by all threads during an specific execution step are the same:

# test_file.py
def test_same_execution_values(thread_comp):
    a = 2
    b = [3, 4, 5]
    c = None
    # Check that the values for a, b, c are the same across tests
    thread_comp(a=a, b=b, c=c)

Contributing

Contributions are very welcome. Tests can be run with tox, please ensure the coverage at least stays the same before you submit a pull request.

License

Distributed under the terms of the MIT license, "pytest-run-parallel" is free and open source software

Issues

If you encounter any problems, please file an issue along with a detailed description.